[zeromq-dev] Package Loss in PUSH/PULL

[Pieter Hintjens]

Yes, when there are disconnections, any data in flight will be lost.
This is unavoidable unless one adds some kind of session layer, i.e.
handshaking for messages sent. And even then, it must operate at the
right level, e.g. if a receiving layer handshakes a message yet the
process crashes before the application can handle it, it will be lost.

First action should be to investigate the cause of those disconnections.

On Sun, Jul 13, 2014 at 12:23 PM, 蒋达晟 <dsjiang753 at hotmail.com> wrote:
> I've confirmed that there are some reconnections, around 1~10 in 1million
> during one minute time window.
>
> I'm curious that if package loss is doomed to happen when reconnect happens
> in sender.
> If so, how to avoid reconnect or avoid package loss without an ack from the
> receiver side in application level?
>
>
> ________________________________
> From: dsjiang753 at hotmail.com
> To: zeromq-dev at lists.zeromq.org
> Date: Sat, 12 Jul 2014 15:08:24 +0000
>
> Subject: Re: [zeromq-dev] Package Loss in PUSH/PULL
>
> I've confirmed that the lost package between sender/receiver is serial.  The
> approximate time of the interval is around 1 ~ 3s
> But I check the FD of sending socket, it doesn't change during the
> procedure.
>
>> From: ph at imatix.com
>> Date: Sat, 12 Jul 2014 16:30:43 +0200
>> To: zeromq-dev at lists.zeromq.org
>> Subject: Re: [zeromq-dev] Package Loss in PUSH/PULL
>>
>> Yes, you can monitor the sending socket for reconnections.
>>
>> On Sat, Jul 12, 2014 at 4:23 PM, 蒋达晟 <dsjiang753 at hotmail.com> wrote:
>> > Sorry for misleading, I use serial id for the sender, so the lost
>> > package
>> > may be serial for the sender-receiver pair.
>> > Is there any way to identify if a re-connection happens?
>> >
>> > ________________________________
>> > From: dsjiang753 at hotmail.com
>> > To: zeromq-dev at lists.zeromq.org
>> > Date: Sat, 12 Jul 2014 14:13:02 +0000
>> >
>> > Subject: Re: [zeromq-dev] Package Loss in PUSH/PULL
>> >
>> > All the process is alive during the procedure.
>> > And I don't think there is a re-connection ever happen due to the lost
>> > package is non-serial.
>> >
>> >
>> >> From: ph at imatix.com
>> >> Date: Sat, 12 Jul 2014 16:01:36 +0200
>> >> To: zeromq-dev at lists.zeromq.org
>> >> Subject: Re: [zeromq-dev] Package Loss in PUSH/PULL
>> >>
>> >> Is there any trace of processes dying and restarting?
>> >>
>> >> On Sat, Jul 12, 2014 at 3:45 PM, 蒋达晟 <dsjiang753 at hotmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I'm using zeromq on a cluster with 1000 machines. I need to transfer
>> >> > data
>> >> > between each peer and I use the PUSH/PULL mode, but surprisingly I
>> >> > found
>> >> > that some of the messages are lost. The percentage is quite low, but
>> >> > I
>> >> > can't
>> >> > figure why this will happen, can anyone help me?
>> >> > The loss rate is about 1MB~10MB every 10TB
>> >> > The pattern is that only a few machines (less than 10) will have this
>> >> > problem on an uncertain period (not the beginning/end, non-serial for
>> >> > the
>> >> > sender, not sure if all on the same receiver).
>> >> > Environment: Redhat 4, zeroMQ (4.03/4.04), using the Java binding.
>> >> >
>> >> > Thanks!
>> >> >
>> >> > Dasheng Jiang
>> >> >
>> >> > _______________________________________________
>> >> > zeromq-dev mailing list
>> >> > zeromq-dev at lists.zeromq.org
>> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >> >
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> > _______________________________________________ zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________ zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>