[zeromq-dev] zeromq / pub sub lost messages

alex. thissideup at riseup.net
Wed Jan 4 11:17:50 CET 2017


The publisher does not know when a subscriber congests, as the pub-sub
pattern is stateless in that regard.

What you could do is to check which message was the last one received at
the subscriber and request all messages from that message onwards from
the publisher. However, that requires a separate connection to the
publisher.

The pub-sub-pattern is mostly only a viable option when the subscriber
is able to work even if it doesn't receive all messages. If your usecase
does not allow that you might be better of using another pattern, like
req-rep or broker.

Best,
alex.

On 04.01.2017 10:43, Philippe Guibert wrote:
> On Wed, Jan 4, 2017 at 10:34 AM, Jens Auer <jens.auer at betaversion.net> wrote:
> 
> Hello Jens,
> 
> Best wishes too,
> 
>> Hi,
>>
>>
>>
>> how do you send the messages?
> 
> I use zmq_msg_send() procedure.
> I also used zmq_send() procedure with and without MSG_NO_WAIT flag.
> I expect that return code sends me an error when dropping message,
> which is not the case.
> 
>> The default HWM is 1000 messages and sending
> 
> I used tcp transport connection, and wireshark. I observed that the
> packets number sent was matching the received messages on the
> subscriber side.
> I deduced the problem was due to an emission problem, not a reception problem.
> 
> I did a counter test by setting ZMQ_SNDHWM to a big value ( in my case 60000).
> I could observe no drops with both ipc and tcp transport connection.
> 
>> is done asynchronously in the IO thread. If you exceed this number of
>> messages in the queue, e.g. by sending a batch of much more than 1000
>> messages you will experience message drop.
> 
> How can I know there are some drop messages ?
> If I make a limitation, I expect to handle the flow congestion, in
> order to resubmit to publisher, but it seems publisher does not return
> me valuable error code.
> 
> Best Regards,
> 
> Philippe
> 
>>
>>
>>
>> Best wishes,
>>
>> Jens
>>
>>
>>
>> Von: zeromq-dev [mailto:zeromq-dev-bounces at lists.zeromq.org] Im Auftrag von
>> Philippe Guibert
>> Gesendet: Mittwoch, 4. Januar 2017 06:51
>> An: zeromq-dev at lists.zeromq.org
>> Betreff: [zeromq-dev] zeromq / pub sub lost messages
>>
>>
>>
>> Hi all,
>>
>> I am using zeromq 4.1.3 version through PUB/SUB operation.
>>
>> Some information about the way it is implemented:
>>
>> - It uses one single PUB application, and single SUB application, in
>> separate process.
>>
>> - Underlying Transport layer is IPC ( I am on the same machine).
>>
>> - I checked that subscription is done far before the first publication
>> message is sent.
>>
>> - Message carried are 120 byte size.
>>
>> - I use zmq_msg_recv() in reception and zmq_msg_send() in transmission.
>>
>> - I check for return code on both sides.
>>
>> I have one PUB application that sends several thousand of messages per
>> second, and I expect that the SUB application will receive all messages.
>>
>> The observation is that while the publisher sends 15000 messages, the
>> subscriber can receive sometimes 15000, but other times 14750, or 11000. It
>> does not seem to be deterministic.
>>
>> I have no error codes related to the zmq_msg_recv() or zmq_msg_send() API.
>>
>> Do you have any suggestions to give on how to handle that issue ?
>>
>> Do you suggest to use pgm protocol instead of ipc ?
>>
>> Are there any tracked issues related to that issue described ?
>>
>> Thanks in advance for any suggestion,
>>
>> Best Regards,
>>
>> Philippe
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 



More information about the zeromq-dev mailing list