[zeromq-dev] PUB/SUB sockets over PGM intermixes multipart messages from different peers

[Andrey Smirnov]

Hi!

We're trying to us ZeroMQ to implement N-to-N PUB-SUB scheme where every
host is both publisher and subscriber. We use PUB/SUB sockets with IPC
transport to route message inside one host and PGM transport to send
messages between hosts.

So on each server we have process that creates PUB socket with PGM transport
and SUB socket on the same PGM endpoint. Local multicast is disabled. Actual
messages sent are multipart (as PUB/SUB requirement) with two parts: tag and
body. Everything works completely fine under low load: messages published on
one host are sent over multicast PGM and received on all other hosts. When
load goes up (we're observing it at around 20 msg/sec) sometimes multipart
messages come intermixed with each other: instead of two multipart messages
[tag1, body1] and [tag2, body2] we receive two multipart messages [tag1,
tag2, body1] and [body2].

I was digging through ZeroMQ sources (we're on 2.1.4 now) to find out the
reason for such behavior. In pgm_receiver.cpp every peer (TSI) is handled
separately, it has its own decoder to do unframing. But received messages
are sent over the same pipe (inout). My current guess is that if two
multipart messages are sent over multicast and PGM packets are intermixed in
such way that first we receive tag1, then tag2, and afterwards body1 and
body2 (they're coming in that order from decoders), ZeroMQ would interpret
that as two multipart messages of length 3 and 1 ([tag1, tag2, body1],
[body2]) due to the MORE flags on tag1, tag2 (being originally first parts
of different multipart messages).

Is this expected behavior? (i.e. ZeroMQ isn't supposed to work in this way?)

My only workaround so far is not to use multipart messages, send only body
and subscribe to '' (drop tags).

Thanks,

-- 
Andrey Smirnov,
Qik Web Team Lead
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110426/9eea122a/attachment.htm>