[zeromq-dev] XREP/ROUTER no longer receive messages atomically in 3.0/4.0

Brian Granger ellisonbg at gmail.com
Wed Sep 21 18:03:25 CEST 2011

On Sat, Sep 17, 2011 at 11:52 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:
> On 09/06/2011 12:23 AM, Pieter Hintjens wrote:
>> On Monday, September 5, 2011, MinRK wrote:
>>     Okay, then I will put my official vote behind making RCVMORE as the
>>     sole-indicator of a contiguous message.  This would mean that any
>>     time LABEL is set, MORE is also set (if I understand LABEL
>>     correctly).  Perhaps this would mean SNDLABEL=6, rather thank
>>     SNDLABEL=4, as it implies SNDMORE (2)?
>> This gets my vote too. The model that seems to be cleanest is that we
>> have multiple parts, indicated by SNDMORE, and these parts can be
>> labels, indicated by SNDLABEL, or data parts. Otherwise it is becomes
>> quite messy. What does zmq_msg_t hold? A label? A part? A message? A
>> data part?
> The problem here is layering.
> The multi-part messages were originally introduced to help with
> multi-hop request/reply (sticking the identities to the message).
> Unfortunately, these were also exposed to the user, which created
> situation where the feature has no clear place in the protocol stack.
> It's used on 0MQ level -- which should mean that it should be invisible
> to the endpoint applications.
> It's used on application level -- which should mean that it's invisible
> to 0MQ.
> The simplest way to deal with mislayering would be to disable
> application level multi-parts  in 0MQ entirely (no more MORE) and use it
> only internally (LABELs). Optionally, an library could be added on top
> of 0MQ to emulate the application-level multi-parts.

Application level multi-part messages are one of the killer features
of 0MQ.  Without them our usage of 0MQ would be nearly impossible.
PLEASE don't remove them.

We use multi-part messages to send a message header (which is small,
typically JSON) and multiple body sections:

[JSON header, body1, body2, ...]

The body sections can be very large (hundreds of MBs) and need to be
sent atomically with the header through our messaging pipes.  Without
multi-part messages we either have to:

1. Do an expensive copy+concat of the header and body parts.
2. Send the header and body parts separately non-atomically, which has
a host of problems for multi-hop systems.

In my mind this usage case is one of the main usage cases for
multi-part messages in the application level.

> The above solution would cause backward compatibility problems however.
> Thus, the current design tries to cut a compromise between clear
> layering and backward compatibility. It's not perfect, but at least it
> doesn't require lower layers (0mq as such) to deal with upper layers'
> protocol entities (MORE).
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

Brian E. Granger
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu and ellisonbg at gmail.com

More information about the zeromq-dev mailing list