[zeromq-dev] multi-part messages

[john skaller]

On 23/01/2012, at 1:58 AM, Pieter Hintjens wrote:
> 
> And we end up with the simple, useful and widely used API we have today.

IMHO the real problem with multipart messages is that creating them
is dynamic and hence error prone, i.e. you have to grab each part in turn
with flow of control.

This would make sense if you could "stream on" parts and save memory.
But you can't.

The problem is like this: if you use say an array, you can put the
pieces into it in any temporal order. This allows you to write
simple code.

If you have to put the pieces into stream, you have to mess up your
code so control flow is in the same sequence as the message.
This results in convoluted code.

this is a general observation, not peculiar to ZMQ. I have exactly
this problem in the Felix compiler. I have to generate C++ code
in a particular order: declare things first, then define them for example.

This makes a shambles of the compiler back end. A lot of code has
to "predict" ahead of time what is actually required and organise it
so the final code generation is linear (i.e. control flows through things
in the order that they have to generate the output).

Just as an example: in Felix you have two kinds of function:
ordinary "C" functions, and applicative objects which are instances
of C++ classes with an "apply" method. Because Felix allows a C
function to be used wherever a function is needed, any C function used
where a Felix function is required must be enclosed in a wrapper.

the optimal time to know this is just as I'm about to generate code
to pass the function.. and then generate a wrapper if there isn't one
already.

But, I cannot do this. Because the wrapper is a C++ class and it should
have been declared much earlier in the file. So .. I have to run
a special "predict which C functions need wrappers" pass.

This really sucks! It can easily be fixed if every logical part of the output
was a separate file, and I could write into each one, then merge
them at the end. Then I could generate the wrappers "on demand".

The point: the current API is not as good as an array.
The only reason to NOT use an in memory data structure would
be the storage cost, but in this case, there is no extra cost
since 0MQ holds the lot in memory before streaming it anyhow.

I'm NOT suggesting to change 0MQ. But I'm thinking I will do the
array version in the Felix binding. Since an array of one message
is just a special case of an array of more than one message ...
only one API call is needed to handle both.


--
john skaller
skaller at users.sourceforge.net