[zeromq-dev] Important: backward incompatible changes for 0MQ/3.0!

Martin Sustrik sustrik at 250bpm.com
Wed Mar 30 11:12:30 CEST 2011


Hi Paul,

>> Btw, note that once a connection is established, the number of
>> allocations is highly optimised. In ideal case, 0mq should be able to
>> run with zero allocations from that point on. It's the remaining
>> out-of-critical-path code that is responsible for almost all the
>> allocations.
>
> Well.. what about zmq_msg_init_data/zmq_msg_init_size?

It's kind of complex, but basically, for very small messages there's no 
allocation -- the message data are kept on stack as a part of zmq_msg_t 
structure. For larger messages and zmq_msg_init_size() there's one chunk 
allocated -- the metadata (such as reference count) are simply preceding 
user data (actual message content) in a single memory chunk. As for 
zmq_msg_init_data() there are two allocations. There's the allocation 
done by user, then there's an allocation of message metadata inside 
zmq_msg_init_data() itself.

> And I'd like to discuss custom allocations for message bodies.
> Somebody wanted to allocate memory on a shared pool. I'd also like
> to overallocate message body buffer, and add some bookkeeping info,
> so I don't need to move message to another place later.

Yes. I was intentionally delaying this discussion not to mix it with the 
other discussion (how to overload *all* allocations in the library).

Given that that thread have more or less died, let's have a look at the 
message body allocation.

> It would be great if API is based on setsockopt if that is at all possible
> (yes I really mean socket-specific). For allocating outgoing messages
> zmq_msg_init_data is mostly ok apart from allocating zmq_content_t,
> which is probably not that bad, if memory allocator is smart enought.

Yes. The send() side of the problem is more or less solved by 
zmq_msg_init_data(). (Can you spell out more clearly what's the problem 
with zmq_content_t?)

As for the recv() side, there's no way at the moment to specify what 
allocation mechanism should be used.

You are right that it should be done on socket level (setting alloc/free 
function pointers using setsockopt, for example). However, let me 
explain what was implicit in your mail, namely, what are the problems 
with alternative approaches:

1. The allocation mechanism cannot be implemented on message level, such 
as setting alloc function pointer when creating a message. The reason is 
that the buffer is quite probably already received from the network at 
the point when you are creating the message to receive it to.

2. Global scope, i.e. saving the alloc function pointer in a global 
variable breaks the model of contexts which allows several instances of 
0mq to work ok within a single process without misinteracting one with 
another.

3. Context scope, i.e. setting the alloc pointer in zmq_init(), is no 
good as you often need to suit the allocation mechanism to a particular 
message feed. So, for example, if you are getting a stream of really 
large messages that you want to receive directly to the shm as well as 
latency-sensitive feed of small control messages you don't want the 
latter to be allocated in shmem as it would negatively effect the latency.

Martin



More information about the zeromq-dev mailing list