[zeromq-dev] Python message passing

[Martin Sustrik]

Hi folks,

There have been several issues in this thread that call for explaining 
the rationale behind design assumptions in 0MQ.

One big issue is serialisation, whether it should be part of the 
messaging system or not.

Once there was a strong feeling that the middleware is responsible for 
serialisation of user data. Given this assumption, middleware had to 
define complex wire-level protocol to push any kind of data into the 
message on one side and get it out on the other side. The protocol had 
to be generic enough to work with any hardware architecture, any OS and 
any high-level language. Different issues had to be taken care of like 
different endianness, lack of unsigned integer types in Java, algorithms 
to marshall pointers etc.

The effort resulted in complex middleware specifications like CORBA. One 
of the reasons why CORBA has declined from the status of ultra-hot 
technology to today's relative obscureness is related to the 
serialisation. Need for serialisation requires complex specifications 
which in turn make the software bloated, suboptimal, hard to code and 
make 100% interoperable and reliable.

In short, with 0MQ we've opted to use wire format as simple as possible 
(size-prefixed BLOB) to ensure easy interoperability, efficiency and 
reduce the code (and bug) bloat.

This design decision also adheres to the UNIX philosophy of "do one 
thing and do it well". Our decision was to make messaging software do 
messaging and allow serialisation libraries (there are plenty of them) 
do the serialisation work on the layer above it.

The other issue touched in thread was how language binding libraries are 
coded.

The decision was to code the libraries by hand. There were two main 
motivations here:

1. Languages differ and generating exactly the same wrapper for each 
language would violate "style" of individual languages making it 
unpleasant and even difficult to use.

2. Hand written wrapper is always more efficient when compared to the 
generated one. This is especially true if fragile concepts like 
zero-copy are to be used.

See my further comments inlined:

> Also, I was wondering if it is possible to wrap the C++ library using 
> SWIG, as this possibly reduces the overhead of the calls increasing 
> effciency ?

My bet would be that SWIG won't make the performance better. Feel free 
to try though.

 > There should however be some sort of hook to allow for low level
 > serialization techniques e.g. easy combination with the google thing
 > (whose name I always forget).

Protocol Buffers maybe? Can you have a hint how do you imagine such a 
hook to look like?

 > Alternately I have been looking at and learning boost to wrap zeromq,
 > as it seems nicer for doing C++ wrappings.

Sure. Have a look at it. However, it should be noted that boost despite 
its name is not hyper-optimised in all of its components. I recall a 
bottleneck in real-world stock quote processing application that turned 
out to be caused by underperforming boost tokeniser.

 > A final thought on this situation: if the string conversion is really
 > what is causing the difference between acceptable and unacceptable
 > timing in python, by which i mean  there is no good algorithmic or
 > "simple dirty trick" method to bring the speed to the acceptable side,
 >  then I've found that a compiled solution is usually the way to go. It
 > may just be your main loop, or your most computationally heavy
 > functions, but it can make all the difference.

Agreed.

 > About serialization, yes that would be nice. But I am thinking message
 > size would be a factor, no?

Factor to parformance, right? Yes, message size is a factor, however, 
the effect is visible only if the difference in size is large (say 1kB 
vs. 4kB per message). If the difference we are considering is say 16 
bytes vs. 32 bytes per message, you probably won't be able to see the 
difference unless you are developing a clustered supercomputer.

 > re SWIG, I haven't used Boost to wrap C++ but I have read it has its
 > advantages. SWIG itself would have a translation layer but wouldn't it
 > save the memcpy() call in the pyZMQ_send() ?

No. The memcpy is related to the lifetime of the string. The wrapper 
creates new string so that 0MQ can deallocate it at will. This is 
actually a place where improvement would be of value. 0MQ C++ API allows 
you to pass in the data buffer - which can be the native python string - 
and a deallocation function that is called asynchronously when the data 
are no longer needed. The challenge is make it work well with Python 
memory management model.

 > I believe that the memcpy or a similar mechanism would be required or a
 > bunch of accounting to track the PyString object and prevent its garbage
 > collection. I will need to look into how other C* I/O backends that use
 > async I/O do it to be sure.

Yes. As noted above, this is a place for improvement. Maybe it's 
possible to refcount the python string to prevent it from garbage 
collection while it is processed by 0MQ?

Martin