[zeromq-dev] Python message passing
sustrik at fastmq.com
Fri Apr 24 09:44:49 CEST 2009
There have been several issues in this thread that call for explaining
the rationale behind design assumptions in 0MQ.
One big issue is serialisation, whether it should be part of the
messaging system or not.
Once there was a strong feeling that the middleware is responsible for
serialisation of user data. Given this assumption, middleware had to
define complex wire-level protocol to push any kind of data into the
message on one side and get it out on the other side. The protocol had
to be generic enough to work with any hardware architecture, any OS and
any high-level language. Different issues had to be taken care of like
different endianness, lack of unsigned integer types in Java, algorithms
to marshall pointers etc.
The effort resulted in complex middleware specifications like CORBA. One
of the reasons why CORBA has declined from the status of ultra-hot
technology to today's relative obscureness is related to the
serialisation. Need for serialisation requires complex specifications
which in turn make the software bloated, suboptimal, hard to code and
make 100% interoperable and reliable.
In short, with 0MQ we've opted to use wire format as simple as possible
(size-prefixed BLOB) to ensure easy interoperability, efficiency and
reduce the code (and bug) bloat.
This design decision also adheres to the UNIX philosophy of "do one
thing and do it well". Our decision was to make messaging software do
messaging and allow serialisation libraries (there are plenty of them)
do the serialisation work on the layer above it.
The other issue touched in thread was how language binding libraries are
The decision was to code the libraries by hand. There were two main
1. Languages differ and generating exactly the same wrapper for each
language would violate "style" of individual languages making it
unpleasant and even difficult to use.
2. Hand written wrapper is always more efficient when compared to the
generated one. This is especially true if fragile concepts like
zero-copy are to be used.
See my further comments inlined:
> Also, I was wondering if it is possible to wrap the C++ library using
> SWIG, as this possibly reduces the overhead of the calls increasing
> effciency ?
My bet would be that SWIG won't make the performance better. Feel free
to try though.
> There should however be some sort of hook to allow for low level
> serialization techniques e.g. easy combination with the google thing
> (whose name I always forget).
Protocol Buffers maybe? Can you have a hint how do you imagine such a
hook to look like?
> Alternately I have been looking at and learning boost to wrap zeromq,
> as it seems nicer for doing C++ wrappings.
Sure. Have a look at it. However, it should be noted that boost despite
its name is not hyper-optimised in all of its components. I recall a
bottleneck in real-world stock quote processing application that turned
out to be caused by underperforming boost tokeniser.
> A final thought on this situation: if the string conversion is really
> what is causing the difference between acceptable and unacceptable
> timing in python, by which i mean there is no good algorithmic or
> "simple dirty trick" method to bring the speed to the acceptable side,
> then I've found that a compiled solution is usually the way to go. It
> may just be your main loop, or your most computationally heavy
> functions, but it can make all the difference.
> About serialization, yes that would be nice. But I am thinking message
> size would be a factor, no?
Factor to parformance, right? Yes, message size is a factor, however,
the effect is visible only if the difference in size is large (say 1kB
vs. 4kB per message). If the difference we are considering is say 16
bytes vs. 32 bytes per message, you probably won't be able to see the
difference unless you are developing a clustered supercomputer.
> re SWIG, I haven't used Boost to wrap C++ but I have read it has its
> advantages. SWIG itself would have a translation layer but wouldn't it
> save the memcpy() call in the pyZMQ_send() ?
No. The memcpy is related to the lifetime of the string. The wrapper
creates new string so that 0MQ can deallocate it at will. This is
actually a place where improvement would be of value. 0MQ C++ API allows
you to pass in the data buffer - which can be the native python string -
and a deallocation function that is called asynchronously when the data
are no longer needed. The challenge is make it work well with Python
memory management model.
> I believe that the memcpy or a similar mechanism would be required or a
> bunch of accounting to track the PyString object and prevent its garbage
> collection. I will need to look into how other C* I/O backends that use
> async I/O do it to be sure.
Yes. As noted above, this is a place for improvement. Maybe it's
possible to refcount the python string to prevent it from garbage
collection while it is processed by 0MQ?
More information about the zeromq-dev