[zeromq-dev] Python message passing

Martin Sustrik sustrik at fastmq.com
Fri Apr 24 15:56:21 CEST 2009


Hi Erich,

> Thanks Martin for the great response and clarification! This email 
> should be reworked into an FAQ entry (or 3) or a wiki page or similar.

Good point. Will do.

> I respectfully disagree. This statement assumes perfect knowledge of 
> both languages and the interface between them. In many cases a good 
> interface generator can incorporate arcane tricks and whatnot that are 
> tedious and ugly to write by hand.  i have personally seen this when 
> using pyrex to wrap c libraries.  I don't know if it is a case of my 
> C/python extension-fu being weak, or pyrex being awesome (most likely a 
> combination), but the pyrex generated interface was much faster and 
> easier to read.

You are welcome to try. However, I wouldn't expect much improvent simply 
because the wrapper is so thin - there's very little to optimise there 
with the exception of the memcpy issue described in previous email. And 
I don't believe any code generator, however smart, would help with that one.

> Have a function: register_serializer(serialize_func, unserialize_func).  
> Then send includes the following python equivelent:
> 
> if iserialize_func is not None: serialize_func(msg.data)
> 
> and an unserialize_func call on the recv side.
>  
> I don't know how useful such a thing would be in languages that don't 
> have decent introspection. If it can be made to work in all the target 
> languages, or even most, then it does make sense from an api user's 
> point of view. With ths register_serializer function I don't have to 
> wrap zeromq send and recv myself. (of course I still could).

This won't work in strongly typed languages (C,C++,Java). By coincidence 
these languages are main focus of the project :)

> A tangent off of this:  I've been wondering for a while why the decision 
> was made to include wrappers for certain very slow languages like 
> python.  It seems to me that all the focus on efficiency and maximizing 
> the use of a big cluster, that python et al undo all of that.

Several reasons:

1. Rapid prototyping.
2. High-perf system written primarily in C can still benefit from 
administrative components written in Python. With administration the 
performance is irrelevant. Ability to write/change code easily is much 
more important. (It would be nice to have Perl binding for exactly this 
reason.)
3. Even python can benefit from features like ability to handle 
thousands of connections at the same time, fast disk offload of 
messages, auto-reconnection as application fail and get restarted etc.

Martin



More information about the zeromq-dev mailing list