[zeromq-dev] Python message passing

[Erich Heine]

Thanks Martin for the great response and clarification! This email should be
reworked into an FAQ entry (or 3) or a wiki page or similar.

On Fri, Apr 24, 2009 at 2:44 AM, Martin Sstrik <sustrik at fastmq.com> wrote:
<snip>

> 1. Languages differ and generating exactly the same wrapper for each
> language would violate "style" of individual languages making it
> unpleasant and even difficult to use.
>

Im glad this decision was made. It is never fun to use a library or whatnot
that "even has python bindings" but those bindings weren't made with the
above in mind.


> 2. Hand written wrapper is always more efficient when compared to the
> generated one. This is especially true if fragile concepts like
> zero-copy are to be used.
>

I respectfully disagree. This statement assumes perfect knowledge of both
languages and the interface between them. In many cases a good interface
generator can incorporate arcane tricks and whatnot that are tedious and
ugly to write by hand.  i have personally seen this when using pyrex to wrap
c libraries.  I don't know if it is a case of my C/python extension-fu being
weak, or pyrex being awesome (most likely a combination), but the pyrex
generated interface was much faster and easier to read.


>
>  > There should however be some sort of hook to allow for low level
>  > serialization techniques e.g. easy combination with the google thing
>  > (whose name I always forget).
>
> Protocol Buffers maybe? Can you have a hint how do you imagine such a
> hook to look like?
>

Protocol Buffers is it!  Here is my idea tho:

Have a function: register_serializer(serialize_func, unserialize_func).
Then send includes the following python equivelent:

if iserialize_func is not None: serialize_func(msg.data)

and an unserialize_func call on the recv side.

I don't know how useful such a thing would be in languages that don't have
decent introspection. If it can be made to work in all the target languages,
or even most, then it does make sense from an api user's point of view. With
ths register_serializer function I don't have to wrap zeromq send and recv
myself. (of course I still could).


>  > Alternately I have been looking at and learning boost to wrap zeromq,
>  > as it seems nicer for doing C++ wrappings.
>
> Sure. Have a look at it. However, it should be noted that boost despite
> its name is not hyper-optimised in all of its components. I recall a
> bottleneck in real-world stock quote processing application that turned
> out to be caused by underperforming boost tokeniser.
>

I have heard about similar things myself. Im wondering if this matters when
interfacing with python tho. I mean I do love the language, but it certainly
is slow :)  Now that I've discovered that the mess which is bjam is not
required to do boost.python stuff, I may make a super simple wrapper today,
mimicking the current python api to see if i hit the wall earlier or not.

A tangent off of this:  I've been wondering for a while why the decision was
made to include wrappers for certain very slow languages like python.  It
seems to me that all the focus on efficiency and maximizing the use of a big
cluster, that python et al undo all of that.

Regards,
Erich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20090424/a7a80e67/attachment.htm>