[zeromq-dev] Non-blocking calls and Twisted

[Laurens Van Houtven]

Right, so I've had a little bit more time to look at this, and there are two
ways of doing it:
    1. ThreadedSelectReactor (aka DreadedSelectReactor,
DreadfulSelectReactor, TerribleSelectReactor and TorturedSelectReactor),
hereafter abbreviated to TSR
    2. Reimplement ZMQ in Python/Twisted (I've picked up this is what the
Erlang bindings did?)

Writing a custom reactor, while it would work, is suboptimal because it
would disable mainloop integration with stuff like Qt, Gtk... (mostly GUI
event loops). The only remaining problem is that you couldn't use the stuff
Twisted supports outside of select() calls (epoll on Linux2.6, kqueue on
BSD, IOCP on win32), because you can only use exactly one reactor (well,
sort-of, but that would get us into way more detail than we probably should
here), and TSR doesn't do any of those things.

The only way to retain them ({epoll, kqueue, iocp}) would be to ignore
ZeroMQ's current implementation of it. I don't really understand ZeroMQ's
internals yet, so I'm not really sure how much of it could be reused
(although presumably quite a bit, through Cython). Twisted will happily give
you bytes and pretend everything's a stream, so TCP would be fairly easy to
implement. I don't think interthread/IPC would ever work -- IPC might,
interthread would just be ugly (and threads are mostly an anti-pattern in
Twisted code to make up for silly things that don't have async interfaces
anyway). Twisted does do multicast, but AFAIK it doesn't do RFC 3208 PGM.

So, you could write Cython extensions that call the C++ code, but I'm not
sure where to tap in to it and just how well that would work, mostly because
I know nothing of 0MQ's C++ internals. Additionally, I'm just looking at ZMQ
because we're using AMQP in production now and I wanted to see what all of
the fuss is about, six-months-in-the-future-me maintaining something that he
started writing six months ago because of a piece of software he used for a
week is not a good place for future-me to be :-)

Like you said, binary protocols are easy, queuing is hard, best case
scenario we can keep both. Where exactly does the code stop requiring real
sockets and does it become happy with bytestrings?

tia
lvh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100413/55a0fc0d/attachment.htm>