[zeromq-dev] Non-blocking calls and Twisted

Brian Granger ellisonbg at gmail.com
Mon Apr 12 22:36:48 CEST 2010


Laurens,


> Glad to hear it, I'm a twisted dev so I'm pretty biased in favor of twisted
> ;-)

Great, so you area ahead of me on this!

> Anyway, I can confirm this would require a new reactor, which would be
> implemented somewhat analogously to twisted.internet.ThreadedSelectReactor.
> I haven't quite sudied the intricacies of zmq's select yet, but I think
> you'd need a subclass of t.i.p.PosixReactorBase and a new
> twisted.internet.interfaces class IReactorZMQ, with methods connectZMQ and
> listenZMQ :-)

Yep, that is the type of thing I was thinking, but you are definitely
more familiar with this part of Twisted than myself.

> The alternative not involving a reactor would mean reimplementing all of
> ZeroMQ in Python, bypassing all of the stuff ZeroMQ does now on real
> sockets. I don't think that's an option, but if anyone feels like they have
> too much time on their hands...

The wire protocol is pretty simple - I don't think that would be too
difficult.  But, the hard part is the messaging queueing algorithms.
That might take a bit more work.  Would be nice to have though...

Cheers,

Brian

> thanks
> Laurens
> On Mon, Apr 12, 2010 at 8:37 PM, Brian Granger <ellisonbg at gmail.com> wrote:
>>
>> Laurens,
>>
>> Perhaps I can rephrase your question and then answer it....
>>
>> How would one use pyzmq with twisted?
>>
>> Because 0MQ/pyzmq implement their own poll/select functions, you will need
>> to
>> create a new reactor type that calls the zmq poll/select function.
>> Then you will need
>> to wrap the pyzmq Socket type into something that Twisted knows how to
>> handle.  I
>> forget the exact details in twisted, but I think this will involve the
>> Transport API.
>> I would love to ship a 0MQ reactor with pyzmq itself.  I have already
>> worked on
>> a pyzmq event loop for tornado, and I use twisted a lot, so this
>> definitely makes sense.
>> I am pretty busy right now, but I am more than willing to help in any way
>> I can.
>>
>> Cheers,
>>
>> Brian
>>
>>
>>
>> On Mon, Apr 12, 2010 at 5:34 AM, Laurens Van Houtven <lvh at laurensvh.be>
>> wrote:
>> > Dear Martin,
>> >
>> > Thanks for your answer :-) Specifically ZMQ_NOBLOCK is interesting. I
>> > don't
>> > really care how 0MQ does stuff under the hood as long as it performs,
>> > but
>> > I'd like a Python API that behaves event-drivenly (i.e. non-blocking IO
>> > and
>> > callbacks) if possible, if only because Twisted contains fully
>> > functional
>> > SSL and HTTP servers (and code I don't have to write is way better than
>> > code
>> > I do have to write).
>> > thanks
>> > Laurens
>> >
>> > On Mon, Apr 12, 2010 at 8:14 AM, Martin Sustrik <sustrik at 250bpm.com>
>> > wrote:
>> >>
>> >> Hi Laurens,
>> >>
>> >> > I've noticed that 0MQ implementations are (at least the ones I've
>> >> > seen)
>> >> > exclusively blocking connection per thread. Is there a particular
>> >> > reason
>> >> > for this? I'd like to use Twisted and talk to 0MQ (from the role as a
>> >> > publisher) asynchronously. Are there any important caveats there?
>> >>
>> >> I am not 100% sure what you are asking about so let me give couple of
>> >> answers and you pick the right one :)
>> >>
>> >> 1. zmq_send/zmq_recv calls are not necessarily blocking. Use
>> >> ZMQ_NOBLOCK
>> >> flag to make them non-blocking.
>> >>
>> >> 2. If you want to handle many physical connections at the same time,
>> >> 0MQ
>> >> is doing this under the hood. Single 0MQ socket can be backed by 10,000
>> >> TCP connections.
>> >>
>> >> 3. If you still need multiple 0MQ sockets you can multiplex using
>> >> zmq_poll() function.
>> >>
>> >> 4. If you want to multiplex between 0MQ sockets and POSIX sockets, you
>> >> can still use zmq_poll().
>> >>
>> >> 5. If you still need to use native multiplexing mechanism, it's not
>> >> possible. There's an explanation on FAQ page:
>> >>
>> >> "Why can't I use standard I/O multiplexing functions such as select()
>> >> or
>> >> poll() on ØMQ sockets?"
>> >>
>> >> "ØMQ socket is not a standard POSIX socket. It would be great if it
>> >> was,
>> >> however, POSIX doesn't provide a mechanism to simulate file descriptors
>> >> in user space. To convert ØMQ sockets into POSIX file descriptors we
>> >> would have either to move ØMQ to kernel-space or hack the kernel to
>> >> provide the functionality needed. In both cases we would have to
>> >> sacrifice portability and stick to a single operating system."
>> >>
>> >> Martin
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> >
>>
>>
>>
>> --
>> Brian E. Granger, Ph.D.
>> Assistant Professor of Physics
>> Cal Poly State University, San Luis Obispo
>> bgranger at calpoly.edu
>> ellisonbg at gmail.com
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>



-- 
Brian E. Granger, Ph.D.
Assistant Professor of Physics
Cal Poly State University, San Luis Obispo
bgranger at calpoly.edu
ellisonbg at gmail.com



More information about the zeromq-dev mailing list