[zeromq-dev] Non-blocking calls and Twisted

Laurens Van Houtven lvh at laurensvh.be
Mon Apr 12 22:25:59 CEST 2010


Glad to hear it, I'm a twisted dev so I'm pretty biased in favor of twisted
;-)

Anyway, I can confirm this would require a new reactor, which would be
implemented somewhat analogously to twisted.internet.ThreadedSelectReactor.
I haven't quite sudied the intricacies of zmq's select yet, but I think
you'd need a subclass of t.i.p.PosixReactorBase and a new
twisted.internet.interfaces class IReactorZMQ, with methods connectZMQ and
listenZMQ :-)

The alternative not involving a reactor would mean reimplementing all of
ZeroMQ in Python, bypassing all of the stuff ZeroMQ does now on real
sockets. I don't think that's an option, but if anyone feels like they have
too much time on their hands...

thanks
Laurens

On Mon, Apr 12, 2010 at 8:37 PM, Brian Granger <ellisonbg at gmail.com> wrote:

> Laurens,
>
> Perhaps I can rephrase your question and then answer it....
>
> How would one use pyzmq with twisted?
>
> Because 0MQ/pyzmq implement their own poll/select functions, you will need
> to
> create a new reactor type that calls the zmq poll/select function.
> Then you will need
> to wrap the pyzmq Socket type into something that Twisted knows how to
> handle.  I
> forget the exact details in twisted, but I think this will involve the
> Transport API.
> I would love to ship a 0MQ reactor with pyzmq itself.  I have already
> worked on
> a pyzmq event loop for tornado, and I use twisted a lot, so this
> definitely makes sense.
> I am pretty busy right now, but I am more than willing to help in any way I
> can.
>
> Cheers,
>
> Brian
>
>
>
> On Mon, Apr 12, 2010 at 5:34 AM, Laurens Van Houtven <lvh at laurensvh.be>
> wrote:
> > Dear Martin,
> >
> > Thanks for your answer :-) Specifically ZMQ_NOBLOCK is interesting. I
> don't
> > really care how 0MQ does stuff under the hood as long as it performs, but
> > I'd like a Python API that behaves event-drivenly (i.e. non-blocking IO
> and
> > callbacks) if possible, if only because Twisted contains fully functional
> > SSL and HTTP servers (and code I don't have to write is way better than
> code
> > I do have to write).
> > thanks
> > Laurens
> >
> > On Mon, Apr 12, 2010 at 8:14 AM, Martin Sustrik <sustrik at 250bpm.com>
> wrote:
> >>
> >> Hi Laurens,
> >>
> >> > I've noticed that 0MQ implementations are (at least the ones I've
> seen)
> >> > exclusively blocking connection per thread. Is there a particular
> reason
> >> > for this? I'd like to use Twisted and talk to 0MQ (from the role as a
> >> > publisher) asynchronously. Are there any important caveats there?
> >>
> >> I am not 100% sure what you are asking about so let me give couple of
> >> answers and you pick the right one :)
> >>
> >> 1. zmq_send/zmq_recv calls are not necessarily blocking. Use ZMQ_NOBLOCK
> >> flag to make them non-blocking.
> >>
> >> 2. If you want to handle many physical connections at the same time, 0MQ
> >> is doing this under the hood. Single 0MQ socket can be backed by 10,000
> >> TCP connections.
> >>
> >> 3. If you still need multiple 0MQ sockets you can multiplex using
> >> zmq_poll() function.
> >>
> >> 4. If you want to multiplex between 0MQ sockets and POSIX sockets, you
> >> can still use zmq_poll().
> >>
> >> 5. If you still need to use native multiplexing mechanism, it's not
> >> possible. There's an explanation on FAQ page:
> >>
> >> "Why can't I use standard I/O multiplexing functions such as select() or
> >> poll() on ØMQ sockets?"
> >>
> >> "ØMQ socket is not a standard POSIX socket. It would be great if it was,
> >> however, POSIX doesn't provide a mechanism to simulate file descriptors
> >> in user space. To convert ØMQ sockets into POSIX file descriptors we
> >> would have either to move ØMQ to kernel-space or hack the kernel to
> >> provide the functionality needed. In both cases we would have to
> >> sacrifice portability and stick to a single operating system."
> >>
> >> Martin
> >> _______________________________________________
> >> zeromq-dev mailing list
> >> zeromq-dev at lists.zeromq.org
> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
>
>
>
> --
> Brian E. Granger, Ph.D.
> Assistant Professor of Physics
> Cal Poly State University, San Luis Obispo
> bgranger at calpoly.edu
> ellisonbg at gmail.com
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100412/123bb24d/attachment.html>


More information about the zeromq-dev mailing list