[zeromq-dev] Twisted integration, part 2: review
nicholas at nichol.as
Fri Jun 4 19:41:39 CEST 2010
On Jun 4, 2010, at 6:31 PM, Laurens Van Houtven wrote:
> Some people might remember the twisted integration idea from a few
> weeks back, that kind of fizzled out for a bunch of reasons, the one
> that we can all help fix is a technical one. I've written a mail
> intended for the Twisted mailing list, and I was wondering if people
> could give it a quick review from the ZeroMQ side of things: I want to
> be sure I'm not misrepresenting or just plain lying about ZeroMQ.
> You can find the mail here: http://paste.pocoo.org/raw/221958/
I quickly summarize the 4 possible different approaches as you suggested.
1. Implement everything in Python
2. Write a new thin layer to the C++ API
3. Use PyZMQ's provided select
4. Use PyZMQ's Poll as a reactor
I agree with your reasoning that option 1 isn't really an option. Nor is option 2, why go duplicating the work already done on PyZMQ. PyZMQ follows 0MQ well and has a pretty thorough test suite.
So, i really believe option 3 and option 4 are the only viable options. I am not so sure if it is a good idea to disregard option 4 at this point. Ie, doesn't Twisted already have the option to choose an EPOLL reactor? I think a reactor based on ZMQ's Poll would be a fine substitute for people just wanting that kind of integration. I am currently doing this in my current approach (a Stackless Python based event loop around 0MQ's poll).
While I would prefer to have my main event loop based around regular EPOLL (for performance reasons) it is difficult to combine them at the moment as you have to handle two different pollers at the same time. There is no way to get a signal from ZMQ yet to easily integrate it in an external event loop. Thus i don't think option 3 is a viable option at this moment, you could do it but you would basically end up with something that looks like busy looping a non blocking PyZMQ select or blocking the Twisted event loop. There has been some discussion on the mailing list so i expect that features that will make this possible in the correct way will probably be added in the near future.
What you could do, for now, is have ZMQ be your main poller, option 4, and than have it also poll for the signal fd of the other poller. Epoll for example, has an fd which is pollable i can imagine that the other reactors have something similar. Another approach is to have the two pollers run in separate threads, but you would still need inter-thread communication for that which will add to the latency.
But, I think the best would be to wait a little bit before throwing this at the Twisted community until ZMQ's Poller/Pollset/wait_fd interface has had a little more thought (and preferably has been implemented). Not only would this open the road to option 3, but I also think it would otherwise be difficult to have a decent discussion with a large group of people when the interface in question is still in flux.
More information about the zeromq-dev