[zeromq-dev] San Francisco meetup today!
Martin Sustrik
sustrik at 250bpm.com
Wed Mar 9 22:53:05 CET 2011
Hi all,
> As you most likely know we had tonight the first (ever?) ZeroMQ meetup
> in San Francisco, kindly hosted by Loggly at 78 1st St in the heart of
> the city.
Maxime, thanks for summarising the discussion!
Also thanks to Jon Gifford for hosting the event and paying the bill in
the brewery :)
> One topic we discussed was how we should handle changes that break
> compatibility of the wire protocol used by ZeroMQ. Given the size of the
> community, it was agreed that if this should happen, it would better be
> done sooner rather than later, as a new major revision (and thus most
> likely in a new repository like zeromq3), in an extensible way and with
> a long-enough period of warning before the change. The rationale behind
> this discussion is that some of the desired features (see below) like
> port multiplexing require some enhancements to the headers sent by
> ZeroMQ.
One interesting point -- at least for me -- was that people are not that
opposed to backward incompatible changes, given they are clearly
delineated, say by increasing the major version number.
The point is that the changes proposed so far would affect only wire
format and C API. That would cause problem to binding maintainers, who
would have to upgrade the bindings, but it would leave non-C users
unaffected.
> * graceful validation of a connection, i.e. have a clearly defined
> behavior when someone tries to connect to incompatible socket types
> together. This would be achieved by transmitting the socket type on
> the wire at connection time, with some check on the server side to
> validate this is an acceptable mix and match;
In more general terms, the wire format has to be extended to contain
more information about the peer. The other peer can that use the
provided information to either accept or reject the connection.
> * port multiplexing, which would allow sharing of a single listening
> port for multiple sockets, using the concept of "sub ports". This
> would obviously only be possible within the same ZMQ context
> (because ultimately only one bind(port) can be requested from the
> kernel for a given port number), but would allow someone to attach
> several sockets to this one listening port, and to connect to
> different sockets on one remote port.
This was the most often voiced requirement AFAICS. It seems that TCP
ports are considered to be rather precious real estate and people are a
bit nervous about using large number of them.
> * integrate handling of timeouts, re-sends w/ de-dup and eventually
> checkpointing in REP/REQ sockets, directly as base socket features
> handled by ZMQ.
>
> Timeouts would make using REP/REQ sockets easier so you don't get
> stuck in the recv() state if you never get an answer.
POSIX defines SO_SNDTIMEO and SO_RCVTIMEO to do this. It's not
implemented by all OSes, nonetheless, we can use this kind of socket
options for 0MQ sockets.
> Re-sends would make ZMQ automatically, at periodic and eventually
> configurable intervals, resend the last message until you get an
> answer (useful if the other end disappears after receiving the
> message but before sending a reply). De-duping (on the receiving
> end) is just a required feature in this case where you're still
> alive but haven't answered *yet*.
Yes.
Another comment was about IPC performance. It was pointed out that IPC
roundtrip in 0MQ (done over UNIX domain sockets) takes tens of
microseconds, while say 29West's LBM claims sub microsecond IPC roundtrip.
> Note that this is a very, very rough summary of what we discussed, and
> may not be entirely accurate -- and for that I apologize. Feel free to
> comment, correct, criticize and contribute to these ideas! ZeroMQ can
> only become better for us users iff we express our needs somehow.
I think the face-to-face meetings really help with people expressing
their opinions, criticism, ideas etc. We should do that more often.
Anyway, in the meantime feel free to discuss on the mailing list.
> Martin also reported that he applied on behalf of ZeroMQ as a Google
> Summer of Code organization, and apparently discussion already started
> on that particular subject on this very list. Having SoC students work
> on ZMQ would be a great thing IMHO, so we should make sure we have some
> project ideas (that should be fine) and that we'll welcome candidates
> with open arms :)
The ideas page is here:
http://www.zeromq.org/topics:google-summer-of-code
Add more ideas, discuss the ideas already there etc.
However, make sure that ideas you propose are suitable for GSoC, i.e.
are more or less self-contained, can be done by a single student during
the summer etc.
> Finally, we briefly talked about automated testing, in particular for
> long-lasting "burn tests" scenarios that would help test core changes to
> ZeroMQ both at the C API level and at the language bindings level. Feel
> free to propose/submit test scenarios and/or their implementations!
There are two problems here AFAICS:
1. Collecting the test cases/implementations -- if you do 0MQ tests,
don't throw them out afterwards, instead share them with others.
2. Actually running the tests. It would be preferable if that can be
automated, however, any non-trivial test case will be distributed
(running on multiple boxes, starting individual applications in
pre-defined order etc.) which makes the whole think quite difficult to
implement. Thoughts?
Martin
Martin
More information about the zeromq-dev
mailing list