[zeromq-dev] Proposal for new features in ZMQ.

[Fabien Ninoles]

I just terminated most of my porting of the P2P layer of our cluster to
ZMQ, (ab-)using a lot the collector device I have presented last week
(see http://www.tzone.org/~fabien/hg/index.cgi/collector/).  I find out
that it shared a lot with two other pattern used in ZMQ, the PUB socket
and the LRU device, and so there is probably some stuff to share to make
them more easy to implement both, which can probably be useful for other
usages.

Here the feature set:

Feature 1 - Add a new msg LABEL flag marking a frame as "labeled".

This flag would allow to distinguish between frames from the original
sender and frame added by REQ and ROUTER sockets.  This will greatly
ease the usage of any socket that connect to a ROUTER socket, including
the REQ socket itself.

Feature 2 - Embedded flags inside the msg_t structure.

By allowing frame flags to be set inside the msg_t structure, it would
be far easier to keep forward compatibility with new flags for proxy
device that need to copy msg frames in a transparent manner.  Just as an
example, a queue socket would simply be a while True:
out.send(in.recv()) loop, which will now take care of the multipart
flags without problem.

Feature 3 - Add a timeout sockopt.

The timeout would allow the recv method to return with a ETIMEOUT error
after a specified amount of time inactivity.  It can also be queried by
device to implement a similar mechanism inside the device polling
scheme.  May be the socket could also reset itself, just like if the
socket were close and reopen again (removing the need for the user to do
it itself).

Feature 4 - Add a ready sockopt.

This sockopt allow a socket to be marked as ready to read a new message.
It will be send on any messages send by the socket as well as the
connection handshake time, and will be reset on the next successful recv
of the socket (this information does not need to be propagated however).
On the other end, sockets could be setup (another sockopt) to only send
to socket which were marked as ready (either in a lb or dist manner).

Just having this option will make the LRU device a simple queue with a
special socket option.  You'll be able to use REP device on the worker
just as usual and could probably even handled disconnection of the
standby workers more efficiently.

Feature 5 - The COLLECTOR socket.

The COLLECTOR socket is related to the PATCH socket just like the REQ
socket is related to XREQ.  COLLECTOR socket would allow reception only
on sockets that it has send a message previously, and only until it
received a READY signal from this socket.  All other messages would be
dropped, although any READY signal will push back the socket in the
eligible list for the next send request.



All those features can be emulated in "client" space, but they are
clearly less efficient than doing it directly in the library itself.
Most of the current existing pattern would be greatly simplified using
them, avoiding weird usage like DEALER socket as worker end point in
LRU, and probably unseen and interesting new patterns could probably
build using them.

I'm volunteer to work on all of them, I would just like to have your
agreement on it before starting.  I'm still a very new to ZMQ and even
to network programming, so I could have miss a lot of things in this
proposal.

Thanks,
Fabien