[zeromq-dev] help/advice for using XREP pattern when running many FSM/co-routine actors in a single thread.

matthew riek matthew.riek at gmail.com
Sun Sep 25 12:14:55 CEST 2011


I have a server that I would dearly like to move from boost::asio to 0mq.
The server manages hundreds of client connections and has complex
asynchronous statefulness for each connected client.  To avoid running
hundreds to thousands of threads in this server, each client connection has
a FSM instance per client request, which all run in a single thread - so it
is very much unlike the actor-per-thread pattern which is well supported by
0mq.  I guess these FSM instances may best be likened to co-routines.

Anyway, I would love help with the following questions - I have looked at
the API and played with zmq for a few days now so I hope I'm not missing the

1) in this context, I can't readily use zmq_poll to "wake up" or schedule a
client FSM instance when there is a message to read for that client.
zmq_poll works wonderfully for device based threads as exemplified in all
the documentation - but in this context I can't see how I can refactor the
FSM manager to block on zmq_poll.  Currently all asynchronous events that
can "wake up" or schedule an FSM to run are placed in a prioritized
thread-safe queue.  Ideally, some kind of callback from zmq API to notify
that a read can be performed on the XREP/ROUTER socket would be ideal!  Is
such a setup possible with the current API?

Things are a little more difficult than this though... It is highly
desirable for a client to have a full read queue (on the server), and not
desire callback notification untill the server FSM(s) for that client are
ready to perform a read (they may be busy performing and write queueing a
time intensive complex query).  So this read notification must be requested
per client.  I understand that this however is suggesting plain-old-sockets,
and gets aways from the abstracted 0mq XREP "socket".  Perhaps a 0mq socket
pattern that may be viewed as "an acceptor and collection of
plain-old-sockets"?  This would not need the XREP fair-queued strategy, as a
properly asynchronous (adjective describing message, not API) socket pattern
queues for each socket rather than multiplexing all client queues.

2) XREP pattern drops messages when an outgoing queue for a connected client
is at HW (for reasons I understand and agree with).  In this context, I need
send to be both non-blocking (The API already allows me to know if a
particular client outgoing queue is full) and to not drop packets.  I have
read about the suggested CBFC and this might work, but callback style
notification that a queue for a particular client is no longer in an
exception (HW) state is far more ideal.  To use CBFC the current state
machines will need quite a bit of refactoring to be able to receive the
credit token from the client - unless the credit token were to come
out-of-band.  The server side FSM's read input, process then write
results... The output can be very large, and clients (which must be
XREQ/DEALER) have multiple RMI like objects that can asynchronously queue up
future requests in the send queue which makes getting the credit token to
the server difficult.  Am I missing something obvious with CBFC?  My current
reckoning is that CBFC is easiest with REQ -> XREP and perhaps more
difficult with XREQ -> XREP.

3) dropped/closed/connected client notification callback would also be
ideal.  I have seen some comments about this possibly coming for 4.0?
Thought I would just state that it is highly desirable in this instance, to
help with justification for this change to the API.

I would dearly love your comments / advice.

P.S. Well done with 0mq :)

Best regards,

Matt Riek.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20110925/f7cc56db/attachment.htm>

More information about the zeromq-dev mailing list