[zeromq-dev] zmq REQ to multiple REP sockets (Pieter Hintjens)

malist mailing.list at manageapp.com
Thu Feb 10 01:01:09 CET 2011


I have been experimenting with zmq and now I realize that this zfl library is exactly what I need. I'm trying to build classic database client-server communication with zmq. 

Here is what I have in my mind. I would appreciate any comments.


In my system one client must be tied and communicate with exactly one (and always the same) server side process. Trouble here is that my programming language has it's own process system and everything runs basically inside one operating system process (or thread, I'm not sure). So I have to use poll loop in my language. 

Trouble here is:
* If I have for example 50 processes then system performance with 50 poll loops would be very bad. Also I want all zmq code in to another operating system thread so that I can utilize multiple processors.

My plan:
* Every client and worker sends XREP-XREP message directly to server address.
* Client and worker machines both have 1 server process loop and all the other (worker and client) processes are paused
* When servers gets information from zfl that worker (or client) has some message already delivered to inproc memory socket then it wakes up that process
* Worker or client reads message from inproc
* Reply does not go back to client zmq socket but to client machine server process (using XREP-XREP addresses)

I have some zmq test code ready, but performance is not very good. I have also implemented this system with apache apr tcp sockes. It works, but performance is not good. Also there is a lot work to do (heartbeat, reconnect) and zmq has it all.

Also what I need here is message queue and as I understand zfl handles it.

server pseudo-code:

zmq_plugin_server_start  // starts zfl server in operating system thread
  prsNum = zmq_plugin_any_messages_incoming() 
     // are there any messages ready in any inproc channel, return client process number to wake it up
  if( prsNum > 0 )
    resume process( prsNum ) // wake up sleeping process
  smll sleep // to give time to other processes and not consume all processor power
until (quit)

Server runs in a tight loop (now about 6% of processor power) and polls if there are any messages coming to any workers or clients. This way I need only one polling process and other (worker or client) processes are paused until they get some new messages.

worker pseudo-code:

zmq_plugin_client_start( myChannel )  // joins to zfl server inproc channel
  pause process(current process)  // go to sleep = do not consume any resources
  while( zmq_plugin_message_receive( myChannel ) > 0 ) // gets message into my language variables
    ... // execute call and prepare replyMessge blob
    zmq_plugin_message_send( myChannel, replyMessge ) 
        // this send message to client machine server address uning zfl and inproc channel
  end while
until ( quit )
zmq_plugin_client_stop( myChannel )  // sends "quit" message to client process to inform that connection is now closed

client pseudo-code:

zmq_plugin_client_start( myChannel )  // joins to zfl server inproc channel
zmq_plugin_message_send( myChannel, requestMessge ) 
pause process(current process)  // go to sleep = do not consume any resources
zmq_plugin_message_receive( myChannel ) 
zmq_plugin_client_stop( myChannel ) // sends "quit" message to worker process so it can die

Both client and server have one polling server looping for incoming messages and waking up paused processes. Client may wake every 0,5 second even if there are no messages waiting to inform user about waiting or break after timeout (my calls can easily be 30 seconds long). Worker processes are paused until client disconnects. 

This way I can have system where you can use client program, close machine, go to holiday in Rio and continue using application 3 weeks later anywhere in the world (after opening ssh tunnel or something).


Pasi Mankinen
Manage Applications

More information about the zeromq-dev mailing list