[zeromq-dev] zeromq & pylons

[Nicholas Piël]

On Aug 4, 2010, at 1:17 AM, Eric Bell wrote:

> I wanted to use the PUBLISH-SUBSCRIBE pattern so that any number of
> consumers could process the results coming from one specific job. For
> example, I could have a consumer whose job was to generate PDF reports,
> or another consumer stored results in a database.

...

> It sounds like I need to implement a "boundary layer" between the
> multi-threaded web server and the single-threaded 0MQ sockets. This
> boundary layer will be in charge of collecting messages on each job and
> vending out information to whomever asks for it. This is a totally
> viable approach, but requires a shift in my thinking ... originally I
> was using 0MQ as both a communications and storage mechanism. If I
> create a "boundary layer" then 0MQ is only a communications mechanism. 


Correct! You will need to figure out the right way to set up the 
'boundary layer'. I think you can the one of the following two 
approaches, but you will need to use something extra than just PUB/SUB.


 * Fully distributed:                        
       
        Pylons --[PUSH+PULL]--> Worker
        Pylons <--[ PUB+SUB ]-- Worker

   Have each Pylons thread generate some sort of UUID, then on a
   browser request PUSH the task downstream from the Pylons thread to
   the workers including the UUID of the thread in the message. You
   don't really want pub/sub or rep/req here because it is more      
   difficult to roundrobin the tasks over FREE workers. This
   functionality comes basically for free when using the pipeline
   pattern. The thread also subscribes to its own channel as identified
   by the generated UUID. When the worker is finished on the job it
   PUBlishes the result to the correct channel as extracted from the
   request message. 

   A fully distributed approach can be difficult when both the Pylons 
   threads AND the workers are volatile. As you will need to set up 
   the connections between the Pylons threads and the workers. You 
   could use something like Zookeeper to do this for you or you might
   even want to create your own node-discovery algorithm based on
   ZeroMQ multicast.


 * Using a Job server:  
  
         Pylons --[REQ]--> JobServer
                           JobServer --[PUSH+PULL]--> Workers
                           JobServer <--[PUSH+PULL]-- Worker
         Pylons <--[REP]-- JobServer

   A more practical approach can be to just use an intermediate 
   server to which both the Pylons threads and the Workers connect to. 
   The JobServer simply forwards incoming requests to workers, collects 
   the result and sends the reply back to the client. When using such a 
   combination of REQ/REP and PUSH/PULL in your JobServer you can even 
   use multiple JobServers for redundancy as REQ/REP can roundrobin your 
   requests.