[zeromq-dev] Question on ZeroMQ patterns

[Felix De Vliegher]

Hi andrew

The router (or splitter, from the EIA book) would attach a unique identifier to each job and store that id and its sub-jobs in Redis. All workers would then ultimately report back to the sink, which aggregates the results of the tasks that belong together. There might be a better approach though, but this is the idea for now :)

Cheers,
Felix

On Friday 29 June 2012 at 12:57, Andrew Hume wrote:

> before i answer, how are you going to implement patterns such as aggregator from teh EIA book?
> i think that means knowing how you identify tasks/jobs and if the tracking and organising of all
> that is going to be centralised or distributed.
> 
> andrew
> 
> On Jun 29, 2012, at 3:08 AM, Felix De Vliegher wrote:
> > Hi list
> > 
> > I'm trying to set up a system where certain jobs can be executed through zeromq, but there are currently a few unknowns in how to tackle certain issues. Basically, I have a Redis queue with jobs. I pick one job from the queue and push it to a broker that distributes it to workers that handle the job.
> > 
> > So far so good, but there's a few extra requirements:
> > - one job can have multiple sub-jobs which might or might not need to be executed in a specific order. "item_update 5" could have "cache_update 5" and "clear_proxies 5" as sub-jobs). I'm currently thinking of using the routing slip pattern (http://www.eaipatterns.com/RoutingTable.html) to do this.
> > - some sub-jobs need to wait for other sub-jobs to finish first.
> > - some jobs need to be published across multiple subscribers, other jobs only need to be handled by one worker.
> > - workers should be divided into groups that will only handle specific tasks (majordomo pattern?)
> > - some workers could forward-publish something themselves to a set of subscribers
> > 
> > Right now, I have the following setup:
> > (Redis queue) <---- (one or more routers | push) -----> (pull | one or more brokers | push) -----> (pull | multiple workers | push) ----> (pull | sink)
> > 
> > 
> > The brokers and the sink are the stable part of the architecture. The routers are responsible for getting a job from the queue, deciding the sub-jobs for each job and attaching the routing slip. What I haven't done yet is implementing majordomo to selectively define workers for a certain service, so every worker can handle every task right now. The requirement that some jobs are pub/sub and other are push/pull also isn't fulfilled.
> > 
> > I was wondering if this is the right approach and if there are better ways of setting up messaging, keeping into account the requirements?
> > 
> > 
> > Kind regards,
> > 
> > Felix De Vliegher
> > Egeniq.com (http://Egeniq.com)
> > 
> > 
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org (mailto:zeromq-dev at lists.zeromq.org)
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> 
> 
> ------------------
> Andrew Hume (best -> Telework) +1 623-551-2845
> andrew at research.att.com (mailto:andrew at research.att.com) (Work) +1 973-236-2014
> AT&T Labs - Research; member of USENIX and LOPSA
> 
> 
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org (mailto:zeromq-dev at lists.zeromq.org)
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev