[zeromq-dev] Question on ZeroMQ patterns

Andrew Hume andrew at research.att.com
Fri Jun 29 12:57:24 CEST 2012


before i answer, how are you going to implement patterns such as aggregator from teh EIA book?
i think that means knowing how you identify tasks/jobs and if the tracking and organising of all
that is going to be centralised or distributed.

	andrew

On Jun 29, 2012, at 3:08 AM, Felix De Vliegher wrote:

> Hi list
> 
> I'm trying to set up a system where certain jobs can be executed through zeromq, but there are currently a few unknowns in how to tackle certain issues. Basically, I have a Redis queue with jobs. I pick one job from the queue and push it to a broker that distributes it to workers that handle the job.
> 
> So far so good, but there's a few extra requirements:
> - one job can have multiple sub-jobs which might or might not need to be executed in a specific order. "item_update 5" could have "cache_update 5" and "clear_proxies 5" as sub-jobs). I'm currently thinking of using the routing slip pattern (http://www.eaipatterns.com/RoutingTable.html) to do this.
> - some sub-jobs need to wait for other sub-jobs to finish first.
> - some jobs need to be published across multiple subscribers, other jobs only need to be handled by one worker.
> - workers should be divided into groups that will only handle specific tasks (majordomo pattern?)
> - some workers could forward-publish something themselves to a set of subscribers
> 
> Right now, I have the following setup:
> (Redis queue) <---- (one or more routers | push) -----> (pull | one or more brokers | push) -----> (pull | multiple workers | push) ----> (pull | sink)
> 
> 
> The brokers and the sink are the stable part of the architecture. The routers are responsible for getting a job from the queue, deciding the sub-jobs for each job and attaching the routing slip. What I haven't done yet is implementing majordomo to selectively define workers for a certain service, so every worker can handle every task right now. The requirement that some jobs are pub/sub and other are push/pull also isn't fulfilled.
> 
> I was wondering if this is the right approach and if there are better ways of setting up messaging, keeping into account the requirements?
> 
> 
> Kind regards,
> 
> Felix De Vliegher
> Egeniq.com
> 
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev


------------------
Andrew Hume  (best -> Telework) +1 623-551-2845
andrew at research.att.com  (Work) +1 973-236-2014
AT&T Labs - Research; member of USENIX and LOPSA




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120629/9d9c34de/attachment.htm>


More information about the zeromq-dev mailing list