[zeromq-dev] Question on ZeroMQ patterns
Felix De Vliegher
felix at egeniq.com
Fri Jun 29 12:08:44 CEST 2012
Hi list
I'm trying to set up a system where certain jobs can be executed through zeromq, but there are currently a few unknowns in how to tackle certain issues. Basically, I have a Redis queue with jobs. I pick one job from the queue and push it to a broker that distributes it to workers that handle the job.
So far so good, but there's a few extra requirements:
- one job can have multiple sub-jobs which might or might not need to be executed in a specific order. "item_update 5" could have "cache_update 5" and "clear_proxies 5" as sub-jobs). I'm currently thinking of using the routing slip pattern (http://www.eaipatterns.com/RoutingTable.html) to do this.
- some sub-jobs need to wait for other sub-jobs to finish first.
- some jobs need to be published across multiple subscribers, other jobs only need to be handled by one worker.
- workers should be divided into groups that will only handle specific tasks (majordomo pattern?)
- some workers could forward-publish something themselves to a set of subscribers
Right now, I have the following setup:
(Redis queue) <---- (one or more routers | push) -----> (pull | one or more brokers | push) -----> (pull | multiple workers | push) ----> (pull | sink)
The brokers and the sink are the stable part of the architecture. The routers are responsible for getting a job from the queue, deciding the sub-jobs for each job and attaching the routing slip. What I haven't done yet is implementing majordomo to selectively define workers for a certain service, so every worker can handle every task right now. The requirement that some jobs are pub/sub and other are push/pull also isn't fulfilled.
I was wondering if this is the right approach and if there are better ways of setting up messaging, keeping into account the requirements?
Kind regards,
Felix De Vliegher
Egeniq.com
More information about the zeromq-dev
mailing list