[zeromq-dev] ZeroMQ application design questions

Marc Demierre demierre.marc at lab.ntt.co.jp
Fri Nov 19 03:44:26 CET 2010


Hello everybody !

I am an intern currently developing a prototype for a distributed video
processing server in Python. I didn't use ZeroMQ in my first prototype
(only subprocesses), but now I would like to adapt it to ZeroMQ to have
better scalability (it will run on a cluster later).

However, I am still a beginner with ZeroMQ and I have some questions
about my application design. I want to separate it in the following
components:

- Controller server: The main component. It will have the following
threads:
    - Main: Manages the progress of currently running jobs and send
updates to the job source (external application). Receives jobs from
Checker, prepares progress tracking and send them to Distributor
    - Checker: Asks the job source regularly to get new processing jobs
and sends it to Main
    - Distributor: Divides the jobs in smaller tasks and give them to
workers
    - Sink: Receives the jobs progress updates and results and send them
to Main
    
- Worker process: processes the videos using C libraries (but it's in
Python too). It has to:
    - Ask the Distributor for new tasks (all tasks does not take the
same time so I think it is the best solution)
    - Process jobs
    - Send updates regularly (e.g. every 5% of the task)
    - Send task result when finished
    
Now I needed to choose which patterns to use, and I am not really sure:

- Workers are simple:
    - REQ socket to ask for jobs
    - One (or two?) PUSH socket for updates and results
  
- Controller server:
    - Sink:
        - One (or two?) PULL sockets to receive updates and results
        - A ? socket to send updates and results to Main
    - Distributor:
    	- A XREP socket to receive workers task requests
        - A ? socket to receive jobs to divide from Main
    - Checker:
        - A ? socket to send the jobs it got to Main
    - Main:
        - A ? socket to receive updates and results from Sink
	- A ? socket to receives jobs from Checker

As you can see, I am not sure about how to plug the pieces together with
ZeroMQ sockets.

My current prototype us some shared states, which are bad, and works
like this:

- Main is the main thread of the application. It manages the jobs
progresses. It acts as the central point and communications between
threads have to go through it
- Checker is created by Main and uses a method of Main to give jobs to
it
- Another thread that is called ProcessManager. It does the following:
    - Creates 3 process-safe queues (multiprocessing module):
        - A tasks queue
        - A progress queue
        - A results queue
    - Creates worker processes with the multiprocessing module
    - Receives the jobs from Main, divides them and put the tasks in the
tasks queue
    - The processes do get() on the queue to receive tasks
    - The processes put updates and results in the queues
    - The thread checks the queues and send updates and progresses to
Main
    
It works but it is not really scalable and I would like to totally avoid
threads calling methods of other threads and setting attributes.

Any advice on how to do this with ZeroMQ ?

A second thing: I wrote most of the programs of The Guide in Python. I
can't contribute them now but I will as soon as my internship is
finished (my boss allowed me to give it under my name after the end of
my contract). Where should I send them and how (pastebin, attached to
email, my Dropbox public folder, ...) ? I will do it in the beginning
January.

Thanks for reading and sorry for the long e-mail.

Marc




More information about the zeromq-dev mailing list