[zeromq-dev] zeromq, pyzmq, pyRpc issue

Justin Israel justinisrael at gmail.com
Thu Aug 30 18:04:21 CEST 2012


I think this functionality would definitely complicate the server a bit more. Sounds like what you would need is a dedicated control channel. Something like a second publish type: publishService(service, control=True). And then the server would have to check for a control request and handle it separately, instead of sending it into the normal work queue. 

That would only be part of the problem. Then you need to explicitly track all of the clients active requests so that you even know where they are in the work queue. Running on which worker, or waiting in line. 
Then, how to kill them? What if a worker is blocking on the call? You might have to completely destroy a thread and start a new one? 

The tracking part might be the easiest since once you have marked a client ID as "killed", the workers can at least check a new request they get before starting it. 



On Aug 30, 2012, at 6:46 AM, Jan Grec <jgrec at redhat.com> wrote:

> Hello everyone!
> 
> I'm currently trying to solve a problem involving python-zmq and pyRpc.
> I'm a zmq newbie.
> 
> PyRpc (http://pypi.python.org/pypi/pyRpc/0.2) (author is CC'd).
> 
> In our example, I deploy one server (S) and (possibly) multiple clients (C).
> 
> Our server is started with N 'worker' threads.
> 
> Client connects to our pyRpc server and server.call()'s one of the
> exported functions.
> 
> Our server handles client request by assigning it to one of the
> available workers (W) using the zmq.QUEUE device to handle queuing
> and distributing requests to workers (workers are connected
> through inproc protocol).
> 
> When client sends a request, zmq.QUEUE assigns them one of the
> workers and sends the request to this worker. The worker then processes
> the request (in the worker thread, not a new thread) and sends results
> back to client.
> 
> The worker will not begin to process additional requests from the
> client until after the first/current request is completed.
> 
> IOW, the worker is blocked and cannot manage any of other of the
> client's requests (requests are waiting in queue).
> 
> _____________________
> |    Server-side      |
> |                     |
> | | W |  | W |  | W | |
> | |___|  |___|  |___| |
> |   |      |      |   |
> |    \     |     /    |
> |     \____|____/     |
> |    |           |    |
> |    | zmq.QUEUE |    |
> |    |___________|    |
> |_____/___|___\_______|
>     /    |    \
>    /     |     \
>  _|_    _|_    _|_
> | C |  | C |  | C |
> 
> The problem then, is how to handle a client's CTRL-C when what we want it to
> mean is 'hey server! stop processing this client request!'?
> 
> Right now, CTRL-C kills the client, as expected, but what needs to be done
> so that the client can also pass the message along to the server that it
> needs to kill the request on the server side as well?
> 
> I've created some example to illustrate the problem:
> simple server: http://pastebin.com/t2mk6jSQ
> simple client: http://pastebin.com/2LyTkpmU
> some notes: http://pastebin.com/371VxiE1
> 
> I'm out of ideas. If you have a solution or idea how to solve this,
> I'd be very thankful.
> 
> Thanks for your help
> Jan Grec
> 
> -- 
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Jan Grec <jgrec at redhat.com>
> Intern
> Red Hat Czech
> 
> "Never" never means never.



More information about the zeromq-dev mailing list