[zeromq-dev] content based routing for a thread pool

[Francesco]

Hi Lei,
I didn't really follow this email thread closely but I happened to read
this:

I think it suggests that as long as the "active" message has not finished,
> meaning it's still sending the "MORE" flag, a ROUTER socket will not
> receive() a frame from a different client. Given that the proxy loop is
> single threaded, that means it won't tend to other clients until the
> current one is finished. If some of the message frames have delays, they
> will surely cause delays in the processing of frames from other clients as
> well.
>
> I think this means the multipart is not meant to be used to carry really
> long repeating parts but rather just one part of the message with different
> "fields".
>

Definitively true. As far as zmq_msg_recv() and zmq_msg_send() are
concerned, they will consider a block of message parts always "all
together". For example you cannot receive only 1 part out of 2 of a ZMQ
message. And the messages cannot be delivered "partially". That's the nice
thing about zmq multipart: the parts are all delivered all together, as a
strong guarantee. This helps to reduce the complexity (you don't need to
think how to deal with a half-received ZMQ message).



> If I want to improve my threadpool to have better handling of queuing of
> the workers, also improved reliability by bringing in heartbeats, I will
> probably need to bring in the PPP. But the example in the guide seems to be
> having the proxy, ppqueue, pushing tasks to the workers. If I want to have
> the queue in the proxy, then I should have the workers "pulling" tasks. In
> the earlier part of the guide, it says this is achieved by having worker
> send a "READY" message to proxy and then proxy should reply with the task.
> So the core of my question here is, the worker can not know how long the
> task queue is in the proxy so it's possible queue is empty. In this case
> what should the proxy do? My guess is not replying to worker but do all the
> other things so that the worker should keep polling and that will be most
> efficient, right?
>

Just my 2 cents here: if the proxy has an empty queue and a worker is
asking for some work to do, then the proxy should just answer "there's
nothing for you" and keep moving. The worker OTOH can implement a simple
sleeping logic (one simple logic I used in production and turned out to
work well enough: after N _consecutive_ loops in which you get " there's
nothing for you", sleep for X msecs, then restart).

HTH,
Francesco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20231113/045dfa2a/attachment.htm>