[zeromq-dev] on scalability of PUB/SUB and PUSH/PULL

Jun Li jltz922181 at gmail.com
Wed Jun 11 11:05:13 CEST 2014


I am using PUB/SUB socket pattern to distribute commands from the
coordinator to the many worker processes, and I also have the PUSH/PULL to
have each worker process to push the processing results to the coordinator.
The coordinator is bound to the PUB socket and also the PULL socket, with
the current context to set to 1 thread.   In my test environment, there
would be one single coordinator process and up to 200 worker processes.

I have just started the scalability testing. But it seems that with 15
worker processes, the end-to-end communication latency is about 15 ms, for
the coordinator to distribute (via PUB) the commands and finally aggregate
the results back (via PULL) from the worker processes. But when I increased
the number of worker processes to 50, I then observed the end-to-end
communication latency of about 80 ms. This implies that as the number of
the worker processes grow, the latency also grows and thus brings up the
scalability issue.

The message size communicated between the coordinator and the worker
processes are not that big, less than 100 Bytes.

While I am planning to measure the latency spent on each hop, I would like
to seek suggestions:

*for a large number of the worker processes to be handled by a single
coordinator with low latency, should the context at the coordinator be set
to >  1 thread?

*Should I use the other socket pattern such as Router/Dealer, instead of
pub/sub and push/pull, in order to address the scalability issue?


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140611/c02aabd4/attachment.htm>

More information about the zeromq-dev mailing list