[zeromq-dev] Distributed application with REQ REPLY sockets

[Justin Azoff]

On Mon, Aug 13, 2018 at 2:07 AM Patrick Reedy <preedy73 at gmail.com> wrote:

> I’m struggling with a pipeline stage that needs to complete quickly, in less than a second, but sometimes is taking 5 or 6 seconds.
>
> In this pipeline stage each of the 50 nodes have to send their to piece of data to all of the 100 main workers - so 50 nodes each sending 100 pieces of data with each piece about 1.5M bytes. The main workers need to collect all of the pieces of data before the pipeline can advance to the next stage.

What kind of network interfaces do you have?  When this stage runs how
much data total does each physical interface send/retrieve?  Could you
be saturating the network?



-- 
- Justin