[zeromq-dev] Migrating IPC-based Pub/Sub System To ZMQ

[Santy, Michael]

Thanks Pieter, I believe that you made valid assumptions about my system.  Just to further fill in the holes:

They'll be 4 data collectors and 3 data processors initially.  I can see use cases where both of these vary between 1 and 8, so I'm trying to make the system flexible, and that's why I labeled them as 1..N.

Each data collector will be sending ~10MB messages at 2-5 Hz, depending on configuration.  The data combiner will aggregate data from multiple data collectors and send a data processor a 100-200MB message every 2-3 seconds.

I'd like the system to keep functioning when one or more data collector and data processors are offline.  The system would have to have a hot spare data combiner to be fault tolerant, as as you mentioned that may not be worth the complexity.

Due to time constraints, workloads should not be re-executed if there's some sort of failure.  It is acceptable to simply drop data (as long as its infrequent enough).

One thing that I didn't understand about your reply was the concept of PUSH/PULL sockets between the combiner and each processor.  I had planned to simply use PUB/SUB and add something to the header of the message that indicated which processor (i.e. processor ID) the message would go to.  Each processor would subscribe to the messages containing the appropriate processor ID.  It's my understanding that the filtering is done on the producer.

I appreciate your insight.

Regards,
Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100816/bca6c767/attachment.htm>