[zeromq-dev] Publish to and Read from a buffer

Brett Viren brett.viren at gmail.com
Wed Mar 18 13:42:50 CET 2020

Hi Ziang,

When faced with similar patterns/needs in my own work I consider what
might be done at different points in the data flow from upstream to
downstream and what implications that might have.  Here are my thought
processes applied to the case you describe:

Would all subscribers want the exact same collection of individual
messages?  If so, then for simplicity, I might develop a publisher that
did the collecting and then emitted larger messages less frequently.  If
this is a high rate stream and the individual messages prior to
collecting were small then this strategy can also improve overall
throughput (see the zeromq wiki for some throughput performance results
as a function of message).

Or, do the many subscribers need different collections?  If so, the
collecting needs to be downstream of the PUB sockets.  There are two
general locations I would consider to add collecting.

The first is to place the collecting functionality into a zeromq "proxy"
or "device" (as it's sometimes called).  This may be a stand-alone
program or an actor function running in a thread of a larger
application.  It has an input SUB and an output PUB and in between it
performs the collection.  This allows the publisher and the subscriber
codes to stay simple.  It allows for a variety of collection mechanism
across the overall network.  In the case of intensive data flows, it
spreads the processing over more cores.  If message size bandwidth
limitations are an issue, these actors can run in the same application
as the original PUBs and use inproc:// transport.

Finally, to provide the benefits above comes with the complexity of more
"moving parts" that may require configuration or deployment mechanisms.
So, finally, I'd consider hard-wiring collection right in the
application that holds the final SUB socket.

What is best depends strongly on many details not yet described.  In my
own case where I had all the detailed requirements I could handle I
found judging which strategy is best is not at all obvious, a'priori.

Trial and testing was needed.

Happy flows, 

Ziang Gao via zeromq-dev <zeromq-dev at lists.zeromq.org> writes:

> Hello,
> I'm new to ZMQ and I recently started a project using the publisher and subscriber model.
> However, in my application, the client who's subscribing is making decisions based on the
> data collected during a small amount of time. Ideally, it would collect data for 1 sec then make
> a decision(during the decision-making period it would still be collecting data for the next
> decision), so I'm thinking that I can publish data into a buffer then grab all the data in there
> at a certain rate.
> Does ZMQ provide a function or model that supports this kind of design? I know that I can
> create a buffer in a separate thread by myself and store the data there, but I would like to
> know if ZMQ had already provided a function for it before I'm heading that way. 
> The language I'm using is Python and OS is Unbuntu 16.04.
> Thank you very much!
> Best regards,
> Ziang
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 832 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20200318/8740ae83/attachment.sig>

More information about the zeromq-dev mailing list