[zeromq-dev] Publish to and Read from a buffer

Ziang Gao ziang.gao at aerodefense.tech
Wed Mar 18 18:57:44 CET 2020

Hi Brett,

Thank you so much for your advice!

In my application, the subscribers collect data with different topics and
have different speeds for processing. One of the problems is that the
processing time is not always the same; Also the publishers are not sending
data at a certain rate, sometimes it may send a large amount of data in a
short time and other times it may not publish data at all.

I have already used a forwarder device between multiple publishers and
subscribers, and since all the subscribers have different behaviors, it
might be better to define a buffer or queue before each subscriber and
collect data in there, so SUBs can keep their own pace. In this case, are
there any libraries or modules you can recommend that can create a queue
right after connecting the port? Or as you suggested, defining queues
within forwarder would be a better design? And how to achieve it? I tried
to use multi-thread to create multiple PUB ports after the forwarder but it
seems only the first PUB is activated.



Ziang Gao

RF Team Lead

Ziang.Gao at AeroDefense.tech | 201.899.1996

AeroDefense.tech <https://aerodefense.tech/>

AirWarden™ Drone & Pilot Detection, Location, & Alert System is the FIRST &
ONLY drone detection product to be recognized by the Department of Homeland
Security SAFETY Act <https://aerodefense.tech/dhs-safety-act-designation>.

On Wed, Mar 18, 2020 at 8:42 AM Brett Viren <brett.viren at gmail.com> wrote:

> Hi Ziang,
> When faced with similar patterns/needs in my own work I consider what
> might be done at different points in the data flow from upstream to
> downstream and what implications that might have.  Here are my thought
> processes applied to the case you describe:
> Would all subscribers want the exact same collection of individual
> messages?  If so, then for simplicity, I might develop a publisher that
> did the collecting and then emitted larger messages less frequently.  If
> this is a high rate stream and the individual messages prior to
> collecting were small then this strategy can also improve overall
> throughput (see the zeromq wiki for some throughput performance results
> as a function of message).
> Or, do the many subscribers need different collections?  If so, the
> collecting needs to be downstream of the PUB sockets.  There are two
> general locations I would consider to add collecting.
> The first is to place the collecting functionality into a zeromq "proxy"
> or "device" (as it's sometimes called).  This may be a stand-alone
> program or an actor function running in a thread of a larger
> application.  It has an input SUB and an output PUB and in between it
> performs the collection.  This allows the publisher and the subscriber
> codes to stay simple.  It allows for a variety of collection mechanism
> across the overall network.  In the case of intensive data flows, it
> spreads the processing over more cores.  If message size bandwidth
> limitations are an issue, these actors can run in the same application
> as the original PUBs and use inproc:// transport.
> Finally, to provide the benefits above comes with the complexity of more
> "moving parts" that may require configuration or deployment mechanisms.
> So, finally, I'd consider hard-wiring collection right in the
> application that holds the final SUB socket.
> What is best depends strongly on many details not yet described.  In my
> own case where I had all the detailed requirements I could handle I
> found judging which strategy is best is not at all obvious, a'priori.
> Trial and testing was needed.
> Happy flows,
> -Brett.
> Ziang Gao via zeromq-dev <zeromq-dev at lists.zeromq.org> writes:
> > Hello,
> >
> > I'm new to ZMQ and I recently started a project using the publisher and
> subscriber model.
> > However, in my application, the client who's subscribing is making
> decisions based on the
> > data collected during a small amount of time. Ideally, it would collect
> data for 1 sec then make
> > a decision(during the decision-making period it would still be
> collecting data for the next
> > decision), so I'm thinking that I can publish data into a buffer then
> grab all the data in there
> > at a certain rate.
> >
> > Does ZMQ provide a function or model that supports this kind of design?
> I know that I can
> > create a buffer in a separate thread by myself and store the data there,
> but I would like to
> > know if ZMQ had already provided a function for it before I'm heading
> that way.
> >
> > The language I'm using is Python and OS is Unbuntu 16.04.
> >
> > Thank you very much!
> >
> > Best regards,
> > Ziang
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20200318/b426eeb5/attachment.htm>

More information about the zeromq-dev mailing list