[zeromq-dev] PUB socket dropping old messages instead of new ones

Auer, Jens jens.auer at cgi.com
Mon Dec 12 08:36:53 CET 2016


Hi Luca,

I didn't look at the Radio/Dish sockets because we use 4.1 and there is not intention to switch to 4.2 soon. I hoped to do it in the xpub class or at least without changing the internal pipe to avoid breaking other things. I am not experienced with implementing lock-free data structures (but it does sound interesting...) and I could not find any reference for this. 

Usually I would use a ringbuffer implement this behavior, but when it overwrites data there is a shared access to the memory which makes it problematic for a lock-free data structure. Ringbuffers are common to implement lock-free queues, but they never overwrite. I found one library which offers a lock-free ringbuffer with overwrite behavior (liblfds), so I will take a look. I suspect that taking a dependency just for this is not desired?

While looking at the source code, I saw that dbuffer used by ypipe_conflate uses a mutex. Shouldn't that be a performance issue? I think it could be implemented with atomic_ptr.

Cheers,
Jens

--
Dr. Jens Auer | CGI | Software Engineer
CGI Deutschland Ltd. & Co. KG
Rheinstraße 95 | 64295 Darmstadt | Germany
T: +49 6151 36860 154
jens.auer at cgi.com
Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter de.cgi.com/pflichtangaben.

CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI Group Inc. and its affiliates may be contained in this message. If you are not a recipient indicated or intended in this message (or responsible for delivery of this message to such person), or you think for any reason that this message may have been addressed to you in error, you may not use or copy or deliver this message to anyone else. In such case, you should destroy this message and are asked to notify the sender by reply e-mail.


> -----Original Message-----
> From: zeromq-dev [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of
> Luca Boccassi
> Sent: 05 December 2016 14:20
> To: ZeroMQ development list
> Subject: Re: [zeromq-dev] PUB socket dropping old messages instead of new ones
> 
> Have you looked into implementing that in the radio/dish sockets?
> 
> Since it's still in DRAFT state, it will be easier to change.
> 
> Changing how the internal pipe works for the existing sockets without breaking
> compatibility might be hard.
> 
> On Mon, 2016-12-05 at 12:47 +0000, Auer, Jens wrote:
> > Hi,
> >
> > I am using a XPUB socket to distribute data between three sites in several
> channels. For our application, it would be better if the socket would drop old
> messages when it reaches the HWM. I searched the mailing list and found some old
> questions asking about this, and an answer that it has been requested but never
> implemented, although it should be fairly easy. So I had a quick look at the source
> code, but I don't think that it is as easy as expected.
> >
> > The internal pipe is implemented with the assumption that each end is not
> modified concurrently, i.e. there are never two concurrent writes are reads.
> Concurrent read and write is of course expected. From what I see, this makes it
> hard to drop the oldest message in the queue because the queue has to be modified
> from the writer thread. Does anybody have an idea on how to do this? I think it
> could be possible to send to a command to the writer thread to drop a message, but
> that does not sound very efficient. If somebody could point me to the right direction
> I can pick up from there.
> >
> > Cheers,
> > Jens
> >
> > --
> > Jens Auer | CGI | Software-Engineer
> > CGI (Germany) GmbH & Co. KG
> > Rheinstraße 95 | 64295 Darmstadt | Germany
> > T: +49 6151 36860 154
> > jens.auer at cgi.com<mailto:jens.auer at cgi.com>
> > Unsere Pflichtangaben gemäß § 35a GmbHG / §§ 161, 125a HGB finden Sie unter
> de.cgi.com/pflichtangaben<http://de.cgi.com/pflichtangaben>.
> >
> > CONFIDENTIALITY NOTICE: Proprietary/Confidential information belonging to CGI
> Group Inc. and its affiliates may be contained in this message. If you are not a
> recipient indicated or intended in this message (or responsible for delivery of this
> message to such person), or you think for any reason that this message may have
> been addressed to you in error, you may not use or copy or deliver this message to
> anyone else. In such case, you should destroy this message and are asked to notify
> the sender by reply e-mail.
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 



More information about the zeromq-dev mailing list