[zeromq-dev] ZMQ - Send std::shared_ptr via inproc pub-sub?

BJovke bjovan at gmail.com
Wed Jun 24 23:38:17 CEST 2020


Great!

On Wed, Jun 24, 2020, 23:29 Thomas Rodgers <rodgert at twrodgers.com> wrote:

> Unless you positively acknowledge receipt there’s no completely reliable
> way to avoid this. In my case it was sufficient to periodically scan the
> table for ‘uncollected’ refs and delete them. The receiving code still
> needs to be prepared for failure.
>
> On Jun 24, 2020, at 1:53 PM, BJovke <bjovan at gmail.com> wrote:
>
> 
> That's a good solution.
>
> But what happens if receiver never receives the message or doesn't take
> the shared_ptr out of the table for some other reason?
>
> It will be there forever.
>
> Yes, you're right, sending a pointer to shared_ptr is not good because
> message is sent asynchronously. It could be done by finding a way to know
> that the message has been received.
>
> On Wed, Jun 24, 2020, 22:31 Thomas Rodgers <rodgert at twrodgers.com> wrote:
>
>>
>>
>> On Jun 24, 2020, at 12:16 PM, Serguei Khasimkhanov <SerKhas at hotmail.com>
>> wrote:
>>
>> 
>> Thank you for the answers. I will reply to everyone in this email:
>>
>>
>> > BJovke: You can send a pointer to your shared_ptr and then the
>> receiver should copy construct (it's not an actual copy but a reference)
>> it's own shared_ptr from it and forget about this pointer.
>>
>> But the problem is that the original shared pointer may get destroyed or
>> replaced before the receiver can make a copy.
>>
>>
>> > Rodrigo: If you can change the class definition, you could use the
>> seldom seen std::enable_shared_from_this.
>>
>> If I send a pointer to the object, and the original shared_ptr to the
>> object goes out of scope, the object will be destroyed, and I won't be able
>> to `std::enable_shared_from_this`.
>>
>>
>> > Sergei: You should serialize your object on publisher side -> send ->
>> receive -> deserialize on subscriber. By object I mean target object on
>> which shared_ptr points.
>>
>> If I serialize a shared_ptr, it won't get copy-constructed, so the
>> reference count won't increase, and the owned pointer could get freed
>> before any receivers receive the message.
>>
>>
>> > Rafal: Don't you think that sending a pointer to a memory is very error
>> prone? e.g.: object may be destroyed before the receiver gets a serialized
>> copy of shared_ptr as ref_count will not consider the fact that there is an
>> instance of pointer in the ZMQ inproc buffer. IMO it would be better to
>> send an object identifier which could be used by a receiver to retrieve
>> data from some data provider.
>>
>> Yes that's what I'm worried about. I don't know when to delete the
>> pointer because I don't know when everyone is done using it. That data
>> provider idea is my backup plan - if a receiver wants a copy of the data,
>> they send a separate data request to the publisher (through a separate
>> socket), and the publisher sends each one a properly-constructed copy of
>> the std::shared_ptr.
>>
>>
>> I have code which has done this. As has been pointed out, this is tricky
>> as you need to ensure the lifetime of the thing being pointed to until the
>> receiver has received the message and taken ownership of the shared
>> reference.
>>
>> The way I ended up managing it was storing the shared_ptr in a table
>> hashed by the pointer value (each hash bucket having it’s own mutex to
>> reduce contention) and then sending the pointer value over the inproc
>> socket as a key into the table. Retrieving the key then moves the
>> shared_ptr out of the table so that the receiver has ownership of the
>> shared_ptr ref.
>>
>>
>> > Bill: Your original question implies (but does not state explicitly)
>> that you want to use 0mq to send shared_ptr’s to other threads in the same
>> process.    If that is the case you need to be cognizant of (
>> https://en.cppreference.com/w/cpp/memory/shared_ptr):
>>
>> Sorry for the confusion, yes it is all the same process. My goal is to
>> have each subscriber to copy-construct their own copy of the shared_ptr.
>>
>>
>> > Harald: shared_pointers are thread safe  (not the underlying data). so
>> you can copy the shared_ptr to the thread, read the date, and forget about
>> it. no need to try some tricks like serialize pointer addresses, you will
>> not succeed that way anyway.
>>
>> The problem is, I cannot just "copy the shared_ptr and forget about it".
>> If it goes out of scope before any subscribers make a copy, its data will
>> be deallocated.
>>
>>
>> ------------------------------
>> *From:* zeromq-dev <zeromq-dev-bounces at lists.zeromq.org> on behalf of
>> Harald Achitz <harald.achitz at gmail.com>
>> *Sent:* June 24, 2020 2:36 PM
>> *To:* ZeroMQ development list <zeromq-dev at lists.zeromq.org>
>> *Subject:* Re: [zeromq-dev] ZMQ - Send std::shared_ptr via inproc
>> pub-sub?
>>
>> shared_pointers are thread safe  (not the underlying data)
>> so you can copy the shared_ptr to the thread, read the date, and forget
>> about it
>> no need to try some tricks like serialize pointer addresses, you will not
>> succeed that way anyway
>>
>> On Wed, 24 Jun 2020 at 20:25, Bill Torpey <wallstprog at gmail.com> wrote:
>>
>> Serguei:
>>
>> Your original question implies (but does not state explicitly) that you
>> want to use 0mq to send shared_ptr’s to other threads in the same process.
>>    If that is the case you need to be cognizant of (
>> https://en.cppreference.com/w/cpp/memory/shared_ptr):
>>
>> The ownership of an object can only be shared with another shared_ptr by
>> copy constructing or copy assigning its value to another shared_ptr.
>> Constructing a new shared_ptr using the raw underlying pointer owned by
>> another shared_ptr leads to undefined behavior.
>>
>>
>>
>> If that is not the case (i.e., if the destination of the send is not
>> another thread in the same process), you need to actually serialize the
>> data, as suggested by *Sergei Nikulov <sergey.nikulov at gmail.com
>> <sergey.nikulov at gmail.com>>*
>>
>> Which is it?
>>
>> B
>>
>> On Jun 24, 2020, at 2:06 PM, Rafał Dudycz <rafal.dudycz at gmail.com> wrote:
>>
>> Hi,
>>
>> Don't you think that sending a pointer to a memory is very error prone?
>> e.g.: object may be destroyed before the receiver gets a serialized copy of
>> shared_ptr as ref_count will not consider the fact that there is an
>> instance of pointer in the ZMQ inproc buffer.
>> IMO it would be better to send an object identifier which could be used
>> by a receiver to retrieve data from some data provider.
>>
>> --
>> Rafal
>>
>>
>> śr., 24 cze 2020 o 19:56 Sergei Nikulov <sergey.nikulov at gmail.com>
>> napisał(a):
>>
>> On Wed, Jun 24, 2020 at 8:11 PM Serguei Khasimkhanov
>> <SerKhas at hotmail.com> wrote:
>> >
>> > Hello all,
>> >
>> > I have a publisher thread that has a bunch of shared_ptrs to large
>> chunks of data. Occasionally I want to share some of those shared_ptrs with
>> subscribers (for read-only.) The problem is that ZeroMQ only allows sending
>> binary data, not C++ objects. How can I accomplish this?
>>
>> You should serialize your object on publisher side -> send -> receive
>> -> deserialize on subscriber.
>> There is no other way except you doing some interprocess communication.
>> Ref. https://www.google.com/search?q=c%2B%2B+serialization
>>
>> >
>> > I was thinking I could allocate a copy of the shared_ptr on the heap,
>> then publish a pointer to it, but since I don't know when the subscribers
>> are done using that pointer, I don't know when to deallocate it.
>> >
>> > - Ser
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>> --
>> Best Regards,
>> Sergei Nikulov
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20200624/a252351b/attachment.htm>


More information about the zeromq-dev mailing list