[zeromq-dev] CZMQ: why aren't zframes reference counted?

[Pieter Hintjens]

Sounds good. My usual strategy is to write such pieces outside CZMQ,
let them mature for a while, and if they remain useful, move them down
to CZMQ. It's easier to throw away code like that. Or, to build new
shortcuts in CZMQ to simplify existing code higher up the stack.

By that I mean, please try to not make code because it could be
useful, rather because you have a real need for it today.

On Wed, Sep 3, 2014 at 2:11 PM, Goswin von Brederlow <goswin-v-b at web.de> wrote:
> Oh, maybe that came across wrong. I don't intent do refcount zmsg nor
> zframe. My intention was to utilize the already existing refcount in
> the zmq_msg_t object, i.e. refcount the data blob itself and only
> those blobs. All the metadata (zframe and zmsg) would still be copied.
> I'm fine with duplicating the metadata. Just duplicating 1MB data
> blobs in messages seems out of the question.
>
> All that realy needs is a function to create a new zframe from an
> existing one (zframe_clone) that will share the underlying zmq_msg_t.
> The rest could be written from existing functions. But I think they
> could be usefull for others so I would like to include them in czmq
> too.
>
> On Mon, Sep 01, 2014 at 05:25:23PM +0200, Pieter Hintjens wrote:
>> OK, a few ideas then. zframe is the wrong level to do reference
>> counting; it should happen at zmsg or a similar multiframe class.
>> Second, you are doing your own queuing and acking, so you can design
>> your own ref counted class on top of zframe. I'd probably not even use
>> zframe then, use zmq_msg directly.
>>
>> On Mon, Sep 1, 2014 at 5:11 PM, Goswin von Brederlow <goswin-v-b at web.de> wrote:
>> > The zmsg_send_keep() would be using ZFRAME_REUSE. I considered that.
>> >
>> > But when I send a message to all (or many) peers it gets put into each
>> > peers outgoing list, prefixed by the peers sequence number, and only
>> > gets removed when the peer ACKs the message.
>> >
>> > Even if I use ZFRAME_REUSE I need a way to know when a message can be
>> > destroyed. That means either reference counting the message itself or
>> > using a shallow copy since the underlying zmq_msg_t already reference
>> > counts (which then means every peer can destory its shallow copy when
>> > it gets an ACK).
>> >
>> > I considered keeping a global list of all pending messages with a list
>> > of peers that are pending and removing peers from the list as they
>> > ACK. But that would result in a much more complex code.
>> >
>> > On Mon, Sep 01, 2014 at 04:14:44PM +0200, Pieter Hintjens wrote:
>> >> There is an option on zframe_send (ZFRAME_REUSE) which side-steps the
>> >> destruction. It's a simple exception that lets you send the same frame
>> >> many times, without the machinery of reference counting, duplicating,
>> >> etc.
>> >>
>> >> On Mon, Sep 1, 2014 at 2:57 PM, Goswin von Brederlow <goswin-v-b at web.de> wrote:
>> >> > Hi,
>> >> >
>> >> > the zframe_t type is based on the zmq_msg_t type. But while zms_msg_t
>> >> > uses reference counting there is no function to clone a zframe_t using
>> >> > a shared zmq_msg_t object. Consequently there is no way to make a
>> >> > shallow copy of a zmsg_t (which holds a zlist of zframe_t).
>> >> >
>> >> > Also sending a zmsg_t is destrutive. I need to send the same message
>> >> > (except different identity frame at the start) to many peers and
>> >> > possibly resend it when a peer reconnects. Currently that means
>> >> > duplicating every message many (100-1000) times which wastes both ram
>> >> > and cpu a lot.
>> >> >
>> >> > Any objections to adding three functions:
>> >> >
>> >> > // create new zframe_t that shares the same content as frame
>> >> > zframe_t * zframe_clone (zframe_t *frame);
>> >> >
>> >> > // create new zmsg_t that is a shallow copy of msg
>> >> > // frame contents are shared, see zframe_clone
>> >> > zmsg_t * zmsg_shallow_dup (zmsg_t *msg);
>> >> >
>> >> > // Send message to socket, but do not destroy after sending.
>> >> > int zmdg_send_keep (zmsg_t *msg, void *dest);
>> >> >
>> >> > MfG
>> >> >         Goswin
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev