[zeromq-dev] Notes from a hackathon

Mon Feb 2 18:11:23 CET 2015

Doron,

You're right - there's a cost there I wasn't considering.  I'm not yet
entirely convinced that the cost makes the abstraction no longer worth it
in certain applications, but it at least shows that it should not be the
most advocated pattern in general.

On Mon, Feb 2, 2015 at 8:55 AM, Doron Somech <somdoron at gmail.com> wrote:

> Joe the problem with the api of zmsg for single frame is that you cannot
> calculate the frame size in advance causing a lot of re allocation and
> copying data around (it's ok for receiving, the problem is sending).
>
>
>
> On Mon, Feb 2, 2015 at 6:35 PM, Joe McIlvain <joe.eli.mac at gmail.com>
> wrote:
>
>> In my opinion, maintaining some kind of multi-frame abstraction at the
>> API level is in fact very useful to applications, even if they are sent
>> atomically on the implementation level as you describe.
>>
>> For example, in CZMQ It's quite useful in program flow to be able to push
>> or pop a single zframe onto or off of a zmsg, then send that zmsg as an
>> entire message.  Rarely do I find myself using zframe_send or zframe_recv
>> with the "send more" and "receive more" flags in a CZMQ application.  The
>> CZMQ "picture" messages are nice, but they are not always the easiest API
>> to integrate into an application.  Having zmsg and zframe remain (or
>> something like them), while removing zframe_send and zframe_recv could ease
>> applications relying on multi-frame semantics into the transition.  The
>> implementation for zmsg would simply serialize the frames in some
>> zmsg-determined way into a single-part zmq_msg to send and deserialize
>> according to the same scheme to receive.  This multi-frame serialization
>> essentially be at the application level rather than at the ZMTP level.
>>
>> In short, I think a multi-frame abstraction is important for
>> applications, but if we remove the ability to send one frame at a time at
>> the fundamental level, this abstraction can remain at a higher level
>> (whether in libzmq, or in CZMQ and other wrappers) and coexist happily with
>> the proposed changes.
>>
>>
>>
>> On Mon, Feb 2, 2015 at 7:37 AM, Doron Somech <somdoron at gmail.com> wrote:
>>
>>> So the new sockets are in the github master, you can take a loot at the
>>> test to see how to use the new routing id field:
>>>
>>> https://github.com/zeromq/libzmq/blob/master/tests/test_client_server.cpp
>>>
>>> Few of the reasons I didn't like multi frames:
>>> * I tried in the past to make both zeromq and NetMQ thread safe, think
>>> of sending from multiple threads or receiving from multiple threads. we
>>> cannot do that with Multipart.
>>> * There are a lot of good transport that already implement message
>>> framing (UDP, WebSockets, SCTP and even HTTP), but because zeromq required
>>> it is own framing it was not easy to add them.
>>> * Multipart, router usage (routing id frame) and not being thread safe
>>> make the learning curve of zeromq hard to beginners. Without three of them
>>> zeromq become much simpler.
>>> * At least with NetMQ single part message is much faster than multipart.
>>> * New stacks, multipart is complicated to implement, with the new API it
>>> will much more easier to implement new stacks (like JSMQ) or any native
>>> stack.
>>>
>>> Pieter I'm looking forward to see how you expose the routing id in the
>>> czmq API.
>>> I also like the czmq API for sending mutlipart messages (the picture
>>> feature) so maybe we can use that to generate single frame which is also
>>> compatible with zproto.
>>>
>>> About the implementation, none of new sockets support any option now.
>>> server behave like mandatory router, so when router is not reachable or
>>> highwater mark is reached an error will be returned.
>>>
>>> As ZMTP 3.1 is still in raw status, what do you think of removing the
>>> multipart from it? maybe the 3.1 will only support the new socket types.
>>>
>>> Anyway I really excited about this change.
>>>
>>>
>>>
>>>
>>>
>>> On Mon, Feb 2, 2015 at 4:17 PM, Thomas Rodgers <rodgert at twrodgers.com>
>>> wrote:
>>>
>>>> What we really want IMO is per-peer metadata, and an API to get/set
>>>>> this. Using messages is a hack.
>>>>
>>>>
>>>> Currently working on that :)
>>>>
>>>> Having two layers that both
>>>>> carry per-message data is... wrong IMO.
>>>>
>>>>
>>>> Protocols supporting 'out of band' data aren't exactly uncommon.
>>>>
>>>> However the key thing is, what's the problem. Then we can discuss
>>>>> solutions to that.
>>>>
>>>>
>>>> I don't have an immediate use-case justifying it. I noted it, mostly
>>>> because it has come up a few times since I started paying attention.
>>>>
>>>> On Mon, Feb 2, 2015 at 7:55 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>>>
>>>>> > It seems to me that in order to remove multi-part messages and
>>>>> introduce new
>>>>> > socket types (e.g. SERVER/CLIENT) that would
>>>>> > necessitate a revision of the wire protocol. If we are going to do
>>>>> that, it
>>>>> > might be worth considering per-message
>>>>> > metadata -
>>>>>
>>>>> We'd have to be very clear about the problem that per-message metadata
>>>>> is aiming for. There is an elegance to delivering blobs and nothing
>>>>> more. Metadata can be added on top using zproto.
>>>>>
>>>>> What we really want IMO is per-peer metadata, and an API to get/set
>>>>> this. Using messages is a hack. If we are sending/receiving data on a
>>>>> per-message basis, that is the message. Having two layers that both
>>>>> carry per-message data is... wrong IMO.
>>>>>
>>>>> However the key thing is, what's the problem. Then we can discuss
>>>>> solutions to that.
>>>>>
>>>>> -Pieter
>>>>> _______________________________________________
>>>>> zeromq-dev mailing list
>>>>> zeromq-dev at lists.zeromq.org
>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150202/535b3ba3/attachment.htm>