[zeromq-dev] Notes from a hackathon

Pieter Hintjens ph at imatix.com
Wed Feb 4 19:53:07 CET 2015


Incidentally, threads like these are the main reason I prefer small
problem driven change over planning and consensus.

I'm going to in any case start removing multipart support from some
higher layers like Malamute and Zyre, as a matter of experimentation,
and probably play with zclient and zserver classes in CZMQ, based on
the new client and server socket types, to explore the session
management.



On Wed, Feb 4, 2015 at 7:51 PM, Pieter Hintjens <ph at imatix.com> wrote:
> The discussion about thread safety was quite short iirc, though that
> contributor did discuss other things... at length. I merged his
> "thread safe socket" change rapidly, then we reverted it after a few
> days, and he disappeared. It was rather brute force and I suspect did
> not work at all, it simply wrapped all accesses to the socket
> structure in mutexes. No discussion at the time of multipart data and
> atomic send/recv.
>
> As for socket safety, I've no strong opinion. I see that many people
> expect that to work and hit errors when it doesn't. I see that nanomsg
> has threadsafe sockets and no multipart. I see that sharing sockets
> across threads would make some actor models simpler, which is nice.
>
>
>
>
>
>
> On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
> <pelletier.michel at gmail.com> wrote:
>> I think Brian has some good points here, there are numerous unrelated issues
>> being discussed in this thread.
>>
>> A few points that I have:
>>
>> Multi part messages have also bothered me.  However as a Python programmer i
>> see Min's points about the expense of buffer creation.  To my knowledge
>> zproto does not (yet) have Python generation support either, or maybe
>> something like generated cffi or ctypes wrappers around the zproto generated
>> C code.  That being said there are a variety of serialization libraries for
>> Python.  With some ctypes and mmap magic they can also be done "zero copy"
>> but it's not pretty:
>>
>> https://gist.github.com/michelp/7522179
>>
>> Multi part envelops are also how multi-hop routing is done.  I don't see how
>> the new ideas handle that.  I don't think we can just say "multi hop routing
>> is bad" and get rid of it.
>>
>> "Thread safe" sockets do not sound appealing to me.  We did that, had a long
>> and contentious discussion with the person championing them, merged it, then
>> reverted it and that person is now no longer in the community.  Pieter was
>> the most vocal opponent to them then and now he wants them back.  Of course,
>> anyone can change their mind, but the only current argument I hear now for
>> them though is improving the performance of short lived threads, but that
>> can be solved, more correctly in my opinion, with thread or connection
>> pools.  If you creating and tearing down threads that rapidly then you have
>> two problems.
>>
>> -Michel
>>
>> On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <bknox at digitalocean.com> wrote:
>>>
>>> After catching up on this thread, I feel like at least three problems are
>>> being conflated into one problem.  I'll state what I see being discussed
>>> from my perspective:
>>>
>>> 1. "Using multi part messages as a way to route to clients from a router
>>> socket is overly complicated and not how new users expect things to work"
>>>
>>> 2. "Using multi part messages for message serialization is costly, and
>>> potentially confusing to others."
>>>
>>> 3. "ZeroMQ sockets are not thread safe."
>>>
>>> While on an implementation level these three problems may be related, on a
>>> conceptual level I don't see them as related.  I may agree with some of
>>> these problem statements and not others.
>>>
>>> For me, my first priority is to always have the ability to get back a nice
>>> agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
>>> socket use compatible with standard io interfaces in Go.  Structure for what
>>> is contained in those bytes is a concern of a different layer.  Sometimes I
>>> use zproto for this (which I like), and other times I don't.
>>>
>>> As a demonstration that the problems are different problems, I solved #1
>>> for myself in goczmq without addressing anything else.
>>>
>>> I would assert some of the confusion in this discussion is that we're
>>> talking about multiple problem statements at the same time.
>>>
>>> Cheers - and it was great meeting people this week!
>>>
>>> Brian
>>>
>>>
>>>
>>>
>>> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>>>
>>>> Ironically, in my testing of high message rate), allowing multipart
>>>> creates significant costs. Multipart is just one way of getting
>>>> zero-copy, and even then only works on writing, not reading.
>>>>
>>>> For high performance brokers like Malamute I'd *really* like to be
>>>> moving blobs around instead of lists of blobs.
>>>>
>>>>
>>>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin <gregg at pointillistic.com>
>>>> wrote:
>>>> > M> Perhaps it is because I spend my days in a higher level language
>>>> > M> like Python, but zproto is not an attractive option.
>>>> >
>>>> > Same here. I will read in detail about it shortly, but it may not make
>>>> > it into my toolbox as a multipart replacement. Multipart looked very
>>>> > cool when I found 0MQ, but I've ended up not using it much. I'm not
>>>> > doing high performance stuff though. Simplicity and ease of use are
>>>> > tops on my list.
>>>> >
>>>> > -- Gregg
>>>> >
>>>> > _______________________________________________
>>>> > zeromq-dev mailing list
>>>> > zeromq-dev at lists.zeromq.org
>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>



More information about the zeromq-dev mailing list