[zeromq-dev] Notes from a hackathon

Pieter Hintjens ph at imatix.com
Wed Feb 4 19:51:29 CET 2015

The discussion about thread safety was quite short iirc, though that
contributor did discuss other things... at length. I merged his
"thread safe socket" change rapidly, then we reverted it after a few
days, and he disappeared. It was rather brute force and I suspect did
not work at all, it simply wrapped all accesses to the socket
structure in mutexes. No discussion at the time of multipart data and
atomic send/recv.

As for socket safety, I've no strong opinion. I see that many people
expect that to work and hit errors when it doesn't. I see that nanomsg
has threadsafe sockets and no multipart. I see that sharing sockets
across threads would make some actor models simpler, which is nice.

On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
<pelletier.michel at gmail.com> wrote:
> I think Brian has some good points here, there are numerous unrelated issues
> being discussed in this thread.
> A few points that I have:
> Multi part messages have also bothered me.  However as a Python programmer i
> see Min's points about the expense of buffer creation.  To my knowledge
> zproto does not (yet) have Python generation support either, or maybe
> something like generated cffi or ctypes wrappers around the zproto generated
> C code.  That being said there are a variety of serialization libraries for
> Python.  With some ctypes and mmap magic they can also be done "zero copy"
> but it's not pretty:
> https://gist.github.com/michelp/7522179
> Multi part envelops are also how multi-hop routing is done.  I don't see how
> the new ideas handle that.  I don't think we can just say "multi hop routing
> is bad" and get rid of it.
> "Thread safe" sockets do not sound appealing to me.  We did that, had a long
> and contentious discussion with the person championing them, merged it, then
> reverted it and that person is now no longer in the community.  Pieter was
> the most vocal opponent to them then and now he wants them back.  Of course,
> anyone can change their mind, but the only current argument I hear now for
> them though is improving the performance of short lived threads, but that
> can be solved, more correctly in my opinion, with thread or connection
> pools.  If you creating and tearing down threads that rapidly then you have
> two problems.
> -Michel
> On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <bknox at digitalocean.com> wrote:
>> After catching up on this thread, I feel like at least three problems are
>> being conflated into one problem.  I'll state what I see being discussed
>> from my perspective:
>> 1. "Using multi part messages as a way to route to clients from a router
>> socket is overly complicated and not how new users expect things to work"
>> 2. "Using multi part messages for message serialization is costly, and
>> potentially confusing to others."
>> 3. "ZeroMQ sockets are not thread safe."
>> While on an implementation level these three problems may be related, on a
>> conceptual level I don't see them as related.  I may agree with some of
>> these problem statements and not others.
>> For me, my first priority is to always have the ability to get back a nice
>> agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
>> socket use compatible with standard io interfaces in Go.  Structure for what
>> is contained in those bytes is a concern of a different layer.  Sometimes I
>> use zproto for this (which I like), and other times I don't.
>> As a demonstration that the problems are different problems, I solved #1
>> for myself in goczmq without addressing anything else.
>> I would assert some of the confusion in this discussion is that we're
>> talking about multiple problem statements at the same time.
>> Cheers - and it was great meeting people this week!
>> Brian
>> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>> Ironically, in my testing of high message rate), allowing multipart
>>> creates significant costs. Multipart is just one way of getting
>>> zero-copy, and even then only works on writing, not reading.
>>> For high performance brokers like Malamute I'd *really* like to be
>>> moving blobs around instead of lists of blobs.
>>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin <gregg at pointillistic.com>
>>> wrote:
>>> > M> Perhaps it is because I spend my days in a higher level language
>>> > M> like Python, but zproto is not an attractive option.
>>> >
>>> > Same here. I will read in detail about it shortly, but it may not make
>>> > it into my toolbox as a multipart replacement. Multipart looked very
>>> > cool when I found 0MQ, but I've ended up not using it much. I'm not
>>> > doing high performance stuff though. Simplicity and ease of use are
>>> > tops on my list.
>>> >
>>> > -- Gregg
>>> >
>>> > _______________________________________________
>>> > zeromq-dev mailing list
>>> > zeromq-dev at lists.zeromq.org
>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

More information about the zeromq-dev mailing list