[zeromq-dev] Notes from a hackathon

Fri Feb 6 16:41:58 CET 2015

Having thought about this for a couple of more days, I want to at least
take a stab at arguing against "threadsafe" sockets -

libzmq's thread safety guarantees, to me anyway, are very clear,
unsurprising and non-controversial - I cannot share a socket with another
thread without a full fence.

The kinds of systems I generally build have very strict requirements on
overall latency, to the point that most of my networking IO is done through
kernel-bypass libraries and NICs that support this, for raw TCP and UDP
multicast. The latency sensitive code that does IO is in it's own thread,
with exclusive access to the NICs which are accessed via kernel bypass.
Coordination with other threads in the same process is done via inproc pair
sockets. Pair sockets + very small messages (small enough that libzmq does
not need to perform allocation) provide a very nice interface to a lock
free queue with low overhead using a single atomic CAS operation. Atomic
operations are cheap, but they are not free (~30 clocks on x86). Adding a
mutex, even one that is never contended, to the socket will essentially
triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
message on the pipe, one atomic CAS to release the mutex). I would like to
have the option to avoid this.

If a wrapper wants thread safe sockets to enable certain use-cases that may
be more idiomatic for the language in question, it can provide the full
fence. AZMQ <https://github.com/zeromq/azmq> does exactly this by default,
but you have the option to opt out of it. It does this because Boost Asio
by default allows it's sockets to be used from multiple threads for async
IO and I need to guard more than just exclusive access to the ZeroMQ socket
a the fence in this case. Putting a mutex inside of the libzmq socket,
essentially doubles the overhead for no gain in useful functionality and
runs completely counter to one of C and C++'s overarching principles:
"don't pay for what you don't use".

If a class of apps really demands short lived exclusive access to a socket,
provide a pool abstraction. The pool is thread safe, obtain a socket,
perform I/O in a single thread, return the socket to the pool.

On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier <pelletier.michel at gmail.com
> wrote:

>
>
> On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens <ph at imatix.com> wrote:
>
>> The discussion about thread safety was quite short iirc, though that
>> contributor did discuss other things... at length. I merged his
>> "thread safe socket" change rapidly, then we reverted it after a few
>> days, and he disappeared. It was rather brute force and I suspect did
>> not work at all, it simply wrapped all accesses to the socket
>> structure in mutexes. No discussion at the time of multipart data and
>> atomic send/recv.
>>
>
> My memory of the conversation at the time is pretty dim, I agree the
> changes were ugly and untested and the contributor was difficult to reason
> with and seemed to want to make the changes based on no real need at all.
>
>
>>
>> As for socket safety, I've no strong opinion. I see that many people
>> expect that to work and hit errors when it doesn't. I see that nanomsg
>> has threadsafe sockets and no multipart. I see that sharing sockets
>> across threads would make some actor models simpler, which is nice.
>>
>
> This is the classic problem with thread safe anything.  Threads are hard,
> and there is a balance between the complexity of making a thread safe
> construct and the skill required of a programmer to use "unsafe" construct
> in a safe manner.  I still think if the concrete problem is very short
> lived threads causing slow joiner problems, then the simple solution is
> pools (troupes of actors?).
>
> -Michel
>
>
>>
>>
>>
>>
>>
>>
>> On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
>> <pelletier.michel at gmail.com> wrote:
>> > I think Brian has some good points here, there are numerous unrelated
>> issues
>> > being discussed in this thread.
>> >
>> > A few points that I have:
>> >
>> > Multi part messages have also bothered me.  However as a Python
>> programmer i
>> > see Min's points about the expense of buffer creation.  To my knowledge
>> > zproto does not (yet) have Python generation support either, or maybe
>> > something like generated cffi or ctypes wrappers around the zproto
>> generated
>> > C code.  That being said there are a variety of serialization libraries
>> for
>> > Python.  With some ctypes and mmap magic they can also be done "zero
>> copy"
>> > but it's not pretty:
>> >
>> > https://gist.github.com/michelp/7522179
>> >
>> > Multi part envelops are also how multi-hop routing is done.  I don't
>> see how
>> > the new ideas handle that.  I don't think we can just say "multi hop
>> routing
>> > is bad" and get rid of it.
>> >
>> > "Thread safe" sockets do not sound appealing to me.  We did that, had a
>> long
>> > and contentious discussion with the person championing them, merged it,
>> then
>> > reverted it and that person is now no longer in the community.  Pieter
>> was
>> > the most vocal opponent to them then and now he wants them back.  Of
>> course,
>> > anyone can change their mind, but the only current argument I hear now
>> for
>> > them though is improving the performance of short lived threads, but
>> that
>> > can be solved, more correctly in my opinion, with thread or connection
>> > pools.  If you creating and tearing down threads that rapidly then you
>> have
>> > two problems.
>> >
>> > -Michel
>> >
>> > On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <bknox at digitalocean.com>
>> wrote:
>> >>
>> >> After catching up on this thread, I feel like at least three problems
>> are
>> >> being conflated into one problem.  I'll state what I see being
>> discussed
>> >> from my perspective:
>> >>
>> >> 1. "Using multi part messages as a way to route to clients from a
>> router
>> >> socket is overly complicated and not how new users expect things to
>> work"
>> >>
>> >> 2. "Using multi part messages for message serialization is costly, and
>> >> potentially confusing to others."
>> >>
>> >> 3. "ZeroMQ sockets are not thread safe."
>> >>
>> >> While on an implementation level these three problems may be related,
>> on a
>> >> conceptual level I don't see them as related.  I may agree with some of
>> >> these problem statements and not others.
>> >>
>> >> For me, my first priority is to always have the ability to get back a
>> nice
>> >> agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
>> >> socket use compatible with standard io interfaces in Go.  Structure
>> for what
>> >> is contained in those bytes is a concern of a different layer.
>> Sometimes I
>> >> use zproto for this (which I like), and other times I don't.
>> >>
>> >> As a demonstration that the problems are different problems, I solved
>> #1
>> >> for myself in goczmq without addressing anything else.
>> >>
>> >> I would assert some of the confusion in this discussion is that we're
>> >> talking about multiple problem statements at the same time.
>> >>
>> >> Cheers - and it was great meeting people this week!
>> >>
>> >> Brian
>> >>
>> >>
>> >>
>> >>
>> >> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <ph at imatix.com>
>> wrote:
>> >>>
>> >>> Ironically, in my testing of high message rate), allowing multipart
>> >>> creates significant costs. Multipart is just one way of getting
>> >>> zero-copy, and even then only works on writing, not reading.
>> >>>
>> >>> For high performance brokers like Malamute I'd *really* like to be
>> >>> moving blobs around instead of lists of blobs.
>> >>>
>> >>>
>> >>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin <gregg at pointillistic.com
>> >
>> >>> wrote:
>> >>> > M> Perhaps it is because I spend my days in a higher level language
>> >>> > M> like Python, but zproto is not an attractive option.
>> >>> >
>> >>> > Same here. I will read in detail about it shortly, but it may not
>> make
>> >>> > it into my toolbox as a multipart replacement. Multipart looked very
>> >>> > cool when I found 0MQ, but I've ended up not using it much. I'm not
>> >>> > doing high performance stuff though. Simplicity and ease of use are
>> >>> > tops on my list.
>> >>> >
>> >>> > -- Gregg
>> >>> >
>> >>> > _______________________________________________
>> >>> > zeromq-dev mailing list
>> >>> > zeromq-dev at lists.zeromq.org
>> >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>> _______________________________________________
>> >>> zeromq-dev mailing list
>> >>> zeromq-dev at lists.zeromq.org
>> >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>
>> >
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150206/edbdb9ac/attachment.htm>