[zeromq-dev] Notes from a hackathon

Thomas Rodgers rodgert at twrodgers.com
Fri Feb 6 18:02:31 CET 2015


>
> Adding a mutex, even one that is never contended, to the socket will
> essentially triple this (one atomic CAS to acquire the mutex, one atomic
> CAS to put the message on the pipe, one atomic CAS to release the mutex).


This is a bit of a "blue sky" view of the cost of acquiring a mutex. For
the adventurous of spirit, chase down the call path of pthread_mutex
sometime in GLIBC. It is substantially more involved than a single pair of
'lock; cmpxchg' instructions, but it tries really hard to make that the
rough cost of the happy path.

On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers <rodgert at twrodgers.com>
wrote:

> Having thought about this for a couple of more days, I want to at least
> take a stab at arguing against "threadsafe" sockets -
>
> libzmq's thread safety guarantees, to me anyway, are very clear,
> unsurprising and non-controversial - I cannot share a socket with another
> thread without a full fence.
>
> The kinds of systems I generally build have very strict requirements on
> overall latency, to the point that most of my networking IO is done through
> kernel-bypass libraries and NICs that support this, for raw TCP and UDP
> multicast. The latency sensitive code that does IO is in it's own thread,
> with exclusive access to the NICs which are accessed via kernel bypass.
> Coordination with other threads in the same process is done via inproc pair
> sockets. Pair sockets + very small messages (small enough that libzmq does
> not need to perform allocation) provide a very nice interface to a lock
> free queue with low overhead using a single atomic CAS operation. Atomic
> operations are cheap, but they are not free (~30 clocks on x86). Adding a
> mutex, even one that is never contended, to the socket will essentially
> triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
> message on the pipe, one atomic CAS to release the mutex). I would like to
> have the option to avoid this.
>
> If a wrapper wants thread safe sockets to enable certain use-cases that
> may be more idiomatic for the language in question, it can provide the full
> fence. AZMQ <https://github.com/zeromq/azmq> does exactly this by
> default, but you have the option to opt out of it. It does this because
> Boost Asio by default allows it's sockets to be used from multiple threads
> for async IO and I need to guard more than just exclusive access to the
> ZeroMQ socket a the fence in this case. Putting a mutex inside of the
> libzmq socket, essentially doubles the overhead for no gain in useful
> functionality and runs completely counter to one of C and C++'s overarching
> principles: "don't pay for what you don't use".
>
> If a class of apps really demands short lived exclusive access to a
> socket, provide a pool abstraction. The pool is thread safe, obtain a
> socket, perform I/O in a single thread, return the socket to the pool.
>
> On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier <
> pelletier.michel at gmail.com> wrote:
>
>>
>>
>> On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>
>>> The discussion about thread safety was quite short iirc, though that
>>> contributor did discuss other things... at length. I merged his
>>> "thread safe socket" change rapidly, then we reverted it after a few
>>> days, and he disappeared. It was rather brute force and I suspect did
>>> not work at all, it simply wrapped all accesses to the socket
>>> structure in mutexes. No discussion at the time of multipart data and
>>> atomic send/recv.
>>>
>>
>> My memory of the conversation at the time is pretty dim, I agree the
>> changes were ugly and untested and the contributor was difficult to reason
>> with and seemed to want to make the changes based on no real need at all.
>>
>>
>>>
>>> As for socket safety, I've no strong opinion. I see that many people
>>> expect that to work and hit errors when it doesn't. I see that nanomsg
>>> has threadsafe sockets and no multipart. I see that sharing sockets
>>> across threads would make some actor models simpler, which is nice.
>>>
>>
>> This is the classic problem with thread safe anything.  Threads are hard,
>> and there is a balance between the complexity of making a thread safe
>> construct and the skill required of a programmer to use "unsafe" construct
>> in a safe manner.  I still think if the concrete problem is very short
>> lived threads causing slow joiner problems, then the simple solution is
>> pools (troupes of actors?).
>>
>> -Michel
>>
>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
>>> <pelletier.michel at gmail.com> wrote:
>>> > I think Brian has some good points here, there are numerous unrelated
>>> issues
>>> > being discussed in this thread.
>>> >
>>> > A few points that I have:
>>> >
>>> > Multi part messages have also bothered me.  However as a Python
>>> programmer i
>>> > see Min's points about the expense of buffer creation.  To my knowledge
>>> > zproto does not (yet) have Python generation support either, or maybe
>>> > something like generated cffi or ctypes wrappers around the zproto
>>> generated
>>> > C code.  That being said there are a variety of serialization
>>> libraries for
>>> > Python.  With some ctypes and mmap magic they can also be done "zero
>>> copy"
>>> > but it's not pretty:
>>> >
>>> > https://gist.github.com/michelp/7522179
>>> >
>>> > Multi part envelops are also how multi-hop routing is done.  I don't
>>> see how
>>> > the new ideas handle that.  I don't think we can just say "multi hop
>>> routing
>>> > is bad" and get rid of it.
>>> >
>>> > "Thread safe" sockets do not sound appealing to me.  We did that, had
>>> a long
>>> > and contentious discussion with the person championing them, merged
>>> it, then
>>> > reverted it and that person is now no longer in the community.  Pieter
>>> was
>>> > the most vocal opponent to them then and now he wants them back.  Of
>>> course,
>>> > anyone can change their mind, but the only current argument I hear now
>>> for
>>> > them though is improving the performance of short lived threads, but
>>> that
>>> > can be solved, more correctly in my opinion, with thread or connection
>>> > pools.  If you creating and tearing down threads that rapidly then you
>>> have
>>> > two problems.
>>> >
>>> > -Michel
>>> >
>>> > On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <bknox at digitalocean.com>
>>> wrote:
>>> >>
>>> >> After catching up on this thread, I feel like at least three problems
>>> are
>>> >> being conflated into one problem.  I'll state what I see being
>>> discussed
>>> >> from my perspective:
>>> >>
>>> >> 1. "Using multi part messages as a way to route to clients from a
>>> router
>>> >> socket is overly complicated and not how new users expect things to
>>> work"
>>> >>
>>> >> 2. "Using multi part messages for message serialization is costly, and
>>> >> potentially confusing to others."
>>> >>
>>> >> 3. "ZeroMQ sockets are not thread safe."
>>> >>
>>> >> While on an implementation level these three problems may be related,
>>> on a
>>> >> conceptual level I don't see them as related.  I may agree with some
>>> of
>>> >> these problem statements and not others.
>>> >>
>>> >> For me, my first priority is to always have the ability to get back a
>>> nice
>>> >> agnostic blob of bytes from ZeroMQ.   This makes it easy to make
>>> ZeroMQ
>>> >> socket use compatible with standard io interfaces in Go.  Structure
>>> for what
>>> >> is contained in those bytes is a concern of a different layer.
>>> Sometimes I
>>> >> use zproto for this (which I like), and other times I don't.
>>> >>
>>> >> As a demonstration that the problems are different problems, I solved
>>> #1
>>> >> for myself in goczmq without addressing anything else.
>>> >>
>>> >> I would assert some of the confusion in this discussion is that we're
>>> >> talking about multiple problem statements at the same time.
>>> >>
>>> >> Cheers - and it was great meeting people this week!
>>> >>
>>> >> Brian
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <ph at imatix.com>
>>> wrote:
>>> >>>
>>> >>> Ironically, in my testing of high message rate), allowing multipart
>>> >>> creates significant costs. Multipart is just one way of getting
>>> >>> zero-copy, and even then only works on writing, not reading.
>>> >>>
>>> >>> For high performance brokers like Malamute I'd *really* like to be
>>> >>> moving blobs around instead of lists of blobs.
>>> >>>
>>> >>>
>>> >>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin <
>>> gregg at pointillistic.com>
>>> >>> wrote:
>>> >>> > M> Perhaps it is because I spend my days in a higher level language
>>> >>> > M> like Python, but zproto is not an attractive option.
>>> >>> >
>>> >>> > Same here. I will read in detail about it shortly, but it may not
>>> make
>>> >>> > it into my toolbox as a multipart replacement. Multipart looked
>>> very
>>> >>> > cool when I found 0MQ, but I've ended up not using it much. I'm not
>>> >>> > doing high performance stuff though. Simplicity and ease of use are
>>> >>> > tops on my list.
>>> >>> >
>>> >>> > -- Gregg
>>> >>> >
>>> >>> > _______________________________________________
>>> >>> > zeromq-dev mailing list
>>> >>> > zeromq-dev at lists.zeromq.org
>>> >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> >>> _______________________________________________
>>> >>> zeromq-dev mailing list
>>> >>> zeromq-dev at lists.zeromq.org
>>> >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> >>
>>> >>
>>> >>
>>> >> _______________________________________________
>>> >> zeromq-dev mailing list
>>> >> zeromq-dev at lists.zeromq.org
>>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> >>
>>> >
>>> >
>>> > _______________________________________________
>>> > zeromq-dev mailing list
>>> > zeromq-dev at lists.zeromq.org
>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> >
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150206/c879713c/attachment.htm>


More information about the zeromq-dev mailing list