[zeromq-dev] Notes from a hackathon
Michel Pelletier
pelletier.michel at gmail.com
Fri Feb 6 21:40:08 CET 2015
Something else that occurred to me today, perhaps incorrectly, is that
there was mention that nanomsg doesn't have multipart messages, but it does
still preserve multipart messages in a way, but supporting scatter and
gather io vectors. Perhaps emulating this well known API pattern is the
right approach to moving forward that still keep everyone happy and brings
us closer to the ability to have thread safety.
-Michel
On Fri, Feb 6, 2015 at 12:28 PM, MinRK <benjaminrk at gmail.com> wrote:
> If I recall correctly, libzmq-3.0 (or was it the old 4.0 experimental
> branch?) preserved multi-hop while separating routing from content by
> making the routing id a sequence rather than a single value. That way, it
> seems that the same push/pop behavior that happens on a ROUTER-DEALER
> device today could still work on CLIENT-SERVER. The push/pop would be
> happening on the routing stack instead of message content.
>
> If a message were something like:
>
> {
> frame_t frames[];
> int nframes;
> routing_id_t route[];
> int nroutes;
> }
>
> it would seem that you could have multi-part content (selfishly, in the
> way that affects me - messages composed of non-contiguous memory),
> multi-hop routing, and send it all with a single `zmq_send_newfangled`
> call, allowing future attempts at making `zmq_send_newfangled` a threadsafe
> call.
>
> There may be reasons this would be super gross and horrible, but it's an
> idea, anyway.
>
> -MinRK
>
>
> On Fri, Feb 6, 2015 at 9:02 AM, Thomas Rodgers <rodgert at twrodgers.com>
> wrote:
>
>> Adding a mutex, even one that is never contended, to the socket will
>>> essentially triple this (one atomic CAS to acquire the mutex, one atomic
>>> CAS to put the message on the pipe, one atomic CAS to release the mutex).
>>
>>
>> This is a bit of a "blue sky" view of the cost of acquiring a mutex. For
>> the adventurous of spirit, chase down the call path of pthread_mutex
>> sometime in GLIBC. It is substantially more involved than a single pair of
>> 'lock; cmpxchg' instructions, but it tries really hard to make that the
>> rough cost of the happy path.
>>
>> On Fri, Feb 6, 2015 at 9:41 AM, Thomas Rodgers <rodgert at twrodgers.com>
>> wrote:
>>
>>> Having thought about this for a couple of more days, I want to at least
>>> take a stab at arguing against "threadsafe" sockets -
>>>
>>> libzmq's thread safety guarantees, to me anyway, are very clear,
>>> unsurprising and non-controversial - I cannot share a socket with another
>>> thread without a full fence.
>>>
>>> The kinds of systems I generally build have very strict requirements on
>>> overall latency, to the point that most of my networking IO is done through
>>> kernel-bypass libraries and NICs that support this, for raw TCP and UDP
>>> multicast. The latency sensitive code that does IO is in it's own thread,
>>> with exclusive access to the NICs which are accessed via kernel bypass.
>>> Coordination with other threads in the same process is done via inproc pair
>>> sockets. Pair sockets + very small messages (small enough that libzmq does
>>> not need to perform allocation) provide a very nice interface to a lock
>>> free queue with low overhead using a single atomic CAS operation. Atomic
>>> operations are cheap, but they are not free (~30 clocks on x86). Adding a
>>> mutex, even one that is never contended, to the socket will essentially
>>> triple this (one atomic CAS to acquire the mutex, one atomic CAS to put the
>>> message on the pipe, one atomic CAS to release the mutex). I would like to
>>> have the option to avoid this.
>>>
>>> If a wrapper wants thread safe sockets to enable certain use-cases that
>>> may be more idiomatic for the language in question, it can provide the full
>>> fence. AZMQ <https://github.com/zeromq/azmq> does exactly this by
>>> default, but you have the option to opt out of it. It does this because
>>> Boost Asio by default allows it's sockets to be used from multiple threads
>>> for async IO and I need to guard more than just exclusive access to the
>>> ZeroMQ socket a the fence in this case. Putting a mutex inside of the
>>> libzmq socket, essentially doubles the overhead for no gain in useful
>>> functionality and runs completely counter to one of C and C++'s overarching
>>> principles: "don't pay for what you don't use".
>>>
>>> If a class of apps really demands short lived exclusive access to a
>>> socket, provide a pool abstraction. The pool is thread safe, obtain a
>>> socket, perform I/O in a single thread, return the socket to the pool.
>>>
>>> On Wed, Feb 4, 2015 at 1:04 PM, Michel Pelletier <
>>> pelletier.michel at gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On Wed, Feb 4, 2015 at 10:51 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>>>
>>>>> The discussion about thread safety was quite short iirc, though that
>>>>> contributor did discuss other things... at length. I merged his
>>>>> "thread safe socket" change rapidly, then we reverted it after a few
>>>>> days, and he disappeared. It was rather brute force and I suspect did
>>>>> not work at all, it simply wrapped all accesses to the socket
>>>>> structure in mutexes. No discussion at the time of multipart data and
>>>>> atomic send/recv.
>>>>>
>>>>
>>>> My memory of the conversation at the time is pretty dim, I agree the
>>>> changes were ugly and untested and the contributor was difficult to reason
>>>> with and seemed to want to make the changes based on no real need at all.
>>>>
>>>>
>>>>>
>>>>> As for socket safety, I've no strong opinion. I see that many people
>>>>> expect that to work and hit errors when it doesn't. I see that nanomsg
>>>>> has threadsafe sockets and no multipart. I see that sharing sockets
>>>>> across threads would make some actor models simpler, which is nice.
>>>>>
>>>>
>>>> This is the classic problem with thread safe anything. Threads are
>>>> hard, and there is a balance between the complexity of making a thread safe
>>>> construct and the skill required of a programmer to use "unsafe" construct
>>>> in a safe manner. I still think if the concrete problem is very short
>>>> lived threads causing slow joiner problems, then the simple solution is
>>>> pools (troupes of actors?).
>>>>
>>>> -Michel
>>>>
>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Feb 4, 2015 at 7:35 PM, Michel Pelletier
>>>>> <pelletier.michel at gmail.com> wrote:
>>>>> > I think Brian has some good points here, there are numerous
>>>>> unrelated issues
>>>>> > being discussed in this thread.
>>>>> >
>>>>> > A few points that I have:
>>>>> >
>>>>> > Multi part messages have also bothered me. However as a Python
>>>>> programmer i
>>>>> > see Min's points about the expense of buffer creation. To my
>>>>> knowledge
>>>>> > zproto does not (yet) have Python generation support either, or maybe
>>>>> > something like generated cffi or ctypes wrappers around the zproto
>>>>> generated
>>>>> > C code. That being said there are a variety of serialization
>>>>> libraries for
>>>>> > Python. With some ctypes and mmap magic they can also be done "zero
>>>>> copy"
>>>>> > but it's not pretty:
>>>>> >
>>>>> > https://gist.github.com/michelp/7522179
>>>>> >
>>>>> > Multi part envelops are also how multi-hop routing is done. I don't
>>>>> see how
>>>>> > the new ideas handle that. I don't think we can just say "multi hop
>>>>> routing
>>>>> > is bad" and get rid of it.
>>>>> >
>>>>> > "Thread safe" sockets do not sound appealing to me. We did that,
>>>>> had a long
>>>>> > and contentious discussion with the person championing them, merged
>>>>> it, then
>>>>> > reverted it and that person is now no longer in the community.
>>>>> Pieter was
>>>>> > the most vocal opponent to them then and now he wants them back. Of
>>>>> course,
>>>>> > anyone can change their mind, but the only current argument I hear
>>>>> now for
>>>>> > them though is improving the performance of short lived threads, but
>>>>> that
>>>>> > can be solved, more correctly in my opinion, with thread or
>>>>> connection
>>>>> > pools. If you creating and tearing down threads that rapidly then
>>>>> you have
>>>>> > two problems.
>>>>> >
>>>>> > -Michel
>>>>> >
>>>>> > On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <bknox at digitalocean.com>
>>>>> wrote:
>>>>> >>
>>>>> >> After catching up on this thread, I feel like at least three
>>>>> problems are
>>>>> >> being conflated into one problem. I'll state what I see being
>>>>> discussed
>>>>> >> from my perspective:
>>>>> >>
>>>>> >> 1. "Using multi part messages as a way to route to clients from a
>>>>> router
>>>>> >> socket is overly complicated and not how new users expect things to
>>>>> work"
>>>>> >>
>>>>> >> 2. "Using multi part messages for message serialization is costly,
>>>>> and
>>>>> >> potentially confusing to others."
>>>>> >>
>>>>> >> 3. "ZeroMQ sockets are not thread safe."
>>>>> >>
>>>>> >> While on an implementation level these three problems may be
>>>>> related, on a
>>>>> >> conceptual level I don't see them as related. I may agree with
>>>>> some of
>>>>> >> these problem statements and not others.
>>>>> >>
>>>>> >> For me, my first priority is to always have the ability to get back
>>>>> a nice
>>>>> >> agnostic blob of bytes from ZeroMQ. This makes it easy to make
>>>>> ZeroMQ
>>>>> >> socket use compatible with standard io interfaces in Go. Structure
>>>>> for what
>>>>> >> is contained in those bytes is a concern of a different layer.
>>>>> Sometimes I
>>>>> >> use zproto for this (which I like), and other times I don't.
>>>>> >>
>>>>> >> As a demonstration that the problems are different problems, I
>>>>> solved #1
>>>>> >> for myself in goczmq without addressing anything else.
>>>>> >>
>>>>> >> I would assert some of the confusion in this discussion is that
>>>>> we're
>>>>> >> talking about multiple problem statements at the same time.
>>>>> >>
>>>>> >> Cheers - and it was great meeting people this week!
>>>>> >>
>>>>> >> Brian
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <ph at imatix.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> Ironically, in my testing of high message rate), allowing multipart
>>>>> >>> creates significant costs. Multipart is just one way of getting
>>>>> >>> zero-copy, and even then only works on writing, not reading.
>>>>> >>>
>>>>> >>> For high performance brokers like Malamute I'd *really* like to be
>>>>> >>> moving blobs around instead of lists of blobs.
>>>>> >>>
>>>>> >>>
>>>>> >>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin <
>>>>> gregg at pointillistic.com>
>>>>> >>> wrote:
>>>>> >>> > M> Perhaps it is because I spend my days in a higher level
>>>>> language
>>>>> >>> > M> like Python, but zproto is not an attractive option.
>>>>> >>> >
>>>>> >>> > Same here. I will read in detail about it shortly, but it may
>>>>> not make
>>>>> >>> > it into my toolbox as a multipart replacement. Multipart looked
>>>>> very
>>>>> >>> > cool when I found 0MQ, but I've ended up not using it much. I'm
>>>>> not
>>>>> >>> > doing high performance stuff though. Simplicity and ease of use
>>>>> are
>>>>> >>> > tops on my list.
>>>>> >>> >
>>>>> >>> > -- Gregg
>>>>> >>> >
>>>>> >>> > _______________________________________________
>>>>> >>> > zeromq-dev mailing list
>>>>> >>> > zeromq-dev at lists.zeromq.org
>>>>> >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>> >>> _______________________________________________
>>>>> >>> zeromq-dev mailing list
>>>>> >>> zeromq-dev at lists.zeromq.org
>>>>> >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>> >>
>>>>> >>
>>>>> >>
>>>>> >> _______________________________________________
>>>>> >> zeromq-dev mailing list
>>>>> >> zeromq-dev at lists.zeromq.org
>>>>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>> >>
>>>>> >
>>>>> >
>>>>> > _______________________________________________
>>>>> > zeromq-dev mailing list
>>>>> > zeromq-dev at lists.zeromq.org
>>>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>> >
>>>>> _______________________________________________
>>>>> zeromq-dev mailing list
>>>>> zeromq-dev at lists.zeromq.org
>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>
>>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150206/824bf063/attachment.htm>
More information about the zeromq-dev
mailing list