[zeromq-dev] Explicit control over socket buffers?

Pieter Hintjens ph at imatix.com
Wed Jun 11 09:56:44 CEST 2014


Well, maybe. You need (afaik) queues in some very specific places to
get performance, and network buffers don't suffice.

A mistake in ZeroMQ's original design was arguably to expose these to
the user rather than making them entirely hidden. However once people
stop trying to stick their fingers in the blender, it does seem to
work very well.

-Pieter

On Tue, Jun 10, 2014 at 8:14 PM, Charles Remes <lists at chuckremes.com> wrote:
> Maybe it’s time to learn a lesson from nanomsg and eliminate SND/RCV queues altogether. AFAIK, nanomsg pushes all queueing down a layer into the OS and let’s the OS handle this task via whatever mechanisms exist (various buffers, network stack, etc).
>
> cr
>
> On Jun 10, 2014, at 1:09 PM, Pieter Hintjens <ph at imatix.com> wrote:
>
>> It's not just imprecision. The measurement can be several orders of
>> magnitude wrong / out of date, particularly when there's a high
>> message rate. We know "full" and "empty" and vague mutterings in
>> between. Simply measuring the pipe state would interfere with its
>> operation. (I believe, happy to be proven wrong).
>>
>> I know only two ways to safely manage the number of messages
>> explicitly. One is from the client side, using credit-based flow
>> control. Second is to offload messages into a single-threaded memory
>> queue (a list) that can be managed explicitly by its owning task.
>>
>> -Pieter
>>
>> On Tue, Jun 10, 2014 at 4:27 PM, Andrew Hume <andrew at research.att.com> wrote:
>>> there is a minor way in which i disagree (and have always done so) with
>>> this:
>>>
>>> there are often potentially many useful ways “internals” can be exposed,
>>> even if approximately. for example, there is little reason why an
>>> approximate
>>> answer to “how many messages are buffered” can’t be generated. and for
>>> nearly
>>> all cases, knowing the answer +/- a handful of messages would be good
>>> enough.
>>> and most of the time, allowing a small degree of imprecision allows an
>>> inexpensive
>>> solution.
>>>
>>> of course, YMMV.
>>>
>>> andrew
>>>
>>> On Jun 9, 2014, at 11:48 PM, Pieter Hintjens <ph at imatix.com> wrote:
>>>
>>> It's the same reason TCP doesn't let you discard packets that are on
>>> the wire. If you expose the internals of the plumbing to the
>>> application then you cannot scale. It makes things considerably more
>>> complex, and potentially slower by orders of magnitude. Even trying to
>>> expose a concept like "how many messages are queued?" breaks things
>>> (libzmq's pipes are filled and emptied asynchronously by two threads
>>> and there is simply no way to measure the current state except by
>>> freezing everything).
>>>
>>> If you want explicit management of message queues, you build this on top.
>>>
>>> -Pieter
>>>
>>> On Mon, Jun 9, 2014 at 9:24 AM, Dmitry Antipov <antipov at dev.rtsoft.ru>
>>> wrote:
>>>
>>> Hello,
>>>
>>> why 0mq doesn't offer the facilities to control socket buffers (except
>>> setting send/receive watermarks and default socket policy to discard/block)?
>>> In particular, someone can ask about functions like 1) zmq_peek, to tell
>>> how many messages are buffered for input/output, and 2) zmq_discard, to
>>> discard all queued messages.
>>>
>>> It's quite possible that such a questions just illustrates some
>>> misunderstanding
>>> about what 0mq sockets are and how they should be used; but can someone
>>> explain
>>> the (obviously by-design) solution to _not_ provide these features?
>>>
>>> Thanks in advance,
>>> Dmitry
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>>
>>> -----------------------
>>> Andrew Hume
>>> 949-707-1964 (VO and best)
>>> 732-420-0907 (NJ)
>>> andrew at research.att.com
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev



More information about the zeromq-dev mailing list