[zeromq-dev] No rate limitation using JZMQ

Benjamin Errouane Benjamin2002 at gmx.de
Sat Dec 19 12:30:44 CET 2015


>
> There is currently a fixed leaky bucket in OpenPGM, there is nothing
> fundamentally stopping anyone changing this to a token bucket or to an
> adaptive scheme.


So this might be the reason I experience loss: The queue is full and I'm
losing the messages that get pushed into to filled queue?

Interestingly if the paket size is chosen to be small (<1 KB) there is no
loss, with bigger pakets loss goes up to 100% fast. Maybe that's because
with small pakets, the per message calculation overhead is big enough,
that enqueuing new messages takes long enough for the transport layer to
keep up. With rising paket sizes the overhead ratio goes down until the
transport-layer cannot keep up with the data that gets enqueued.

How does Zmq handle the NORM implementation? Does it get feed by the same
Zmq-Queue? Would the loss I'm experiencing right now be prevented by using
the NORM implementation in Zmq?

Cheers

2015-12-19 0:29 GMT+01:00 Steven McCoy <steven.mccoy at miru.hk>:

> On 18 December 2015 at 12:44, Benjamin Errouane <Benjamin2002 at gmx.de>
> wrote:
>
>> A new parameter was added in PGM that can set the individual limits for
>>> Original Data (ODATA) and Repair Data (RDATA) packets thus reserving
>>> bandwidth for repairs and ancillary data.
>>
>>
>> Are these parameters somehow accessible in JZMQ? I would have thought the
>> setRate-function would use these limits in some way. But if I'm
>> understanding your statements correctly the throttling is achieved in the
>> zmq-Framework rather than in the OpenPGM-framework, right?
>>
>
> ZeroMQ has a small subset of the socket options OpenPGM makes available.
> This is driven by demand, thus the C++ API and the Java API would both have
> to be modified.
>
>
>
>> The general recommendation however for ZeroMQ users has been to implement
>>> a level of application rate limiting before feeding into ZeroMQ, however it
>>> can be coarse limiting as the underlying framework implements fine grained
>>> throttling.
>>
>>
>> I thought about limiting the rate at application level but this is not
>> really the preferred solution, because the network should be adaptable to
>> changes in the network topology. What kind of throttling does Zmq
>> implement? Right now I'm not experiencing any throttling, but this may be
>> due to the application currently pushing messages at maximum rate.
>>
>
> There is currently a fixed leaky bucket in OpenPGM, there is nothing
> fundamentally stopping anyone changing this to a token bucket or to an
> adaptive scheme.
>
>
> One more question regarding the currently implemented rate limitation. Do
>> I understand it correctly that the rate is limited for a single buffer, so
>> if I send one huge message, the rate is limited. But the rate is not
>> limited between different messages, so sending a lot of tiny messages is
>> not rate-limited the same way?
>
>
> The rate limit is for everything sent on the wire including protocol
> messages.
>
> Generally you will find better throughput using less large packets,
> jumbograms preferred, especially on Windows platforms.  However the library
> has really only been designed at low latency high packet rate scenarios.
>
> --
> Steve-o
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20151219/b75ea93e/attachment.htm>


More information about the zeromq-dev mailing list