[zeromq-dev] QoS support

[Erich Heine]

On Sat, Nov 14, 2009 at 12:52 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:

> Hi Erich,
>
>
>  Sorry I should have revised my statements better, I hope this clears them
>> up.
>>
>
> This is pretty interesting scenario. I have no experience with industrial
> automation whatsoever, so your insight is very valuable...


My experience is minimal so far, just learning about these things... :)

>
>
> Ok. There are producers producing messages with different priorities,
> however, all messages are published to the same topic hierarchy. You are
> right, there's no way to implement such scenario using 0mq without a need to
> create a separate transport for each priority level.
>
> I must admit I have no straightforward answer at the moment. Do you have
> any idea of how the thing should be implemented?
>
>
Not at the moment, I'm still in the "define the problem" stage :)  It is not
easy tho.  I'm sort of toying with tying into various tc queues
automatically, but this is very linux-centric (actually worse, it is pretty
version specific for linux, at least 3 different methods required depending
on kernel version)

One more question: Can you think of a scenario where there's large number of
> priority levels? Say 256? Or 10000? The point is that it's rather acceptable
> to have 3 connections per feed, however, it's not acceptable to have 10000
> of them.
>
> Yes and no. It depends on whether each signal can be ranked.  For instance
Consumer X wants  signals (NAME{prio (higher number better prio)}) A{3},
B{3}, C{2}, D{1}, and Consumer Y wants signals A{3}, B{2}, D{1} E{1}.  We
could prioritize and send out these as: A{3} B{3} C{2} D{1} E{1} or as A{6},
B{5}, C{2} D{2} E{1} (or the latter condensed into 4 actual queues..).
In the later scenario, many queue levels could emerge.

Of course one thing to remember with queueing theory is that at a certain
point increasing the number of queues worsens the situation. Basically it is
an optimization problem.


>  Point 2, is a slightly different take on the same notion. It assumes that
>> some applications will not partake in the system QoS of their own accord. In
>> such cases, some QoS mechanisms on the host will go a long way. Say in the
>> example above there is an app that aggregates security video feeds and sends
>> them along the network. For whatever reason this app runs on the same box as
>> the control apps (its a beefy box at a small plant or something). The video
>> system doesn't do QoS with the control system. So we need to have host based
>> QoS here, with priority queueing making the video a lower priority via
>> system intervention. The video may or may not be tagged on egress, that is a
>> separate issue, what we are doing here is providing a guarantee to the the
>> control system that it will be able to do its own thing correctly.  Further
>> say each individual sensor and camera steam comes in on the LAN. We need
>> ingress controls to prioritize handling of sensor data over video data.
>>
>
> Right. What's the part that's not doable by configuring host firewall and
> switch?
>
> Nothing. It is just a host configuration versus a configuration starting at
the gateway. Im arguing for host based, end-to-end QoS, this is just an
example of where it is needed. (also depending on system, bandwidth
allocation is different from firewall, but the idea is the same).

Regards,
Erich
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20091118/21565c5b/attachment.htm>