[zeromq-dev] contributing to 0MQ

[Erich Heine]

Hi all,

Just figured I'd drop a quick note about the research I am involved in. We
don't do research on 0mq, but it is used in our research. It's slightly off
topic for "subjects to research" but sometimes it helps just to know what
others are doing. First a bit on what we are looking at:

We are dong some experiments on a converged network as it relates to the
power-grid control space. By converged we mean SCADA, other sensors,
"engineering" (random data), and enterprise traffic all on the same pipe.
This space has some pretty interesting requirements, such as very tight loop
constraints (approaching and including real-time). Due to the remote nature
of many of the assets in the communication loops, there is a constraint on
simple over-provisioning. Further, simple over-provisioning does not allow
for certain SLAs to be reached anyway.

How we use 0mq:

* As a generic messaging layer: some of our work involves coordinating
endpoints, which we do via 0mq in a dedicated diffserv channel. We do this
because it gets us messages without the pain of dealing with sockets
directly, and more importantly, without the crap associated with a lot of
messaging systems.

* As a traffic generator. Seriously, the perf tests are the only way we have
found yet which reliably fills a network pipe without also making cpu usage
go to 100% (useful for determining if our results are because of our
networking stuff or because of our real-time cpu scheduling).

Near-future work:

*  0mq will be the transport for some work with "data diodes" -- hardware
enforced one-way data flows. These will probably be a requirement for some
power stuff in the future, and 0mq already has the semantics in it.

* We have a real-time kernel and a real-time network scheduler, and are
looking to maybe tie 0mq in with that, so as to avoid our own messy message
handling stuff.

Cool things unrelated to our research:

* I was at the middleware 2010 conference, and using 0mq implemented a
couple of the algorithms being presented as they were being presented.  This
is another of the cool  things resulting from 0mq not getting to high in the
conceptual stack. Im certain that cool concepts like gossip protocols and
whatnot could easily be implemented/researched on top of 0mq.

Anyway, thats how 0mq is currently being used in my corner of academia, hope
that helps others think of cool research stuff.

Regards,
Erich

PS sorry for the messy braindump presentation above :/

On Thu, Apr 15, 2010 at 12:35 PM, Martin Sustrik <sustrik at 250bpm.com> wrote:

> Hi Indefons, John,
>
> > I am not a member of iMatix nor a developer on the 0MQ team, so am
> > being a bit rude by responding to your note - Martin S., please
> > forgive me!
>
> Well, you've worked on the VMS port plus COBOL and Fortran bindings, no?
> That makes you part of the team :)
>
> Actually, if anyone has any research ideas or would like to participate
> on research please shout! I am happy to see academic research on 0MQ
> happening. Although we've done a lot of research there was never enough
> time (or will) to publish it in more formal way than email or ad-hoc
> whitepaper placed on the website. That makes 0MQ look like it's missing
> formal theoretical background.
>
> > One of the most difficult areas with software the nature of 0MQ is
> > exactly the one you are thinking of: "Latency monitoring and
> > adjustment". Given reasonable SLAs between
> > systems/nodes/threads/whatever, it would be great to 1) examine the
> > QoS against the SLAs and 2) be in a position to monitor the state of
> > the system; even better, change parameters on the fly to alleviate
> > things such as congestion, hung clients, or whatever.
>
> Agreed. However, it may prove hard for a person without direct
> experience with production environment to do research of such research.
> It's up to Indefons to decide anyway.
>
> Some other topics off the top of my head:
>
> 1. Theoretical foundation for latency and throughput measurement. I've
> wrote a short whitepaper on measuring jitter:
>
> http://www.zeromq.org/whitepapers:measuring-jitter
>
> Similar documents for latency and throughput are missing still. This
> kind of research requires some experience with statistics.
>
> 2. As QoS in 0MQ is delegated to the networking layer, so any research
> on QoS would require messing with underlying infrastructure. Say "How to
> set up your network to get the best latencies." Similar theme was
> researched by Patrik Csokas (bachelor degree paper):
>
> http://www.zeromq.org/local--files/area:whitepapers/bc_csokas.pdf
>
> 3. Very interesting research would be formally describing and getting
> the lockless algorithms used to back 0MQ queues. However, that seems a
> way off from your interests.
>
> 4. Interesting theme is unifying messaging API with Berkeley socket API.
> I can even imagine doing this work in the scope of IETF. The project
> could result in implementing a small wrapper library that would override
> the Berkeley socket API and allow to use 0MQ via standard POSIX APIs.
>
> 5. The topic of scaling messaging solutions to the Internet scale is big
> and underresearched area. Topic so wide would require a lot of
> commitment though. If you are interested in this I can send you a
> whitepaper we've written about it.
>
> If you want to go some other way, feel free to discuss it here. You'll
> probably get a lot of advice once the focus of the research is more clear.
>
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100415/3950e7ca/attachment.htm>