[zeromq-dev] Preliminary results of voting

Hak Mem while1hak at gmail.com
Tue Sep 2 10:42:20 CEST 2008


Hi -


1. Wildcard at the end off the string is ok to get started
2. Having precompiled subjects would work to also get started...

In case #1, the utility of non-terminal wildcards in the selector string
goes up the more 0MQ is used e.g. subscribe to all fills - trade.*.fills
instead of trade.*. Furthermore, I would say if you are going to relax any
of the above constraints - #1 should be relaxed ahead of #2. The primary
drawback of #2 is the onerous nature of precompilation rather than loss of
functionality - this would typically mean a saturday morning cycling of the
0MQ infrastructure when there is no traffic as no market is open.

/clamberon{soapbox}

The survey was great idea and I am glad that the FastMQ team is open and
actively seeks feedback. However, behavioral economics and history
(graphical desktops, mp3 players) has shown that there is a knowledge gap
between what people need and what they think they need.

I also voted for one of my favorite features - but I would rather that the
0MQ team focus on the features that will lead to the widest adoption and the
rise of a vibrant user and developer community around it. I would posit that
conferring longevity (open source immortality?) on 0MQ is a far more
critical goal.

Some background - I work in an smallish trading organization (<200 people)
where we own licenses to Tibco/Elvin and LBM's 29 west in addition to market
data system built on another messaging system whose API is hidden to us.
Nothing would delight us more than a *viable* open source solution which
will be there for a while and free us from vendor lockin or the threat of
end of product lifecycle event (e.g. elvin).

I would suggest that topic based routing and a windows port take the highest
precedent. Even more critical in the financial space would be the ability to
interface 0MQ to an RTD server. Sexy - no. Mundane - yes. Useful - very.
This "feature" will enable pumping of data in to Excel - you know the value
of that from walking on any trading floor. We hacked an RTD interface for
Tibco ourselves about 5 years ago and now locked into that vendor since all
our spreadsheets use that. No other vendor has had the foresight to provide
an RTD interface - I wonder if these vendors are blind or just ......

The surprise that Martin expressed at the result of the voting shows that
voters - while well meaning - are clueless about hardware latencies and
costs. In our world, even if we are coloed at the various exchanges (and we
are coloed at almost every one of them) data still takes 100s of us to get
to us. Shaving 10-20us at the extreme cost and *complexity* of moving to IB
or Myrinet stacks is not worth it.

If 0MQ can gain critical acceptance  - then rest of the features - e.g. -
multi language support will be added by the community. In the case of the
Spread toolkit - which is a very camel to horses comparison here - Perl/Tcl
and Mysql support was added by the community.

Hence, I reiterate, the features that take precedence are those which will
gain 0MQ wider acceptance. In this case the interests of the the user
community and FastMQ are aligned.  Everyone wins if 0MQ wins.

Just my $0.02 cents..
/clamberoff

-h


After a week of voting on prospective 0MQ features we have some
> preliminary results.
>
> Topic routing seems to be the most desired feature (6 votes). In a way
> it pairs with another desired feature, IP multicast support (4 votes) to
> provide an efficient publish/subscribe framework. We are working on
> these features, however, it'll take some time to get the functionality
> right. In the meantime it would be useful to know how much functionality
> are users willing to sacrifice to get extremely low latency and
> extremely high throughput. Specifically:
>
> 1. Would it be enough to use wildcard only on the end of the selector
> string? (I.e. "animals.mammals.*" but not "animals.*.dogs".)
> 2. Would it be sufficient to have the topic hierarchy precompiled in
> advance and not to be able to add new topics on the fly?
>
> Opinions are welcome!
>
> The next desired item - surprisingly - is to lower the latency from
> current 30 us to 10 us (5 votes). I have no idea what conclusion to make
> from this. 30 us is as low as you can get on commodity hardware and
> standard networking stack. Are people really willing to buy expensive
> networking hardware and use non-standard networking stacks (consider
> what network administrators would say if faced with significant amount
> of non-IP traffic on the company network) - all this just to lower the
> latency by 20 us? Comments on this issue would be helpful.
>
> Anyway, for those striving for ultra-low latency, I would suggest
> visiting Myricom's stall at High Performance on Wall Street conference
> (New York, September 22nd) and asking for current performance figures
> for 0MQ on the top of SDP-MX/Myricom-10GbE stack. It should be slightly
> over 10 us...
>
> Porting to Windows is obviously a concern (5 votes). Although
> non-conformance of Windows with respect to any possible standard makes
> the thing particularly annoying to implement, we'll work on this and
> deliver Windows version of 0MQ in form of binaries (to avoid building
> problems with mutually incompatible version of MSVC).
>
> Support for different programming languages is - as expected -
> desirable. We'll publish C, Python and Java extensions for 0MQ during
> next few weeks.
>
> Solving C10K problem seems to be a concern (3 votes). Next major version
> of 0MQ will contain epoll (dev/poll, kqueue) -based solution.
>
> There are 3 votes for XMPP support. Once again, it would be interesting
> to have some feedback on this requirement. Does anybody need XMPP
> messages transferred in microseconds, several million of them a second?
> And if these are the requirements of your system, why have you chosen
> XML-based (and thus inherently slow) XMPP over a binary protocol?
>
> Regards.
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



On Tue, Sep 2, 2008 at 3:57 AM, Ritesh Bansal <rbansal at gmail.com> wrote:

>
>
> ---------- Forwarded message ----------
> From: Martin Sustrik <sustrik at fastmq.com>
> Date: Tue, Sep 2, 2008 at 2:38 AM
> Subject: [zeromq-dev] Preliminary results of voting
> To: zeromq-dev at lists.zeromq.org
>
>
> Hi all,
>
> After a week of voting on prospective 0MQ features we have some
> preliminary results.
>
> Topic routing seems to be the most desired feature (6 votes). In a way
> it pairs with another desired feature, IP multicast support (4 votes) to
> provide an efficient publish/subscribe framework. We are working on
> these features, however, it'll take some time to get the functionality
> right. In the meantime it would be useful to know how much functionality
> are users willing to sacrifice to get extremely low latency and
> extremely high throughput. Specifically:
>
> 1. Would it be enough to use wildcard only on the end of the selector
> string? (I.e. "animals.mammals.*" but not "animals.*.dogs".)
> 2. Would it be sufficient to have the topic hierarchy precompiled in
> advance and not to be able to add new topics on the fly?
>
> Opinions are welcome!
>
> The next desired item - surprisingly - is to lower the latency from
> current 30 us to 10 us (5 votes). I have no idea what conclusion to make
> from this. 30 us is as low as you can get on commodity hardware and
> standard networking stack. Are people really willing to buy expensive
> networking hardware and use non-standard networking stacks (consider
> what network administrators would say if faced with significant amount
> of non-IP traffic on the company network) - all this just to lower the
> latency by 20 us? Comments on this issue would be helpful.
>
> Anyway, for those striving for ultra-low latency, I would suggest
> visiting Myricom's stall at High Performance on Wall Street conference
> (New York, September 22nd) and asking for current performance figures
> for 0MQ on the top of SDP-MX/Myricom-10GbE stack. It should be slightly
> over 10 us...
>
> Porting to Windows is obviously a concern (5 votes). Although
> non-conformance of Windows with respect to any possible standard makes
> the thing particularly annoying to implement, we'll work on this and
> deliver Windows version of 0MQ in form of binaries (to avoid building
> problems with mutually incompatible version of MSVC).
>
> Support for different programming languages is - as expected -
> desirable. We'll publish C, Python and Java extensions for 0MQ during
> next few weeks.
>
> Solving C10K problem seems to be a concern (3 votes). Next major version
> of 0MQ will contain epoll (dev/poll, kqueue) -based solution.
>
> There are 3 votes for XMPP support. Once again, it would be interesting
> to have some feedback on this requirement. Does anybody need XMPP
> messages transferred in microseconds, several million of them a second?
> And if these are the requirements of your system, why have you chosen
> XML-based (and thus inherently slow) XMPP over a binary protocol?
>
> Regards.
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>

> ---------- Forwarded message ----------
> From: Martin Sustrik <sustrik at fastmq.com>
> Date: Tue, Sep 2, 2008 at 2:38 AM
> Subject: [zeromq-dev] Preliminary results of voting
> To: zeromq-dev at lists.zeromq.org
>
>
> Hi all,
>
> After a week of voting on prospective 0MQ features we have some
> preliminary results.
>
> Topic routing seems to be the most desired feature (6 votes). In a way
> it pairs with another desired feature, IP multicast support (4 votes) to
> provide an efficient publish/subscribe framework. We are working on
> these features, however, it'll take some time to get the functionality
> right. In the meantime it would be useful to know how much functionality
> are users willing to sacrifice to get extremely low latency and
> extremely high throughput. Specifically:
>
> 1. Would it be enough to use wildcard only on the end of the selector
> string? (I.e. "animals.mammals.*" but not "animals.*.dogs".)
> 2. Would it be sufficient to have the topic hierarchy precompiled in
> advance and not to be able to add new topics on the fly?
>
> Opinions are welcome!
>
> The next desired item - surprisingly - is to lower the latency from
> current 30 us to 10 us (5 votes). I have no idea what conclusion to make
> from this. 30 us is as low as you can get on commodity hardware and
> standard networking stack. Are people really willing to buy expensive
> networking hardware and use non-standard networking stacks (consider
> what network administrators would say if faced with significant amount
> of non-IP traffic on the company network) - all this just to lower the
> latency by 20 us? Comments on this issue would be helpful.
>
> Anyway, for those striving for ultra-low latency, I would suggest
> visiting Myricom's stall at High Performance on Wall Street conference
> (New York, September 22nd) and asking for current performance figures
> for 0MQ on the top of SDP-MX/Myricom-10GbE stack. It should be slightly
> over 10 us...
>
> Porting to Windows is obviously a concern (5 votes). Although
> non-conformance of Windows with respect to any possible standard makes
> the thing particularly annoying to implement, we'll work on this and
> deliver Windows version of 0MQ in form of binaries (to avoid building
> problems with mutually incompatible version of MSVC).
>
> Support for different programming languages is - as expected -
> desirable. We'll publish C, Python and Java extensions for 0MQ during
> next few weeks.
>
> Solving C10K problem seems to be a concern (3 votes). Next major version
> of 0MQ will contain epoll (dev/poll, kqueue) -based solution.
>
> There are 3 votes for XMPP support. Once again, it would be interesting
> to have some feedback on this requirement. Does anybody need XMPP
> messages transferred in microseconds, several million of them a second?
> And if these are the requirements of your system, why have you chosen
> XML-based (and thus inherently slow) XMPP over a binary protocol?
>
> Regards.
> Martin
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20080902/062f20e5/attachment.htm>


More information about the zeromq-dev mailing list