[zeromq-dev] scalability of zeromq in terms fof queues and topics

Bhavin Turakhia bhavin.t at directi.com
Thu Apr 8 16:01:40 CEST 2010


Hi Martin

Thanks for your response. It is indeed helpful. However it is quite esoteric
that the topic x subscriber lookup is an O(1) lookup. My quick question here
though is that is it a practical O(1) lookup or in a practical scenario with
a large quantity (say 1 million topics and 1 million subscribers) it would
degrade. For instance - a hash table is theoretically an O(1) lookup but not
in practice :)

w.r.t the second note - I will keep you posted once we do some testing. If
anyone is interested we could also setup some hardware which can be used to
run these tests

Bhavin

> -----Original Message-----
> From: Martin Sustrik [mailto:sustrik at 250bpm.com]
> Sent: Thursday, April 08, 2010 6:42 PM
> To: Bhavin Turakhia
> Cc: zeromq-dev at lists.zeromq.org
> Subject: Re: [zeromq-dev] scalability of zeromq in terms fof queues and
> topics
> 
> Hi Bhavin,
> 
> > I went through every performance test and paper on the zeromq.org
> site. I
> > could not find any benchmark and performance metric that spoke about
> > zeromq's performance and scalability from the standpoint of number of
> topics
> > and consumers etc. I have the following queries concerning this -
> >
> > * is there any benchmark or test report that shows how scalability is
> > impacted with increase in number of topics (I am assuming since more
> topics
> > may require more processing due to parsing and filtering)
> 
> There's no such test. However, the complexity of filtering algorithm
> with respect to total number of topics is O(1).
> 
> Complexity w.r.t. number of active subscriptions is O(1).
> 
> The only parameter of the algorithm that has O(n) complexity is the
> length of the longest active subscription.
> 
> In other words, number of topics and subscriptions is irrelevant to the
> performance. The only relevant parameter is the average length of the
> topic string.
> 
> > * is there any upper limit for topics - what should we keep in mind
> on a
> > node w.r.t number of topics
> 
> No, there's no limit.
> 
> > * same w.r.t subscribers. The performance tests talk about latencies
> and
> > messages per second. How does that change with number of consumer end
> > points? Are there any limits / limitations / non-linear scaling
> issues
> > concerning increase in number of consumers connected to a single
> publisher?
> 
> Once again, there are no such tests - the obvious reason is the amount
> of hardware needed.
> 
> Anyway, with scenarios where message is delivered to _one_ endpoint
> (REQ/REP for instance) the is O(1), i.e. number of connections is
> irrelevant thanks to O(1) scheduler written by Dhammika Pathirana.
> 
> However, if message is delivered to many endpoints (say PUB/SUB) and
> you
> are using TCP rather than multicast, some work has obviously be done to
> send the message down each connection. Performance results would be
> interesting here...
> 
> In case you'll decide to run the tests yourself, please do contribute
> them to the community!
> 
> Thanks.
> Martin
-------------- next part --------------
A non-text attachment was scrubbed...
Name: winmail.dat
Type: application/ms-tnef
Size: 4038 bytes
Desc: not available
Url : http://lists.zeromq.org/pipermail/zeromq-dev/attachments/20100408/8faf7b1f/attachment.bin 


More information about the zeromq-dev mailing list