[zeromq-dev] Can a Trading Platform rely on ØMQ?

Martin Sustrik sustrik at 250bpm.com
Wed Sep 7 08:51:09 CEST 2011


Hi Anatoly,

>       1. Is there a built in mechanism to e.g. start off loading
> messages to disk, in case a queue is close to be overflowed? Or to just
> STOP instead of discard messages silently?

There's ZMQ_SWAP option in 2.x versions.

Applying backpressure ("STOP") works fro REQ/REP and PUSH/PULL patterns.

With PUB/SUB, applying backpressure combined with slow/dead subscriber 
can lead to unbounded latencies, even to deadlock of the whole message 
distribution system.

>       2. Mad Black Box is something that looks the closest to what we
> need, however we already have publishers themselves sharded. Would
> sharding them further into different Subscriber's "PUSH"ers be necessary
> to avoid a "subscriber's overflow"? ( e.g. processing side would be
> slower, as it deserializes [probably google protobufs] and persists
> messages to disk )

I guess you are speaking of market data here. If the publisher is 
overloaded, think of creating a more complex topology with devices in 
the middle to distribute the load.

>       3. In case the answer to 3 is YES, does not an additional internal
> sharding introduce a new point of failure ( e.g. a thread dies, etc.. ),
> in which case is there a some kind of built in recovery / retry
> mechanism ( similar to PGM, but a bit of a higher level, since we are
> dealing with ZMTP messages )?

If the point that stores the message dies, the message is lost. That 
applies to PGM or any other mechanism. The only option is to store the 
messages on a disk, with obvious performance penalty. Even then, if the 
disk dies, the messages are lost. To prevent that you have to store them 
on RAID, SAN or somesuch. If the whole RAID is destroyed etc.

Martin



More information about the zeromq-dev mailing list