[zeromq-dev] Multiple push connections - no failover?

Thomas Guyot-Sionnest thomas at blinkx.com
Fri Mar 25 20:46:09 CET 2011


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

I have a setup in which a large number of clients connect to a pool of
streamer devices (4 currently), to which the "processors" connect and
process the messages. In the design, it is assumed for reliability
purpose that the clients can be short-lived - or even crash - and
therefore need to flush their data as soon as possible. On the other
end, the processors flush the device queue as fast as possible and write
the data to a safe storage location prior to processing them.

The theory was that as long as there is one streamer device in the
system, and the first stage processing can flush the messages as fast as
they come in, the data will end up on safe storage within seconds and
will be protected against restarts, crashes and power failure.

Now I realized that shutting down - even gracefully (SIGTERM) - a device
doen't make the clients fail over to the other devices. Instead they
appear to queue up the messages to that device until it gets back up.
The effects are very visible; if I shut down one device, I see only 75%
of the traffic on the other end. If I shut down two, it goes down to
50%. When I turn the devices back up I see a surge for a few seconds
while the queues get flushed, then it return to normal.

Is it normal behavior? When using multiple devices in fair-queuing,
shouldn't there be a single global queue that distribute the requests to
the alive connections only?

Thanks

- --
Thomas Guyot
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk2M8QEACgkQ6dZ+Kt5Bchbu9ACgv7s2Mh3mISgRfjCN2EWg4ORa
MmAAoLhc9yEJxbk/zx1nDrsBMEUaxwdZ
=ppu3
-----END PGP SIGNATURE-----




More information about the zeromq-dev mailing list