[zeromq-dev] Implementing persistent queues

Brian Candler B.Candler at pobox.com
Wed Apr 14 17:11:30 CEST 2010

I am interested in local persistent queueing of messages in the case where
the remote server is unreachable.

The application I have in mind is a RADIUS server which forwards RADIUS
accounting packets to a central data warehouse for processing.  However the
data warehouse and/or the intervening network connection may not be
available 100% of the time, so may need to be spooled for later delivery.:

    Acct request                  zeromq
NAS -------------> RADIUS ----------------------> warehouse
    <------------- server
    Acct response     ^

Since the RADIUS server has sent an acknowledgement back to the NAS, the
data should be spooled to persistent storage.  It doesn't need to be 100%
bomb-proof but in the normal case it should survive a process stop/start.

Another similar application might be forwarding syslogs over zeromq.

There's a tantalising hint under zmq_setsockopt(3) that ZMQ_SWAP might
perform some sort of disk offload, but it seems this might only be for
messages above the high water mark, and/or may or may not persist if the
client is terminated and restarted.

I tried it with the Ruby bindings, but that wasn't very productive:

    s.setsockopt(SWAP, 1)

results in:

    remote_swap.rb:32:in `setsockopt': Invalid argument (RuntimeError)
	    from remote_swap.rb:32

I also can't see anywhere in the libzmq C++ source where the swap option is
actually used (other than being parsed in options.cpp), but that's probably
just me being dense.

It seems to me that what I want is:

(1) some sort of local spooler/disk queue application which I can fire
the zeromq stream at. Is there existing code for this?

(2) some way of knowing that the warehouse has received a particular
message, so that I know it's safe to delete it from the disk spool. I'm not
worried about occasional duplicates.

talks about disconnection events and gap messages, but I can't find these
in the 2.0.6 code base.

    $ grep -iR gap .

So perhaps I should implement some sort of reliable delivery protocol:
basically an ACK returned for each message after it has been received and
processed by the warehouse. (*)

Is there an existing 'best practice' for doing reliable delivery over
zeromq? I could, for example, use the REQ/REP pattern, but it looks like
it could be difficult to get decent throughput (AFAICS I would need
multiple threads at each end and multiple connected sockets). Or would it
be better to piggy-back something onto P2P, with a message stream in one
direction and an ACK stream in the other?

I'm sure this must be something which has been done many times before.



(*) A concrete example of what I'm thinking of is called "QoS 1" in MQTT:

More information about the zeromq-dev mailing list