[zeromq-dev] Not sure if ZeroMQ is right for me

[Sean Robertson]

It sounds like you would have good luck combining the two. RabbitMQ is
best for the actual "MQ" part of the equation, storing messages on the
brokers, while ZeroMQ is suited for passing the messages between the
nodes - server and brokers, brokers and clients. Since you've looked
at the Titanic pattern, just replace the "Disk" part with "RabbitMQ"
and you should be on your way. (And just to be pedantic, if you do
that I'd recommend flipping your terminology to fit the naming scheme:
clients -> workers, server -> client)

On Fri, Dec 13, 2013 at 4:14 PM, Van Klaveren, Brian N.
<bvan at slac.stanford.edu> wrote:
> Hi,
>
> I’m trying to determine if ZeroMQ is right for me.
>
> I have a local batch system of a few thousand nodes, several external batch systems distributed globally, and a server which distributes those jobs to their respective batch system.
>
> Upon startup and job completion, a job will notify the server that it has started/ended (traditionally done through email, although this has many problems, which is why we’re moving away from it).
> Typically jobs are staggered, but occasionally thousands of jobs all start up at once (still trying to explain to a coworker why querying oracle from 1100 hosts yesterday was bad idea)
>
> What I think I’d like to do is to have a system where the server and clients talk through a broker preferentially.
> There will be a main broker and a failover broker.
> When the server goes down, or unresponsive, the broker should deliver those to a persistent-queue client (similar to titanic service protocol?), and that persistent queue should keep those messages until the server is reconnected.
> When a client isn’t responsive (happens semi-frequently when you are dealing with thousands of batch machines), messages originating from the server should be delivered to the persistent queue, where it will retry at two intervals, after which the client is disconnected, server notified, and messages are discarded. (persistent queue will have write-back caching)
>
> For firewall reasons and other networking reasons, it may be necessary to have an intermediate broker (and possibly a persistent queue) at each of the external batch locations.
>
> Server implementation would probably use jeromq since the rest of the server is written in java, clients would likely use pyczmq.
>
> For message security, I was thinking I’d just pre-distribute a symmetric to each batch system that the jobs would use to encrypt their message (but leave any necessary metadata in the clear).
>
> I couldn’t really determine if there is a library/framework or something that already exists that would make this easy, although it seems like it shouldn’t be too hard with RabbitMQ. However, the support of zeromq and minimalism of the library is desirable to me and possible users. Even if I didn’t use RabbitMQ for the whole thing, I’d probably end up using it for the persistent queue at least. Should I even bother with ZeroMQ?
>
> Thanks,
> Brian
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev