[zeromq-dev] Message-on-expiration hacked into Redis

Lourens Naudé lourens at methodmissing.com
Thu Apr 5 08:45:08 CEST 2012


Hi Steffen,

I've got similar experiments, but for all state changes, at
https://github.com/methodmissing/redis/commits/zmq

Here's some usage examples : https://gist.github.com/a767acd4ca8df0e9f528

- Lourens

On Thu, Apr 5, 2012 at 6:59 AM, Steffen Mueller
<zeromq at steffen-mueller.net>wrote:

> Dear all,
>
> I thought you'd find this entertaining or possibly even useful as a
> fun-with-0MQ example: I hacked support for dispatching 0MQ PUB messages
> from a Redis server whenever a Redis key (string only for now) expires.
>
> It's quite limited functionality, should probably be changed to support
> all Redis types, and likely stands no chance of being incorporated
> upstream, though. It does what I need it to do (extremely high
> read/write load session cache which expires old sessions into a cold
> storage, a fat mysql, from where they can be promoted back into the hot
> cache if necessary).
>
> https://github.com/tsee/redis/tree/msg_on_expire
>
> One detail that I am still wondering about is the type of socket to use:
>
> - There will be many Redis instances on many servers (it's
> single-threaded and did I say there's extremely high read/write load?).
> - Eventhough the hot cache is not write-through (yay!), I expect a
> rather high insert load on mysql, so expect to require more than one
> worker that inserts expired sessions into the database.
> - All the Redis instances have slaves for resiliency.
> - This needs to be able to fail over elegantly without losing too much
> data.
>
> Thus
>
> => I will probably switch to PUSH in the expiry hook with multiple
> insert workers PULLing.
>
> => I could have a worker on the same machine as Redis, one worker per
> Redis instance, but while that would provide more than enough insert
> throughput, it doesn't allow batching of inserts as much for improved
> database load.
>
> => How can the 0MQ mesh of connections recover if a Redis instance fails
> over to the slave? ==> Redis should reconnect. But of a worker is
> added/removed, the worker needs to reconnect. Who should be binding
> instead of connecting?
>
> What's the general strategy for resilience and failover in such setups
> with 0MQ?
>
> BTW: Did I miss a bit of the 0MQ API that allows checking for dropped
> connections to trigger reconnects?
>
> --Steffen
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120405/75dc51f6/attachment.htm>


More information about the zeromq-dev mailing list