[zeromq-dev] Welcome to the "zeromq-dev" mailing list

Tim Crowder crowdert at yahoo-inc.com
Fri May 30 21:03:08 CEST 2014


Hi Jeremy-

You might take a look at existing systems for how they handle it.
Apache Kafka, for instance, always maintains a backlog of messages,
via fast-append to logfiles. Clients keep track of the "offset" of the
last message they processed, and actively pull new messages from
the publisher.

Even if you know when the connection dropped, it's hard to know
which message was completely processed (vs delivered) last.
So, without client pulls, you need explicit client acknowledgement of
the last processed message.
This means that you have to keep saving/buffering messages until all
clients acknowledge them, then you can discard messages up to that point.

Cheers!
.timrc

________________________________
From: zeromq-dev-bounces at lists.zeromq.org [zeromq-dev-bounces at lists.zeromq.org] on behalf of Jeremy Richemont [jrichemont at gmail.com]
Sent: Friday, May 30, 2014 7:21 AM
To: ZeroMQ development list
Subject: Re: [zeromq-dev] Welcome to the "zeromq-dev" mailing list

Thanks, Charles. I did, in fact, find that pattern. The problem is it does not match what I am trying to do. That pattern for when you have state + deltas. What I have is a continuous message stream which, once started to client x must be preserved even if client x dies for a bit (not forever of course, I put an SLA of 1 million messages/client) and then reconnects, every message it missed is replayed, in order, then the live stream resumes.

It needs to handle n clients, any of which may drop and reconnect so each one will need an independent message cache. PUB/SUB will not do for this because I may need to send messages 10 - 100 to client x on reconnect but 50 - 200 to client y.

Asking for state is a good idea - ask for missing updates in my case - but the question remains; how does the server know the client is no longer available and it must therefore start backing up messages from a PUB socket? The client can't tell it over OOB because it died already.

If I could just query PUB and get a list of clients plus a notification when one drops that'd solve the problem I think. But how to do that?

Jeremy

On 30 May 2014 15:00, Charles Remes <lists at chuckremes.com<mailto:lists at chuckremes.com>> wrote:
Take a look at the Clone pattern in the zguide.

http://zguide.zeromq.org/page:all#Reliable-Pub-Sub-Clone-Pattern

This might be what you need.

cr

On May 29, 2014, at 11:20 AM, Jeremy Richemont <jrichemont at gmail.com<mailto:jrichemont at gmail.com>> wrote:

>
> Hi. I am struggling to work out how to use zmq to implement the architecture I need. I have a classic publish/subscribe situation except that once client x has subscribed to a topic I need the topic data to be sent to it to be cached if the client dies and resent on reconnect. The data order is important and I can't miss messages should the client be offline for a while.
>
> The PUB/SUB pattern doesn't seem to know about individual clients and will just stop sending to client x if it dies. Plus I can't find out this has happened and cache the messages, or know when it reconnects.
>
> To try to get around this I used the REQ/REP pattern so the clients can announce themselves and have some persistence but this is not ideal for a couple of reasons:
>
> 1) The clients must constantly ask "got any data for me?" which offends my sensibilities
>
> 2) What happens if there's no data to send to client x but there is to client y? Without zmq I'd have had a thread per client and simply block the one with no data but I can't block client x without also blocking client y in a single thread.
>
> Am I trying to shove a round peg in a square hole, here? Is there some way I can get feedback from PUB saying 'failed to send to client x'? so I can cache the messages instead? Or is there some other pattern I should be using?
>
> Otherwise it's back to low level tcp for me...
>
> Many thanks;
>
> Jeremy
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org<mailto:zeromq-dev at lists.zeromq.org>
http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140530/456f8cab/attachment.htm>


More information about the zeromq-dev mailing list