[zeromq-dev] Welcome to the "zeromq-dev" mailing list

Jeremy Richemont jrichemont at gmail.com
Mon Jun 2 14:59:59 CEST 2014


Hi Tim. I agree, a message can only be deemed to have been 'sent' if an
acknowledgement for it is received. As soon as no ack is received the
server must start backing up messages - but only for that client. It's
perfectly possible to have one client up-to-date, one that's catching up
from 200 messages ago and another catching up from 1000 messages ago. The
server will need to know about each client - either because it's been
configured to expect a certain set of clients or through some discovery
mechanism whereby the client connects and tells the server about itself.

I kind of like the latter version as it's more flexible but it is more
complex to implement.

The situation is also complicated by the fact that the server handles ten
independent message streams. In a way a client may subscribe to all ten but
in practice I think I'd keep it one to one. A system requiring all ten
streams would create ten clients to subscribe to each.

Cheers;

Jeremy


On 30 May 2014 20:03, Tim Crowder <crowdert at yahoo-inc.com> wrote:

>  Hi Jeremy-
>
> You might take a look at existing systems for how they handle it.
> Apache Kafka, for instance, always maintains a backlog of messages,
> via fast-append to logfiles. Clients keep track of the "offset" of the
> last message they processed, and actively pull new messages from
> the publisher.
>
> Even if you know when the connection dropped, it's hard to know
> which message was completely processed (vs delivered) last.
> So, without client pulls, you need explicit client acknowledgement of
> the last processed message.
> This means that you have to keep saving/buffering messages until all
> clients acknowledge them, then you can discard messages up to that point.
>
> Cheers!
> .timrc
>
>  ------------------------------
> *From:* zeromq-dev-bounces at lists.zeromq.org [
> zeromq-dev-bounces at lists.zeromq.org] on behalf of Jeremy Richemont [
> jrichemont at gmail.com]
> *Sent:* Friday, May 30, 2014 7:21 AM
> *To:* ZeroMQ development list
> *Subject:* Re: [zeromq-dev] Welcome to the "zeromq-dev" mailing list
>
>   Thanks, Charles. I did, in fact, find that pattern. The problem is it
> does not match what I am trying to do. That pattern for when you have state
> + deltas. What I have is a continuous message stream which, once started to
> client x must be preserved even if client x dies for a bit (not forever of
> course, I put an SLA of 1 million messages/client) and then reconnects,
> every message it missed is replayed, in order, then the live stream
> resumes.
>
>  It needs to handle n clients, any of which may drop and reconnect so
> each one will need an independent message cache. PUB/SUB will not do for
> this because I may need to send messages 10 - 100 to client x on reconnect
> but 50 - 200 to client y.
>
>  Asking for state is a good idea - ask for missing updates in my case -
> but the question remains; how does the server know the client is no longer
> available and it must therefore start backing up messages from a PUB
> socket? The client can't tell it over OOB because it died already.
>
>  If I could just query PUB and get a list of clients plus a notification
> when one drops that'd solve the problem I think. But how to do that?
>
>  Jeremy
>
> On 30 May 2014 15:00, Charles Remes <lists at chuckremes.com> wrote:
>
>> Take a look at the Clone pattern in the zguide.
>>
>> http://zguide.zeromq.org/page:all#Reliable-Pub-Sub-Clone-Pattern
>>
>> This might be what you need.
>>
>> cr
>>
>> On May 29, 2014, at 11:20 AM, Jeremy Richemont <jrichemont at gmail.com>
>> wrote:
>>
>> >
>> > Hi. I am struggling to work out how to use zmq to implement the
>> architecture I need. I have a classic publish/subscribe situation except
>> that once client x has subscribed to a topic I need the topic data to be
>> sent to it to be cached if the client dies and resent on reconnect. The
>> data order is important and I can't miss messages should the client be
>> offline for a while.
>> >
>> > The PUB/SUB pattern doesn't seem to know about individual clients and
>> will just stop sending to client x if it dies. Plus I can't find out this
>> has happened and cache the messages, or know when it reconnects.
>> >
>> > To try to get around this I used the REQ/REP pattern so the clients can
>> announce themselves and have some persistence but this is not ideal for a
>> couple of reasons:
>> >
>> > 1) The clients must constantly ask "got any data for me?" which offends
>> my sensibilities
>> >
>> > 2) What happens if there's no data to send to client x but there is to
>> client y? Without zmq I'd have had a thread per client and simply block the
>> one with no data but I can't block client x without also blocking client y
>> in a single thread.
>> >
>> > Am I trying to shove a round peg in a square hole, here? Is there some
>> way I can get feedback from PUB saying 'failed to send to client x'? so I
>> can cache the messages instead? Or is there some other pattern I should be
>> using?
>> >
>> > Otherwise it's back to low level tcp for me...
>> >
>> > Many thanks;
>> >
>> > Jeremy
>> >
>>   > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140602/52401257/attachment.htm>


More information about the zeromq-dev mailing list