[zeromq-dev] Tie Breaker
Kenneth Adam Miller
kennethadammiller at gmail.com
Wed Jul 29 02:07:04 CEST 2015
There's no need to take any of those options because in my case, because
the producer polls every time before sending.
I think I realize with your first sentence what I missed when I wasn't
vocalizing my issue. I think I now know that it's not because the other
clients aren't concurrent that there is an issue, it's the lack of
isolating the location in code for polling into doing both read and write
polling in one call. If it were one call, then the send lockup could never
happen, but right now, it occurs in sequence, where if you return from
polling for read, you read, do work, and then poll before sending that back
out. In any case, I think I am highly confident I have the correct
implementation path now.
On Tue, Jul 28, 2015 at 7:43 PM, Justin Karneges <justin at affinix.com> wrote:
> You can poll for read and write at the same time, so there shouldn't be
> any issue with both peers sending simultaneously and then failing to read.
>
> Note that if you can't write, then you need to think about how to manage
> your outbound messages. The three basic choices you have are:
> 1) Backpressure (while waiting to write, stop doing anything that causes
> new messages to be produced)
> 2) Drop messages
> 3) Queue every message, indefinitely (caution to the wind!)
>
> The third option is almost never what you want. It sucks having to choose
> between the first and second options but that's reality. Once you've
> decided on your strategy then you'll know what code to write.
>
> On Tue, Jul 28, 2015, at 04:28 PM, Kenneth Adam Miller wrote:
>
> I can't use ZeroMQ for this, because I know if I could this wouldn't be a
> problem. But since there are so many systems engineers on the channel and
> fundamentally it's what ZMQ aims to solve, I think it's the appropriate
> place.
>
> For a scalability benchmark meant to emulate our real world scenario, I
> have a cycle created between a producer and a consumer, where the producer
> spams a fifo object that the consumer is listening on. The consumer does
> some validation work on the data that it was hit with and sends it back out
> over a different fifo object that the producer is concurrently listening to
> (rather it's a chain of four processes linked in this way).
>
> <- process_echo <-
> Producer consumer
> -> process_echo ->
>
> This works fine from the perspective of the producer, because listening
> and writing requires no coordination between threads. But the producer and
> echoing processes were written singled threaded, and from the perspective
> that it would poll only for input and not for when it writes.
>
> Well, as my testing went along I discovered that I had to make both the
> producer and consumer poll on output so that no data could be lost when the
> sender goes to send and the fifo isn't being read from fast enough.
> (possibly that's wrong)
>
> But using poll on the sender has caused hangups on the system as a whole.
> Poll returns for readers, I think, when someone has already written or is
> writing. And for writers, when someone is blocking (whether polling or in
> read) to read.
>
> But the problem that this solves brings up a different problem: if the
> single threaded programs poll to write, then the cycle could deadlock where
> every side is polling to write, before even one message makes it fully
> round circle.
>
> My thought is, I could break it by making the consumer concurrent in
> reading and writing, that way, lockup on send cannot happen, because there
> will always be a reader to allow execution to continue in the echo tool and
> consumer. But that's hard because the code base is large.
>
> I said all of that just to ask-does anybody have any alternative
> suggestions regarding how to approach this exact problem?
>
> All fifo objects are in non-blocking mode.
> *_______________________________________________*
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150728/c2cc0fc0/attachment.htm>
More information about the zeromq-dev
mailing list