[zeromq-dev] Tie Breaker

[Kenneth Adam Miller]

That's ok, this is only for a simulation test. I'm not focused on
congestion issues right now. I'm getting a measure of how much the
implementation decisions have improved scalability in both space and speed
across the entire software system.

On Tue, Jul 28, 2015 at 8:12 PM, Justin Karneges <justin at affinix.com> wrote:

>  Sure thing. And yeah, if you're always polling forever for writability
> before writing, then you effectively have a backpressure strategy.
>
> Note that if you begin polling for read and write at the same time, then
> you open the door to potentially processing new data while you've still got
> pending outbound data to write, and then you will face those three choices
> I mentioned.
>
> On Tue, Jul 28, 2015, at 05:07 PM, Kenneth Adam Miller wrote:
>
> Thank you, by the way.
>
> On Tue, Jul 28, 2015 at 8:07 PM, Kenneth Adam Miller <
> kennethadammiller at gmail.com> wrote:
>
> There's no need to take any of those options because in my case, because
> the producer polls every time before sending.
>
> I think I realize with your first sentence what I missed when I wasn't
> vocalizing my issue. I think I now know that it's not because the other
> clients aren't concurrent that there is an issue, it's the lack of
> isolating the location in code for polling into doing both read and write
> polling in one call. If it were one call, then the send lockup could never
> happen, but right now, it occurs in sequence, where if you return from
> polling for read, you read, do work, and then poll before sending that back
> out. In any case, I think I am highly confident I have the correct
> implementation path now.
>
> On Tue, Jul 28, 2015 at 7:43 PM, Justin Karneges <justin at affinix.com>
> wrote:
>
>
> You can poll for read and write at the same time, so there shouldn't be
> any issue with both peers sending simultaneously and then failing to read.
>
> Note that if you can't write, then you need to think about how to manage
> your outbound messages. The three basic choices you have are:
> 1) Backpressure (while waiting to write, stop doing anything that causes
> new messages to be produced)
> 2) Drop messages
> 3) Queue every message, indefinitely (caution to the wind!)
>
> The third option is almost never what you want. It sucks having to choose
> between the first and second options but that's reality. Once you've
> decided on your strategy then you'll know what code to write.
>
> On Tue, Jul 28, 2015, at 04:28 PM, Kenneth Adam Miller wrote:
>
> I can't use ZeroMQ for this, because I know if I could this wouldn't be a
> problem. But since there are so many systems engineers on the channel and
> fundamentally it's what ZMQ aims to solve, I think it's the appropriate
> place.
>
> For a scalability benchmark meant to emulate our real world scenario, I
> have a cycle created between a producer and a consumer, where the producer
> spams a fifo object that the consumer is listening on. The consumer does
> some validation work on the data that it was hit with and sends it back out
> over a different fifo object that the producer is concurrently listening to
> (rather it's a chain of four processes linked in this way).
>
>                  <-   process_echo <-
> Producer                                      consumer
>                 ->    process_echo ->
>
> This works fine from the perspective of the producer, because listening
> and writing requires no coordination between threads. But the producer and
> echoing processes were written singled threaded, and from the perspective
> that it would poll only for input and not for when it writes.
>
> Well, as my testing went along I discovered that I had to make both the
> producer and consumer poll on output so that no data could be lost when the
> sender goes to send and the fifo isn't being read from fast enough.
> (possibly that's wrong)
>
> But using poll on the sender has caused hangups on the system as a whole.
> Poll returns for readers, I think, when someone has already written or is
> writing. And for writers, when someone is blocking (whether polling or in
> read) to read.
>
> But the problem that this solves brings up a different problem: if the
> single threaded programs poll to write, then the cycle could deadlock where
> every side is polling to write, before even one message makes it fully
> round circle.
>
> My thought is, I could break it by making the consumer concurrent in
> reading and writing, that way, lockup on send cannot happen, because there
> will always be a reader to allow execution to continue in the echo tool and
> consumer. But that's hard because the code base is large.
>
> I said all of that just to ask-does anybody have any alternative
> suggestions regarding how to approach this exact problem?
>
> All fifo objects are in non-blocking mode.
>    *_______________________________________________*
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>    *_______________________________________________*
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150728/0ea22074/attachment.htm>