[zeromq-dev] Tie Breaker

[Kenneth Adam Miller]

Thank you, by the way.

On Tue, Jul 28, 2015 at 8:07 PM, Kenneth Adam Miller <
kennethadammiller at gmail.com> wrote:

> There's no need to take any of those options because in my case, because
> the producer polls every time before sending.
>
> I think I realize with your first sentence what I missed when I wasn't
> vocalizing my issue. I think I now know that it's not because the other
> clients aren't concurrent that there is an issue, it's the lack of
> isolating the location in code for polling into doing both read and write
> polling in one call. If it were one call, then the send lockup could never
> happen, but right now, it occurs in sequence, where if you return from
> polling for read, you read, do work, and then poll before sending that back
> out. In any case, I think I am highly confident I have the correct
> implementation path now.
>
> On Tue, Jul 28, 2015 at 7:43 PM, Justin Karneges <justin at affinix.com>
> wrote:
>
>>  You can poll for read and write at the same time, so there shouldn't be
>> any issue with both peers sending simultaneously and then failing to read.
>>
>> Note that if you can't write, then you need to think about how to manage
>> your outbound messages. The three basic choices you have are:
>> 1) Backpressure (while waiting to write, stop doing anything that causes
>> new messages to be produced)
>> 2) Drop messages
>> 3) Queue every message, indefinitely (caution to the wind!)
>>
>> The third option is almost never what you want. It sucks having to choose
>> between the first and second options but that's reality. Once you've
>> decided on your strategy then you'll know what code to write.
>>
>> On Tue, Jul 28, 2015, at 04:28 PM, Kenneth Adam Miller wrote:
>>
>> I can't use ZeroMQ for this, because I know if I could this wouldn't be a
>> problem. But since there are so many systems engineers on the channel and
>> fundamentally it's what ZMQ aims to solve, I think it's the appropriate
>> place.
>>
>> For a scalability benchmark meant to emulate our real world scenario, I
>> have a cycle created between a producer and a consumer, where the producer
>> spams a fifo object that the consumer is listening on. The consumer does
>> some validation work on the data that it was hit with and sends it back out
>> over a different fifo object that the producer is concurrently listening to
>> (rather it's a chain of four processes linked in this way).
>>
>>                  <-   process_echo <-
>> Producer                                      consumer
>>                 ->    process_echo ->
>>
>> This works fine from the perspective of the producer, because listening
>> and writing requires no coordination between threads. But the producer and
>> echoing processes were written singled threaded, and from the perspective
>> that it would poll only for input and not for when it writes.
>>
>> Well, as my testing went along I discovered that I had to make both the
>> producer and consumer poll on output so that no data could be lost when the
>> sender goes to send and the fifo isn't being read from fast enough.
>> (possibly that's wrong)
>>
>> But using poll on the sender has caused hangups on the system as a whole.
>> Poll returns for readers, I think, when someone has already written or is
>> writing. And for writers, when someone is blocking (whether polling or in
>> read) to read.
>>
>> But the problem that this solves brings up a different problem: if the
>> single threaded programs poll to write, then the cycle could deadlock where
>> every side is polling to write, before even one message makes it fully
>> round circle.
>>
>> My thought is, I could break it by making the consumer concurrent in
>> reading and writing, that way, lockup on send cannot happen, because there
>> will always be a reader to allow execution to continue in the echo tool and
>> consumer. But that's hard because the code base is large.
>>
>> I said all of that just to ask-does anybody have any alternative
>> suggestions regarding how to approach this exact problem?
>>
>> All fifo objects are in non-blocking mode.
>>   *_______________________________________________*
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150728/c04ebe6a/attachment.htm>