[zeromq-dev] message loss at high send rate in Jyre

Adam Wynne awynne at gmail.com
Sat Sep 5 14:42:53 CEST 2015


Hi Trevor,

Thanks for the message.  How do I tell jyre to use jzmq?  I don't see any
System.loadLibrary("libjzmq") call anywhere in jyre or anything in the pom
that could be used to switch between the two.

thanks
Adam

On Fri, Sep 4, 2015 at 3:40 PM Trevor Bernard <trevor.bernard at gmail.com>
wrote:

> Adam,
>
> I was able to get it running with JZMQ. I updated Jyre so it's so that
> jeromq and jzmq can be used interchangeably. Also, if you're building
> your own JNI wrapper, you can likely use this as a base:
> https://github.com/trevorbernard/zmq-jni . It uses CMake instead of
> autotools and it's a very simple wrapper. It also integrates cmake
> into the maven build process.
>
> -Trev
>
>
> On Fri, Sep 4, 2015 at 10:49 AM, Wynne Adam (CR/RTC3.1-NA)
> <Adam.Wynne at us.bosch.com> wrote:
> > Hi Trev,
> >
> > Thanks much for the help.  I am currently writing a jni wrapper for
> libzyre b/c Pieter thought that was the fastest way to take advantage of
> improvements in libzyre.  But I'm also really interested to try the
> performance testing with jyre/jzmq as well.  Let me know if you get such a
> test running.
> >
> > Adam
> >
> >
> > -----Original Message-----
> > From: zeromq-dev-bounces at lists.zeromq.org [mailto:
> zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Trevor Bernard
> > Sent: Thursday, September 03, 2015 11:10 AM
> > To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
> > Subject: Re: [zeromq-dev] message loss at high send rate in Jyre
> >
> > Adam,
> >
> > I was able to reliably recreate your error using JeroMQ on my MBP though
> I have yet to try on a Linux. When I did use JZMQ with libzmq-4.1.3, I
> would get: Resource temporarily unavailable
> > (src/signaler.cpp:301) after a certain point in time. I'm wondering if
> this is mac specific? I'll set up a VM and retry the tests.
> >
> > -Trev
> >
> > On Thu, Sep 3, 2015 at 11:59 AM, Trevor Bernard <
> trevor.bernard at gmail.com> wrote:
> >> Hi Adam,
> >>
> >> Building JZMQ is pretty straight forward:
> >>
> >> $ ./autogen.sh && ./configure && make && sudo ldconfig && mvn install
> >> -Dgpg.skip=true
> >>
> >> I submitted a few patches to jyre this morning that made jeromq the
> >> default. Also, if you want to test with JZMQ, you can just use the
> >> follow branch in my fork:
> >> https://github.com/trevorbernard/jyre/tree/jzmq
> >>
> >> You will need to mvn install jyre as well.
> >>
> >> I made some changes to your test here:
> >> https://github.com/trevorbernard/jyre-standalone-benchmark
> >>
> >> I made the counters thread safe and I changed the gradle build to look
> >> for artifacts in ~/.m2/repository
> >>
> >> Let me know how your testing goes.
> >>
> >> -Trev
> >>
> >> On Thu, Sep 3, 2015 at 9:19 AM, Wynne Adam (CR/RTC3.1-NA)
> >> <Adam.Wynne at us.bosch.com> wrote:
> >>> Hi All,
> >>>
> >>> After talking with Pieter, I'm going to first do the test with a JNI
> >>> wrapper for the zre lib.  If anyone has built such a wrapper and is
> >>> willing to share, please let me know :)
> >>>
> >>> Thanks
> >>> Adam
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: zeromq-dev-bounces at lists.zeromq.org
> >>> [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Utsav
> >>> Drolia
> >>> Sent: Wednesday, September 02, 2015 10:17 AM
> >>> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
> >>> Subject: Re: [zeromq-dev] message loss at high send rate in Jyre
> >>>
> >>> Could you check where the messages do get dropped?
> >>> Specifically, is there a buffer overflow at the pipe between the
> ZreInterface and the Agent?
> >>>
> >>>> On Sep 2, 2015, at 9:46 AM, Wynne Adam (CR/RTC3.1-NA) <
> Adam.Wynne at us.bosch.com> wrote:
> >>>>
> >>>> I simplified a bit and now explicitly wait for all responders to JOIN
> but I'm getting the same results.  I can't see any real difference between
> your test and mine.  I think my test is correct unless I'm doing something
> really dumb that I can't see.
> >>>>
> >>>> Incidentally, if I lower the pause to 0, the ZreInterface.send()
> starts blocking, which I think is the behavior we'd expect.
> >>>>
> >>>> We prefer to use the pure java version of Zyre so that we can use it
> in Android.  We'll use the C library for other platforms.
> >>>>
> >>>>
> >>>> Best regards
> >>>>
> >>>> Adam Wynne
> >>>> CR/RTC3.1-NA
> >>>>
> >>>> Tel. +1(412)390-3211
> >>>>
> >>>>
> >>>> -----Original Message-----
> >>>> From: zeromq-dev-bounces at lists.zeromq.org
> >>>> [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Pieter
> >>>> Hintjens
> >>>> Sent: Wednesday, September 02, 2015 3:30 AM
> >>>> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
> >>>> Subject: Re: [zeromq-dev] message loss at high send rate in Jyre
> >>>>
> >>>> It might be a Jyre issue yet the most likely seems an issue in your
> test case (the loss is so dramatic). What I'd do is simplify it, and see
> when it starts happening.
> >>>>
> >>>> If you are able to write a JNI layer then we could eventually wrap
> the Zyre C API in Java.
> >>>>
> >>>> On Tue, Sep 1, 2015 at 11:10 PM, Wynne Adam (CR/RTC3.1-NA) <
> Adam.Wynne at us.bosch.com> wrote:
> >>>>> OK.  Do you think it's a problem in Jyre or JeroMQ?
> >>>>>
> >>>>>
> >>>>> Best regards
> >>>>>
> >>>>> Adam Wynne
> >>>>> CR/RTC3.1-NA
> >>>>>
> >>>>> Tel. +1(412)390-3211
> >>>>>
> >>>>>
> >>>>> -----Original Message-----
> >>>>> From: zeromq-dev-bounces at lists.zeromq.org
> >>>>> [mailto:zeromq-dev-bounces at lists.zeromq.org] On Behalf Of Pieter
> >>>>> Hintjens
> >>>>> Sent: Tuesday, September 01, 2015 2:19 PM
> >>>>> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
> >>>>> Subject: Re: [zeromq-dev] message loss at high send rate in Jyre
> >>>>>
> >>>>> So here's a C test case that does this:
> >>>>> https://gist.github.com/hintjens/d54d25c30e0339fb0a87
> >>>>>
> >>>>> I do get message loss if I (a) start broadcasting before all nodes
> are ready or (b) don't wait for all messages to return. Otherwise it works
> even with no pause between message sends.
> >>>>>
> >>>>> -Pieter
> >>>>>
> >>>>> On Tue, Sep 1, 2015 at 11:25 AM, Pieter Hintjens <ph at imatix.com>
> wrote:
> >>>>>> Hi Adam,
> >>>>>>
> >>>>>> I'm recreating this test case in C/Zyre to see how it performs.
> >>>>>>
> >>>>>> -Pieter
> >>>>>>
> >>>>>> On Mon, Aug 31, 2015 at 4:39 PM, Wynne Adam (CR/RTC3.1-NA)
> >>>>>> <Adam.Wynne at us.bosch.com> wrote:
> >>>>>>> Hi Everyone,
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> We are evaluating Jyre for use in a project and are doing some
> >>>>>>> benchmark testing.  We found that at higher send rates, some
> >>>>>>> messages appear to get lost.  I have created a github project
> >>>>>>> with multiple “nodes” running in their own threads to demonstrate
> >>>>>>> the issue.  I included a README in the project that describes the
> >>>>>>> test scenario and how to run it so it’s easy to run the test.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> https://github.com/awynne/jyre-standalone-benchmark
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Please let me know if you have any ideas and/or are able to look
> into it.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Best regards
> >>>>>>>
> >>>>>>> Adam Wynne
> >>>>>>>
> >>>>>>> Bosch Research and Technology Ctr.
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> zeromq-dev mailing list
> >>>>>>> zeromq-dev at lists.zeromq.org
> >>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>>>>>>
> >>>>> _______________________________________________
> >>>>> zeromq-dev mailing list
> >>>>> zeromq-dev at lists.zeromq.org
> >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>>>> _______________________________________________
> >>>>> zeromq-dev mailing list
> >>>>> zeromq-dev at lists.zeromq.org
> >>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>>> _______________________________________________
> >>>> zeromq-dev mailing list
> >>>> zeromq-dev at lists.zeromq.org
> >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>>> _______________________________________________
> >>>> zeromq-dev mailing list
> >>>> zeromq-dev at lists.zeromq.org
> >>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>>
> >>> _______________________________________________
> >>> zeromq-dev mailing list
> >>> zeromq-dev at lists.zeromq.org
> >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>> _______________________________________________
> >>> zeromq-dev mailing list
> >>> zeromq-dev at lists.zeromq.org
> >>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150905/497d7edf/attachment.htm>


More information about the zeromq-dev mailing list