[zeromq-dev] ZMQ performance diff b/w RHEL7 and RHEL8

[Jasper Jaspers]

Here’s the info you requested. What issues are you seeing?


RHEL 7:

   uname -a

      Linux cas00 3.10.0-1160.11.1.el7.oe.x86_64 #1 SMP Mon Jan 18 19:15:21
EST 2021 x86_64 x86_64 x86_64 GNU/Linux



   lsb_release -a

      LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-
noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-
4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch

      Distributor ID:             RedHatEnterpriseServer

      Description: Red Hat Enterprise Linux Server release 7.9 (Maipo)

      Release:        7.9

      Codename:  Maipo



RHEL8:

   uname -a

      Linux cas00 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06
EST 2022 x86_64 x86_64 x86_64 GNU/Linux



   lsb_release -a

      LSB Version:    :core-4.1-amd64:core-4.1-noarch

      Distributor ID: RedHatEnterprise

      Description:    Red Hat Enterprise Linux release 8.5 (Ootpa)

      Release:        8.5

      Codename:       Ootp


On Saturday, April 9, 2022, Bill Torpey <wallstprog at gmail.com> wrote:

> Hi Jasper:
>
> I’d be VERY interested in knowing what version of RH7 you’re running:
>
> uname -a
>
> lsb_release -a
>
> I’m seeing weird network performance on CentOS 7, as of kernel version
> 1062 (part of 7.7) —earlier kernel version 957 (part of 7.6) is fine.
>
> Curious if that’s related to what you are seeing.  I’ve been trying to run
> this down for a while, and any clues would be most welcome.
>
> Thanks!
>
> Bill
>
> > On Apr 8, 2022, at 9:56 PM, Jasper Jaspers <jaspers01995 at gmail.com>
> wrote:
> >
> > I’m noticing a difference in performance between zeromq running on Linux
> RHEL7 and RHEL8. I’m running with zmq version 4.3.4.
> >
> > I have a sample program to replicate the issues were seeing on our full
> system. The application uses a simple REQ/REP pattern with an INPROC
> socket.  The client repeatedly (every 50 milliseconds) sends 3 message
> parts ( 100, 200, and 500000 bytes) and then waits for the reply. The
> server receives the message parts and sends a basic reply message of 1 byte
> back. I’m also changing the thread policy/priority (SCHED_FIFO) to minimize
> latency due to other things running on the node. Im capturing the time it
> takes for each client transaction to complete.
> >
> > When this is only app running on the node each message transaction takes
> about 1-10 microseconds.
> >
> >  To simulate other lower priority load on the node I’m running a
> compilation which puts a bunch of cpu usage on every core. When this is
> running in the background I’m seeing a difference between RHEL7 and RHEL8.
> >
> > Here’s results from running test for about 10 minutes:
> >
> > On RHEL7 I saw 5 transactions that took over 1 milliseconds, with the
> highest being 2.5 milliseconds.
> >
> > On RHEL8 I saw about 100 transactions that took over 1 milliseconds with
> most around 10 milliseconds and the highest around 20 milliseconds.
> >
> > After seeing this I looked into where the time was being spent and found
> that for each transaction above 1 milliseconds all the time was spent in
> the client’s zmq_rcvmsg call waiting for the reply.
> >
> > Given that the application is running with real time priority I wouldn’t
> expect there to be that much additional latency due to other lower priority
> applications running on the node.  Do you agree with that assessment? And
> if so do you think the issue is within zmq or within the Linux OS?  Given
> that this works much better in RHEL7 I’ll leaning toward the OS. If so, any
> ideas what I could look for, or do you suggest contacting red hat directly
> for support ?
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20220409/114e50dd/attachment.htm>