[zeromq-dev] ZMQ performance diff b/w RHEL7 and RHEL8

Bill Torpey wallstprog at gmail.com
Mon Apr 11 16:16:34 CEST 2022


Jasper:

Thanks!  Unfortunately it looks like our problems are different — mine began with CentOS 7.7, while you seem to be fine with 7.9 and are only seeing issues with CentOS 8 (if I understand you correctly).

The problem I’m seeing is very poor performance  — network transfers at 6KB/s (vs. 500K-1MB/s) — and appears to be specific to Samba.  What is odd is that the exact same configuration runs fine if I boot with an earlier version of the kernel (-957 or below).

It’s nevertheless possible that the two issues are related — if I’m able to come up with any clues, I’ll be sure to post them back here.

Thanks again for the help!

Bill

> On Apr 9, 2022, at 8:03 PM, Jasper Jaspers <jaspers01995 at gmail.com> wrote:
> 
>> Here’s the info you requested. What issues are you seeing?
>> 
>> 
>> 
>> RHEL 7:
>> 
>>    uname -a
>> 
>>       Linux cas00 3.10.0-1160.11.1.el7.oe.x86_64 #1 SMP Mon Jan 18 19:15:21 EST 2021 x86_64 x86_64 x86_64 GNU/Linux
>> 
>>  
>> 
>>    lsb_release -a
>> 
>>       LSB Version: :core-4.1-amd64:core-4.1-noarch:cxx-4.1-amd64:cxx-4.1-noarch:desktop-4.1-amd64:desktop-4.1-noarch:languages-4.1-amd64:languages-4.1-noarch:printing-4.1-amd64:printing-4.1-noarch
>> 
>>       Distributor ID:             RedHatEnterpriseServer
>> 
>>       Description: Red Hat Enterprise Linux Server release 7.9 (Maipo)
>> 
>>       Release:        7.9
>> 
>>       Codename:  Maipo
>> 
>>  
>> 
>> RHEL8:
>> 
>>    uname -a
>> 
>>       Linux cas00 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64 x86_64 x86_64 GNU/Linux
>> 
>>  
>> 
>>    lsb_release -a
>> 
>>       LSB Version:    :core-4.1-amd64:core-4.1-noarch
>> 
>>       Distributor ID: RedHatEnterprise
>> 
>>       Description:    Red Hat Enterprise Linux release 8.5 (Ootpa)
>> 
>>       Release:        8.5
>> 
>>       Codename:       Ootp
>> 
> 
> On Saturday, April 9, 2022, Bill Torpey <wallstprog at gmail.com <mailto:wallstprog at gmail.com>> wrote:
> Hi Jasper:
> 
> I’d be VERY interested in knowing what version of RH7 you’re running:
> 
> uname -a
> 
> lsb_release -a
> 
> I’m seeing weird network performance on CentOS 7, as of kernel version 1062 (part of 7.7) —earlier kernel version 957 (part of 7.6) is fine.
> 
> Curious if that’s related to what you are seeing.  I’ve been trying to run this down for a while, and any clues would be most welcome.
> 
> Thanks!
> 
> Bill
> 
> > On Apr 8, 2022, at 9:56 PM, Jasper Jaspers <jaspers01995 at gmail.com <mailto:jaspers01995 at gmail.com>> wrote:
> > 
> > I’m noticing a difference in performance between zeromq running on Linux RHEL7 and RHEL8. I’m running with zmq version 4.3.4. 
> > 
> > I have a sample program to replicate the issues were seeing on our full system. The application uses a simple REQ/REP pattern with an INPROC socket.  The client repeatedly (every 50 milliseconds) sends 3 message parts ( 100, 200, and 500000 bytes) and then waits for the reply. The server receives the message parts and sends a basic reply message of 1 byte back. I’m also changing the thread policy/priority (SCHED_FIFO) to minimize latency due to other things running on the node. Im capturing the time it takes for each client transaction to complete. 
> > 
> > When this is only app running on the node each message transaction takes about 1-10 microseconds. 
> > 
> >  To simulate other lower priority load on the node I’m running a compilation which puts a bunch of cpu usage on every core. When this is running in the background I’m seeing a difference between RHEL7 and RHEL8. 
> > 
> > Here’s results from running test for about 10 minutes:
> > 
> > On RHEL7 I saw 5 transactions that took over 1 milliseconds, with the highest being 2.5 milliseconds. 
> > 
> > On RHEL8 I saw about 100 transactions that took over 1 milliseconds with most around 10 milliseconds and the highest around 20 milliseconds. 
> > 
> > After seeing this I looked into where the time was being spent and found that for each transaction above 1 milliseconds all the time was spent in the client’s zmq_rcvmsg call waiting for the reply. 
> > 
> > Given that the application is running with real time priority I wouldn’t expect there to be that much additional latency due to other lower priority applications running on the node.  Do you agree with that assessment? And if so do you think the issue is within zmq or within the Linux OS?  Given that this works much better in RHEL7 I’ll leaning toward the OS. If so, any ideas what I could look for, or do you suggest contacting red hat directly for support ?
> > 
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
> > https://lists.zeromq.org/mailman/listinfo/zeromq-dev <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20220411/24c7a64f/attachment.htm>


More information about the zeromq-dev mailing list