[zeromq-dev] ZMQ performance diff b/w RHEL7 and RHEL8

[Bill Torpey]

Hi Jasper:

I’d be VERY interested in knowing what version of RH7 you’re running:

uname -a

lsb_release -a

I’m seeing weird network performance on CentOS 7, as of kernel version 1062 (part of 7.7) —earlier kernel version 957 (part of 7.6) is fine.

Curious if that’s related to what you are seeing.  I’ve been trying to run this down for a while, and any clues would be most welcome.

Thanks!

Bill

> On Apr 8, 2022, at 9:56 PM, Jasper Jaspers <jaspers01995 at gmail.com> wrote:
> 
> I’m noticing a difference in performance between zeromq running on Linux RHEL7 and RHEL8. I’m running with zmq version 4.3.4. 
> 
> I have a sample program to replicate the issues were seeing on our full system. The application uses a simple REQ/REP pattern with an INPROC socket.  The client repeatedly (every 50 milliseconds) sends 3 message parts ( 100, 200, and 500000 bytes) and then waits for the reply. The server receives the message parts and sends a basic reply message of 1 byte back. I’m also changing the thread policy/priority (SCHED_FIFO) to minimize latency due to other things running on the node. Im capturing the time it takes for each client transaction to complete. 
> 
> When this is only app running on the node each message transaction takes about 1-10 microseconds. 
> 
>  To simulate other lower priority load on the node I’m running a compilation which puts a bunch of cpu usage on every core. When this is running in the background I’m seeing a difference between RHEL7 and RHEL8. 
> 
> Here’s results from running test for about 10 minutes:
> 
> On RHEL7 I saw 5 transactions that took over 1 milliseconds, with the highest being 2.5 milliseconds. 
> 
> On RHEL8 I saw about 100 transactions that took over 1 milliseconds with most around 10 milliseconds and the highest around 20 milliseconds. 
> 
> After seeing this I looked into where the time was being spent and found that for each transaction above 1 milliseconds all the time was spent in the client’s zmq_rcvmsg call waiting for the reply. 
> 
> Given that the application is running with real time priority I wouldn’t expect there to be that much additional latency due to other lower priority applications running on the node.  Do you agree with that assessment? And if so do you think the issue is within zmq or within the Linux OS?  Given that this works much better in RHEL7 I’ll leaning toward the OS. If so, any ideas what I could look for, or do you suggest contacting red hat directly for support ?
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev