[zeromq-dev] ZMQ performance diff b/w RHEL7 and RHEL8

[Jasper Jaspers]

I’m noticing a difference in performance between zeromq running on Linux
RHEL7 and RHEL8. I’m running with zmq version 4.3.4.

I have a sample program to replicate the issues were seeing on our full
system. The application uses a simple REQ/REP pattern with an INPROC
socket.  The client repeatedly (every 50 milliseconds) sends 3 message
parts ( 100, 200, and 500000 bytes) and then waits for the reply. The
server receives the message parts and sends a basic reply message of 1 byte
back. I’m also changing the thread policy/priority (SCHED_FIFO) to minimize
latency due to other things running on the node. Im capturing the time it
takes for each client transaction to complete.

When this is only app running on the node each message transaction takes
about 1-10 microseconds.

 To simulate other lower priority load on the node I’m running a
compilation which puts a bunch of cpu usage on every core. When this is
running in the background I’m seeing a difference between RHEL7 and RHEL8.

Here’s results from running test for about 10 minutes:

On RHEL7 I saw 5 transactions that took over 1 milliseconds, with the
highest being 2.5 milliseconds.

On RHEL8 I saw about 100 transactions that took over 1 milliseconds with
most around 10 milliseconds and the highest around 20 milliseconds.

After seeing this I looked into where the time was being spent and found
that for each transaction above 1 milliseconds all the time was spent in
the client’s zmq_rcvmsg call waiting for the reply.

Given that the application is running with real time priority I wouldn’t
expect there to be that much additional latency due to other lower priority
applications running on the node.  Do you agree with that assessment? And
if so do you think the issue is within zmq or within the Linux OS?  Given
that this works much better in RHEL7 I’ll leaning toward the OS. If so, any
ideas what I could look for, or do you suggest contacting red hat directly
for support ?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20220408/d7306771/attachment.htm>