[zeromq-dev] Dealer socket pkts are not drained by I/O thread
donthamsetty sivaprasad
shivp at rocketmail.com
Wed Nov 17 05:04:00 CET 2021
Thanks Osiris for the response .Actually it has been found that out_batch_size is not problem . I have verified the flow. the following is the problem.
We have a sender and receiver running on core 0 and core 1 of arm CPU respectively on a linux system. ZMQ I/O threads affinity is said to core 1.The communication is Dealer=>Dealer TCP . The sendhwm is set 10000.After running for couple of hours with zmq_msg_send on sender side and zmq_poll on receiver side, pkts are not sent . On debugging it has been found , pkts are enqueued in the pipe but not drained by I/O thread. The read notification is not by send_activate_read or it could have been lost somehow.. I have confirmed that mailbox of I/O thread does not have anything . To confirm the theory , I have changed the flush code of the zmq::pipe_t to do send_activate_read for every 100 msgs as a backup and problem is NOT seen . Please let me know if there as any chance of compare and swap operation used in ypipe.hpp not working in a multi core system OR any leads on this ..
The following is the present state of the pipe and internal Queue.(gdb) p *$88$89 = {<zmq::object_t> = {_vptr.object_t = 0x5cd4950 <vtable for zmq::pipe_t+8>, _ctx = 0xf2d005a0, _tid = 458}, <zmq::array_item_t<1>> = { _vptr.array_item_t = 0x5cd49a8 <vtable for zmq::pipe_t+96>, _array_index = 0}, <zmq::array_item_t<2>> = { _vptr.array_item_t = 0x5cd49b8 <vtable for zmq::pipe_t+112>, _array_index = 0}, <zmq::array_item_t<3>> = { _vptr.array_item_t = 0x5cd49c8 <vtable for zmq::pipe_t+128>, _array_index = 0}, _in_pipe = 0xc0c3640, _out_pipe = 0xc1e3b78, _in_active = true, _out_active = true, _hwm = 10000, _lwm = 500, _in_hwm_boost = -1, _out_hwm_boost = -1, _msgs_read = 0, _msgs_written = 3286676, _peers_msgs_read = 3280000, _peer = 0xc1a26f8, _sink = 0xc1a25f4, _state = zmq::pipe_t::active, _delay = true, _router_socket_routing_id = {_data = 0x0, _size = 0, _owned = true}, _server_socket_routing_id = 0, _conflate = false, _endpoint_uri = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xc1959a4 "tcp://127.0.0.1:62032"}}}(gdb) p $89._peer$90 = (zmq::pipe_t *) 0xc1a26f8(gdb) p *$90$91 = {<zmq::object_t> = {_vptr.object_t = 0x5cd4950 <vtable for zmq::pipe_t+8>, _ctx = 0xf2d005a0, _tid = 2}, <zmq::array_item_t<1>> = { _vptr.array_item_t = 0x5cd49a8 <vtable for zmq::pipe_t+96>, _array_index = -1}, <zmq::array_item_t<2>> = { _vptr.array_item_t = 0x5cd49b8 <vtable for zmq::pipe_t+112>, _array_index = -1}, <zmq::array_item_t<3>> = { _vptr.array_item_t = 0x5cd49c8 <vtable for zmq::pipe_t+128>, _array_index = -1}, _in_pipe = 0xc1e3b78, _out_pipe = 0xc0c3640, _in_active = false, _out_active = true, _hwm = 1000, _lwm = 5000, _in_hwm_boost = -1, _out_hwm_boost = -1, _msgs_read = 3283609, _msgs_written = 0, _peers_msgs_read = 0, _peer = 0xbffb858, _sink = 0xc0c6130, _state = zmq::pipe_t::active, _delay = true, _router_socket_routing_id = {_data = 0x0, _size = 0, _owned = true}, _server_socket_routing_id = 0, _conflate = false, _endpoint_uri = {static npos = <optimized out>, _M_dataplus = {<std::allocator<char>> = {<__gnu_cxx::new_allocator<char>> = {<No data fields>}, <No data fields>}, _M_p = 0xf763c478 ""}}}
(gdb) p *$102================================>yqueue state.$109 = {<zmq::ypipe_base_t<zmq::msg_t>> = { _vptr.ypipe_base_t = 0x5c47490 <vtable for zmq::ypipe_t<zmq::msg_t, 256>+8>}, _queue = { _begin_chunk = 0xc202480, _begin_pos = 153, _back_chunk = 0xcd40300, _back_pos = 148, _end_chunk = 0xcd40300, _end_pos = 149, _spare_chunk = {_ptr = 0x0}}, _w = 0xcd42800, _r = 0xc204ac0, _f = 0xcd42800, _c = {_ptr = 0xcd42800}}
Thanks Siva
On Tuesday, November 16, 2021, 09:53:28 PM GMT+5:30, Osiris Pedroso <opedroso at gmail.com> wrote:
Hi Siva,
What Stephane meant was to write some code that shows the issue you are experiencing.
Yes, it's more work, but it would speed up wonders how fast you get a solution for your issue.
Best regards,Osiris
On Tue, Nov 16, 2021, 10:14 donthamsetty sivaprasad via zeromq-dev <zeromq-dev at lists.zeromq.org> wrote:
Stephane .Thanks for the response.To simplify the question ,If more than 8192(output_batch_size) bytes is lying in the dealer socket pipe , will the I/O thread get more than one notification to drain the pipe.? My understanding sender will not notify the I/O thread unless pipe is completely drained.
Please correct me if i am wrong .
ThanksSiva On Tuesday, November 16, 2021, 02:35:37 PM GMT+5:30, Stephane Vales <vales at ingenuity.io> wrote:
Hello Siva,
Rather than pointing the code by itself, maybe it would be better if you could write some minimal examples testing your assumptions. Then, if they are confirmed, you could use them to expose problems in this list or to create issues on github.
Cheers.
Stéphane.
Le 16 nov. 2021 à 07:51, donthamsetty sivaprasad via zeromq-dev <zeromq-dev at lists.zeromq.org> a écrit :
Hi ,The stream engine ( ZMQ dealer=>ZMQ dealer using TCP connection) in the I/O thread at a time reads at most 8172 from the below code. But the writer sends the notification to the I/O thread only if it completely drains the queue in ypie.hpp: flush() based on the compare and swap _c_ptr .
src/config.hpp: out_batch_size = 8192,
src/stream_engine.cpp: while (_outsize < static_cast<size_t> (out_batch_size)) {src/stream_engine.cpp: size_t n = _encoder->encode (&bufptr, out_batch_size - _outsi
Could you please confirm if this is a real problem ? If more than 8172 bytes are lying in pipe and I?O threads drains only 8172 , How does writer sends read notification to I/O thread since it is not fully drained?
Could you please answer this ?
ThanksSiva On Sunday, November 14, 2021, 08:49:10 AM GMT+5:30, donthamsetty sivaprasad via zeromq-dev <zeromq-dev at lists.zeromq.org> wrote:
The version used is zeromq 4.3.1
Thanks in advance .Response is highly appreciated.
ThanksSiva
On Sunday, November 14, 2021, 08:42:00 AM GMT+5:30, donthamsetty sivaprasad via zeromq-dev <zeromq-dev at lists.zeromq.org> wrote:
Hi ,The pipe_t between Dealer socket and I/O thread has been stuck in the sense that msgs are continuously written into the pipe but not drained. And the sender is not doing the flush because the compare and swap pointer is NOT null.I have gone through the code. After the sender has notified the I/O thread for read , I/O thread reads atmost output_batch_size(8072).. Please correct me if I am wrong . If the I/O thread does not fully drain the pipe for any reason , the sender will not notify again since sender depends on c_ptr (cas ptr) being NULL , which will not be set to NULL unless the I/O thread drains the pipe fully.Basically the case is sender is NOT nitifying the I/O thread sicne the c_ptr is not NULL at all and I/O thread is not reading because it will not get the notifcation ..The pipe is continuously filled and messages are not sent out to the ultimate destination due to this..
OR The above situtation could be there is some race in multi core environment where compare and swap operation is not operating consistently . Our system is ARM v7 system .
Please let me know if there are known problems.
THanksSiva_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
_______________________________________________
zeromq-dev mailing list
zeromq-dev at lists.zeromq.org
https://lists.zeromq.org/mailman/listinfo/zeromq-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20211117/75d999fb/attachment.htm>
More information about the zeromq-dev
mailing list