[zeromq-dev] zeromq protocol_error handling

James Harvey jamesdillonharvey at gmail.com
Fri May 21 20:38:28 CEST 2021


Hi Bill,

I will check/reply to rest of points later ( im in the pub ) but that is
the point. The protocol_error stops everything so no more reconnect from
the pub socket. Its effectively a zombie as it's terminated but still the
endpoint is registered on the socket.

Cheers

James


On Fri, 21 May 2021, 18:43 Bill Torpey, <wallstprog at gmail.com> wrote:

> Hi James:
>
> A couple of questions:
>
> - Is the SUB socket attempting to reconnect?  (Default is yes).
>
> - Are you activating any of the socket options added by recent changes?
> IIRC none of the new options (e.g., ZMQ_RECONNECT_STOP_CONN_REFUSED)  have
> any effect by default — they need to be activated explicitly.
>
> - Are you tracing socket events?  If not, you should give that a try — it
> will tell you what is going on “under the covers”. You can find an example
> at
> https://github.com/nyfix/OZ/blob/4627b0364be80de4451bf1a80a26c00d0ba9310f/src/transport.c#L1549
>
> I’ll try to take a look when I have some time, but not sure when that will
> be …
>
> Regards,
>
> Bill
>
> On May 21, 2021, at 10:04 AM, James Harvey <jamesdillonharvey at gmail.com>
> wrote:
>
> Thanks Bill
>
> I pulled the latest libzmq and the issue still occurs.
>
> I have tracked it down to the protocol_error handling.  In the case of a
> ZMQ_SUB connecting to a ZMQ_REQ a protocol_error happens (expected) and the
> session is terminated.
>
> The termination does not remove that connection endpoint from the socket.
> This means subsequent calls to socket->connect on the same endpoint (after
> the correct service has resumed) are no ops because SUB can only have one
> connection to a single endpoint.
>
>
> The change below fixes my issue but I'm not sure if it's correct for other
> protocol errors.  I haven't worked on the sessions/pipes before.    I
> noticed in gdb the second session has a _pipe but is not fully created.
>
> https://github.com/zeromq/libzmq/blob/master/src/session_base.cpp#L487
>
>         case i_engine::protocol_error:
> //            if (_pending) {
>             if (_pending || handshaked_) {  // <<<  if handshaked we
> should also terminate pipes.
>                 if (_pipe)
>                     _pipe->terminate (false);
>                 if (_zap_pipe)
>                     _zap_pipe->terminate (false);
>             } else {
>                 terminate ();
>             }
>
> I am happy to create a pull request to discuss if I am on the right track?
>
> I have test code to recreate.
>
> #include "testutil.hpp"
> #include "testutil_unity.hpp"
> #include <iostream>
> #include <stdlib.h>
> SETUP_TEARDOWN_TESTCONTEXT
> char end[] = "tcp://127.0.0.1:55667";
>
> void test_pubreq ()
> {
>
> // SUB up and connect to 55557
>     void *sub = test_context_socket (ZMQ_SUB);
>     TEST_ASSERT_SUCCESS_ERRNO (zmq_setsockopt (sub, ZMQ_SUBSCRIBE, "", 0));
>     TEST_ASSERT_SUCCESS_ERRNO (zmq_connect (sub, end));
>
> // REQ is up incorrectly on 55667
>     void *req = test_context_socket (ZMQ_REQ);
>     TEST_ASSERT_SUCCESS_ERRNO (zmq_bind (req, end));
>     msleep(1000);
>     TEST_ASSERT_SUCCESS_ERRNO (zmq_unbind (req, end));
> // REQ is down
> // At this point the SUB socket has a protocol_error on 55667 (so no
> reconnect) but the socket thinks it still connected to 55667
>
>     msleep(1000);
>
> // PUB correctly comes up on 55667
>     void *pub = test_context_socket (ZMQ_PUB);
>     TEST_ASSERT_SUCCESS_ERRNO (zmq_bind (pub, end));
>
> // NOTE: If we force a disconnect here it works.
> //    TEST_ASSERT_SUCCESS_ERRNO (zmq_disconnect (sub, end));
>
> // Connect again fails
>     TEST_ASSERT_SUCCESS_ERRNO (zmq_connect (sub, end));
>
>     msleep(100);
>
>     send_string_expect_success (pub, "Hello", 0);
>
>     msleep(100);
>
>     recv_string_expect_success (sub, "Hello", 0);
>
>     msleep(100);
>
>     test_context_socket_close (pub);
>     test_context_socket_close (req);
>     test_context_socket_close (sub);
>
> }
>
> int main (void)
> {
>     setup_test_environment ();
>
>     UNITY_BEGIN ();
>     RUN_TEST (test_pubreq);
>     return UNITY_END ();
> }
>
> On Thu, May 20, 2021 at 4:56 PM Bill Torpey <wallstprog at gmail.com> wrote:
>
>> Sorry — meant to get back to you sooner, but it’s been a crazy week.
>>
>> You don’t say what version you’re running, but there have been some
>> changes in that area not that long ago — check these out and see if they
>> help:
>>
>> https://github.com/zeromq/libzmq/pull/3831
>>
>> https://github.com/zeromq/libzmq/pull/3960
>>
>> https://github.com/zeromq/libzmq/pull/4053
>>
>> Good luck.
>>
>> Bill
>>
>>
>> On May 20, 2021, at 10:26 AM, James Harvey <jamesdillonharvey at gmail.com>
>> wrote:
>>
>> Hi,
>>
>> I will try and simplify my previous long email.
>>
>> If a stream gets into a protocol error state  (e.g tcp SUB connect to
>> REQ)
>>
>> Should the information (connection is terminated) be passed somehow back
>> to the parent socket so if connect() is called again it attempts to connect
>> rather than a no-op.
>>
>> OR
>>
>> Should we add a protocol error event to socket monitor so the calling
>> process can handle it  by calling disconnect/connect
>>
>> Just want some clarification so I work on the correct code.
>>
>> Thanks
>>
>> James
>>
>> On Thu, May 13, 2021 at 4:48 PM James Harvey <jamesdillonharvey at gmail.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have a rare/random bug that causes my ZMQ_SUB socket to fail for a
>>> certain endpoint with no way to track/notify.  Yes it's because a SUB
>>> connects to a REQ socket but once you start to use zeromq for lots of
>>> transient systems in a large company this kind of thing will happen
>>> occasionally.
>>>
>>> The process happens like this:
>>>
>>>   - ZMQ_PUB binds on 1.2.3.4:44444 (ephemeral)
>>>   - ZMQ_SUB connects to 1.2.3.4:44444 (data flows)
>>>   - ZMQ_PUB goes down
>>>   - Unrelated process (ZMQ_REQ) comes up and grabs the same
>>> 1.2.3.4:44444 as its ephemeral
>>>   - ZMQ_SUB has not yet been told to disconnect so it reconnects to the
>>> ZMQ_REQ
>>>   - protocol error happens and the connection is terminated in the
>>> session/engine
>>>   - Now a good ZMQ_PUB comes up and binds on 1.2.3.4:44444
>>>   - ZMQ_SUB gets new instruction to connect()
>>>   - connect() just returns noop.
>>>     - The socket_base thinks it still has a valid endpoint and SUB only
>>> connects once to each endpoint.
>>>   - At this point there are no errors and no data flowing.
>>>
>>> My question is, should the protocol_error in the session propagate up to
>>> remove the endpoint from the socket?
>>>
>>> If yes I can look at adding that, if no do you have any suggestions?
>>>
>>> Thanks for your time
>>>
>>> James
>>>
>>> Some links to the code:
>>>
>>> If socket is SUB and the endpoint is present dont connect.
>>> https://github.com/zeromq/libzmq/blob/master/src/socket_base.cpp#L901
>>>
>>> terminate with no reconnect on protocol_error
>>> https://github.com/zeromq/libzmq/blob/master/src/session_base.cpp#L486
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20210521/b798e93c/attachment.htm>


More information about the zeromq-dev mailing list