[zeromq-dev] ZMQ reconnect/ephemeral ports

Bill Torpey wallstprog at gmail.com
Sat Sep 16 20:34:04 CEST 2017 

Hi Luca:

Just a gentle reminder to add an issue so this can be tracked (or let me know if you’d prefer that I do that).

Thanks!

Bill


> On Sep 2, 2017, at 1:21 PM, Luca Boccassi <luca.boccassi at gmail.com> wrote:
> 
> On Sat, 2017-09-02 at 12:02 -0400, Bill Torpey wrote:
>> Thanks again, Luca!
>> 
>> For now, I’m going to go with disabling reconnect on the “data”
>> sockets — that seems to be the best solution for my use case
>> (connecting to endpoints that were returned by the peer binding to an
>> unspecified (“wildcard”) port — e.g., "tcp://<interface>:*" in ZMQ).
>> 
>> This assumes that ZMQ will completely forget about the endpoint
>> if/when it is disconnected, if it is set not to reconnect.  Otherwise
>> I might run afoul of ZMQ’s silently ignoring connections to endpoints
>> that it already knows about:  https://github.com/zeromq/libzmq/issues
>> /788 <https://github.com/zeromq/libzmq/issues/788> (e.g., in the case
>> where another process later happens to be assigned the same ephemeral
>> port).
>> 
>> I’ve done a quick scan of the libzmq code (v4.2.2) and it doesn’t
>> appear that the endpoint is removed in the case of a (terminal)
>> disconnect.  If you can confirm/deny this behavior, that would be
>> helpful.  Failing that, I guess I’ll need to test this in the
>> debugger — any hints on how best to do this would also be much
>> appreciated.
>> 
>> Regards,
>> 
>> Bill
> 
> Yes it doesn't look like it removes the endpoint - I guess it's a
> corner case that's missed. I'll open an issue.
> 
> BTW all these things are very quick and easy to try with Python on
> Linux. Just install pyzmq, open a python3 terminal and:
> 
> import zmq
> ctx = zmq.Context.instance()
> rep = ctx.socket(zmq.REP)
> rep.bind("tcp://127.0.0.1:12345")
> req = ctx.socket(zmq.REQ)
> req.connect("tcp://127.0.0.1:12345")
> req.send_string("hello")
> rep.recv()
> rep.send_string("hallo")
> req.recv()
> rep.unbind("tcp://127.0.0.1:12345")
> rep.close()
> rep = ctx.socket(zmq.REP)
> rep.bind("tcp://127.0.0.1:12345")
> req.send_string("hello")
> rep.recv()
> rep.send_string("hallo")
> req.recv()
> rep.unbind("tcp://127.0.0.1:12345")
> rep.close()
> req.close()
> rep = ctx.socket(zmq.REP)
> rep.bind("tcp://127.0.0.1:12345")
> req = ctx.socket(zmq.REQ)
> req.setsockopt(zmq.RECONNECT_IVL, -1)req.connect("tcp://127.0.0.1:12345")
> req.send_string("hello")
> rep.recv()
> rep.send_string("hallo")
> req.recv()
> rep.unbind("tcp://127.0.0.1:12345")
> rep.close()
> rep = ctx.socket(zmq.REP)
> rep.bind("tcp://127.0.0.1:12345")
> req.send_string("hello")
> rep.recv()
> 
> This last one won't receive the message
> 
>>> On Sep 1, 2017, at 6:19 PM, Luca Boccassi <luca.boccassi at gmail.com>
>>> wrote:
>>> 
>>> On Fri, 2017-09-01 at 18:03 -0400, Bill Torpey wrote:
>>>> Thanks Luca!  That was very helpful.
>>>> 
>>>> Although it leads to a couple of other questions:
>>>> 
>>>> - Can I assume that a ZMQ disconnect of a tcp endpoint would only
>>>> occur if the underlying TCP socket is closed by the OS? Or are
>>>> there
>>>> conditions in which ZMQ will proactively disconnect the TCP
>>>> socket
>>>> and try to reconnect?
>>> 
>>> Normally that's the case - you can set up heartbeating with the
>>> appropriate options and that will kill a connection if there's no
>>> answer
>>> 
>>>> - I see that there is a sockopt (ZMQ_RECONNECT_IVL) that can be
>>>> set
>>>> to -1 to disable reconnection entirely.  In my case, the the
>>>> “data”
>>>> socket pair will *always* connect to an ephemeral port, so I
>>>> *never*
>>>> want to reconnect.  Would this be a reasonable option in my case,
>>>> do
>>>> you think?
>>> 
>>> If that makes sense for your application, go for it - in these
>>> cases
>>> the only way to be sure is to test it and see how it works
>>> 
>>>> - Would there be any interest in a patch that would disable
>>>> reconnects (controlled by sockopt) for ephemeral ports only?  I’m
>>>> guessing that reconnecting mostly makes sense with well-known
>>>> ports,
>>>> so something like this may be of general interest?
>>> 
>>> If by ephemeral port you mean anything over 1024, then actually in
>>> most
>>> applications I've seen it's always useful to reconnect, and the
>>> existing option should be enough for those cases where it's not
>>> desired
>>> - we don't want to duplicate functionality
>>> 
>>>> Thanks again!
>>>> 
>>>> Bill 
>>>> 
>>>>> On Sep 1, 2017, at 5:30 PM, Luca Boccassi <luca.boccassi at gmail.
>>>>> com>
>>>>> wrote:
>>>>> 
>>>>> On Fri, 2017-09-01 at 16:59 -0400, Bill Torpey wrote:
>>>>>> I'm curious about how ZMQ handles re-connection.  I
>>>>>> understand
>>>>>> that
>>>>>> re-connection is supposed to happen "automagically" under the
>>>>>> covers,
>>>>>> but that poses an interesting question.
>>>>>> 
>>>>>> To make a long story short, the application I'm working on
>>>>>> uses
>>>>>> pub/sub sockets over TCP. and works like follows:
>>>>>> 
>>>>>> At startup:
>>>>>> 1.  connects to a proxy/broker at a well-known address, using
>>>>>> a
>>>>>> pub/sub socket pair ("discovery");
>>>>>> 2.  subscribes to a well-known topic using the "discovery"
>>>>>> sub
>>>>>> socket;
>>>>>> 3.  binds a different pub/sub socket pair ("data") and
>>>>>> retrieves
>>>>>> the
>>>>>> actual endpoints assigned;
>>>>>> 4.  publishes the "data" endpoints from step 3 on the
>>>>>> "discovery"
>>>>>> pub
>>>>>> socket; 
>>>>>> 
>>>>>> When the application receives a message on the "discovery"
>>>>>> sub
>>>>>> socket, it connects the "data" socket pair to the endpoints
>>>>>> specified
>>>>>> in the "discovery" message.
>>>>>> 
>>>>>> So far, this seems to be working relatively well, and allows
>>>>>> the
>>>>>> high-volume, low-latency "data" messages to be sent/received
>>>>>> directly
>>>>>> between peers, avoiding the extra hop caused by a
>>>>>> proxy/broker
>>>>>> connection.  The discovery messages use the proxy/broker, but
>>>>>> since
>>>>>> these are (very) low-volume the extra hop doesn't
>>>>>> matter.  The
>>>>>> use of
>>>>>> the proxy also eliminates the "slow joiner" problem that can
>>>>>> happen
>>>>>> with other configurations.
>>>>>> 
>>>>>> My question is what happens when one of the "data" peer
>>>>>> sockets
>>>>>> disconnects.  Since ZMQ (apparently) keeps trying to
>>>>>> reconnect,
>>>>>> what
>>>>>> would prevent another process from binding to the same
>>>>>> ephemeral
>>>>>> port?  
>>>>>> 
>>>>>> - Can I assume that if the new application at that port is
>>>>>> not a
>>>>>> ZMQ
>>>>>> application, that the reconnect will (silently) fail, and
>>>>>> continue to
>>>>>> be retried?
>>>>> 
>>>>> The ZMTP handshake would fail, so yes.
>>>>> 
>>>>>> - What if the new application at that port *IS* a ZMQ
>>>>>> application?  Would the reconnect succeed?  And if so, what
>>>>>> would
>>>>>> happen if it's a *DIFFERENT* ZMQ application, and the
>>>>>> messages
>>>>>> that
>>>>>> it's sending/receiving don't match what the original
>>>>>> application
>>>>>> expects?
>>>>> 
>>>>> Depends on how you handle it in your application. If you have
>>>>> security
>>>>> concerns, then use CURVE with authentication so that only
>>>>> authorised
>>>>> peers can connect.
>>>>> 
>>>>>> It's reasonable for the application to publish a disconnect
>>>>>> message
>>>>>> when it terminates normally, and the connected peers can
>>>>>> disconnect
>>>>>> that endpoint.  But, applications don't always terminate
>>>>>> normally
>>>>>> ;-)
>>>>> 
>>>>> That's a common pattern. But the application needs to handle
>>>>> unexpected
>>>>> data somewhat gracefully. What that means is entirely up to the
>>>>> application - as far as the library is concerned, if the
>>>>> handshake
>>>>> succeeds then it's all good (hence the use case for CURVE).
>>>>> 
>>>>>> Any guidance, hints or tips would be much appreciated --
>>>>>> thanks
>>>>>> in
>>>>>> advance!
>>>>> 
>>>>> -- 
>>>>> Kind regards,
>>>>> Luca Boccassi_______________________________________________
>>>>> zeromq-dev mailing list
>>>>> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org
>>>>>> <mailto:zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.
>>>>> zeromq.org>>
>>>>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
>>>>> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>>
>>>> 
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>>>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
>>> 
>>> -- 
>>> Kind regards,
>>> Luca Boccassi_______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org <mailto:zeromq-dev at lists.zeromq.org>
>>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> <https://lists.zeromq.org/mailman/listinfo/zeromq-dev>
>> 
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> -- 
> Kind regards,
> Luca Boccassi_______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev

More information about the zeromq-dev mailing list