[zeromq-dev] Cleaning up file descriptors for dead router peers

Sat Jun 27 01:58:15 CEST 2015

Hi All,

I've gota trivial bit of code to reproduce this issue on a single host
using iptables to simulate network partition.

https://s3-us-west-2.amazonaws.com/marcin-zmq-example/zmq_test.cpp

The file has comments on how to run the executable, but the short version
is that you start a ZMQ_ROUTER listener which accepts connections from
other peers, and remembers their identities and pings them every 5 seconds.
Then, you start a number of peers which connect to this router and start
pinging it every few seconds.

Once you use the iptables command (also in the comments in the file), the
router can't ping the peers, and the peers can't ping the router. The file
descriptors and connections remain open forever on both sides.

Furthermore, when you undo the iptables block, the connections never come
back.

On Thu, Jun 25, 2015 at 3:35 PM, Marcin Romaszewicz <marcin at brkt.com> wrote:

> I've done some testing of github head version with the heartbeat code, and
> it didn't work for me, I have monotonically increasing file descriptor
> counts, but I'm not sure if I set up my test scenario properly. I had the
> following setup.
>
> 1 ZMQ_ROUTER socket with
>   ZMQ_HEARTBEAT_IVL = 3000 (3 seconds)
>   ZMQ_HEARTBEAT_TTL = 30000 (30 seconds)
>   ZMQ_HEARTBEAT_TIMEOUT = 30000 (30 seconds)
>
> However, due to logistical reasons, my clients which were connecting to
> this ZMQ socket were on ZMQ 4.1.2. Was this a valid test scenario? It would
> take me a couple of days to set up the AMI's to test Router(4.2.0) <->
> client(4.2.0)
>
> Another question:
> If I switch a router socket into ZMQ_ROUTER_RAW mode, send it a disconnect
> fame (peer identity followed by empty frame), then switch off RAW mode,
> would I be doing something completely unsupported, or is it worth a try? My
> tests take a very long time and a lot of work to set up right now, so I'm
> reluctant to try something if it's probably a waste of time.
>
> Thanks,
> -- Marcin
>
>
> On Wed, Jun 24, 2015 at 1:01 PM, Pieter Hintjens <ph at imatix.com> wrote:
>
>> The underlying sockets should indeed error out. Presumably the code
>> isn't handling this properly.
>>
>>
>> On Wed, Jun 24, 2015 at 8:16 PM, Marcin Romaszewicz <marcin at brkt.com>
>> wrote:
>> > Thanks, this probably would solve our problem, however, I'm reluctant to
>> > deploy the bleeding edge from your git repo into our production systems,
>> > even if it does work on my test cluster.
>> >
>> > When I detect that a peer is dead with my own heartbeats, why is it that
>> > attempting to send data to the dead peer doesn't force some kind of
>> > connection cleanup or reset? The underlying os sockets should error out
>> > eventually.
>> >
>> > On Wed, Jun 24, 2015 at 10:52 AM, Pieter Hintjens <ph at imatix.com>
>> wrote:
>> >>
>> >> For what it's worth, we just merged a pull request that adds
>> >> connection heartbeating. It could be fun to see if this solves your
>> >> problem. (In theory it should...)
>> >>
>> >> https://github.com/zeromq/libzmq/pull/1448
>> >>
>> >>
>> >> On Wed, Jun 24, 2015 at 6:48 PM, Marcin Romaszewicz <marcin at brkt.com>
>> >> wrote:
>> >> > Yes, you can easily reproduce this by pulling a network cable or
>> >> > shutting
>> >> > the host down before it can do any sort of TCP connection cleanup.
>> I'm
>> >> > seeing it in AWS when instances get terminated, because they're
>> given so
>> >> > little time to respond to TERM that connections aren't cleaned up.
>> >> >
>> >> > The iptables approach which Francis mentioned should work as well.
>> >> >
>> >> > I'll see if I can come up with a simple example of reproducing this.
>> It
>> >> > might be even possible to repro this on a single machine simply by
>> >> > suspending a peer.
>> >> >
>> >> > -- Marcin
>> >> >
>> >> > On Wed, Jun 24, 2015 at 2:47 AM, Pieter Hintjens <ph at imatix.com>
>> wrote:
>> >> >>
>> >> >> Do you think there's any way to reproduce this in the lab, e.g.
>> >> >> killing a peer before it can shut down TCP properly?
>> >> >>
>> >> >> On Tue, Jun 23, 2015 at 10:08 PM, Marcin Romaszewicz <
>> marcin at brkt.com>
>> >> >> wrote:
>> >> >> > Hi All,
>> >> >> >
>> >> >> > I've got an issue with ZMQ_ROUTER sockets which I'm having a hard
>> >> >> > time
>> >> >> > working around, and I'd love some advice, but I suspect the
>> answer is
>> >> >> > that
>> >> >> > what I want to do isn't possible.
>> >> >> >
>> >> >> > Say I have a router socket listening on a port, and I have peers
>> >> >> > connecting
>> >> >> > and disconnecting randomly over TCP. These peers have random
>> >> >> > identities
>> >> >> > for
>> >> >> > all intents and purposes.
>> >> >> >
>> >> >> > Most of the time, a peer will disconnect "cleanly", meaning the
>> TCP
>> >> >> > connection is terminated via FIN or RST packets, ZMQ cleans up the
>> >> >> > file
>> >> >> > descriptor.
>> >> >> >
>> >> >> > However, some of the time, my peer will die silently, effectively
>> due
>> >> >> > to
>> >> >> > network outage or power outage or something.
>> >> >> >
>> >> >> > In these cases, the router socket keeps the file descriptor around
>> >> >> > forever.
>> >> >> > I know that the peer is dead because all my peers heartbeat to
>> each
>> >> >> > other,
>> >> >> > and the heartbeats have gone away. I thought that trying to send
>> some
>> >> >> > data
>> >> >> > to a dead peer would tear down that connection, since the
>> underlying
>> >> >> > TCP
>> >> >> > socket would eventually start erroring, but it doesn't, zmq must
>> be
>> >> >> > dropping
>> >> >> > my packet before sending it to the underlying socket.
>> >> >> >
>> >> >> > The socket monitor tells me that someone has connected to the
>> router
>> >> >> > socket
>> >> >> > on on its bound port with a specific file descriptor, but I've
>> got so
>> >> >> > many
>> >> >> > of these coming in that I can't associate a specific file
>> descriptor
>> >> >> > with a
>> >> >> > specific peer.
>> >> >> >
>> >> >> > TCP keep-alives don't work all that well in raising errors in a
>> dead
>> >> >> > connection.
>> >> >> >
>> >> >> > What I know on the app side due to my heartbeats is that peer XYZ
>> is
>> >> >> > dead.
>> >> >> > I'd like to tell the router socket to close the underlying file
>> >> >> > descriptor.
>> >> >> > What I know via the monitor is that I have a bunch of file
>> >> >> > descriptors
>> >> >> > open,
>> >> >> > but I can't map them to peers. If I could, I'd just call
>> os.close()
>> >> >> > on
>> >> >> > that
>> >> >> > file descriptor and hopefully ZMQ would handle this gracefully.
>> >> >> >
>> >> >> > Eventually, in a few hours of uptime, my process hits the os file
>> >> >> > descriptor
>> >> >> > limit, and stops receiving new connections on the zeromq level. I
>> can
>> >> >> > have
>> >> >> > the process quit when it detects this, but that forces all the
>> >> >> > functioning
>> >> >> > peers to reconnect and re-do some work, so I'd like to avoid it.
>> >> >> >
>> >> >> > I scanned the previous discussions about it, and there has been
>> >> >> > mention
>> >> >> > of
>> >> >> > exposing this somehow, but I don't see anything along these lines
>> in
>> >> >> > the
>> >> >> > latest API. (looking at 4.1.2 release).
>> >> >> >
>> >> >> > Any suggestions on how I could work around this?
>> >> >> >
>> >> >> > I'm thinking of extending the socket monitor to have a new event
>> >> >> > type,
>> >> >> > like
>> >> >> > ZMQ_PEER_CONNECT/DISCONNECT which passes back the peer ID and file
>> >> >> > descriptor, but I've not gone through the zmq code enough yet to
>> know
>> >> >> > how
>> >> >> > much work this would be.
>> >> >> >
>> >> >> > Thanks in advance,
>> >> >> > -- Marcin
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > _______________________________________________
>> >> >> > zeromq-dev mailing list
>> >> >> > zeromq-dev at lists.zeromq.org
>> >> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >> >> >
>> >> >> _______________________________________________
>> >> >> zeromq-dev mailing list
>> >> >> zeromq-dev at lists.zeromq.org
>> >> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > zeromq-dev mailing list
>> >> > zeromq-dev at lists.zeromq.org
>> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >> >
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150626/83f8c6c8/attachment.htm>