[zeromq-dev] client not reconnecting properly or syn-fin death spiral

Stephen Hemminger shemminger at vyatta.com
Thu Oct 18 04:08:43 CEST 2012


In our application which looks a little like the clonvsrv2 pattern
in the guide. 
If client is rebooted, it gets stuck and never reconnects.

This all using Zmq packages in Debian squeeze-backports (ie 2.2.0+dfsg-2)

Server:
     pub= zocket_new(ctx, ZMQ_PUB);
     zsockopt_set_hwm(pub, 8192);
     zsocket_bind(pub, server_pub_endpoint);

     req = zsocket_new(ctx, ZMQ_ROUTER);
     zsocket_bind(req, server_sync_endpoint);


Client:
     s1 = zsocket_new(ctx, ZMQ_SUB);
     zsocket_set_identity(s1, configured_uuid);
     zsocket_connect(s1, server_pub_endpoint);
 
     s2 = zsocket_new(ctx, ZMQ_DEALER);
     zsocket_set_identity(s2, configure_uuid);
     zsocket_connect(s2, server_sync_endpoint);

After a brief period 173 TCP sockets stuck in TIME-WAIT on the server
and still no connection. It looks like the client is trying to make new
connection for synchronization and the TCP tuple on the server is still in use
or being rejected.

Is ZMQ trying to choose its own ephemeral ports? There appears to be no port
randomization?

Eventually, all the TIME-WAIT sockets clear and the client finds a comfortable
connection.

Each handshake looks like:
8:47:37.782009 IP 192.168.1.31.49339 > 192.168.1.41.5904: Flags [S], seq 4021932813, win 14600, options [mss 1460,sackOK,TS val 113438 ecr 0,nop,wscale 7], length 0
18:47:37.782280 IP 192.168.1.41.5904 > 192.168.1.31.49339: Flags [S.], seq 269340484, ack 4021932814, win 14480, options [mss 1460,sackOK,TS val 166082 ecr 113438,nop,wscale 7], length 0
18:47:37.783683 IP 192.168.1.31.49339 > 192.168.1.41.5904: Flags [.], ack 1, win 115, options [nop,nop,TS val 113438 ecr 166082], length 0
18:47:37.783695 IP 192.168.1.31.49339 > 192.168.1.41.5904: Flags [P.], seq 1:39, ack 1, win 115, options [nop,nop,TS val 113438 ecr 166082], length 38
18:47:37.783906 IP 192.168.1.41.5904 > 192.168.1.31.49339: Flags [.], ack 39, win 114, options [nop,nop,TS val 166083 ecr 113438], length 0
18:47:37.784068 IP 192.168.1.41.5904 > 192.168.1.31.49339: Flags [F.], seq 1, ack 39, win 114, options [nop,nop,TS val 166083 ecr 113438], length 0
18:47:37.784314 IP 192.168.1.31.49339 > 192.168.1.41.5904: Flags [F.], seq 39, ack 2, win 115, options [nop,nop,TS val 113438 ecr 166083], length 0
18:47:37.784539 IP 192.168.1.41.5904 > 192.168.1.31.49339: Flags [.], ack 40, win 114, options [nop,nop,TS val 166083 ecr 113438], length 0
18:47:37.931600 IP 192.168.1.31.49340 > 192.168.1.41.5904: Flags [S], seq 2846485645, win 14600, options [mss 1460,sackOK,TS val 113452 ecr 0,nop,wscale 7], length 0
18:47:37.931933 IP 192.168.1.41.5904 > 192.168.1.31.49340: Flags [S.], seq 875941597, ack 2846485646, win 14480, options [mss 1460,sackOK,TS val 166097 ecr 113452,nop,wscale 7], length 0
18:47:37.932205 IP 192.168.1.31.49340 > 192.168.1.41.5904: Flags [.], ack 1, win 115, options [nop,nop,TS val 113453 ecr 166097], length 0
18:47:37.932226 IP 192.168.1.31.49340 > 192.168.1.41.5904: Flags [P.], seq 1:39, ack 1, win 115, options [nop,nop,TS val 113453 ecr 166097], length 38
18:47:37.932462 IP 192.168.1.41.5904 > 192.168.1.31.49340: Flags [.], ack 39, win 114, options [nop,nop,TS val 166097 ecr 113453], length 0
18:47:37.932686 IP 192.168.1.41.5904 > 192.168.1.31.49340: Flags [F.], seq 1, ack 39, win 114, options [nop,nop,TS val 166098 ecr 113453], length 0
18:47:37.932945 IP 192.168.1.31.49340 > 192.168.1.41.5904: Flags [F.], seq 39, ack 2, win 115, options [nop,nop,TS val 113453 ecr 166098], length 0
18:47:37.933184 IP 192.168.1.41.5904 > 192.168.1.31.49340: Flags [.], ack 40, win 114, options [nop,nop,TS val 166098 ecr 113453], length 0
18:47:38.080230 IP 192.168.1.31.49341 > 192.168.1.41.5904: Flags [S], seq 3167428905, win 14600, options [mss 1460,sackOK,TS val 113467 ecr 0,nop,wscale 7], length 0
18:47:38.080465 IP 192.168.1.41.5904 > 192.168.1.31.49341: Flags [S.], seq 2712872645, ack 3167428906, win 14480, options [mss 1460,sackOK,TS val 166112 ecr 113467,nop,wscale 7], length 0
18:47:38.080724 IP 192.168.1.31.49341 > 192.168.1.41.5904: Flags [.], ack 1, win 115, options [nop,nop,TS val 113467 ecr 166112], length 0
18:47:38.080754 IP 192.168.1.31.49341 > 192.168.1.41.5904: Flags [P.], seq 1:39, ack 1, win 115, options [nop,nop,TS val 113467 ecr 166112], length 38
18:47:38.080973 IP 192.168.1.41.5904 > 192.168.1.31.49341: Flags [.], ack 39, win 114, options [nop,nop,TS val 166112 ecr 113467], length 0
18:47:38.081227 IP 192.168.1.41.5904 > 192.168.1.31.49341: Flags [F.], seq 1, ack 39, win 114, options [nop,nop,TS val 166112 ecr 113467], length 0
18:47:38.081484 IP 192.168.1.31.49341 > 192.168.1.41.5904: Flags [F.], seq 39, ack 2, win 115, options [nop,nop,TS val 113467 ecr 166112], length 0



More information about the zeromq-dev mailing list