[zeromq-dev] Destroying 0MQ context gets indefinitely stuck/hangs, despite linger=0
Tomas Krajca
tomas at repositpower.com
Thu May 18 07:00:12 CEST 2017
Hi Luca,
Sorry for the delay.
> If I understand correctly the pattern is:
>
> 1) ROUTER binds over TCP and enables CURVE
> 2) REQ connects over TCP with CURVE
> 3) REQ sends a message
> 4) REQ waits for a response that never arrives
>
> What is the timeout value, and how is it checked (poll, socket option,
> etc)?
> Can you tell if the ROUTER receives the request and sends a reply?
>
The timeout value is 20s but it doesn't seem to matter. I use pyzmq
which I think uses the standard zmq_poll() on that socket. The issue is
that the REQ never manages to send its message to the ROUTER. It looks
like it gets into an invalid state before sending that message. The REQ
does a successful CURVE handshake and then goes completely silent (it
doesn't even send the INITIATE message).
It's probably good to note that we create and close 1000s of those REQ
sockets every hour, each of them sending and receiving only one message.
So I thought perhaps two of those REQs eventually end up with the same
identity (we don't set identity explicitely, it's automatically set by
libzmq) which then confuses libzmq or the ROUTER. According to the
documentation, if there are two REQs with the same identity, the second
REQ gets ignored. That seems to fit as the second REQ successfully
authenticates via CURVE but then it gets ignored. I tried to work on a
reproduction but couldn't reproduce the issue even if I set the identity
manually to exactly the same string. Also, the tcpdump capture of the
REQ with duplicate identity seems different to the capture I got for my
issue.
The capture for my issue finishes before the INITIATE message:
00000000 ff 00 00 00 00 00 00 00 01 7f ........ ..
00000000 ff 00 00 00 00 00 00 00 01 7f 03 ........
...
0000000A 03 00 43 55 52 56 45 00 00 00 00 00 00 00 00 00 ..CURVE.
........
0000001A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
0000002A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
0000003A 00 00 00 00 00 00 ......
0000000B 00 43 55 52 56 45 00 00 00 00 00 00 00 00 00 00 .CURVE..
........
0000001B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
0000002B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
0000003B 00 00 00 00 00 .....
00000040 04 c8 05 48 45 4c 4c 4f 01 00 00 00 00 00 00 00 ...HELLO
........
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
00000090 00 00 d2 cd f4 27 96 45 4e 1b 58 d6 61 d0 45 55 .....'.E
N.X.a.EU
000000A0 85 2b 5a fd b4 1e 04 9a c6 d3 1d 15 18 ad f9 68 .+Z.....
.......h
000000B0 bd 64 00 00 00 00 00 00 00 01 0b e7 47 33 d5 66 .d......
....G3.f
000000C0 f1 5a 1a f6 29 88 c0 20 2a 7e 46 32 5e 5e 20 0c .Z..)..
*~F2^^ .
000000D0 4a 0b 40 98 c5 b5 5c 76 ec 8b 08 47 de c6 dd 17 J. at ...\v
...G....
000000E0 ad 4e 5a 54 97 7d 21 a3 1e 3a 78 f6 78 da 30 e4 .NZT.}!.
.:x.x.0.
000000F0 57 14 d7 d5 a7 b9 de 4e 06 f4 04 6d ff 9a ca bd W......N
...m....
00000100 27 72 33 cc 1f af ca 89 89 5a 'r3..... .Z
00000040 04 a8 07 57 45 4c 43 4f 4d 45 bd 7d 68 e7 c4 11 ...WELCO
ME.}h...
00000050 f0 bb b5 39 ce 24 cb 5d e1 bc ce 34 7e ba f8 63 ...9.$.]
...4~..c
00000060 1f c1 f0 4e 73 b5 f8 ad 39 97 5d 84 3d 9a 41 e8 ...Ns...
9.].=.A.
00000070 58 4f fc b3 fd 26 06 5a ac b4 6a 95 21 7a 87 6d XO...&.Z
..j.!z.m
00000080 be 57 21 f5 b4 92 a5 9b 9f 7c 25 02 a8 a8 ae f5 .W!.....
.|%.....
00000090 f6 b4 b6 af 21 a5 cb 9f b6 ba 12 a5 be 36 49 3c ....!...
.....6I<
000000A0 50 06 c8 68 7c d5 e9 61 24 7c a1 66 50 ba 4c 59 P..h|..a
$|.fP.LY
000000B0 e1 27 3e 55 66 be 84 f3 fd 2c df 2e 10 6d 2e 88 .'>Uf...
.,...m..
000000C0 53 ac 32 b4 70 0a 49 01 e7 d7 88 db 61 00 dd e9 S.2.p.I.
....a...
000000D0 b0 3a 8b 0b 8e fb 07 e8 53 d3 3d 5c 95 06 30 91 .:......
S.=\..0.
000000E0 f1 81 85 61 2d 33 e4 70 5d 43 ...a-3.p
]C
While the duplicate REQ sends the INITIATE message:
00000000 ff 00 00 00 00 00 00 00 06 7f ........ ..
00000000 ff 00 00 00 00 00 00 00 01 7f 03 ........ ...
0000000A 03 00 43 55 52 56 45 00 00 00 00 00 00 00 00 00 ..CURVE. ........
0000001A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0000002A 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
0000003A 00 00 00 00 00 00 ......
0000000B 00 43 55 52 56 45 00 00 00 00 00 00 00 00 00 00 .CURVE..
........
0000001B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
0000002B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........
........
0000003B 00 00 00 00 00 .....
00000040 04 c8 05 48 45 4c 4c 4f 01 00 00 00 00 00 00 00 ...HELLO ........
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00000060 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........
00000090 00 00 93 a5 83 2c af a7 b7 ba 6d b9 15 c4 c4 fa .....,.. ..m.....
000000A0 3a 73 5b 7a e3 f0 19 b2 f0 4b 2a 0b 8c dd 96 52 :s[z.... .K*....R
000000B0 3e 03 00 00 00 00 00 00 00 01 0f 9d 49 37 87 87 >....... ....I7..
000000C0 86 68 35 2a 5d 1f 83 df 0a 40 8b e6 1e c9 72 0b .h5*]... . at ....r.
000000D0 32 4b 6c b5 27 cc 38 07 e9 d3 ce 45 2c 87 82 92 2Kl.'.8. ...E,...
000000E0 78 70 18 54 23 8b b5 b1 e5 f9 27 61 4d 44 15 dd xp.T#... ..'aMD..
000000F0 af 1c ea 9a 47 1e fa 57 a5 a5 8f e8 72 67 c8 25 ....G..W ....rg.%
00000100 2b 3a 93 c8 a7 30 36 13 23 36 +:...06. #6
00000040 04 a8 07 57 45 4c 43 4f 4d 45 9a ba 8a 25 96 d6 ...WELCO
ME...%..
00000050 97 00 5f 91 6f 36 f8 0e 13 ac 1e 9d 27 1e f8 78 .._.o6..
....'..x
00000060 db 8c a2 87 88 dc 03 55 4c 6c 40 ec 76 2f 1e 2e .......U
Ll at .v/..
00000070 c9 6a 0f 05 d6 10 d5 28 01 38 b7 e0 8f 52 91 f0 .j.....(
.8...R..
00000080 af d9 ea 4d 59 51 bd f6 b1 d2 38 81 a8 6f a7 43 ...MYQ..
..8..o.C
00000090 e1 26 80 3c 17 61 38 5e 53 9a 51 d7 5f f9 2e cb .&.<.a8^
S.Q._...
000000A0 23 04 64 82 5f 9b 97 86 1b 36 0c 5e c4 65 28 f1 #.d._...
.6.^.e(.
000000B0 5a 15 64 da ab a9 2c 9f 07 a2 01 d9 52 c2 88 c7 Z.d...,.
....R...
000000C0 74 cd bb 70 da 1d 05 24 1f 18 74 2d 96 86 51 4d t..p...$
..t-..QM
000000D0 cc 13 a6 9a 29 db 31 2c 6c 8f 48 f3 32 81 c2 d6 ....).1,
l.H.2...
000000E0 67 56 24 6a 64 94 ff de 85 17 gV$jd... ..
0000010A 06 00 00 00 00 00 00 01 26 08 49 4e 49 54 49 41 ........ &.INITIA
0000011A 54 45 e3 2f c3 db 51 d6 02 88 ff 6e ed 97 d5 36 TE./..Q. ...n...6
0000012A 23 ff 94 0e ad 84 17 de cd cf 8e c0 54 ab 50 eb #....... ....T.P.
0000013A 01 87 8d a3 a0 93 38 eb 39 c4 1f b0 37 a3 61 65 ......8. 9...7.ae
0000014A db d3 90 64 4b 28 c3 aa 22 15 e7 cd 54 9e 81 75 ...dK(.. "...T..u
0000015A 04 b2 2c 1e a8 db 93 15 23 f0 78 97 b0 81 b5 44 ..,..... #.x....D
0000016A 08 f0 cc 89 e3 77 f7 22 f9 c1 ca fb db 7a 9e 77 .....w." .....z.w
0000017A 35 6d 00 00 00 00 00 00 00 02 84 2d 03 34 cb cf 5m...... ...-.4..
0000018A d1 84 06 de 69 d8 93 78 b2 04 28 08 b0 29 66 18 ....i..x ..(..)f.
0000019A a8 27 71 6c 1c a1 a6 09 04 e4 87 94 6a 3e e9 c7 .'ql.... ....j>..
000001AA 18 dc 66 de 08 4a 25 f4 56 51 4e f4 69 ee f4 bd ..f..J%. VQN.i...
000001BA eb c9 43 dd 0a 2d c9 67 5f 19 ac 3a 2f 1c b2 05 ..C..-.g _..:/...
000001CA cc 5f 1f 67 97 41 6f fc 68 a6 ea 02 0d a4 0e 0c ._.g.Ao. h.......
000001DA 5f ea c1 6d 80 03 fb 85 6b 16 12 81 9c 6f 36 c7 _..m.... k....o6.
000001EA b1 2a 3f 97 fb 4d 88 11 2f 07 82 d8 fc cb 9f a4 .*?..M.. /.......
000001FA 7e 38 77 be fd 5a 85 39 da 1a 63 6a 1c f7 f4 52 ~8w..Z.9 ..cj...R
0000020A 10 e9 1c 3c 93 a2 3b a2 3d 62 b0 76 09 9b 6e b3 ...<..;. =b.v..n.
0000021A 71 be 53 cf dd 73 fe 86 3e 50 69 44 eb ae fc 79 q.S..s.. >PiD...y
0000022A 35 de 9a 95 67 a9 dc 01 5c ce ca c5 71 93 e0 5...g... \...q..
000000EA 04 41 05 52 45 41 44 59 00 00 00 00 00 00 00 01 .A.READY
........
000000FA 2d 4d 97 72 bc fd be 5e 17 b5 6f cb 2b 9c 6e ac -M.r...^
..o.+.n.
0000010A e0 da 99 4c 7d f9 78 76 b8 fe 85 b1 88 64 0e 99 ...L}.xv
.....d..
0000011A 5b 02 1f b0 2c ae 59 c5 27 e0 a7 56 c4 1b 7a a7 [...,.Y.
'..V..z.
0000012A c6 85 36 ..6
> Message: 1
> Date: Mon, 15 May 2017 13:39:26 +0100
> From: Luca Boccassi <luca.boccassi at gmail.com>
> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
> Subject: Re: [zeromq-dev] zeromq-dev Digest, Vol 14, Issue 7
> Message-ID: <1494851966.15080.2.camel at gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> On Mon, 2017-05-15 at 11:57 +1000, Tomas Krajca wrote:
>> Hi Luca,
>>
>> Having a single/shared context didn't help. As soon as the REQ
>> client
>> timed out, 0MQ seemed to get confused and started leaking file
>> handles.
>> It ended up with 100s of those [eventfd] open file descriptors.
>>
>> I am not sure if it's an issue with the reaper. My feeling is that
>> the
>> core issue is the REQ client going silent after successfully
>> establishing the CURVE authentication. I have no idea if 0MQ hits
>> some
>> system limit or if there is a bug of some sort but that's the odd
>> thing
>> for me - successful CURVE handshake/authentication and then silence.
>>
>> For now, I've got a cron job that restarts stuck workers so it's not
>> that urgent/critical. Anyway, I've got some time to do a bit more
>> digging or testing but I don't quite know where to start.
>>
>> Thanks,
>> Tomas
>
> Ok, thanks for confirming this.
>
> I would recommend 2 following steps:
>
> 1) Try with the latest libzmq master and see if the problem still
> happens
> 2) If it does, try to have a minimal test case that reproduces the
> issue with just libzmq - removing the layers of bindings helps a lot
> when trying to reproduce a problem.
>
> If I understand correctly the pattern is:
>
> 1) ROUTER binds over TCP and enables CURVE
> 2) REQ connects over TCP with CURVE
> 3) REQ sends a message
> 4) REQ waits for a response that never arrives
>
> What is the timeout value, and how is it checked (poll, socket option,
> etc)?
> Can you tell if the ROUTER receives the request and sends a reply?
>
>>> Date: Thu, 11 May 2017 11:38:35 +0100
>>> From: Luca Boccassi <luca.boccassi at gmail.com>
>>> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
>>> Subject: Re: [zeromq-dev] Destroying 0MQ context gets indefinitely,
>>> stuck/hangs despite linger=0
>>> Message-ID: <1494499115.4886.3.camel at gmail.com>
>>> Content-Type: text/plain; charset="utf-8"
>>>
>>> On Wed, 2017-05-10 at 15:21 +1000, Tomas Krajca wrote:
>>>> Hi Luca and thanks for your reply.
>>>>
>>>> > Note that these are two well-known anti-patterns. The context
>>>> is
>>>> > intended to be shared and be unique in an application, and
>>>> live
>>>> for as
>>>> > long as the process does, and the sockets are meant to be
>>>> long
>>>> lived as
>>>> > well.
>>>> >
>>>> > I would recommend refactoring and, at the very least, use a
>>>> single
>>>> > context for the duration of your application.
>>>> >
>>>>
>>>> I always thought that having separate context was safer. I will
>>>> refactor the application to use one context for all the
>>>> clients/sockets
>>>> and see if it makes any difference.
>>>>
>>>> I wonder if that's going eliminate the initial problem though. If
>>>> the
>>>> sockets really get somehow stuck/into an inconsistent state, then
>>>> I
>>>> imagine they will just "leak" and stay in that context forever,
>>>> possibly
>>>> preventing the app from a proper termination.
>>>
>>> There could be an unknown race with the reaper. It should help in
>>> that
>>> case.
>>>
>>>> The client usually is long lived for as long as the app lives but
>>>> in
>>>> this particular app, it's a bit more special in that the separate
>>>> tasks
>>>> just use the clients to fetch some data in a standardized way, do
>>>> their
>>>> computation and exit. These tasks are periodically spawned by
>>>> celery.
>>>>
>>>>> Message: 1
>>>>> Date: Mon, 08 May 2017 11:58:42 +0100
>>>>> From: Luca Boccassi <luca.boccassi at gmail.com>
>>>>> To: ZeroMQ development list <zeromq-dev at lists.zeromq.org>
>>>>> Cc: "developers at repositpower.com" <developers at repositpower.com>
>>>>> Subject: Re: [zeromq-dev] Destroying 0MQ context gets
>>>>> indefinitely
>>>>> stuck/hangs despite linger=0
>>>>> Message-ID: <1494241122.11089.5.camel at gmail.com>
>>>>> Content-Type: text/plain; charset="utf-8"
>>>>>
>>>>> On Mon, 2017-05-08 at 11:08 +1000, Tomas Krajca wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> I have come across a weird/bad bug, I believe.
>>>>>>
>>>>>> I run libzmq 4.1.6 and pyzmq 16.0.2. This happens on both
>>>>>> Centos
>>>>>> 6
>>>>>> and
>>>>>> Centos 7.
>>>>>>
>>>>>> The application is a celery worker that runs 16 worker
>>>>>> threads.
>>>>>> Each
>>>>>> worker thread instantiates a 0MQ-based client, gets data and
>>>>>> then
>>>>>> closes
>>>>>> this client. The 0MQ-based client creates its own 0MQ context
>>>>>> and
>>>>>> terminates it on exit. Nothing is shared between the threads
>>>>>> or
>>>>>> clients,
>>>>>> every client processes only one request and then it's fully
>>>>>> terminated.
>>>>>>
>>>>>> The client itself is a REQ socket which uses CURVE
>>>>>> authentication
>>>>>> to
>>>>>> authenticate with a ROUTER socket on the server side. The REQ
>>>>>> socket
>>>>>> has
>>>>>> linger=0. Almost always, the REQ socket issues request, gets
>>>>>> back
>>>>>> response, closes the socket, destroys its context, all is
>>>>>> good.
>>>>>> Once
>>>>>> every one or two days though, the REQ socket times out when
>>>>>> waiting
>>>>>> for
>>>>>> the response from the ROUTER server, it then successfully
>>>>>> closes
>>>>>> the
>>>>>> socket but indefinitely hangs when it goes on to destroy the
>>>>>> context.
>>>>>
>>>>> Note that these are two well-known anti-patterns. The context
>>>>> is
>>>>> intended to be shared and be unique in an application, and live
>>>>> for
>>>>> as
>>>>> long as the process does, and the sockets are meant to be long
>>>>> lived as
>>>>> well.
>>>>>
>>>>> I would recommend refactoring and, at the very least, use a
>>>>> single
>>>>> context for the duration of your application.
>>>>>
>>>>>> This runs in a data center on 1Gb/s LAN so the responses
>>>>>> usually
>>>>>> finish
>>>>>> in under a second, the timeout is 20s. My theory is that the
>>>>>> socket
>>>>>> gets
>>>>>> into a weird state and that's why it times out and blocks the
>>>>>> context
>>>>>> termination.
>>>>>>
>>>>>> I ran a tcpdump and it turns out that the REQ client
>>>>>> successfully
>>>>>> authenticates with the ROUTER server but then it goes
>>>>>> completely
>>>>>> silent
>>>>>> for those 20 odd seconds.
>>>>>>
>>>>>> Here is a tcpdump capture of a stuck REQ client -
>>>>>> https://pastebin.com/HxWAp6SQ. Here is a tcpdump capture of a
>>>>>> normal
>>>>>> communication - https://pastebin.com/qCi1jTp0. This is a full
>>>>>> backtrace
>>>>>> (after SIGABRT signal to the stuck application) -
>>>>>> https://pastebin.com/jHdZS4VU
>>>>>>
>>>>>> Here is ulimit:
>>>>>>
>>>>>> [root at auhwbesap001 tomask]# cat /proc/311/limits
>>>>>> Limit Soft Limit Hard Limit
>>>>>> Units
>>>>>> Max cpu time unlimited unlimited
>>>>>> seconds
>>>>>> Max file size unlimited unlimited
>>>>>> bytes
>>>>>> Max data size unlimited unlimited
>>>>>> bytes
>>>>>> Max stack size 8388608 unlimited
>>>>>> bytes
>>>>>> Max core file size 0 unlimited
>>>>>> bytes
>>>>>> Max resident set unlimited unlimited
>>>>>> bytes
>>>>>> Max processes 31141 31141
>>>>>> processes
>>>>>> Max open files 8196 8196
>>>>>> files
>>>>>> Max locked memory 65536 65536
>>>>>> bytes
>>>>>> Max address space unlimited unlimited
>>>>>> bytes
>>>>>> Max file locks unlimited unlimited
>>>>>> locks
>>>>>> Max pending signals 31141 31141
>>>>>> signals
>>>>>> Max msgqueue size 819200 819200
>>>>>> bytes
>>>>>> Max nice priority 0 0
>>>>>> Max realtime priority 0 0
>>>>>> Max realtime
>>>>>> timeout unlimited unlimited us
>>>>>>
>>>>>>
>>>>>> The application doesn't seem to get over any of the limits,
>>>>>> it
>>>>>> usually
>>>>>> hovers between 100 and 200 open file handlers.
>>>>>>
>>>>>> I tried to swap the REQ socket for a DEALER socket but that
>>>>>> didn't
>>>>>> help,
>>>>>> the context eventually hung as well.
>>>>>>
>>>>>> I also tried to set ZMQ_BLOCKY to 0 and/or ZMQ_HANDSHAKE_IVL
>>>>>> to
>>>>>> 100ms
>>>>>> but the context still eventually hung.
>>>>>>
>>>>>> I looked into the C++ code of libzmq but would need some
>>>>>> guidance
>>>>>> to
>>>>>> troubleshoot this as I am primarily a python programmer.
>>>>>>
>>>>>> I think we had a similar issue back in 2014 -
>>>>>> https://lists.zeromq.org/pipermail/zeromq-dev/2014-September/
>>>>>> 0267
>>>>>> 52.h
>>>>>> tml. From
>>>>>> memory, the tcpdump capture also showed the client/REQ going
>>>>>> silent
>>>>>> after the successful initial CURVE authentication but at that
>>>>>> time
>>>>>> the
>>>>>> server/ROUTER application was crashing with an assertion.
>>>>>>
>>>>>> I am happy to do any more debugging.
>>>>>>
>>>>>> Thanks in advance for any help/pointers.
>>>>>
>>>>> -------------- next part --------------
>>>>> A non-text attachment was scrubbed...
>>>>> Name: signature.asc
>>>>> Type: application/pgp-signature
>>>>> Size: 488 bytes
>>>>> Desc: This is a digitally signed message part
>>>>> URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments
>>>>> /201
>>>>> 70508/fd178ae0/attachment-0001.sig>
>>>>>
>>>>> ------------------------------
>>>>
>>>> <http://www.repositpower.com/>
>>>>
>>>> *Tomas Krajca *
>>>> Software architect
>>>> m. 02 6162 0277
>>>> e. tomas at repositpower.com
>>>> <https://twitter.com/RepositPower>
>>>> <https://www.facebook.com/Reposit-Power-1423585874607903/>
>>>> <https://www.linkedin.com/company/reposit-power>
>>>> _______________________________________________
>>>> zeromq-dev mailing list
>>>> zeromq-dev at lists.zeromq.org
>>>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> -------------- next part --------------
>>> A non-text attachment was scrubbed...
>>> Name: signature.asc
>>> Type: application/pgp-signature
>>> Size: 488 bytes
>>> Desc: This is a digitally signed message part
>>> URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/201
>>> 70511/b79e65b7/attachment-0001.sig>
>>>
>>> ------------------------------
>>>
>>> Subject: Digest Footer
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>> ------------------------------
>>>
>>> End of zeromq-dev Digest, Vol 14, Issue 7
>>> *****************************************
>>>
>>
>>
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: signature.asc
> Type: application/pgp-signature
> Size: 488 bytes
> Desc: This is a digitally signed message part
> URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20170515/647441f2/attachment-0001.sig>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> https://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
> ------------------------------
>
> End of zeromq-dev Digest, Vol 14, Issue 9
> *****************************************
>
--
<http://www.repositpower.com/>
*Tomas Krajca *
Software architect
m. 02 6162 0277
e. tomas at repositpower.com
<https://twitter.com/RepositPower>
<https://www.facebook.com/Reposit-Power-1423585874607903/>
<https://www.linkedin.com/company/reposit-power>
More information about the zeromq-dev
mailing list