[zeromq-dev] zeromq-dev Digest, Vol 27, Issue 57 >> [Release 206 at RHEL 5.1]
Martin Sustrik
sustrik at 250bpm.com
Thu Mar 18 14:21:24 CET 2010
Hm,
Looks like some problem with epoll-style multiplexing.
Can you do the following:
1. Build debug version of 0MQ:
export CXXFLAGS="-g -O0"
./configure
make
2. Run you application under gdb.
3. When it crashes type "bt" to get backtrace.
4. Send me the error and the backtrace.
Thanks.
Martin
Nelson Tsai wrote:
> Hi Martin,
>
> Here is req/rpy case:
>
> *** glibc detected *** ./0mq: malloc(): memory corruption:
> 0x0000000016c55a60 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3d6346ef54]
> /lib64/libc.so.6(__libc_malloc+0x7d)[0x3d634706dd]
> /usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x3d73abd12d]
> /usr/local/lib/libzmq.so.
> 0
> (_ZNSt6vectorIPN3zmq7epoll_t12poll_entry_tESaIS3_EE13_M_insert_auxEN9__gnu_cxx17__normal_iteratorIPS3_S5_EERKS3_
> +0xeb)[0x2b10de8aa1ab]
> /usr/local/lib/libzmq.so.0(_ZN3zmq7epoll_t5rm_fdEPv+0x67)
> [0x2b10de8a9aa7]
> /usr/local/lib/libzmq.so.0(_ZN3zmq12zmq_engine_t6unplugEv+0xd)
> [0x2b10de8c05cd]
> /usr/local/lib/libzmq.so.0(_ZN3zmq12zmq_engine_t5errorEv+0xf3)
> [0x2b10de8c07a3]
> /usr/local/lib/libzmq.so.0(_ZN3zmq12zmq_engine_t8in_eventEv+0x116)
> [0x2b10de8c0af6]
> /usr/local/lib/libzmq.so.0(_ZN3zmq7epoll_t4loopEv+0x11e)[0x2b10de8a9c8e]
> /usr/local/lib/libzmq.so.0(_ZN3zmq8thread_t14thread_routineEPv+0x37)
> [0x2b10de8bc457]
> /lib64/libpthread.so.0[0x3d640061b5]
> /lib64/libc.so.6(clone+0x6d)[0x3d634cd39d]
> ======= Memory map: ========
> 00400000-0041c000 r-xp 00000000 fd:00
> 9888246 /root/dev/0mq/dist/Debug/GNU-Linux-
> x86/0mq
> 0061b000-0061c000 rw-p 0001b000 fd:00
> 9888246 /root/dev/0mq/dist/Debug/GNU-Linux-
> x86/0mq
> 16c50000-16c71000 rw-p 16c50000 00:00 0
> 41e27000-41e28000 ---p 41e27000 00:00 0
> 41e28000-42828000 rw-p 41e28000 00:00 0
> 3d63000000-3d6301a000 r-xp 00000000 fd:00
> 12570909 /lib64/ld-2.5.so
> 3d63219000-3d6321a000 r--p 00019000 fd:00
> 12570909 /lib64/ld-2.5.so
> 3d6321a000-3d6321b000 rw-p 0001a000 fd:00
> 12570909 /lib64/ld-2.5.so
> 3d63400000-3d63544000 r-xp 00000000 fd:00
> 12570910 /lib64/libc-2.5.so
> 3d63544000-3d63744000 ---p 00144000 fd:00
> 12570910 /lib64/libc-2.5.so
> 3d63744000-3d63748000 r--p 00144000 fd:00
> 12570910 /lib64/libc-2.5.so
> 3d63748000-3d63749000 rw-p 00148000 fd:00
> 12570910 /lib64/libc-2.5.so
> 3d63749000-3d6374e000 rw-p 3d63749000 00:00 0
> 3d63800000-3d63882000 r-xp 00000000 fd:00
> 12570917 /lib64/libm-2.5.so
> 3d63882000-3d63a81000 ---p 00082000 fd:00
> 12570917 /lib64/libm-2.5.so
> 3d63a81000-3d63a82000 r--p 00081000 fd:00
> 12570917 /lib64/libm-2.5.so
> 3d63a82000-3d63a83000 rw-p 00082000 fd:00
> 12570917 /lib64/libm-2.5.so
> 3d64000000-3d64015000 r-xp 00000000 fd:00
> 12570912 /lib64/libpthread-2.5.so
> 3d64015000-3d64214000 ---p 00015000 fd:00
> 12570912 /lib64/libpthread-2.5.so
> 3d64214000-3d64215000 r--p 00014000 fd:00
> 12570912 /lib64/libpthread-2.5.so
> 3d64215000-3d64216000 rw-p 00015000 fd:00
> 12570912 /lib64/libpthread-2.5.so
> 3d64216000-3d6421a000 rw-p 3d64216000 00:00 0
> 3d64800000-3d64807000 r-xp 00000000 fd:00
> 12570913 /lib64/librt-2.5.so
> 3d64807000-3d64a07000 ---p 00007000 fd:00
> 12570913 /lib64/librt-2.5.so
> 3d64a07000-3d64a08000 r--p 00007000 fd:00
> 12570913 /lib64/librt-2.5.so
> 3d64a08000-3d64a09000 rw-p 00008000 fd:00
> 12570913 /lib64/librt-2.5.so
> 3d65800000-3d6589d000 r-xp 00000000 fd:00
> 12570914 /lib64/libglib-2.0.so.0.1200.3
> 3d6589d000-3d65a9c000 ---p 0009d000 fd:00
> 12570914 /lib64/libglib-2.0.so.0.1200.3
> 3d65a9c000-3d65a9e000 rw-p 0009c000 fd:00
> 12570914 /lib64/libglib-2.0.so.0.1200.3
> 3d6b400000-3d6b404000 r-xp 00000000 fd:00
> 12570919 /lib64/libgthread-2.0.so.0.1200.3
> 3d6b404000-3d6b603000 ---p 00004000 fd:00
> 12570919 /lib64/libgthread-2.0.so.0.1200.3
> 3d6b603000-3d6b604000 rw-p 00003000 fd:00
> 12570919 /lib64/libgthread-2.0.so.0.1200.3
> 3d72200000-3d7220d000 r-xp 00000000 fd:00
> 12570815 /lib64/libgcc_s-4.1.1-20070105.so.1
> 3d7220d000-3d7240c000 ---p 0000d000 fd:00
> 12570815 /lib64/libgcc_s-4.1.1-200Aborted
>
>
> And this is pub/sub case:
>
> *** glibc detected *** ./0mq: malloc(): memory corruption:
> 0x000000000c33ba80 ***
> ======= Backtrace: =========
> /lib64/libc.so.6[0x3d6346ef54]
> /lib64/libc.so.6(__libc_malloc+0x7d)[0x3d634706dd]
> /usr/lib64/libstdc++.so.6(_Znwm+0x1d)[0x3d73abd12d]
> /usr/local/lib/libzmq.so.
> 0
> (_ZNSt6vectorIPN3zmq7epoll_t12poll_entry_tESaIS3_EE13_M_insert_auxEN9__gnu_cxx17__normal_iteratorIPS3_S5_EERKS3_
> +0xeb)[0x2b61a85401ab]
> /usr/local/lib/libzmq.so.0(_ZN3zmq7epoll_t5rm_fdEPv+0x67)
> [0x2b61a853faa7]
> /usr/local/lib/libzmq.so.0(_ZN3zmq12zmq_engine_t6unplugEv+0xd)
> [0x2b61a85565cd]
> /usr/local/lib/libzmq.so.0(_ZN3zmq12zmq_engine_t5errorEv+0xf3)
> [0x2b61a85567a3]
> /usr/local/lib/libzmq.so.0(_ZN3zmq12zmq_engine_t8in_eventEv+0x116)
> [0x2b61a8556af6]
> /usr/local/lib/libzmq.so.0(_ZN3zmq7epoll_t4loopEv+0x11e)[0x2b61a853fc8e]
> /usr/local/lib/libzmq.so.0(_ZN3zmq8thread_t14thread_routineEPv+0x37)
> [0x2b61a8552457]
> /lib64/libpthread.so.0[0x3d640061b5]
> /lib64/libc.so.6(clone+0x6d)[0x3d634cd39d]
> ======= Memory map: ========
> 00400000-0041c000 r-xp 00000000 fd:00
> 9888246 /root/dev/0mq/dist/Debug/GNU-Linux-
> x86/0mq
> 0061b000-0061c000 rw-p 0001b000 fd:00
> 9888246 /root/dev/0mq/dist/Debug/GNU-Linux-
> x86/0mq
> 0c336000-0c357000 rw-p 0c336000 00:00 0
> 416fe000-416ff000 ---p 416fe000 00:00 0
> 416ff000-420ff000 rw-p 416ff000 00:00 0
> 3d63000000-3d6301a000 r-xp 00000000 fd:00
> 12570909 /lib64/ld-2.5.so
> 3d63219000-3d6321a000 r--p 00019000 fd:00
> 12570909 /lib64/ld-2.5.so
> 3d6321a000-3d6321b000 rw-p 0001a000 fd:00
> 12570909 /lib64/ld-2.5.so
> 3d63400000-3d63544000 r-xp 00000000 fd:00
> 12570910 /lib64/libc-2.5.so
> 3d63544000-3d63744000 ---p 00144000 fd:00
> 12570910 /lib64/libc-2.5.so
> 3d63744000-3d63748000 r--p 00144000 fd:00
> 12570910 /lib64/libc-2.5.so
> 3d63748000-3d63749000 rw-p 00148000 fd:00
> 12570910 /lib64/libc-2.5.so
> 3d63749000-3d6374e000 rw-p 3d63749000 00:00 0
> 3d63800000-3d63882000 r-xp 00000000 fd:00
> 12570917 /lib64/libm-2.5.so
> 3d63882000-3d63a81000 ---p 00082000 fd:00
> 12570917 /lib64/libm-2.5.so
> 3d63a81000-3d63a82000 r--p 00081000 fd:00
> 12570917 /lib64/libm-2.5.so
> 3d63a82000-3d63a83000 rw-p 00082000 fd:00
> 12570917 /lib64/libm-2.5.so
> 3d64000000-3d64015000 r-xp 00000000 fd:00
> 12570912 /lib64/libpthread-2.5.so
> 3d64015000-3d64214000 ---p 00015000 fd:00
> 12570912 /lib64/libpthread-2.5.so
> 3d64214000-3d64215000 r--p 00014000 fd:00
> 12570912 /lib64/libpthread-2.5.so
> 3d64215000-3d64216000 rw-p 00015000 fd:00
> 12570912 /lib64/libpthread-2.5.so
> 3d64216000-3d6421a000 rw-p 3d64216000 00:00 0
> 3d64800000-3d64807000 r-xp 00000000 fd:00
> 12570913 /lib64/librt-2.5.so
> 3d64807000-3d64a07000 ---p 00007000 fd:00
> 12570913 /lib64/librt-2.5.so
> 3d64a07000-3d64a08000 r--p 00007000 fd:00
> 12570913 /lib64/librt-2.5.so
> 3d64a08000-3d64a09000 rw-p 00008000 fd:00
> 12570913 /lib64/librt-2.5.so
> 3d65800000-3d6589d000 r-xp 00000000 fd:00
> 12570914 /lib64/libglib-2.0.so.0.1200.3
> 3d6589d000-3d65a9c000 ---p 0009d000 fd:00
> 12570914 /lib64/libglib-2.0.so.0.1200.3
> 3d65a9c000-3d65a9e000 rw-p 0009c000 fd:00
> 12570914 /lib64/libglib-2.0.so.0.1200.3
> 3d6b400000-3d6b404000 r-xp 00000000 fd:00
> 12570919 /lib64/libgthread-2.0.so.0.1200.3
> 3d6b404000-3d6b603000 ---p 00004000 fd:00
> 12570919 /lib64/libgthread-2.0.so.0.1200.3
> 3d6b603000-3d6b604000 rw-p 00003000 fd:00
> 12570919 /lib64/libgthread-2.0.so.0.1200.3
> 3d72200000-3d7220d000 r-xp 00000000 fd:00
> 12570815 /lib64/libgcc_s-4.1.1-20070105.so.1
> 3d7220d000-3d7240c000 ---p 0000d000 fd:00
> 12570815 /lib64/libgcc_s-4.1.1-200Aborted
>
>
>
> Thanks and Regards,
> Nelson
>
>
> On Mar 18, 2010, at 7:00 PM, zeromq-dev-request at lists.zeromq.org wrote:
>
>> Send zeromq-dev mailing list submissions to
>> zeromq-dev at lists.zeromq.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> or, via email, send a message with subject or body 'help' to
>> zeromq-dev-request at lists.zeromq.org
>>
>> You can reach the person managing the list at
>> zeromq-dev-owner at lists.zeromq.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of zeromq-dev digest..."
>>
>>
>> Today's Topics:
>>
>> 1. Re: [zeromq-announce] Release 206 at RHEL 5.1 (Martin Sustrik)
>> 2. Re: C# interface for Zmq 2.x (Martin Sustrik)
>> 3. Re: P2P updates or another method? (Martin Sustrik)
>> 4. Re: Load balancing REQ/REP sockets (Martin Sustrik)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Thu, 18 Mar 2010 08:18:37 +0100
>> From: Martin Sustrik <sustrik at 250bpm.com>
>> Subject: Re: [zeromq-dev] [zeromq-announce] Release 206 at RHEL 5.1
>> To: zeromq-dev at lists.zeromq.org
>> Cc: zeromq-announce at lists.zeromq.org
>> Message-ID: <4BA1D3CD.7020509 at 250bpm.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Nelson,
>>
>>> We're now suffering one show stop issue of release 206:
>>>
>>> 1.) we could build the release success and okay to build our c++
>>> program with the library
>>> 2.) we could startup our c++ program smoothly
>>> 3.) but any time server got the message from client, it crashed and
>>> memory dump!!
>>>
>>> ***no matter using pub/sub or req/rpy, we got the same issue!!
>>>
>>> Does anybody else have the same experience with solution?
>>>
>>>
>>> Nelson
>>> p.s. same c++ program works fine with release 20b2 at the same OS/
>>> Host
>> Can you provide the backtrace from the crash?
>>
>> Martin
>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Thu, 18 Mar 2010 08:23:59 +0100
>> From: Martin Sustrik <sustrik at 250bpm.com>
>> Subject: Re: [zeromq-dev] C# interface for Zmq 2.x
>> To: Jeff Dik <s450r1 at gmail.com>
>> Cc: zeromq-dev at mail.imatix.com
>> Message-ID: <4BA1D50F.4030507 at 250bpm.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Hi Jeff,
>>
>>> I made some C# bindings a few weeks ago, but just committed it and
>>> pushed it: http://github.com/s450r1/zeromq2/blob/master/bindings/clr/Zmq.cs
>> Nice! Can you attach a licnese to the code so that it can be used
>> safely? LGPL would be preferable.
>>
>>> However, it's only been tested to work with the 0MQ from a few weeks
>>> ago, before the language bindings were kicked out of the core 0MQ
>>> repository :-) I've been meaning to update it with the latest
>>> changes
>>> to 0MQ, but haven't had a chance yet (as I have 6 month old twin
>>> girls).
>> Wow. Congratulations!
>>
>>> Feel free to use that as a starting point if you wish.
>>> There's a few examples in
>>> http://github.com/s450r1/zeromq2/tree/master/bindings/clr. See the
>>> Rakefile for how to build (or use rake[1] to build).
>>>
>>> [1]: http://rake.rubyforge.org/
>> Martin
>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Thu, 18 Mar 2010 08:26:31 +0100
>> From: Martin Sustrik <sustrik at 250bpm.com>
>> Subject: Re: [zeromq-dev] P2P updates or another method?
>> To: Jeff Dik <s450r1 at gmail.com>
>> Cc: zeromq-dev at mail.imatix.com
>> Message-ID: <4BA1D5A7.1070103 at 250bpm.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Jeff,
>>
>>> I've been wanting to use 0MQ at work, but can't quite get the
>>> behavior
>>> I want. I think I want P2P type connections, but run into the lack
>>> of
>>> auto-connect code[1].
>>>
>>> What I'm trying to get is for an application to send lots of messages
>>> and have those stored on a queue until a client connects and receives
>>> the messages. Is that handled best with P2P, or am I overlooking
>>> something else?
>> Yes, P2P is in experimental stage and lacks auto-reconnect as for now.
>>
>> However, you can use DOWNSTREAM/UPSTREAM as a temporary substitute in
>> your scenario.
>>
>> Martin
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Thu, 18 Mar 2010 08:41:15 +0100
>> From: Martin Sustrik <sustrik at 250bpm.com>
>> Subject: Re: [zeromq-dev] Load balancing REQ/REP sockets
>> To: ellisonbg at gmail.com
>> Cc: "zeromq-dev at lists.zeromq.org" <zeromq-dev at lists.zeromq.org>
>> Message-ID: <4BA1D91B.6090302 at 250bpm.com>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> Hi Brian,
>>
>>>> The whole XREQ/XREP business is still messy. If you absolutely
>>>> need it at
>>>> the moment you can use XREQ to connect and load-balance messages
>>>> to mutliple
>>>> REPs. That way you'll get what you need without a need to do any
>>>> special
>>>> processing on the REP side.
>>> I have tried the XREQ -> multiple REP socket topology. The requests
>>> are load balanced to the REP sockets and there can be multiple
>>> requests in the air at a time. BUT, the notion of load balancing
>>> is a
>>> bit odd. The load balancing logic is here:
>>>
>>> http://github.com/sustrik/zeromq2/blob/master/src/lb.cpp#L90
>>>
>>> The idea is that the next send will happen on the current+1 active
>>> pipe. Basically, this round-robins the requests to the different REP
>>> sockets. But, in my mind, this strategy is not a completely useful
>>> load balancing. Here is why. Some requests may take longer/shorter
>>> for the REP socket to process. With the current approach, a REP
>>> socket that handles a very short request, will have to sit there
>>> doing
>>> nothing until its turn comes again. Likewise, a REP socket that gets
>>> a very time consuming request will continue to get new requests,
>>> though it is busy.
>>>
>>> A more useful and dynamic load balancing approach would be would be
>>> the following:
>>>
>>> * Look at the current + 1 pipe. If it is not servicing a request,
>>> use it.
>>> * If the current + 1 pipe is busy, see if the current + 2 pipe is
>>> servicing a request. If not use it.
>>> * Continue through the active pipes. If all are busy, use the
>>> current
>>> + 1 anyways.
>>>
>>> Is there a way of telling if a pipe is servicing a request? Would it
>>> make sense to implement this? This more dynamic load balancing would
>>> be incredibly useful.
>> Yes. The meachism intended for this kind of thing is "queue limits"
>> mechanism recently ported from 0MQ/1.0.
>>
>> You can set ZMQ_HWM socket option on a socket, specifying max number
>> of
>> messages it can hold. Once the limit is reached for specific
>> connection
>> it stops accepting new messages and load balancing skips it in the
>> round-robin.
>>
>> However, the number of requests "on the fly" can be pretty high. There
>> can be messages in the sender's queue, in secder's TCP tx window, on
>> the
>> network (queue on a switch etc.), in receiver's rx window and in
>> receiver's queue.
>>
>> What you are speaking about is making it work in lock-step fashion,
>> i.e.
>> no new message is sent until the reply is received. This should work
>> on
>> a REQ socket. If it does not, it should be fixed.
>>
>> With XREQ socket the situation is a bit more complex. The user can
>> send
>> many requests without waiting for a reply. Some of the requests are
>> routed through a single connection. At the end of the connection there
>> may be another load-balancer, such as zmq_queue. If it worked in
>> lock-step fashion. The second load-balancer would be completely
>> useless
>> as there would be at most one request processed at a time.
>>
>> Thoughts?
>> Martin
>>
>>
>> ------------------------------
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>> End of zeromq-dev Digest, Vol 27, Issue 57
>> ******************************************
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
More information about the zeromq-dev
mailing list