[zeromq-dev] Efficiency for very few short messages

dan smith dan25311 at gmail.com
Tue Jan 29 16:06:00 CET 2013


Jason,

Thanks for the suggestion. I will apply the lbb broker pattern right away
to that problem and will share the results. To me it is a good news that
this is a design issue...

Dan


On Tue, Jan 29, 2013 at 12:58 AM, Jason Smith
<jason.nevar.smith at gmail.com>wrote:

> Hi Dan,
>
> I have found the issue with the processing times.
>
>
> 	for(iequation = 0 ; iequation < nequation ; iequation++)	
> 	{
> 		zmq_msg_t msg;
> 		rc = zmq_msg_init (&msg);
> 		rc = zmq_msg_init_size(&msg, 8);
> 		memset (zmq_msg_data (&msg), 'A', 8);
>
> 		ithread = messageCounter % nthread ;  <---- RIGHT HERE
>
> 		messageCounter++ ;
> 		void * socket = socketsSend[ithread];
> 		rc = zmq_sendmsg (socket, &msg, 0);
> 		zmq_msg_close(&msg);
> 	}
>
> The code above doesn't take into account time it takes for the
> passed equation. It treats them all as being of equal "work" which they
> don't appear to be. This means that some threads will sit around waiting
> for a very long time while others are still busy with three or four items
> on their queue.
>
> This is where a load balancing pattern would be very handy. Search for the
> line with "lbbroker: Load-balancing broker in C" in the zguide for
> an explanation and example code. (http://zguide.zeromq.org/page:all).
>
> The short of it is, have your application have a req socket in it. Send on
> that req socket to a router (frontend) in another thread. All this threads
> job is to do is work out which thread is not busy (first in the list if
> need be) and then route the packet to that thread. This is done through
> another router (backend) socket connected to each "Worker" thread that you
> currently have. These then do the work and message back the result to the
> router (backend) which then knows it can pass the result all the way back
> to the requesting "client" (frontend). The reason for the second thread to
> determine where the work has to be sent is because you won't know until its
> being worked on how long something will take in this case. Predetermining
> this is causing the issues with regards to only a 3 to 5 times speed up on
> my machine.
>
> The zguide has a wonderful diagram of this. Its very simplistic and
> doesn't handle crashes, or overloading, etc. These would have to be worked
> into the end solution based on your environments needs.
>
> If I get a chance tonight I might knock something up using your example.
> Depends on how much packing I get done, haha.
>
> The way I found this was the issue is simply counting the time each thread
> was "waiting" and "processing" found that some were super busy processing
> while others were just sitting around. So you guess was right about the
> sockets just sitting there in some threads. The time being "wasted" however
> is sadly a design issue at this point, not so much ZeroMQ ;)
>
> Hope that helps.
>
> Lastly as a bonus, this load balancing pattern means you would be able to
> add as many front ends and back-ends as you saw fit. Only the "balancer" is
> static in this design.
>
> - J
>
>
> On 29 January 2013 16:30, dan smith <dan25311 at gmail.com> wrote:
>
>>
>> Hi Jason,
>>
>> Thanks a lot for devoting your time to my problem. My expertise is
>> negligible in this area.
>>
>> Looks like that symptom might be CPU dependent ? I tried it just on a
>> quad-core laptop, it has 16G memory though.
>>
>> This problem is really important so I started to evaluate alternative
>> solutions. I found lock-free queues , more specifically lock-free
>> single-producer - single-consumer circular queues. I was impressed by the
>> latency: I could send 10 000 000 (ten millions) 8 bytes messages in one
>> second. It is a very simple thing , there are many versions of it. Latency
>> is in the 100 nanoseconds range. I do not know the reasons but looks like
>> it is faster for this kind of communication.
>>
>> Using it I could reach 30 % speedup for the real problem so the parallel
>> version is faster by now at least, still not fast enough though...
>>
>> Now the problem is how to notify quickly the threads that data is coming.
>>
>> I will test both solutions on a better machine with more cores. Maybe if
>> we have got just few messages, they spend some time in a cache or
>> something. If this is the case, is there a way to forward them to the CPU
>> more quickly? Any further input will be appreciated.
>>
>> Thank you again,
>>
>> Dan
>>
>>
>>
>>
>> On Mon, Jan 28, 2013 at 6:26 PM, Jason Smith <jason.nevar.smith at gmail.com
>> > wrote:
>>
>>> Hi Dan,
>>>
>>> Just tested the debug version and it does drop but not as much as you
>>> listed. Also of note I have been testing on 64 bit windows 7, i7-2600 with
>>> a large amount of Ram. The next test for me will be to look at where the
>>> time is taken up, however thought I would report on what I have seen so
>>> far.
>>>
>>> - J
>>>
>>>
>>> On 29 January 2013 11:16, Jason Smith <jason.nevar.smith at gmail.com>wrote:
>>>
>>>> Hi Dan,
>>>>
>>>> Here's something I have found with your code. Testing here I see the
>>>> same speed up for all numbers of equations. I am using the release version
>>>> of the dll however. About to test the debug version of the dll to see if I
>>>> get different behaviour.
>>>>
>>>> - J
>>>>
>>>>
>>>> On 23 January 2013 13:56, dan smith <dan25311 at gmail.com> wrote:
>>>>
>>>>> Jason,
>>>>>
>>>>> Thanks a lot for taking a look at it.
>>>>>
>>>>> As for the "while(nfinish > 0" loop, my experience is that it does not
>>>>> have significant effect on the time. If I remove it and allow the threads
>>>>> to die, the difference is negligible. In the real application the threads
>>>>> needs to remain alive of course, I just tried to check that the thread
>>>>> closing is not the reason.
>>>>>
>>>>> Closing the sockets in threads might not be the reason either, a
>>>>> terminating message is sent back to the main thread before that.
>>>>>
>>>>> I use zeromq-3.2.2.
>>>>>
>>>>> In the real application I am sending a pointer, here the 8 As simulate
>>>>> that.
>>>>>
>>>>> I am looking forward to your further comments very much. Hope that I
>>>>> am the one who made some mistake and there is a solution for sending few
>>>>> small messages at the latency that I measured for large number of messages
>>>>> (that was under 1 microseconds which would be cool)
>>>>>
>>>>>
>>>>> On Tue, Jan 22, 2013 at 8:13 PM, Jason Smith <
>>>>> jason.nevar.smith at gmail.com> wrote:
>>>>>
>>>>>>
>>>>>> On 23 January 2013 11:42, dan smith <dan25311 at gmail.com> wrote:
>>>>>>
>>>>>>> while(nfinish > 0)
>>>>>>
>>>>>>
>>>>>> Haven't had a chance to compile this here. For some reason have a
>>>>>> linker issue on my work machine.
>>>>>>
>>>>>> At first glance the "while(nfinish > 0)" loop assumes sequential
>>>>>> thread completion for best time. For example you only know of thread
>>>>>> 7 finishing only until 1 through to 6 have completed. Don't know if this is
>>>>>> affecting things drastically or not. Maybe switching to polling here and
>>>>>> updating a "completed" vector list might work better.
>>>>>>
>>>>>> Another area I would look into is the linger of the sockets, it
>>>>>> shouldn't affect closing them down within the thread however its something
>>>>>> to consider.
>>>>>>
>>>>>> When I get a chance I would be looking to place more asserts in to
>>>>>> make sure messages were doing what I thought they were (send and receive
>>>>>> calls return values). Then I would be checking the timing of any close down
>>>>>> code.
>>>>>>
>>>>>> Hope this helps in the meantime.
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> zeromq-dev mailing list
>>>>>> zeromq-dev at lists.zeromq.org
>>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> zeromq-dev mailing list
>>>>> zeromq-dev at lists.zeromq.org
>>>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20130129/1b4bdb80/attachment.htm>


More information about the zeromq-dev mailing list