[zeromq-dev] patch: handle idle connections

[Martin Sustrik]

Dhammika,

Wow! You are the first person out there who ever dared to touch 0MQ core 
code :)

And there's actually a token in the bug tracker to do this kind of thing:

http://jira.fastmq.org/browse/ZMQ-130

Your patch goes it the right direction by eliminating the need to browse 
over large number of pipes that are idle anyway, however, on the other 
hand it introduces a new problem of it's own:

-        typedef std::vector <pipe_t*> pipes_t;
+        typedef std::list <pipe_t*> pipes_t;

This means that every time pipe is moved from one list to another one 
deallocation (free) and one allocation (malloc) is done - list stores 
its elements in dynamically allocated slots.

This may become a problem in scenarios where lot of pipes are swiftly 
moving between active and idle states (mostly reqest/reply or ping-pong 
style scenarios).

To actually measure the performance impact you would need a multicore 
box where allocations/deallocations are done on different CPU cores. In 
that case the overhead is not-negligible even with optimised memory 
allocators like tcmalloc.

Thus, to overcome this problem we've optimised 0MQ heavily to allocate 
as little as possible. At the current stage there's an 
allocation/deallocation pair done once per 256 messages.

With list<> implementation of active/idle lists you have to do 1 
allocation per message (in the worst case scenario) rising the total 
from 0.003 alloc/dealloc pairs per message to 1.003 alloc/dealloc pairs 
per message.

Possible solution would be to implement a simple double-linked list that 
would be able to move elements between two instances efficiently (just 
rewiring the pointers in the list rather than allocating/deallocating 
new element (see attached diagram):

my_list_t <pipe_t*> active;
my_list_t <pipe_t*> idle;

//  Insert an element to the list.
//  Allocation happens here.
active.push_back (pipe);

// Move an element from one list to another.
// No alloc/dealloc happens.
my_list_t <pipe_t*>::iterator it = active.begin ();
active.move_back (it, idle);

This kind of solution would require allocation only when new connection 
is established. There will be no additional allocations/deallocations on 
the critical path.

Thoughts?
Martin

Dhammika Pathirana wrote:
> Hi,
> 
> A patch to improve handling idle connections.
> I did bit of testing on linux.
> 
> Dhammika
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: list.png
Type: image/png
Size: 13537 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20090324/82aff135/attachment.png>