[zeromq-dev] patch: handle idle connections

[Pieter Hintjens]

On Wed, Mar 25, 2009 at 2:57 PM, Martin Sustrik <sustrik at fastmq.com> wrote:

> Scenario: 10,000s of connections, only few of them being active at any
> specific time.

Right.  So the active bitmap would be compressed.  I assume any bit
which is not active is idle?  Are the connections numbered
sequentially or is the idle bitmap also sparse?

> Requirements:
> - connections are sorted into two disjunct sets (active, idle)

If the index is the socket number, note that it is not sequential in either set.

Do you need to scan idle connections?  What operations do you do on
this set except "set active"?

> Actually, I have an idea. The algorithm is completely domain specific, but
> simple and very fast...

You don't care about ordering active connections, and you have a fixed
limit for A?  Then it would work nicely.

There is still a problem that you'll get gaps in the A table when
connections go inactive.  You either need to scan over holes, or
compress.

What I'd do is use linked lists but without memory allocation.  That
is, you allocate a block and in that, make linked lists.  I.e. array
of structures containing next pointer, and data.  Use two blocks, one
for active, one for idle.  Use offset pointers (i.e. byte offset from
start of block) rather than real memory pointers.  Then if you need to
expand the table you reallocate and copy.

This lets you do everything rapidly with no copying, allocation,
scanning, compression...

-Pieter