[zeromq-dev] Slow DEALER Recv in Go

[Cormac Relf]

Hi all,  

I’m building a homogenous cluster database (think a simplified Riak, just for fun), and I’ve set up over TCP:

DEALER[server A, for A], DEALER[server B, for A], DEALER[server C, for A] <-> ROUTER[A]
DEALER[server A, for B], DEALER[server B, for B], DEALER[server C, for B] <-> ROUTER[B]
DEALER[server A, for C], DEALER[server B, for C], DEALER[server C, for C] <-> ROUTER[C]

Hence, if you want to send a message to N random nodes and collect responses (as my database commands), you pick N random nodes from the member list (thoughtfully implemented at https://github.com/hashicorp/memberlist) and then you grab their DEALER (which on NodeJoin has connected to the appropriate ROUTER), send, and receive from it.

However, profiling using timers shows a lot of time being spent in DEALER.RecvMessage(). If I run a thousand sends and recv them all, it takes 40 microseconds each, but when I run ApacheBench for a single request and reply on demand as it has to work, the average time to receive is over 180 microseconds. When you run a read operation, currently I query all known nodes for the key and do a read repair, so reads have a lot of dealer/router round trips involved, sometimes as many as 8 on a 3-node cluster.

I can’t figure our where the bottleneck is, because in my experience, ØMQ sockets are very fast. Even sending each message to the ROUTER takes only 6 microseconds; processing it at the other end (i.e. a DB read) only takes 30 microseconds, so all up we should be waiting less than 40 microseconds for three of them to come back, not 500-600 on average as the timers report. I’m beginning to think it’s the Go scheduler, even.

Take a look at my code, it’s at github.com/cormacrelf/mec-db (http://github.com/cormacrelf/mec-db)  - the networking code resides in peers/peers.go (https://github.com/cormacrelf/mec-db/blob/master/peers/peers.go), the aforementioned replies are in store/store.go (https://github.com/cormacrelf/mec-db/blob/master/store/store.go) Listen().  

--  
Cormac Relf

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140121/c1e5fa57/attachment.htm>