[zeromq-dev] help with a design question using zeromq

[Koren Shoval]

Hi,

I have a scenario that I'd like to get some feedback on. Though I've read a
large part of the guide and played around, I'm still quite a notice with
ZMQ.

My use case is this: (kind of long, sorry -- not sure how to explain better)

a user talks to an 'external' service and asks to get data.
the 'external' service talks to a 'data' service and asks for data (not
using zmq)

i'm attempting to leverage zmq to do fast and stable transmission of data
between 'external' and multiple 'data' services.

there can and will be up to 2000 clients talking to the 'external' service
and each asking for data.
each client request will go to a different 'data' service and it will serve
the request (that part is not done in zmq at the moment)

the 'external' service does not need to send anything to the 'data' service.

what i've done so far, is use a dealer at each point in a p2p kind of way,
where i open a tcp dealr socket in the 'external' and (not using zmq) tell
the 'data' service to send data to the 'external' service's zmq tcp socket.
i do that with another zmq dealer socket in the 'data' service side.

so the bottom line is that for each client --> external service <-- data
service
i open a two way dealer connection in tcp.

no router and no other sockets are used.
i feel like i'm wasting tcp resources, too many ports and too many active
connections.

i thought about using a router or doing something like major domo, but it
seems to be too costly to send an identity on each message block. (data
returned to each client request is avg. 50MB)
and i used 64KB buffers and sent binary messages.

also, to do what with major domo. i'd have to kind of create a unique
identity to each client request. to i'd have 2000 services. and all just
for p2p behavior.

thought about router tcp with inproc dealers to reuse a single tcp socket
for all communications between an 'external' service and a single 'data'
service. but i could not get that to work, since i can't share the state
between threads and each dealer works with one client request. it was
complicated to get working.

i also tried to abuse pub/sub and treat each client request as a 'topic'.
but i feel like multicast is a waste of network resource here, since there
will only be 1:1 subscription to broadcast here.

any help will be appreciated.

Thanks,
Koren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150720/4c7906b9/attachment.htm>