[zeromq-dev] help with a design question using zeromq
Kenneth Adam Miller
kennethadammiller at gmail.com
Sun Jul 19 23:21:48 CEST 2015
I don't have any problem or question explicitly stated, or even that there
is some needed improvement that you don't have, so it's hard to know how to
be helpful. The one thing I would say that I have experience with and that
I could also help with is the idea of your external service managing
requests from potentially N different users. Eventually, N will grow to be
too large to be effectively handled by one machine, so you'll need to load
balance. That's what the malamute project is about, so if you want if you
want to address the idea of scalability there, I can help you.
On Sun, Jul 19, 2015 at 5:12 PM, Koren Shoval <koren99 at gmail.com> wrote:
> Hi,
>
> I have a scenario that I'd like to get some feedback on. Though I've read
> a large part of the guide and played around, I'm still quite a notice with
> ZMQ.
>
> My use case is this: (kind of long, sorry -- not sure how to explain
> better)
>
> a user talks to an 'external' service and asks to get data.
> the 'external' service talks to a 'data' service and asks for data (not
> using zmq)
>
> i'm attempting to leverage zmq to do fast and stable transmission of data
> between 'external' and multiple 'data' services.
>
> there can and will be up to 2000 clients talking to the 'external' service
> and each asking for data.
> each client request will go to a different 'data' service and it will
> serve the request (that part is not done in zmq at the moment)
>
> the 'external' service does not need to send anything to the 'data'
> service.
>
> what i've done so far, is use a dealer at each point in a p2p kind of way,
> where i open a tcp dealr socket in the 'external' and (not using zmq) tell
> the 'data' service to send data to the 'external' service's zmq tcp socket.
> i do that with another zmq dealer socket in the 'data' service side.
>
> so the bottom line is that for each client --> external service <-- data
> service
> i open a two way dealer connection in tcp.
>
> no router and no other sockets are used.
> i feel like i'm wasting tcp resources, too many ports and too many active
> connections.
>
> i thought about using a router or doing something like major domo, but it
> seems to be too costly to send an identity on each message block. (data
> returned to each client request is avg. 50MB)
> and i used 64KB buffers and sent binary messages.
>
> also, to do what with major domo. i'd have to kind of create a unique
> identity to each client request. to i'd have 2000 services. and all just
> for p2p behavior.
>
> thought about router tcp with inproc dealers to reuse a single tcp socket
> for all communications between an 'external' service and a single 'data'
> service. but i could not get that to work, since i can't share the state
> between threads and each dealer works with one client request. it was
> complicated to get working.
>
> i also tried to abuse pub/sub and treat each client request as a 'topic'.
> but i feel like multicast is a waste of network resource here, since there
> will only be 1:1 subscription to broadcast here.
>
> any help will be appreciated.
>
> Thanks,
> Koren
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150719/12a5005a/attachment.htm>
More information about the zeromq-dev
mailing list