[zeromq-dev] help with a design question using zeromq

[Koren Shoval]

the question is, whether using a dedicated TCP channel per request a good
idea or is there a better way to do this kind of comm. with zeromq.
it does work, but i am not sure that it is the correct way to do it.
assuming there is such a thing.
i was hoping to be able to reuse or share the TCP sockets between multiple
requests...

as for your suggestion about malamute. I'll take a look at it.

Thanks,
Koren

On Mon, Jul 20, 2015, 00:21 Kenneth Adam Miller <kennethadammiller at gmail.com>
wrote:

> I don't have any problem or question explicitly stated, or even that there
> is some needed improvement that you don't have, so it's hard to know how to
> be helpful. The one thing I would say that I have experience with and that
> I could also help with is the idea of your external service managing
> requests from potentially N different users. Eventually, N will grow to be
> too large to be effectively handled by one machine, so you'll need to load
> balance. That's what the malamute project is about, so if you want if you
> want to address the idea of scalability there, I can help you.
>
> On Sun, Jul 19, 2015 at 5:12 PM, Koren Shoval <koren99 at gmail.com> wrote:
>
>> Hi,
>>
>> I have a scenario that I'd like to get some feedback on. Though I've read
>> a large part of the guide and played around, I'm still quite a notice with
>> ZMQ.
>>
>> My use case is this: (kind of long, sorry -- not sure how to explain
>> better)
>>
>> a user talks to an 'external' service and asks to get data.
>> the 'external' service talks to a 'data' service and asks for data (not
>> using zmq)
>>
>> i'm attempting to leverage zmq to do fast and stable transmission of data
>> between 'external' and multiple 'data' services.
>>
>> there can and will be up to 2000 clients talking to the 'external'
>> service and each asking for data.
>> each client request will go to a different 'data' service and it will
>> serve the request (that part is not done in zmq at the moment)
>>
>> the 'external' service does not need to send anything to the 'data'
>> service.
>>
>> what i've done so far, is use a dealer at each point in a p2p kind of
>> way, where i open a tcp dealr socket in the 'external' and (not using zmq)
>> tell the 'data' service to send data to the 'external' service's zmq tcp
>> socket. i do that with another zmq dealer socket in the 'data' service side.
>>
>> so the bottom line is that for each client --> external service <-- data
>> service
>> i open a two way dealer connection in tcp.
>>
>> no router and no other sockets are used.
>> i feel like i'm wasting tcp resources, too many ports and too many active
>> connections.
>>
>> i thought about using a router or doing something like major domo, but it
>> seems to be too costly to send an identity on each message block. (data
>> returned to each client request is avg. 50MB)
>> and i used 64KB buffers and sent binary messages.
>>
>> also, to do what with major domo. i'd have to kind of create a unique
>> identity to each client request. to i'd have 2000 services. and all just
>> for p2p behavior.
>>
>> thought about router tcp with inproc dealers to reuse a single tcp socket
>> for all communications between an 'external' service and a single 'data'
>> service. but i could not get that to work, since i can't share the state
>> between threads and each dealer works with one client request. it was
>> complicated to get working.
>>
>> i also tried to abuse pub/sub and treat each client request as a 'topic'.
>> but i feel like multicast is a waste of network resource here, since there
>> will only be 1:1 subscription to broadcast here.
>>
>> any help will be appreciated.
>>
>> Thanks,
>> Koren
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150719/48587fd3/attachment.htm>