[zeromq-dev] help with a design question using zeromq

[Kenneth Adam Miller]

Yeah, if you just have all the same requests going to the same socket,
ZeroMQ will do message framing for you, so supposing that you can
encapsulate a request into one large message, it's easy because you just
write your code into a loop. You can use a the czmq library to easily
parallize this so that messages get handed off to be worked on by different
cores.

Malamute is good because it allows you to add additional machines in easily
so that it scales.

On Sun, Jul 19, 2015 at 7:26 PM, Koren Shoval <koren99 at gmail.com> wrote:

> the question is, whether using a dedicated TCP channel per request a good
> idea or is there a better way to do this kind of comm. with zeromq.
> it does work, but i am not sure that it is the correct way to do it.
> assuming there is such a thing.
> i was hoping to be able to reuse or share the TCP sockets between multiple
> requests...
>
> as for your suggestion about malamute. I'll take a look at it.
>
> Thanks,
> Koren
>
> On Mon, Jul 20, 2015, 00:21 Kenneth Adam Miller <
> kennethadammiller at gmail.com> wrote:
>
>> I don't have any problem or question explicitly stated, or even that
>> there is some needed improvement that you don't have, so it's hard to know
>> how to be helpful. The one thing I would say that I have experience with
>> and that I could also help with is the idea of your external service
>> managing requests from potentially N different users. Eventually, N will
>> grow to be too large to be effectively handled by one machine, so you'll
>> need to load balance. That's what the malamute project is about, so if you
>> want if you want to address the idea of scalability there, I can help you.
>>
>> On Sun, Jul 19, 2015 at 5:12 PM, Koren Shoval <koren99 at gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I have a scenario that I'd like to get some feedback on. Though I've
>>> read a large part of the guide and played around, I'm still quite a notice
>>> with ZMQ.
>>>
>>> My use case is this: (kind of long, sorry -- not sure how to explain
>>> better)
>>>
>>> a user talks to an 'external' service and asks to get data.
>>> the 'external' service talks to a 'data' service and asks for data (not
>>> using zmq)
>>>
>>> i'm attempting to leverage zmq to do fast and stable transmission of
>>> data between 'external' and multiple 'data' services.
>>>
>>> there can and will be up to 2000 clients talking to the 'external'
>>> service and each asking for data.
>>> each client request will go to a different 'data' service and it will
>>> serve the request (that part is not done in zmq at the moment)
>>>
>>> the 'external' service does not need to send anything to the 'data'
>>> service.
>>>
>>> what i've done so far, is use a dealer at each point in a p2p kind of
>>> way, where i open a tcp dealr socket in the 'external' and (not using zmq)
>>> tell the 'data' service to send data to the 'external' service's zmq tcp
>>> socket. i do that with another zmq dealer socket in the 'data' service side.
>>>
>>> so the bottom line is that for each client --> external service <-- data
>>> service
>>> i open a two way dealer connection in tcp.
>>>
>>> no router and no other sockets are used.
>>> i feel like i'm wasting tcp resources, too many ports and too many
>>> active connections.
>>>
>>> i thought about using a router or doing something like major domo, but
>>> it seems to be too costly to send an identity on each message block. (data
>>> returned to each client request is avg. 50MB)
>>> and i used 64KB buffers and sent binary messages.
>>>
>>> also, to do what with major domo. i'd have to kind of create a unique
>>> identity to each client request. to i'd have 2000 services. and all just
>>> for p2p behavior.
>>>
>>> thought about router tcp with inproc dealers to reuse a single tcp
>>> socket for all communications between an 'external' service and a single
>>> 'data' service. but i could not get that to work, since i can't share the
>>> state between threads and each dealer works with one client request. it was
>>> complicated to get working.
>>>
>>> i also tried to abuse pub/sub and treat each client request as a
>>> 'topic'. but i feel like multicast is a waste of network resource here,
>>> since there will only be 1:1 subscription to broadcast here.
>>>
>>> any help will be appreciated.
>>>
>>> Thanks,
>>> Koren
>>>
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>>
>>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150719/364169bc/attachment.htm>