[zeromq-dev] Malamute broker project

[Pieter Hintjens]

When you say services have state, you mean that a given client always
has to reach the same service instance?

On Mon, Mar 2, 2015 at 9:39 AM, Doron Somech <somdoron at gmail.com> wrote:
> Today we also use some kind of consistent hashing to distribute the load
> with service requests (where services are not state less and exclusivity
> needed), how can we do it with malamute?
>
> On Sun, Mar 1, 2015 at 10:00 PM, Kenneth Adam Miller
> <kennethadammiller at gmail.com> wrote:
>>
>> So, in order to manage a mutual exchange of address between two concurrent
>> parties, I thought that on each side I would have a producer produce to a
>> topic that the opposite side was subscribed to. That means that each side is
>> both a producer and a consumer.
>>
>> I have the two entities running in parallel. The front end client connects
>> to the malamute broker, and subscribes to the backendEndpoints topic, and
>> then producing it's endpoint to the frontendEndpoints topic.
>>
>> The opposite side does the same thing, with the back end subscribing to
>> the frontendEndpoints and producing to backendEndpoints.
>>
>>
>> The problem is that if the front end and back end are in their own threads
>> then only the thread that completes the mlm_set_producer and
>> mlm_set_consumer call proceed. The one that didn't make it that far will
>> hang at that mlm_set_x pair point...
>>
>> code:
>>
>>   std::cout << "connectToFrontEnd" << std::endl;
>>   mlm_client_t *frontend_reader = mlm_client_new();
>>   assert(frontend_reader);
>>   mlm_client_t *frontend_writer = mlm_client_new();
>>   assert(frontend_writer);
>>   int rc=mlm_client_connect (frontend_reader,  "tcp://127.0.0.1:9999" ,
>> 1000, "reader/secret");
>>   assert(rc==0);
>>   rc=mlm_client_connect (frontend_writer,  "tcp://127.0.0.1:9999" ,  1000,
>> "writer/secret");
>>   assert(rc==0);
>>   std::cout << "frontend mlm clients connected" << std::endl;
>>
>>   mlm_client_set_consumer(frontend_reader, "backendEndpoints", "*");
>>   mlm_client_set_producer(frontend_writer, "frontendEndpoints");
>>   std::cout << "frontend client producers and consumers set" << std::endl;
>>
>>
>> The code looks exactly* the same for the backend, but with some variable
>> and other changes.
>>
>>   std::cout << "connectToBackEnd" << std::endl;
>>   mlm_client_t *backend_reader = mlm_client_new();
>>   assert(backend_reader);
>>   mlm_client_t *backend_writer = mlm_client_new();
>>   assert(backend_writer);
>>   int rc=mlm_client_connect(backend_reader,"tcp://127.0.0.1:9999", 1000,
>> "reader/secret");
>>   assert(rc==0);
>>   rc=mlm_client_connect(backend_writer,"tcp://127.0.0.1:9999", 1000,
>> "writer/secret");
>>   assert(rc==0);
>>   std::cout << "backend mlm clients connected" << std::endl;
>>
>>   mlm_client_set_consumer(backend_reader, "frontendEndpoints", "*");
>>   mlm_client_set_producer(backend_writer, "backendEndpoints");
>>   std::cout << "backend client producers and consumers set" << std::endl;
>>
>> I only ever will see either "frontend client produces and consumers set"
>> or "backend client producers and consumers set".
>>
>> On Sun, Mar 1, 2015 at 2:00 PM, Pieter Hintjens <ph at imatix.com> wrote:
>>>
>>> My assumption is that a broker that's doing a lot of service requests
>>> won't be showing costs of regular expression matching, compared to the
>>> workload.
>>>
>>>
>>> On Sun, Mar 1, 2015 at 7:49 PM, Doron Somech <somdoron at gmail.com> wrote:
>>> >> I did it in actors and then moved it back into the main server as it
>>> >> was complexity for nothing (at that stage). I'd rather design against
>>> >> real use than against theory.
>>> >
>>> > Don't you worry about the matching performance which will happen on the
>>> > main
>>> > thread? Also a usage I can see is to use exact matching (string
>>> > comparison)
>>> > over regular expression (I usually use exact matching), this is way I
>>> > think
>>> > the plugin model fits the service as well.
>>> >
>>> > On Sun, Mar 1, 2015 at 8:09 PM, Pieter Hintjens <ph at imatix.com> wrote:
>>> >>
>>> >> On Sun, Mar 1, 2015 at 5:52 PM, Doron Somech <somdoron at gmail.com>
>>> >> wrote:
>>> >>
>>> >> > So I went over the code, really liked it. Very simple.
>>> >>
>>> >> Thanks. I like the plugin model, especially neat using CZMQ actors.
>>> >>
>>> >> > I have a question regarding services, for each stream you are using
>>> >> > a
>>> >> > dedicate thread (actors) and one thread for managing mailboxes.
>>> >> > However
>>> >> > (if
>>> >> > I understood correctly) for services you are doing the processing
>>> >> > inside
>>> >> > the
>>> >> > server thread, why didn't you use an actor for each service or actor
>>> >> > to
>>> >> > manage all services? I think the matching of services can be
>>> >> > expensive
>>> >> > and
>>> >> > block the main thread.
>>> >>
>>> >> I did it in actors and then moved it back into the main server as it
>>> >> was complexity for nothing (at that stage). I'd rather design against
>>> >> real use than against theory.
>>> >>
>>> >> -Pieter
>>> >> _______________________________________________
>>> >> zeromq-dev mailing list
>>> >> zeromq-dev at lists.zeromq.org
>>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > zeromq-dev mailing list
>>> > zeromq-dev at lists.zeromq.org
>>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>> >
>>> _______________________________________________
>>> zeromq-dev mailing list
>>> zeromq-dev at lists.zeromq.org
>>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>>
>>
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>