[zeromq-dev] adaptable net

[Matjaž Ostroveršnik]

Hi all,

In last days I am studying the mlm server architecture. Nice and clean 
design. Congrats to the architects.

My objective is set to prepare adaptable net of nodes with client and 
worker functionality (i.e. service pattern in mlm semantics) that
can communicate among themselves. It is always client (consumer of the 
results of the service)-worker(provider of the service) communication, 
but nodes have both client and worker functionality.

There is/are one or more brokers between clients and workers. 
Communication goes client-broker-worker-broker-client. Brokers should be 
as simple as possible.

A. Broker:

 1. There are two or more brokers in the net.
 2. Brokers hold no vital information. If one broker goes down (regular
    / irregular shutdown) net must readapt. No payload is lost.
 3. broker is go between clients and workers from the security reasons
    (clients must not have direct access to the workers and vice versa)
     1. brokers allow only connection of registered clients/workers
 4. brokers distribute work between workers offering the same service
    (load balancing)
     1. e.g. round-robin
 5. provide alternative paths between clients and workers. (resistance
    to failure)
     1. C1->B1->W1->B1-C1 if there are several brokers the same pair
        C1/W1 can communicate via different brokers, assuming that all 
        workers and all clients are connected to all brokers.
 6. source of information about other brokers in the net
     1. i.e. client/worker connects to one predefined broker,
        periodically it distributes the list of other brokers to its
        workers/clients
     2. brokers have periodically "quorum" when they exchange
        information about them selves (i.e. list of all active brokers)
     3. initially each node is configured with at least one broker
        (others are provided dynamically)
     4. initially each broker is configured with at least one peer
        broker (is there mechanism to make broadcast on WAN?)

B. Clients:

 1. connected to one broker initially, gradually they "see" whole set of
    brokers
 2. if one broker fails, client uses different path (via retransmitting
    the message)
 3. clients distribute work between brokers

C. Workers:

 1. - each worker provides one service type
 2. - there can be several workers offering the same service

D. Messages:

 1. payload is opaque (size cca 4KB)
 2. meta-data : sender, receiver, unique-msg id, possible duplicate,
    original message id (for replys), retry cnt
 3. if they are expired and until there is an option for retry client
    retries them

E. Message exchange:

 1. it is always initiated by the client part of the node and goes to
    the service worker, which replys
 2. if message is not return in predefined period of time (msg exchange
    timeout) it is retried (predefined number of retries)
 3. client can select the broker to send the message through, but worker
    always returns message via arriving path(broker)
 4. message exchange C-B-W-B-C is fast (sub seccond)

Current mlm server is by my opinion platform for this task. Am I too 
optimistic?
Existing functionality:
   - A.1-2
   - A.4 (?) believe so
   - C.*
   - E.1 (how to do reply?)
   - E.3-4
Already added functionality:
   - A.3.1 (curve)
   - D.1  (blobs)
TODO functionality:
   - A.5
   - A.6 (the toughest stuff)
   - B.1-3
   - D.2-3
   - E.2

Questions:

 1. Async request-reply pattern. Is it already supported? Code talks
    about replys, but I am unsure. Hints?
 2. D.2-3, E.2: Is current message (mlm_msg_t) suitable for expansion
    (e.g. unique-id, original-unique-id, retry-cnt)?Or should I do it in
    some other way?
 3. A.5: clients / workers should connect to different brokers (e.g.
    array of mlm_client actors )
 4. A.6: Currently I do not have solution for A.6. Is zgossip the right
    solution?
 5. B.1-3 - Strictly related to A.6 solution

What is your opinion on this?
Would you do something in a different way?
How you would tackle A.6?

Thanks in advance

Matjaž

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20160409/4f0d3f40/attachment.htm>