[zeromq-dev] Kivix: Beacon like service discovery

Pieter Hintjens ph at imatix.com
Thu Dec 19 21:42:20 CET 2013


I know you intend to support lots of protocols but... my advice would
be to drop everything except ZMTP, ZRE
(http://rfc.zeromq.org/spec:20), and layered protocols on top of that.
Also, to use Jyre as your starting point, and make it interoperate
with Zyre. You get access to a far larger potential market, as there
are already people making ZMTP work on microcontrollers (e.g. over
PicoTCP).


On Thu, Dec 19, 2013 at 9:32 PM, Robert Gallas <gallas.robert at gmail.com> wrote:
> Aaah thanks ... Will use Jyre probably as quick reference but will wrestle
> with C as well. Was doing some 15 years ago. As well I plan to use this
> pattern on some Arduino / Atmel microcontrollers ... sometimes :-)
>
> Robert
>
>
> 2013/12/19 Pieter Hintjens <ph at imatix.com>
>>
>> Robert,
>>
>> You might even look at the Jyre project, which is a (slightly out of
>> date) Java implementation of the Zyre protocols.
>>
>> -Pieter
>>
>> On Thu, Dec 19, 2013 at 9:08 PM, Robert Gallas <gallas.robert at gmail.com>
>> wrote:
>> > Pieter, Andrew
>> >
>> >
>> > Thanks for pointers. I'm not the best C source reader, but beacon was
>> > well
>> > written and not that much difficult to understand. I'll try to look
>> > around
>> > in Zyre as well.
>> >
>> > I have run quickly through http://markburgess.org/BookOfPromises.pdf and
>> > it
>> > seems that Kivix is targeting same area. By coincidence it seems that
>> > what
>> > Promise Theory calls "promise" in Kivix is implemented as "capability".
>> > And
>> > has a form of URN urn:kivix:demo:mathapp.
>> >
>> > Bundles of promises seems like failover functionality. Which brinks me
>> > to
>> > question why have bundle of promises and not just start more nodes with
>> > same
>> > capability / promise. But that is not question for this thread and
>> > probably
>> > not for ZeroMQ community.
>> >
>> > I'll look at Zyre and will read Promise Theory draft. Hope I'll be able
>> > to
>> > come up with simple "just enough" API or else I'd rather drop this
>> > failover
>> > feature.
>> >
>> > Many thanks again
>> > Robert
>> >
>> >
>> >
>> > 2013/12/19 Andrew Hume <andrew at research.att.com>
>> >>
>> >> this also fits in nicely with promise theory (mark burgess, uni of
>> >> oslo)
>> >> which gives a different take on how to reason about cooperating nodes.
>> >>
>> >> On Dec 19, 2013, at 3:01 AM, Pieter Hintjens <ph at imatix.com> wrote:
>> >>
>> >> One thing you could look at is how we did distributed log collection
>> >> in Zyre. Any node can be a log collector. When nodes discover each
>> >> other, and interconnect, they indicate if they collect logs, and then
>> >> their peers connect back, over ZeroMQ, and stream log events to them.
>> >>
>> >> -Pieter
>> >>
>> >> On Wed, Dec 18, 2013 at 8:27 PM, Robert Gallas
>> >> <gallas.robert at gmail.com>
>> >> wrote:
>> >>
>> >> Thanks Pieter,
>> >>
>> >> For PoC purpose only node discovery is implemented. ZeoMQ (JeroMQ) will
>> >> be
>> >> used as core node interconnection protocol. Other protocols are present
>> >> just
>> >> as PoC of hexagonal application design possibilities.
>> >>
>> >> So far I can live with unreliable Beacon style heartbeat for monitoring
>> >> purposes.
>> >>
>> >> Node management is more interesting topic. To be honest I have not very
>> >> clear vision yet. Kivix is meant to be set of autonomous nodes. But
>> >> nodes
>> >> form functional topology only if they cooperate with each other.
>> >> Autonomous
>> >> nodes design is therefore in direct contradiction to cooperating nodes.
>> >> In
>> >> traditional style of system management there are agents installed on
>> >> every
>> >> node where we need to support manageability. There are three basic
>> >> types
>> >> of
>> >> node functionality failures:
>> >>
>> >> 1. managed functionality fails, agent runs and we can contact the agent
>> >> 2. management agent fails
>> >> 3. failure at network level
>> >>
>> >> Only in first scenario management agent can help to resolve failure
>> >> situation. And there is of course topic of who discovers failed node.
>> >> Basically it's the same situation as with central broker vs brokerless
>> >> architecture. Having central management console is something I would
>> >> like
>> >> to
>> >> avoid.
>> >>
>> >> One possible way of solving node manageability is to have functionality
>> >> requesting node to report failure. Then along with listening to service
>> >> urn
>> >> to show up on network, node can start to broadcast resource starvation
>> >> message. Then only really required functionality is reported to be
>> >> failed.
>> >> Basically there is no need for urn:com.example.kivix.mathapp to be
>> >> reported
>> >> as failed if there is no node requesting that functionality.
>> >>
>> >> Then there is question how to respond to node failure. There can be
>> >> latent
>> >> nodes on network which can implement more than one capability. Then in
>> >> case
>> >> of resource starvation broadcast message, node can takeover the
>> >> responsibility and start to provide required capability. This kind of
>> >> behaviour can be then used in self-load-balancing nodes. If there is
>> >> too
>> >> much messages waiting in dealer queue, dealer can broadcast resource
>> >> starvation message and if there is possibility, some nodes can switch
>> >> to
>> >> worker mode with required capability.
>> >>
>> >> But again there are categories of applications where there are very
>> >> different requirements about level of manageability.
>> >>
>> >> At this time I'm not decided which approach to begin with. Kivix is
>> >> only
>> >> sideproduct of another application. I'm implementing features only if
>> >> requirement is driven by that application or if there is some
>> >> interesting
>> >> idea like Beacon worth to try out as PoC.
>> >>
>> >> If you can point me to some resource where this topic is discussed I
>> >> would
>> >> be glad. I'd like to avoid reinventig the wheel.
>> >>
>> >> Robert
>> >>
>> >>
>> >> 2013/12/18 Pieter Hintjens <ph at imatix.com>
>> >>
>> >>
>> >> Hi Robert,
>> >>
>> >> This is neat! Have you used ZeroMQ for instance for monitoring and
>> >> managing nodes?
>> >>
>> >> -Pieter
>> >>
>> >> On Tue, Dec 17, 2013 at 10:24 PM, Robert Gallas
>> >> <gallas.robert at gmail.com>
>> >> wrote:
>> >>
>> >> Hello,
>> >>
>> >> I'd like to announce simple beacon implementation in Kivix framework.
>> >> Inspiration comes from beacon implementation in czmq. So far as PoC
>> >> only.
>> >> There is no API or protocol similarities implemented yet. Example is
>> >> given
>> >> at http://gabert.github.io/index.html?art=exno5_6
>> >>
>> >> Many Thanks
>> >> Robert
>> >>
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>
>> >>
>> >>
>> >> -----------------------
>> >> Andrew Hume
>> >> 949-707-1964 (VO and best)
>> >> 732-420-0907 (NJ)
>> >> andrew at research.att.com
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >>
>> >
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>



More information about the zeromq-dev mailing list