[zeromq-dev] design communication protocols

Brian Knox briank at talksum.com
Sat Aug 18 19:28:59 CEST 2012


Cem -

I just wanted to say I agree that MessagePack and Protobuf are both very
capable serdes libraries - I've worked with them both before.  Some
languages don't deal that well with JSON and they're excellent choices.

On Sat, Aug 18, 2012 at 12:32 PM, Cem Karan <cfkaran2 at gmail.com> wrote:

> I agree with Brian that using a serialization library is a good idea, but
> they'll only verify that the archived byte string can be deserialized
> correctly.  To verify that your system won't get into a wedged state,
> you'll need to use something stronger, such as spin (thank you for
> mentioning it Andrew!).
>
> Also, how concerned are you with malicious actors?  Guarding against
> random corruption is pretty easy, but if you know that there will be
> malicious actors on your network, then you really DO need something that
> can verify your model, AND you need to spend some time designing a real
> threat model.  That is, what capabilities will the malicious actors have?
>  Can they only corrupt packets, or can they see the state of the entire
> network, and inject/destroy packets at will?  Is there more than one
> attacker?  Do they have an unlimited amount of power/energy, or do they
> have to make choices about when/where they attack?
>
> Finally, I'm going to mention MessagePack (http://msgpack.org/) and
> Google Protocol Buffers (https://developers.google.com/protocol-buffers/)
> as alternatives to JSON.  They are only serialization libraries, but I've
> noticed that MessagePack + blosc (http://blosc.pytables.org/trac) make
> for a fairly powerful way of serializing & compressing data.  I won't post
> any numbers on transmission rates because the rest of my code is
> sufficiently horrible that it will skew the results, but even with HIGHLY
> non-optimal code, I'm pleased with the speeds/compression ratios I'm
> getting.
>
> Thanks,
> Cem Karan
>
> On Aug 17, 2012, at 7:57 AM, Brian Knox wrote:
>
> > Andrea:
> >
> > We heavily utilize JSON for our communications protocols.  In cases
> where we are using things such as MDP, which is multi-frame (
> http://rfc.zeromq.org/spec:7), we use JSON for the message body of the
> multi frame message.
> >
> > I see you are using Python - a really nice library for designing message
> "models" that can be serialized to JSON in python is "structures":
> >
> > https://github.com/j2labs/structures
> >
> > It does type validation, which is very handy for ensuring your messages
> are what you expect.
> >
> > Additionally, the "ujson" json library for Python is pretty fast:
> >
> > http://pypi.python.org/pypi/ujson/
> >
> > We combine this with multi frame protocols because we do not want to
> have to parse the JSON frame for things like routing information.
> >
> > Brian
> >
> > On Fri, Aug 17, 2012 at 7:29 AM, andrea crotti <
> andrea.crotti.0 at gmail.com> wrote:
> > 2012/8/17 Andrew Hume <andrew at research.att.com>:
> > > yes!
> > > you have discovered that devising a distributed computation is almost
> > > isomorphic
> > > to designing the protocol the distributed parts use to communicate.
> > >
> > > this is, in general, quite hard, and i know of no useful shortcuts
> other
> > > than
> > > to try and build on existing patterns (such as in teh Guide).
> > > i have used one method which, although somewhat tedious, did actually
> work.
> > >
> > > the protocol part of teh code was written in pseudocode with m4 macros.
> > > when you "compiled" the pseudocode, it produced two outputs:
> > > 1)  working C code
> > > 2) working Spin code (this language used to be called Promela).
> > >
> > > Spin is a protocol verifier which can handle fairly large protocols.
> > > the build process verified that the protocol was "correct" (taht is,
> > > it had no deadlocks etc) and then built the executable.
> > >
> > > there were occasional missteps in ensuring the two parts were
> equivalent,
> > > but i will say we never had a protocol botch. that is, we never
> deadlock'ed
> > > or wedged because of a protocol error. these had all been caught
> > > during compiling.
> > >
> > > obviously, none of this was specific to spin; any protocol verifier
> would
> > > work, i think.
> > >
> >
> > Uhm that's really interesting, I used spin a long time ago in
> > university but I didn't even think it might be useful now, and it's
> > probably a very neat idea to formally verify the protocols..
> >
> > Would you mind to share a simple example maybe?  I will use Python and
> > might generate the spin code from there too, but would still be nice
> > to see something running..
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120818/384576f1/attachment.htm>


More information about the zeromq-dev mailing list