[zeromq-dev] design communication protocols

[Cem Karan]

I agree with Brian that using a serialization library is a good idea, but they'll only verify that the archived byte string can be deserialized correctly.  To verify that your system won't get into a wedged state, you'll need to use something stronger, such as spin (thank you for mentioning it Andrew!).

Also, how concerned are you with malicious actors?  Guarding against random corruption is pretty easy, but if you know that there will be malicious actors on your network, then you really DO need something that can verify your model, AND you need to spend some time designing a real threat model.  That is, what capabilities will the malicious actors have?  Can they only corrupt packets, or can they see the state of the entire network, and inject/destroy packets at will?  Is there more than one attacker?  Do they have an unlimited amount of power/energy, or do they have to make choices about when/where they attack?  

Finally, I'm going to mention MessagePack (http://msgpack.org/) and Google Protocol Buffers (https://developers.google.com/protocol-buffers/) as alternatives to JSON.  They are only serialization libraries, but I've noticed that MessagePack + blosc (http://blosc.pytables.org/trac) make for a fairly powerful way of serializing & compressing data.  I won't post any numbers on transmission rates because the rest of my code is sufficiently horrible that it will skew the results, but even with HIGHLY non-optimal code, I'm pleased with the speeds/compression ratios I'm getting.

Thanks,
Cem Karan

On Aug 17, 2012, at 7:57 AM, Brian Knox wrote:

> Andrea:
> 
> We heavily utilize JSON for our communications protocols.  In cases where we are using things such as MDP, which is multi-frame (http://rfc.zeromq.org/spec:7), we use JSON for the message body of the multi frame message.
> 
> I see you are using Python - a really nice library for designing message "models" that can be serialized to JSON in python is "structures":
> 
> https://github.com/j2labs/structures
> 
> It does type validation, which is very handy for ensuring your messages are what you expect.
> 
> Additionally, the "ujson" json library for Python is pretty fast:
> 
> http://pypi.python.org/pypi/ujson/
> 
> We combine this with multi frame protocols because we do not want to have to parse the JSON frame for things like routing information.
> 
> Brian
> 
> On Fri, Aug 17, 2012 at 7:29 AM, andrea crotti <andrea.crotti.0 at gmail.com> wrote:
> 2012/8/17 Andrew Hume <andrew at research.att.com>:
> > yes!
> > you have discovered that devising a distributed computation is almost
> > isomorphic
> > to designing the protocol the distributed parts use to communicate.
> >
> > this is, in general, quite hard, and i know of no useful shortcuts other
> > than
> > to try and build on existing patterns (such as in teh Guide).
> > i have used one method which, although somewhat tedious, did actually work.
> >
> > the protocol part of teh code was written in pseudocode with m4 macros.
> > when you "compiled" the pseudocode, it produced two outputs:
> > 1)  working C code
> > 2) working Spin code (this language used to be called Promela).
> >
> > Spin is a protocol verifier which can handle fairly large protocols.
> > the build process verified that the protocol was "correct" (taht is,
> > it had no deadlocks etc) and then built the executable.
> >
> > there were occasional missteps in ensuring the two parts were equivalent,
> > but i will say we never had a protocol botch. that is, we never deadlock'ed
> > or wedged because of a protocol error. these had all been caught
> > during compiling.
> >
> > obviously, none of this was specific to spin; any protocol verifier would
> > work, i think.
> >
> 
> Uhm that's really interesting, I used spin a long time ago in
> university but I didn't even think it might be useful now, and it's
> probably a very neat idea to formally verify the protocols..
> 
> Would you mind to share a simple example maybe?  I will use Python and
> might generate the spin code from there too, but would still be nice
> to see something running..
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> 
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 1581 bytes
Desc: not available
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20120818/8460195d/attachment.bin>