[zeromq-dev] Notes from a hackathon

[James Gatannah]

This ties into the heart of some related discussions we've been having
at work recently. We have a bunch of new hires who are trying to come
to grips with a messaging protocol that's evolved over the past year
or so as we've gotten more familiar with 0mq and adapted it to our
changing needs.

It's really just meandering commentary about the way I perceive the
proposal as it relates to the way we're currently using the library
(in C++, C#, python, and clojure via jzmq). It may be a complete waste
of time, or it may be a validation of the proposed changes.

Our current incarnation looks something like:
Frame 1: address header (originally for pub/sub)
Frame 2: bundle of serialized data (with a bunch of HTTP-style
metadata, like Verb, URI, headers, the "actual" message body etc).
Normal router/dealer identity frame sequences.

New team members would very much like to split that apart into 7 or 8
more frames, just so the "actual" message body is totally isolated. We
could probably accomplish something similar using zproto, though I'd
much rather just cram everything into one single frame.

We can't do either, mainly due to time constraints around changing the
C++ layer.

Pieter Hintjens <ph at imatix.com> wrote:

> I think we can either get the routing id as a message property, and/or
> add a "reply to message" semantic that does this implicitly. The
> second is easier for simple servers, the first is better for realistic
> servers.
>
> Since it only applies to server sockets, I think we would benefit from
> a zserver_reply () method (or similar).

For our app, we have "clients" that have to establish the initial
connection (they use dealer sockets. The central messaging hub has the
router). After that, those "clients" really switch to being "servers."

We're attaching the identity frame(s) to each incoming message so we
can use them to send the response. We're also tracking them on the
"server" side when a "client" first connects so the "server" (which
will usually act more like a client) can send off its requests.

We decided on Friday to just start to thinking of everything as a
"peer" and just think in terms of "message events" instead of thinking
in terms of "client/server" and "request/response".

We haven't run into scenarios where peers drop connections, but we're
also barely at the PoC stage.


Arnaud Kapp <kapp.arno at gmail.com> wrote:
>
> I think thread-safe socket would be cool. As in great to have in some
> particular situation, but most of the case it shouldn't be needed.

This would be a huge win for us.

We have a ridiculously complicated pair of event loops in our central
message hub(s) to make the multi-threaded/asynchronous pieces work
correctly. I wrote it, and I'm absolutely positive that I'll regret it
in a couple of months.

That's on the clojure side, where multi-threading is about as simple
as it can get.

We can get away with single-threading in the C++ parts.

The .NET parts are going to be really ugly.

The python pieces...well, I keep being forced to say "I told you so" about them.


> IMO framing is very good because it allows you to send (and receive)
> various kind of data *easily*.

>From our perspective:

We really want to keep a single messaging protocol at every layer. The
details of that protocol keep evolving as our understanding grows, but
we've managed to maintain backwards compatibility despite my best
efforts to break it.

And that's been a very good thing.

We started with this approach because we needed to have a string that
subscribers could use. If that had just been a standard part of our
main message body (which, at the time, was JSON), there wouldn't have
been any good/standard way to put it at the front of the message.

Yes, there are obviously ways to make that work. They aren't
approaches that we've ever had time to put into practice.

It's an ugly truth about startups: we never have time to do "internal"
things correctly because we're always too busy working on the next set
of must-have features.

> Date: Tue, 3 Feb 2015 12:00:52 -0800
MinRK <benjaminrk at gmail.com> wrote:

> As far as I can tell, I also use the multihop case that Pieter's blog post
> suggests "does not seem to happen in practice" every day, so maybe my use
> of zeromq is not typical.

We aren't using it yet, but it's a major part of our road-map.

That doesn't mean we'll ever actually implement this, but it seems
pretty likely that we will.

I can definitely see the point that the blog post makes about this.
We'd be better off doing our own routing/messaging-frame/custom
message pieces in the middle.

At the same time...when it becomes time to make this happen, I can
almost guarantee that it will be a desperate last-minute feature that
I have to implement over the course of one weekend. I'll be thrilled
to use the library that I've already written to use the current
architecture.

(Thanks for never breaking backwards compatibility!!)

Date: Tue, 3 Feb 2015 23:50:54 +0100
Arnaud Kapp <kapp.arno at gmail.com> wrote:

> However, a good read through the guide really
> help understanding how things work.

It does. My experience has been that convincing people to read that
guide is painful.

It's a travesty of injustice. The Guide is one of the best, most
readable tech manuals that I've ever read.

But half the people I work with absolutely refuse to read it. Instead
they treat the whole thing like voodoo and spend their days
recompiling, hoping the stack traces will go away.

Or maybe that wireshark will magically show them that their sockets
have started working when they're doing it wrong and can't find the
time to RTFM.

I don't believe my current job is atypical. Except that possibly I
work with people who are far smarter than average. We live (and work)
in a society that has forgotten how to read.

I'd normally rather dumb things down at the app level than at the
low-level socket level (and I consider 0mq the low-level interface
these days). But I'm also tired of telling people that they just need
to suck it up and cope with the NULL message frame to/from the Dealer
socket because I'm not going to make my Router socket cope with their
"just this once" special magical case that should ignore that
particular convention.

> I don't have much experience teaching, and none teaching ZMQ.

I wouldn't say that I have "much" experience in either area. But this
sort of thing is definitely a pain point.

It's been more painful than convincing the C++ guy that he shouldn't
NULL-terminate his strings.

We've spent enough time coping with many of these issues that I got
twitchy when I first read the announcement. I probably laughed a
little hysterically when I mentioned it to my colleagues. This is
going to totally invalidate an incredible volume of work.

After I got over the twitches, I realized that this is a Very Good
Thing (TM). Deleted source code is thoroughly debugged.

Now I just have to convince our C++ guy...

Regards,
James