[zeromq-dev] Notes from a hackathon

[MinRK]

On Tue, Feb 3, 2015 at 2:04 PM, Pieter Hintjens <ph at imatix.com> wrote:

> Thanks for the questions. I'd try to flatter by saying how great these
> questions are, except I know you're all way too smart to fall for
> that. :)
>
> Let me answer a bunch of questions in one go:
>
> * Sorry for not posting this here. I've written a more detailed
> analysis of the motivation for this thread (and experiments):
> http://hintjens.com/blog:84.
>
> * The main goal with this is simplicity and accuracy. I'm tired of
> teaching people ZeroMQ constructs that are clearly over-designed and
> disconnected from reality in subtle ways. Each apology for friction
> ("sorry, but don't use REQ-REP, they are kind of weird", "sorry,
> IDENTITY is not really identity", "sorry, message sometimes means
> frame and sometimes message") really signals costs up and down the
> stacks.
>
> * Sessions require protocol commands that are invisible to "normal"
> use. That means either protocol support (e.g. command frames) or some
> layering on top of ZeroMQ blobs.
>
> * Picture sending solves a cross-language problem, namely how to
> construct zproto-style commands without using dedicated codecs. While
> the API is C-ish, the functionality is not. I'd expect that to be
> wrapped in bindings.
>
> * Per-language serialization is an anti-pattern. It's common, and fine
> as such, yet ZeroMQ should IMO always strive to be cross-language.
>
> * REQ-REP between threads may make sense. We certainly use it (ZAP,
> for authentication).
>
> * Is this discussion C4 compatible? Yes, no, this is chatter, not a
> plan or roadmap. Until pull requests hit the road, it's all vapour.
> However the problems I restated in my blog article are real, to many
> people, and should be aired.
>
> * Multipart does easy serialization, yes. Not in C though. We've
> learned how to do variable list serialization with zproto-compatible
> picture sending in CZMQ. Is this a sufficient alternative? I think so.
> zsock_send() was essential to making actors cheap enough to use
> widely. egrep zsock_send *.c in malamute-core/src.
>

Perhaps it is because I spend my days in a higher level language like
Python, but zproto is not an attractive option. I will trust you that it
makes more sense when writing C or C++ libraries. I care a lot more about
multi-part content than routing frames being part of content, which I
acknowledge as a source of this confusion. I don't see anything in this
proposal that makes removing multipart *content* make any sense. Its
removal doesn't seem to provide any benefit. The new server socket removes
the *need* for it in a simple case, which I agree is a benefit, but there
seems to be no benefit to removing the capability itself.

>From a brief twitter discussion, Pieter pointed out that I might be able to
keep `send/recv_multipart` in pyzmq, and reimplement zero-copy framing with
`zmq_send`. I'm not sure how that works without SNDMORE, and I'm not sure
that I can get that and preserve access to `zmq_free_fn`, which I need to
synchronize garbage collection, but it's an interesting idea.


>
> * Locking on socketsL: we did in fact have a pull request for this
> once. It was... nasty. We reverted it. We breathed a sigh of relief.
> However looking back, the problem was valid and has never been solved
> since.
>
> * If you're doing multipart a lot in an application, you can replace
> that with zproto. You already have contracts then, why not make them
> explicit and documented? That's always a win in my experience.
>
> * Multi-hop request reply is a strange beast and I'm keen to
> understand those architectures that use it. My suspicion is that we've
> shifted our vision of what ZeroMQ "is" over the years. My own
> experience seems plausible, yet I distrust it. So, the stories in my
> article are please for falsification, not attempts to convince.
>

I use multi-hop (never more than 2-3, not generic comlicated hops), but I
had the same code working when zeromq-3.0 split routing information from
content (I think they were called 'command frames' or something?). I think
the main difference between this and the earlier attempt was the earlier
attempt used the same mechanism for subscriptions and routing frames, as
well as allowing the routing information itself to be multipart. My
multi-hop architecture is simple enough that I don't expect that I would
have difficulty moving the routing information to the application level,
instead of letting zeromq handle it.

-MinRK


>
> * I think we have learned to not break existing code. It's going to be
> interesting to run multiple models at once. We do this in CZMQ and it
> works OK. There isn't much overhead.
>
> * Authentication and routing information should be (MUST be) two
> totally separate layers. CURVE certainly does not use the routing ID
> (even if it's still misnamed as "IDENTITY"). You can have the same
> client on N connections (obviously). You authenticate a client using a
> mechanism-specific key: PLAIN user id, or CURVE public key.
>
> -Pieter
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150203/f094ce2d/attachment.htm>