[zeromq-dev] Notes from a hackathon

[Michel Pelletier]

I think Brian has some good points here, there are numerous unrelated
issues being discussed in this thread.

A few points that I have:

Multi part messages have also bothered me.  However as a Python programmer
i see Min's points about the expense of buffer creation.  To my knowledge
zproto does not (yet) have Python generation support either, or maybe
something like generated cffi or ctypes wrappers around the zproto
generated C code.  That being said there are a variety of serialization
libraries for Python.  With some ctypes and mmap magic they can also be
done "zero copy" but it's not pretty:

https://gist.github.com/michelp/7522179

Multi part envelops are also how multi-hop routing is done.  I don't see
how the new ideas handle that.  I don't think we can just say "multi hop
routing is bad" and get rid of it.

"Thread safe" sockets do not sound appealing to me.  We did that, had a
long and contentious discussion with the person championing them, merged
it, then reverted it and that person is now no longer in the community.
Pieter was the most vocal opponent to them then and now he wants them
back.  Of course, anyone can change their mind, but the only current
argument I hear now for them though is improving the performance of short
lived threads, but that can be solved, more correctly in my opinion, with
thread or connection pools.  If you creating and tearing down threads that
rapidly then you have two problems.

-Michel

On Wed, Feb 4, 2015 at 3:37 AM, Brian Knox <bknox at digitalocean.com> wrote:

> After catching up on this thread, I feel like at least three problems are
> being conflated into one problem.  I'll state what I see being discussed
> from my perspective:
>
> 1. "Using multi part messages as a way to route to clients from a router
> socket is overly complicated and not how new users expect things to work"
>
> 2. "Using multi part messages for message serialization is costly, and
> potentially confusing to others."
>
> 3. "ZeroMQ sockets are not thread safe."
>
> While on an implementation level these three problems may be related, on a
> conceptual level I don't see them as related.  I may agree with some of
> these problem statements and not others.
>
> For me, my first priority is to always have the ability to get back a nice
> agnostic blob of bytes from ZeroMQ.   This makes it easy to make ZeroMQ
> socket use compatible with standard io interfaces in Go.  Structure for
> what is contained in those bytes is a concern of a different layer.
> Sometimes I use zproto for this (which I like), and other times I don't.
>
> As a demonstration that the problems are different problems, I solved #1
> for myself in goczmq without addressing anything else.
>
> I would assert some of the confusion in this discussion is that we're
> talking about multiple problem statements at the same time.
>
> Cheers - and it was great meeting people this week!
>
> Brian
>
>
>
>
> On Wed, Feb 4, 2015 at 12:50 AM, Pieter Hintjens <ph at imatix.com> wrote:
>
>> Ironically, in my testing of high message rate), allowing multipart
>> creates significant costs. Multipart is just one way of getting
>> zero-copy, and even then only works on writing, not reading.
>>
>> For high performance brokers like Malamute I'd *really* like to be
>> moving blobs around instead of lists of blobs.
>>
>>
>> On Wed, Feb 4, 2015 at 12:41 AM, Gregg Irwin <gregg at pointillistic.com>
>> wrote:
>> > M> Perhaps it is because I spend my days in a higher level language
>> > M> like Python, but zproto is not an attractive option.
>> >
>> > Same here. I will read in detail about it shortly, but it may not make
>> > it into my toolbox as a multipart replacement. Multipart looked very
>> > cool when I found 0MQ, but I've ended up not using it much. I'm not
>> > doing high performance stuff though. Simplicity and ease of use are
>> > tops on my list.
>> >
>> > -- Gregg
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20150204/30f0feb5/attachment.htm>