[zeromq-dev] Announce: CZMQ 2.0 coming up

[Pieter Hintjens]

The zero copy API was frankly, nasty, and spread to multiple classes.
I'd asked the question at the time, whether there was any science
showing that it was faster, and worth the mess. Optimization for its
own sake means nothing, without figures showing the benefit. Anything
that makes the API messy or complex is a problem.

I'd recommend using the low-level ZMQ API for zero-copy at this stage
until / unless we can find a nicer way of defining an API.

Here are my specific concerns with the code:

* The names are long and nasty; adding _zero_copy to everything breaks
the CZMQ style of short clear names.
* Using callbacks is always delicate, and you could clearly see the
test cases becoming unreadable, which is a red flag.
* There were no measurements, no figures, which undermines any claim
of better performance.
* The semantics spread across multiple classes, instead of being in a
single class. That's not good API design.
* It makes no sense to define the deallocator callback in every single object.

What I'd suggest, and this isn't a lot of work, is to make a good test
case that shows the cost of copying, and then optimize that using a
single, new class with the right design. The natural design would be
based around the deallocator, not the data, IMO.

If someone can prove they need zero-copy, and provide a decent test
case, I'll try to design a simple API that does the job.

-Pieter


On Tue, Sep 17, 2013 at 9:31 AM, Laurent Alebarde <l.alebarde at free.fr> wrote:
> Hi Pieter,
>
> On my side, I would prefer that you keep zframe and zmsg zero-copy methods.
> My development philosophy is "do not loose on performance unless you are
> obliged to do so for legitimate reasons". They are not more complex than
> other things and are usefull.
>
> May I suggest to keep it until the next class is available ? And have a
> deprecation period of time ?
>
> Cheers,
>
>
> Laurent.
>
>
> Le 17/09/2013 07:48, Pieter Hintjens a écrit :
>
> Hi all,
>
> Just a heads-up to CZMQ users that I've been working on a new version,
> which will come out as 2.0.0 rc1 soon. There are two main goals: there
> were a number of API improvements I wanted to make (like the new
> zpoller class), plus various internal cleanups. Two, to support
> ZMQ/4.0 security out of the box, especially CURVE certificates, and
> authentication.
>
> Attached is a detailed list of the changes. The first set is waiting
> to be merged at https://github.com/zeromq/czmq/pull/230.
>
> There should be no impact for existing applications, except a number
> of deprecated methods are gone now. If that hurts anyone, let me know
> and we'll add them back.
>
> I'll write up an article showing how to use the security classes.
>
> -Pieter
>
>
> Bug fixes and cleanups of codebase:
>
> * zconfig_load () failed if the ZPL file contained a blank line.
>
> * zconfig_locate () allows and ignores initial '/' on pathname.
>
> * Added zsys_vprintf () to centralize all variable string formatting;
>   this code was duplicated many times across the project.
>
> * Updated zmsg interface documentation to be more detailed.
>
> * Removed deprecated zstr_sendfm () method.
>
> * Removed deprecated zstr_sendf () method.
>
> * Removed definition of ZMQ_IGNERR and added zloop_set_tolerant ()
>   to configure a specific poller as error-tolerant.
>
> * Removed deprecated zframe and zmsg zero-copy methods; this API was
>   too complex and split over several classes, and IMO if we want this
>   optimization it should be in a new zero-copy class with a clean API.
>
> * Removed deprecated zsocket zero-copy methods, for above reason.
>
> * Removed deprecated zlist_copy method.
>
> * Renamed zframe_print_to_stream to zframe_fprint and kept old name
>   as deprecated macro.
>
> * Removed (not deprecated) zctx_set_hwm () as its semantics were not
>   consistent, and confusing to people.
>
> * Removed (not deprecated) zctx_hwm (). We now have separate HWMs for
>   normal socket SND and RCV, and for inter-thread pipes. There is no
>   need to provide accessor methods for these.
>
> New API methods to simplify common messaging tasks:
>
> * Added zpoller class to provide new minimalist interface to zmq_poll.
>   This class makes it very simple to read off a set of sockets, which
>   is an 80% usecase.
>
> * Added zstr_sendx () to send multiple strings in one call. Inspired
>   by PyZMQ's multipart send method.
>
> * Added zstr_recvx () to receive multiple strings in one call. We do
>   a lot of multipart string reads; this makes it simpler for simple
>   cases.
>
> * Added zclock_timestr () to provide date/time as printable string.
>
> * Added zmsg_append () that nullifies caller's frame reference; this
>   is more consistent with the API style than zmsg_add (), which is
>   deprecated.
>
> New classes used for security infrastructure:
>
> * Added zchunk class (from FileMQ), for managing memory blocks.
>   This is a simpler model than zframes, which are optimized for
>   message data.
>
> * Added zhash_comment () to add comments to a serialized zhash file.
>
> * Added zhash_refresh () to reload a backing file if modified on disk.
>
> * Added zconfig_comment () to add comments to saved config file, and
>   added code to save such comments from file when loading it.
>
> * Moved zfile (char *filename) methods to zsys, to make place for
>   a new zfile class based on the CLASS object model. The old methods
>   are still provided but are deprecated.
>
> * Added zfile class (from FileMQ), for managing files. This provides
>   a rich file class with metadata and operations. This is a component
>   for directory management (zdir).
>
> * Added zdir class (from FileMQ), for managing directories. This lets
>   us work with filesystem directories. I wanted this to be able to
>   load certificates for security authentication.
>
> * The zdir and zfile classes are slightly stripped down from their
>   FileMQ originals; no symbolic links, no directory difference tools.
>
> * Unfortunately to keep the old zfile_size and zfile_time methods, I
>   had to use zfile_cursize and zfile_modified in the new class. This
>   is a little clumsy.
>
> * Added zsys_file_modified () to return timestamp for a filename.
>
> * Added zcert class to work with Curve certificates. A certificate is
>   a public + secret key pair, plus metadata like name, email, etc. and
>   is saved as two files, a public text file and a secret text file.
>   Certificates are stored in ZPL (ZMQ RFC 4) configuration file format.
>
> * Added zauth class to automate ZAP authenticators. This class does a
>   simple plug and play authentication of clients against certificates
>   stored in the $HOME/.curve directory. This can be extended over time
>   with other designs.
>
> * Extended zsockopt generation to allow use of binary keys in CURVE
>   key options (e.g. zsocket_set_curve_secretkey_
> bin ()).
>
> * Added addons/makecert.c tool to generate CURVE certificates.
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>