[zeromq-dev] ZeroMQ 4.2 release, planning

Tue Nov 1 15:07:31 CET 2016

Status update:

- v2 APIs are gone from CZMQ:
  https://github.com/zeromq/czmq/pull/1531
  https://github.com/zeromq/czmq/pull/1532
- PR is out to bump the libtool version and changelog for libzmq:
  https://github.com/zeromq/libzmq/pull/2184
- PR is out to backport the zmq_msg_t fix to 4.1:
  https://github.com/zeromq/zeromq4-1/pull/155

Once it's all merged I will tag 4.2.0~rc1 first, then release 4.1.6 from
zeromq4-1 since quite a few fixes have accumulated. Then I'll send PRs
to prepare for CZMQ 4.0.0~rc1.

After the RCs are out, I'll work on the changelogs/NEWS files (help is
appreciated!) as they have fallen dramatically behind.

I'll also prepare more formal release notes for the stable rels, the RCs
will have just a quick note since they are RCs.

On Mon, 2016-10-31 at 23:47 +0000, Luca Boccassi wrote:
> Cool!
> 
> I can take care of it if you like. Tentative plan:
> 
> Tomorrow push an RC1 for libzmq, then the pr to CZMQ to retire v2 APIs,
> then the RC1 for CZMQ.
> 
> If it's all good then a couple days later the finals. I would really like
> to make it for the debian 9 transition freeze which is Saturday.
> 
> On Oct 31, 2016 22:23, "Doron Somech" <somdoron at gmail.com> wrote:
> 
> > Sorry, yes, lets do it :)
> >
> > On Oct 31, 2016 11:44 PM, "Luca Boccassi" <luca.boccassi at gmail.com> wrote:
> >
> >> Ping :-)
> >>
> >> On Oct 28, 2016 18:48, "Luca Boccassi" <luca.boccassi at gmail.com> wrote:
> >>
> >>> I have sent a solution for the alignment problem that solves the sigbus
> >>> problem without breaking ABI compat (plus follow-up for VC++ - sorry
> >>> Windows guys https://github.com/zeromq/libzmq/pull/2179 ).
> >>>
> >>> I tested the alignment and sigbus problem on x86_64 by enabling
> >>> alignment check with:
> >>>
> >>> __asm__("pushf\norl $0x40000,(%rsp)\npopf");
> >>>
> >>> All was fine.
> >>>
> >>> I ran tests built from the zeromq4-1 repository against a shared lib
> >>> from the head of libzmq repo, and they all run fine minus the
> >>> ZMQ_REQ_CORRELATE one but that option was borken anyway.
> >>>
> >>> This allows us to do a release now, and then when we are ready we can do
> >>> the ABI breakage, without blocking 4.2. Which is nice since it means it
> >>> might make it for Debian 9!
> >>>
> >>> So, Doron et al, shall we do the bump this weekend?
> >>>
> >>> On Thu, 2016-10-20 at 17:12 -0500, Thomas Rodgers wrote:
> >>> > I will have some time most likely the week of Nov6 (off for a week of
> >>> C++
> >>> > Committee 'fun') to test different message size alternatives. I'll
> >>> follow
> >>> > up with my results here for consideration the next time we are
> >>> inclined to
> >>> > break the ABI compatibility :)
> >>> >
> >>> > On Sunday, October 16, 2016, Brian Knox <bknox at digitalocean.com>
> >>> wrote:
> >>> >
> >>> > > A new stable version would definitely help me in my quest to get
> >>> ZeroMQ
> >>> > > support enabled by default in rsyslog in distros.
> >>> > >
> >>> > > On Sun, Oct 16, 2016 at 2:40 PM Doron Somech <somdoron at gmail.com>
> >>> wrote:
> >>> > >
> >>> > >> I say lets bump.
> >>> > >>
> >>> > >> On Oct 15, 2016 20:32, "Luca Boccassi" <luca.boccassi at gmail.com>
> >>> wrote:
> >>> > >>
> >>> > >>> As Thomas said, false sharing would be a real issue with 96.
> >>> > >>>
> >>> > >>> So given a release is long due, at this point I'd say to drop this
> >>> for
> >>> > >>> the moment.
> >>> > >>>
> >>> > >>> What do we do for the change to union for zmq_msg_t? Bump ABI
> >>> version or
> >>> > >>> not?
> >>> > >>>
> >>> > >>> On Thu, 2016-10-06 at 09:53 +0300, Doron Somech wrote:
> >>> > >>> > No new socket type, I worked at the time on binary message type,
> >>> might
> >>> > >>> > complete it sometime, but it is not urgent.
> >>> > >>> >
> >>> > >>> > If there is a lot of performance penalty we can give it up, I
> >>> will
> >>> > >>> > find another solution for the Radio-Dish.
> >>> > >>> >
> >>> > >>> > What about 96 bytes? same penalty?
> >>> > >>> >
> >>> > >>> > Regarding the binding, I'm not sure.
> >>> > >>> >
> >>> > >>> > On Sat, Oct 1, 2016 at 9:14 PM, Luca Boccassi <
> >>> luca.boccassi at gmail.com>
> >>> > >>> wrote:
> >>> > >>> > > On Tue, 2016-09-27 at 09:41 +0300, Doron Somech wrote:
> >>> > >>> > >> Sorry for the late response, increasing the msg_t structure
> >>> will be
> >>> > >>> > >> great, however this will require changing a lot of binding.
> >>> > >>> > >
> >>> > >>> > > I think I remember we need it for the new socket types, is that
> >>> > >>> correct?
> >>> > >>> > >
> >>> > >>> > > There is a large performance penalty (intuitively due to not
> >>> fitting
> >>> > >>> > > into a single cache line anymore, but haven't ran
> >>> perf/cachegrind),
> >>> > >>> and
> >>> > >>> > > the throughput with vsm type messages goes down by 4% (min)
> >>> and 20%
> >>> > >>> > > (max) for TCP, and 36% (min) 38 (max) for inproc, which is
> >>> quite a
> >>> > >>> lot,
> >>> > >>> > > so we need to be sure it's worth it.
> >>> > >>> > >
> >>> > >>> > > Regarding the bindings, after a quick search on the Github
> >>> org, I
> >>> > >>> could
> >>> > >>> > > only see:
> >>> > >>> > >
> >>> > >>> > > https://github.com/zeromq/lzmq/blob/master/src/lua/lzmq/
> >>> > >>> ffi/api.lua#L144
> >>> > >>> > > https://github.com/zeromq/clrzmq4/blob/master/lib/zmq.cs#L28
> >>> > >>> > > https://github.com/zeromq/pyczmq/blob/master/pyczmq/zmq.py#L
> >>> 177
> >>> > >>> > >
> >>> > >>> > > Other bindings just import zmq.h. Did I miss any?
> >>> > >>> > >
> >>> > >>> > >> Sorry for disappearing, baby and full time job is a lot :-),
> >>> > >>> hopefully
> >>> > >>> > >> I'm back...
> >>> > >>> > >
> >>> > >>> > > No worries, perfectly understandable :-)
> >>> > >>> > >
> >>> > >>> > >> On Mon, Aug 29, 2016 at 6:46 PM, Luca Boccassi <
> >>> > >>> luca.boccassi at gmail.com> wrote:
> >>> > >>> > >> > Sorry, I meant if we go with (1), not (2), we might bump
> >>> the size
> >>> > >>> as
> >>> > >>> > >> > well, since we are already doing another ABI-breaking
> >>> change.
> >>> > >>> > >> >
> >>> > >>> > >> > I agree on the solution as well.
> >>> > >>> > >> >
> >>> > >>> > >> > On Mon, 2016-08-29 at 17:12 +0200, Pieter Hintjens wrote:
> >>> > >>> > >> >> I'm confused between the (1) and (2) choices, and can't
> >>> see where
> >>> > >>> > >> >> bumping the message size fits.
> >>> > >>> > >> >>
> >>> > >>> > >> >> Nonetheless, I think bumping the size, fixing the alignment
> >>> > >>> issues,
> >>> > >>> > >> >> and bumping the ABI version is the best solution here.
> >>> > >>> > >> >>
> >>> > >>> > >> >> On Fri, Aug 26, 2016 at 12:33 PM, Luca Boccassi <
> >>> > >>> luca.boccassi at gmail.com> wrote:
> >>> > >>> > >> >> > I've given some more thoughts and testing to the
> >>> alignment
> >>> > >>> issue. I can
> >>> > >>> > >> >> > reproduce the problem by enabling alignment checks on
> >>> x86 too.
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > But most importantly, I think we cannot get away from
> >>> bumping
> >>> > >>> the ABI
> >>> > >>> > >> >> > with this fix, however we rearrange it, simply because
> >>> > >>> applications need
> >>> > >>> > >> >> > to be rebuilt against the new header to be fixed. A
> >>> simple
> >>> > >>> rebuild of
> >>> > >>> > >> >> > the libzmq.so is not enough. And the way to do this is
> >>> to bump
> >>> > >>> the ABI
> >>> > >>> > >> >> > so that distros can schedule transitions and rebuilds
> >>> and so
> >>> > >>> on.
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > So the choice list is now restricted to:
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > 1) Bump ABI
> >>> > >>> > >> >> > 2) Revert the fix and leave everything broken on sparc64
> >>> and
> >>> > >>> some
> >>> > >>> > >> >> > aarch64 (rpi3 seems not to be affected, must depend on
> >>> the SoC
> >>> > >>> flavour)
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > If we go with 2, we might as well get 2 birds with one
> >>> stone
> >>> > >>> and bump
> >>> > >>> > >> >> > the zmq_msg_t size to 128 as we have talked about in the
> >>> past.
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > Doron, this would help with the new UDP based socket
> >>> types
> >>> > >>> right?
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > Pros of bumping msg size:
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > - we can get rid of the malloc() in the lmsg type case
> >>> as all
> >>> > >>> the data
> >>> > >>> > >> >> > will fit
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > Cons:
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > - for the vsm/cmsg type cases (for most architectures
> >>> anyway)
> >>> > >>> it won't
> >>> > >>> > >> >> > fit anymore into a single cacheline
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > Given all this, I'd say we should go for it.
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > Opinions?
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > On Sat, 2016-08-13 at 16:59 +0100, Luca Boccassi wrote:
> >>> > >>> > >> >> >> Hello,
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> Trying to give some thoughts again on the libzmq 4.2
> >>> release.
> >>> > >>> It's
> >>> > >>> > >> >> >> really long overdue!
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> The main issue from my point of view is this change:
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> https://github.com/zeromq/libzmq/commit/
> >>> > >>> d9fb1d36ff2008966af538f722a1f4ab158dbf64
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> -typedef struct zmq_msg_t {unsigned char _ [64];}
> >>> zmq_msg_t;
> >>> > >>> > >> >> >>  +/* union here ensures correct alignment on
> >>> architectures
> >>> > >>> that require
> >>> > >>> > >> >> >> it, e.g.
> >>> > >>> > >> >> >>  + * SPARC
> >>> > >>> > >> >> >>  + */
> >>> > >>> > >> >> >>  +typedef union zmq_msg_t {unsigned char _ [64]; void
> >>> *p; }
> >>> > >>> zmq_msg_t;
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> This is flagged by the common ABI checkers tools as an
> >>> ABI
> >>> > >>> breakage
> >>> > >>> > >> >> >> (see: http://abi-laboratory.pro/trac
> >>> ker/timeline/zeromq/ ).
> >>> > >>> And it makes
> >>> > >>> > >> >> >> sense from this point of view: if some applications on
> >>> some
> >>> > >>> > >> >> >> architectures are broken due to wrong alignment, they
> >>> would
> >>> > >>> need to be
> >>> > >>> > >> >> >> rebuilt, and the way to ensure that is to bump the ABI
> >>> > >>> "current" digit
> >>> > >>> > >> >> >> to make sure maintainers do a rebuild.
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> On the other hand, signaling an ABI breakage is a pain,
> >>> and a
> >>> > >>> cause of
> >>> > >>> > >> >> >> major churn for packagers and maintainers. It means for
> >>> > >>> example a new
> >>> > >>> > >> >> >> package has to be created (eg: libzmq5 -> libzmq6), and
> >>> a
> >>> > >>> transition has
> >>> > >>> > >> >> >> to be started and all reverse dependencies need to be
> >>> > >>> rebuilt. And if
> >>> > >>> > >> >> >> this is pointless for all save a few corner cases (eg
> >>> SPARC64
> >>> > >>> as for
> >>> > >>> > >> >> >> above) it's all quite frustrating.
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> So we have a choice to make before we release 4.2, four
> >>> > >>> possibilities as
> >>> > >>> > >> >> >> far as I can see:
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> 1) Ignore the ABI checkers and get yelled at by
> >>> maintainers
> >>> > >>> and
> >>> > >>> > >> >> >> packagers. Also the SPARC64 users will most likely NOT
> >>> get
> >>> > >>> their bug
> >>> > >>> > >> >> >> fixed
> >>> > >>> > >> >> >> 2) Bump ABI revision to 6 and get yelled at by
> >>> maintainers
> >>> > >>> and packagers
> >>> > >>> > >> >> >> 3) Revert the above change and postpone it to when we
> >>> have a
> >>> > >>> more
> >>> > >>> > >> >> >> generally useful reason to break ABI (bump zmq_msg_t
> >>> from 64
> >>> > >>> to 128
> >>> > >>> > >> >> >> bytes for example, Doron?)
> >>> > >>> > >> >> >> 4) Try to be clever and revert the above change and use
> >>> > >>> something like
> >>> > >>> > >> >> >> #pragma pack(8). This will fool the ABI checkers (I
> >>> tried
> >>> > >>> it), and given
> >>> > >>> > >> >> >> that typedef is only used externally to allocate the
> >>> right
> >>> > >>> size it
> >>> > >>> > >> >> >> shouldn't actually affect anything, apart from the
> >>> users of
> >>> > >>> SPARC64
> >>> > >>> > >> >> >> which should get the bugfix with this too. This is very
> >>> > >>> sneaky :-)
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> CC'ing Lazslo, the Debian maintainer, given what we
> >>> choose to
> >>> > >>> do might
> >>> > >>> > >> >> >> result in a lot of work for him :-)
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> Opinions?
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> Kind regards,
> >>> > >>> > >> >> >> Luca Boccassi
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >> On Tue, 2016-05-03 at 10:39 +0200, Pieter Hintjens
> >>> wrote:
> >>> > >>> > >> >> >> > Hi all,
> >>> > >>> > >> >> >> >
> >>> > >>> > >> >> >> > I'm just throwing some ideas on the table. We have a
> >>> good
> >>> > >>> package of
> >>> > >>> > >> >> >> > work on master and it's probably time to make a 4.2
> >>> release.
> >>> > >>> > >> >> >> >
> >>> > >>> > >> >> >> > Luca has already back-ported the enable/disable draft
> >>> > >>> design from
> >>> > >>> > >> >> >> > zproject (CZMQ et al). Yay! So we can now release
> >>> stable
> >>> > >>> master
> >>> > >>> > >> >> >> > safely, while continuing to refine and extend the
> >>> draft API
> >>> > >>> sections.
> >>> > >>> > >> >> >> >
> >>> > >>> > >> >> >> > I propose:
> >>> > >>> > >> >> >> >
> >>> > >>> > >> >> >> > - to end with the stable fork policy; this was needed
> >>> years
> >>> > >>> ago when
> >>> > >>> > >> >> >> > we had massively unstable masters. It's no longer a
> >>> problem.
> >>> > >>> > >> >> >> > - to use the github release function for libzmq
> >>> releases
> >>> > >>> and deprecate
> >>> > >>> > >> >> >> > the separate delivery of tarballs.
> >>> > >>> > >> >> >> > - we aim to make a 4.2.0 rc asap, then fix any issues
> >>> we
> >>> > >>> get, with
> >>> > >>> > >> >> >> > patch releases as usual.
> >>> > >>> > >> >> >> > - we backport the release function to older maintained
> >>> > >>> releases (4.1,
> >>> > >>> > >> >> >> > 3.2) so that their tarballs are provided by github
> >>> instead
> >>> > >>> of
> >>> > >>> > >> >> >> > downloads.zeromq.org.
> >>> > >>> > >> >> >> >
> >>> > >>> > >> >> >> > Problems:
> >>> > >>> > >> >> >> >
> >>> > >>> > >> >> >> > - this will break a few things that depend on
> >>> > >>> downloads.zeromq.org. To
> >>> > >>> > >> >> >> > be fixed as we go.
> >>> > >>> > >> >> >> > - github tarballs are not identical to source
> >>> tarballs,
> >>> > >>> particularly
> >>> > >>> > >> >> >> > they lack `configure`. I propose changing our
> >>> autotools
> >>> > >>> build
> >>> > >>> > >> >> >> > instructions so they always start with `./autogen,sh`
> >>> no
> >>> > >>> matter where
> >>> > >>> > >> >> >> > the sources come from.
> >>> > >>> > >> >> >> >
> >>> > >>> > >> >> >> > I think this will work and also let us gracefully
> >>> > >>> deprecate/switch off
> >>> > >>> > >> >> >> > the downloads box.
> >>> > >>> > >> >> >> >
> >>> > >>> > >> >> >> > -Pieter
> >>> > >>> > >> >> >> > _______________________________________________
> >>> > >>> > >> >> >> > zeromq-dev mailing list
> >>> > >>> > >> >> >> > zeromq-dev at lists.zeromq.org
> >>> > >>> > >> >> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >>
> >>> > >>> > >> >> >
> >>> > >>> > >> >> >
> >>> > >>> > >> >> >
> >>> > >>> > >> >> > _______________________________________________
> >>> > >>> > >> >> > zeromq-dev mailing list
> >>> > >>> > >> >> > zeromq-dev at lists.zeromq.org
> >>> > >>> > >> >> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>> > >>> > >> >> _______________________________________________
> >>> > >>> > >> >> zeromq-dev mailing list
> >>> > >>> > >> >> zeromq-dev at lists.zeromq.org
> >>> > >>> > >> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>> > >>> > >> >
> >>> > >>> > >> >
> >>> > >>> > >
> >>> > >>>
> >>> > >>> _______________________________________________
> >>> > >> zeromq-dev mailing list
> >>> > >> zeromq-dev at lists.zeromq.org
> >>> > >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>> > >
> >>> > >
> >>> > _______________________________________________
> >>> > zeromq-dev mailing list
> >>> > zeromq-dev at lists.zeromq.org
> >>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >>>
> >>>
> >>>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20161101/62a7bce0/attachment.sig>