[zeromq-dev] C4 and CI

Mon Nov 16 15:12:29 CET 2015

John - thanks for sharing your experiences with trying out the C4 process!
It's definitely an interesting situation when "the build system" that
determines if a PR has broken a build or not can't be simple or in one spot.

I think the key point to identify is "what is the criteria for a PR to be
accepted" in this situation. If you have multiple people using multiple
community run systems for performing automated tests, you'll probably want
to take into account the possibility that some subset of these systems may
be not functioning at any given point in time.  Since you don't want that
to halt all progress on your project, identifying the subset of things that
MUST happen (vs SHOULD happen) for a PR to be merged will be important.

Cheers,
Brian

On Mon, Nov 16, 2015 at 3:31 AM, Pieter Hintjens <ph at imatix.com> wrote:

> On-topic, I think, as it relates to our own development practice.
>
> Though I'm not sure what your question actually is... :)
>
> What we've learned, over time, is that the less "official" pieces
> there are, the better. So we have no official bindings, no official
> implementations, and fewer and fewer official packages. Distributing
> power and freedom works better. The only real exception is the
> branding, which my company holds, to prevent rule-breaking projects
> calling themselves "ZeroMQ."
>
> Even the rules are chosen by each project and not all ZeroMQ projects
> follow C4.
>
> For CI, we use a growing mix of systems, and the success/failure
> information is fed into the pull request, yet it's not stressful. I
> often merge broken PRs simply because it keeps the flow of activity
> going, and is more fun for everyone than waiting.
>
> -Pieter
>
>
> On Mon, Nov 16, 2015 at 7:49 AM, Peter Krey <peterjkrey at gmail.com> wrote:
> > this question is appropriate for stackoverflow, not the zeromq mailing
> list.
> >
> > On Sun, Nov 15, 2015 at 10:35 PM, John Morris <john at zultron.com> wrote:
> >>
> >> Hello list,
> >>
> >> We've been using parts of C4 to manage the Machinekit project,
> >> open-source machine control software.  I love the idea of C4, and really
> >> buy into its basic ideas of reducing friction of the development process
> >> in order to scale the developer and, following that, the user community,
> >> the size of which is a primary indicator of project success.
> >>
> >> However, it turns out C4 is a very challenging idea that most people
> >> seem to have trouble swallowing in its entirety, as I have.
> >> Accordingly, we really only follow parts of C4, which is a
> >> disappointment to me.
> >>
> >> Part of this is my own fault.  I run a Buildbot for the project that
> >> builds many different configurations, i.e. combinations of OS, hardware
> >> architecture and real-time thread environment.  It's especially because
> >> of the latter that using a CI system on public infrastructure, like
> >> Travis CI, isn't possible:  running regression tests under the several
> >> RT thread systems (RT_PREEMPT, Xenomai, RTAI) requires special kernel
> >> support unavailable in those environments, so it's necessary to run the
> >> CI system on bare metal or a VM with custom kernels.  In order to
> >> support many OS, arch and RT environments, my Buildbot is extremely
> >> complex and essentially unreproducible.  As a result, despite my clear
> >> communication that it's a system contributed by a third-party (my
> >> company), the community tends to see it as the project's "official" CI
> >> system, dictating "officially-supported" configurations and providing
> >> the "official" package stream.  I shouldn't have to tell this audience
> >> how this undermines C4, and besides this CI system is a SPOF for the
> >> project and is taking too much of my own energy to maintain.
> >>
> >> So, taking the last two years' lessons learned about what (IMO) a C4
> >> community needs in a CI infrastructure (esp. when public CI services
> >> don't make sense), I have a plan for a CI system that seems a better fit
> >> with the spirit of C4, and solves other practical issues at the same
> time.
> >>
> >> Key to the idea is scalability by distributing the burden across many
> >> members of the community.  A trivially-reproducible CI system, such as a
> >> Buildbot instance in a Docker container (either on private hardware or
> >> the cloud), may be set up by any community member to build/test/package
> >> one single particular favorite configuration, for example Debian Jessie
> >> on RPi2 with RT_PREEMPT kernel pointing at the official git repo master
> >> branch.  Set up instructions should be as short as "install Docker;
> >> clone this git repo; edit the configuration; run the Docker container".
> >>
> >> For each PR, the system builds the code for and tests it in the
> >> configured environment (these duties could be separated).  The
> >> build/test results are then aggregated (exactly how is TBD) with those
> >> of other contributed CI systems (with different configurations), and the
> >> PR is updated with a (or a list of) pass/fail status(es), which both
> >> Contributors and Maintainers may use to check for and diagnose problems.
> >>
> >> For each merge, the system builds binary packages and updates a package
> >> repository.  This repo may be published and advertised to other
> >> community members interested in the same configuration.
> >>
> >> In this way, anyone is able to set up a CI system for a particular
> >> configuration.  If many people do so, the burden of running a CI system
> >> for many target configurations will be distributed across many community
> >> members.
> >>
> >> I'm so enamored with this idea partly because of how it fits with C4:
> >>
> >> - Distributing the CI system across the community scales up the number
> >> of configurations built and tested without overburdening any one
> >> community member.
> >> - No one person dictates what configurations are officially supported by
> >> the project:  anyone can contribute build results for any configuration,
> >> and while Contributors and Maintainers will see when a PR breaks the
> >> configuration, ultimately it's up to that configuration's champion to
> >> work with the community to ensure it continues to build.
> >> - Third-party stabilization forks become trivial to set up and publish
> >> packages for; simply set up a new CI system and point it at the fork's
> >> repo on GitHub.  This especially suits vendors wanting to ship
> >> Machinekit on their machine controller hardware.
> >>
> >> As of now, I've implemented many parts that would go into this system,
> >> but many other parts are missing, and it's not a top priority for me.
> >> That puts this idea in the "gedankenexperiment" category, but I'm still
> >> curious how ZeroMQ community members will react, assuming you've made it
> >> this far into the mail!
> >>
> >> An update since I started writing this:  My Buildbot's ARM builder
> >> broke, and I've decided not to resuscitate.  Some others in the project
> >> have decided to set up a new CI system based around OpenSUSE's public
> >> OBS instance.  My earlier experiments with building Debian packages on
> >> OBS were ugly, but it turns out it can be made to work.  Building on OBS
> >> satisfies some of my proposed requirements for a C4 project CI system,
> >> especially that it is trivially reproducible.  However, in practice, it
> >> will take a lot of work to nail down the entire build flow, and the
> >> question of running unit tests in various environments is unresolved.
> >>
> >>         John
> >> _______________________________________________
> >> zeromq-dev mailing list
> >> zeromq-dev at lists.zeromq.org
> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> >
> >
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> >
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20151116/88efad9b/attachment.htm>