[zeromq-dev] C4 and CI

Mon Nov 16 15:36:44 CET 2015

At DomCode (excellent conference, beautiful city!) someone asked me to
argue the case for fast merges (before CI is done, and without code
review). Here's what I said.

You have four possible cases:

1. PR is excellent, works, and trouble free. A fast merge gives
contributor a pleasant kick.

2. PR is good, has things to fix. A fast merge encourages others to
come fix. We now get 2+ people working on a problem. This is an even
better outcome than (1).

3. PR is mediocre, cannot be fixed, author has good intentions. A fast
merge followed by a revert gives contributor incentive to try again.

4. PR is bad, cannot be fixed, and author is a troll. A fast merge
records this and forces others to revert it. We now get existing
contributors defending against vandalism, and trolls confronted
rapidly and effectively.

In all four cases, delaying the merge for any reason turns out to be
the worse strategy.

So my advice is treat CI as informational and allow your community to
build and extend it as one tool for improving the contract the
software exports (along with test cases and documentation).

-Pieter

On Mon, Nov 16, 2015 at 3:12 PM, Brian Knox <bknox at digitalocean.com> wrote:
> John - thanks for sharing your experiences with trying out the C4 process!
> It's definitely an interesting situation when "the build system" that
> determines if a PR has broken a build or not can't be simple or in one spot.
>
> I think the key point to identify is "what is the criteria for a PR to be
> accepted" in this situation. If you have multiple people using multiple
> community run systems for performing automated tests, you'll probably want
> to take into account the possibility that some subset of these systems may
> be not functioning at any given point in time.  Since you don't want that to
> halt all progress on your project, identifying the subset of things that
> MUST happen (vs SHOULD happen) for a PR to be merged will be important.
>
> Cheers,
> Brian
>
> On Mon, Nov 16, 2015 at 3:31 AM, Pieter Hintjens <ph at imatix.com> wrote:
>>
>> On-topic, I think, as it relates to our own development practice.
>>
>> Though I'm not sure what your question actually is... :)
>>
>> What we've learned, over time, is that the less "official" pieces
>> there are, the better. So we have no official bindings, no official
>> implementations, and fewer and fewer official packages. Distributing
>> power and freedom works better. The only real exception is the
>> branding, which my company holds, to prevent rule-breaking projects
>> calling themselves "ZeroMQ."
>>
>> Even the rules are chosen by each project and not all ZeroMQ projects
>> follow C4.
>>
>> For CI, we use a growing mix of systems, and the success/failure
>> information is fed into the pull request, yet it's not stressful. I
>> often merge broken PRs simply because it keeps the flow of activity
>> going, and is more fun for everyone than waiting.
>>
>> -Pieter
>>
>>
>> On Mon, Nov 16, 2015 at 7:49 AM, Peter Krey <peterjkrey at gmail.com> wrote:
>> > this question is appropriate for stackoverflow, not the zeromq mailing
>> > list.
>> >
>> > On Sun, Nov 15, 2015 at 10:35 PM, John Morris <john at zultron.com> wrote:
>> >>
>> >> Hello list,
>> >>
>> >> We've been using parts of C4 to manage the Machinekit project,
>> >> open-source machine control software.  I love the idea of C4, and
>> >> really
>> >> buy into its basic ideas of reducing friction of the development
>> >> process
>> >> in order to scale the developer and, following that, the user
>> >> community,
>> >> the size of which is a primary indicator of project success.
>> >>
>> >> However, it turns out C4 is a very challenging idea that most people
>> >> seem to have trouble swallowing in its entirety, as I have.
>> >> Accordingly, we really only follow parts of C4, which is a
>> >> disappointment to me.
>> >>
>> >> Part of this is my own fault.  I run a Buildbot for the project that
>> >> builds many different configurations, i.e. combinations of OS, hardware
>> >> architecture and real-time thread environment.  It's especially because
>> >> of the latter that using a CI system on public infrastructure, like
>> >> Travis CI, isn't possible:  running regression tests under the several
>> >> RT thread systems (RT_PREEMPT, Xenomai, RTAI) requires special kernel
>> >> support unavailable in those environments, so it's necessary to run the
>> >> CI system on bare metal or a VM with custom kernels.  In order to
>> >> support many OS, arch and RT environments, my Buildbot is extremely
>> >> complex and essentially unreproducible.  As a result, despite my clear
>> >> communication that it's a system contributed by a third-party (my
>> >> company), the community tends to see it as the project's "official" CI
>> >> system, dictating "officially-supported" configurations and providing
>> >> the "official" package stream.  I shouldn't have to tell this audience
>> >> how this undermines C4, and besides this CI system is a SPOF for the
>> >> project and is taking too much of my own energy to maintain.
>> >>
>> >> So, taking the last two years' lessons learned about what (IMO) a C4
>> >> community needs in a CI infrastructure (esp. when public CI services
>> >> don't make sense), I have a plan for a CI system that seems a better
>> >> fit
>> >> with the spirit of C4, and solves other practical issues at the same
>> >> time.
>> >>
>> >> Key to the idea is scalability by distributing the burden across many
>> >> members of the community.  A trivially-reproducible CI system, such as
>> >> a
>> >> Buildbot instance in a Docker container (either on private hardware or
>> >> the cloud), may be set up by any community member to build/test/package
>> >> one single particular favorite configuration, for example Debian Jessie
>> >> on RPi2 with RT_PREEMPT kernel pointing at the official git repo master
>> >> branch.  Set up instructions should be as short as "install Docker;
>> >> clone this git repo; edit the configuration; run the Docker container".
>> >>
>> >> For each PR, the system builds the code for and tests it in the
>> >> configured environment (these duties could be separated).  The
>> >> build/test results are then aggregated (exactly how is TBD) with those
>> >> of other contributed CI systems (with different configurations), and
>> >> the
>> >> PR is updated with a (or a list of) pass/fail status(es), which both
>> >> Contributors and Maintainers may use to check for and diagnose
>> >> problems.
>> >>
>> >> For each merge, the system builds binary packages and updates a package
>> >> repository.  This repo may be published and advertised to other
>> >> community members interested in the same configuration.
>> >>
>> >> In this way, anyone is able to set up a CI system for a particular
>> >> configuration.  If many people do so, the burden of running a CI system
>> >> for many target configurations will be distributed across many
>> >> community
>> >> members.
>> >>
>> >> I'm so enamored with this idea partly because of how it fits with C4:
>> >>
>> >> - Distributing the CI system across the community scales up the number
>> >> of configurations built and tested without overburdening any one
>> >> community member.
>> >> - No one person dictates what configurations are officially supported
>> >> by
>> >> the project:  anyone can contribute build results for any
>> >> configuration,
>> >> and while Contributors and Maintainers will see when a PR breaks the
>> >> configuration, ultimately it's up to that configuration's champion to
>> >> work with the community to ensure it continues to build.
>> >> - Third-party stabilization forks become trivial to set up and publish
>> >> packages for; simply set up a new CI system and point it at the fork's
>> >> repo on GitHub.  This especially suits vendors wanting to ship
>> >> Machinekit on their machine controller hardware.
>> >>
>> >> As of now, I've implemented many parts that would go into this system,
>> >> but many other parts are missing, and it's not a top priority for me.
>> >> That puts this idea in the "gedankenexperiment" category, but I'm still
>> >> curious how ZeroMQ community members will react, assuming you've made
>> >> it
>> >> this far into the mail!
>> >>
>> >> An update since I started writing this:  My Buildbot's ARM builder
>> >> broke, and I've decided not to resuscitate.  Some others in the project
>> >> have decided to set up a new CI system based around OpenSUSE's public
>> >> OBS instance.  My earlier experiments with building Debian packages on
>> >> OBS were ugly, but it turns out it can be made to work.  Building on
>> >> OBS
>> >> satisfies some of my proposed requirements for a C4 project CI system,
>> >> especially that it is trivially reproducible.  However, in practice, it
>> >> will take a lot of work to nail down the entire build flow, and the
>> >> question of running unit tests in various environments is unresolved.
>> >>
>> >>         John
>> >> _______________________________________________
>> >> zeromq-dev mailing list
>> >> zeromq-dev at lists.zeromq.org
>> >> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > zeromq-dev mailing list
>> > zeromq-dev at lists.zeromq.org
>> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>> >
>> _______________________________________________
>> zeromq-dev mailing list
>> zeromq-dev at lists.zeromq.org
>> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
>
>
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>