[zeromq-dev] [PROPOSAL] git repo for test cases

[Pieter Hintjens]

Hi all,

This is a request for comments on a better organization for test cases.

This is specifically for reported issues, not regression testing.
Currently, we have no consistent location for test cases, which means
that:

* Anyone wanting to test a fix for an issue has to search for test cases
* Test cases are often actually absent, leading to speculative fixes
(e.g. downstreamed patches that aren't properly testable)

We need, to properly maintain 0MQ, test cases in C (so that we can
disregard language binding issues), and ported to both 3.0 and 2.1.

We also need space for listings, crash dumps, notes, scripts, etc.
that go along with a test case.

Current practice is to put test cases on a gist or pastebin and then
link to this from the issue tracker (if we're lucky) or else just
provide that link via IRC or email.

One suggestion is to systematically use a gist for every issue, since
a gist is a full repo and can hold all the files we need. This almost
works but gists don't allow multiple authors, nor can they be usefully
named.

So my proposal is to create a test repo, where every issue has a
directory, containing all needed files.

I've made an example at https://github.com/zeromq/issues

The simplest process is that when people want to submit a test case,
they fork the issue repo, add their changes, and submit a pull
request.

-Pieter