[zeromq-dev] Testing framework

[john skaller]

On 23/01/2012, at 11:56 PM, Ian Barber wrote:

> On Mon, Jan 23, 2012 at 12:26 PM, Staffan Gimåker <staffan at spotify.com> wrote:
> Hi,
> 
> Are there any plans to migrate libzmq to use a more fully-featured testing framework (e.g. googletest)? The current automake test system is a bit of a pain to work with since it's missing stuff like a nice way to group related tests, a standardized way to do fixtures, sensible messages from failed assertions, etc.
> 
> Aside from an added (compile-time) dependency, are there any reasons why the automake way is preferable?
> 
> /S
> 
> It's proved tricky to unit test, though there have been some attempts - if you check the list archive you'll find a few discussions, Steve-o's is probably the best summary of issues: http://comments.gmane.org/gmane.network.zeromq.devel/11587

 I am struggling with a closely related problem:
how to check the Guide examples in Felix actually work. 

My belief here: for systems like 0MQ units tests are all but worthless.
The behaviour of the functions depends critically other functions,
so you can't really test any of them independently. Worse, you cannot
test communications and asynchronous behaviour with an isolated
subroutine.

Rather, testing must consist of scalable multi-process examples,
because that's what 0MQ tries to deliver. You need to run a simple
subscription service with N = 3 subscribers to check it works at all
and then you need to HAMMER it by pushing N up until it breaks.

Let me say that again: the primary goal is NOT testing that the
networks work. The thesis is that they always fail. The goal is to
see how and when they fail. If you don't hammer 0MQ you aren't
testing anything at all.

Conversely to this, and as a byproduct, you must monitor performance,
so you end up with a table that shows reliable behaviour at low
values of N associated with a certain performance metric which
can be seen to degrade until either reliability is lost or the network
collapses.

My concept at the moment is to design a language which can
launch and manage processes. Something a bit like the usual
shell script (since that what shells do). Something that can at least
implement the nice pictures in the Guide (i.e. linearise the diagrams).
Something that can timeout servers, report if processes are running,
kill a suite of processes.

Down the track, extend this to support remote machines.
[Obviously by transmitting the specs to a management server
on those machines using 0MQ .. :]

Now I'm biased but the easiest way to design a good language is to
extend an existing one, and there is no language better at doing
this than Felix, since it has a user defined dynamically loaded grammar
at the front end, is a fully general statically typed programming language,
and generates high performance machine binaries at the other end.

You would also use, for example, Python, to similar effect, although
you cannot have a nice language front end (you'd just have to use
dictionaries etc for the specifications).

If anyone is interested in helping with the design send me an email.
There's some work in the implementation but the design is much 
more important.

--
john skaller
skaller at users.sourceforge.net