[zeromq-dev] How should test cases be written?

[André Caron]

Hi Goswing,

The test suite in libzmq uses "assert" in to check results, which tends to
crash the entire test suite that's running in the same program.  In the
worst case, you get a failure on the first test and you have no idea if the
rest of the tests pass.  In my experience, getting a partial success report
is really helpful in order to rule out some areas of the code as "less
suspect", so I would tend to recommend to put as few test cases in the same
program as possible (until the overhead of starting separate executables
starts to become impractical).

Cheers,

André


On Mon, Jan 20, 2014 at 8:12 AM, Pieter Hintjens <ph at imatix.com> wrote:

> We don't really have formal rules for test cases. It could be fun to
> document what we do in practice.
>
> Technically a test must fail or pass, it may not hang.
>
> Here are the guidelines I use for my test code:
>
> - should be written to be reusable by developers looking for examples
> - should thus be in C, more than C++
> - should test every single API functionality that is documented
> - should be tuned to not do long waits or pauses
> - should use 127.0.0.1 as interface for all tests
>
> -Pieter
>
> On Sat, Jan 18, 2014 at 7:19 AM, Goswin von Brederlow <goswin-v-b at web.de>
> wrote:
> > Hi,
> >
> > having written a patch for connection notification on STREAM sockets I
> > wonder about the test case for stream sockets.
> >
> > Currently there are 3 tests with STREAM sockets (tested in that order):
> >
> > 1) test_stream: (uses recv/send)
> >    + test connecting a STREAM and DEALER socket
> >    + test connecting a STREAM and STREAM socket
> >
> >    Tests connection notification, sending messages, closing sockets
> >    and disconnect notification all in one.
> >
> > 2) test_stream_disconnect_notifications:  (uses recv/send_msg)
> >    + test connecting a STREAM and STREAM socket with polling
> >
> > 3) test_stream_empty: (uses recv/send_msg)
> >    connecting a STREAM and DEALER socket
> >    + testing zmq_msg_close on an empty message?
> >
> > As it is test_stream will fail if anything is wrong with STREAM
> > sockets regardless of what it is.
> >
> > I wonder: Is the line number of the failure enough of a clue what is
> > wrong? Or should there be more tests that test things step by step?
> >
> > Esspecially test_stream contains 2 completly seperate tests:
> > STREAM/DEALER and STREAM/STREAM. Shouldn't that at least be split?
> >
> >
> > Secondly I wonder if the tests should care about blocking. They don't
> > set a timeout on the sockets, don't use non-blocking calls and most
> > don't set LINGER to 0 before closing sockets and destroying the
> > context. So if anything goes wrong, like the connect notification not
> > being send, the test simply hangs. It doesn't fail. Shouldn't they?
> >
> >
> > So what is the rule for this in the zmq project?
> >
> > MfG
> >         Goswin
> > _______________________________________________
> > zeromq-dev mailing list
> > zeromq-dev at lists.zeromq.org
> > http://lists.zeromq.org/mailman/listinfo/zeromq-dev
> _______________________________________________
> zeromq-dev mailing list
> zeromq-dev at lists.zeromq.org
> http://lists.zeromq.org/mailman/listinfo/zeromq-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.zeromq.org/pipermail/zeromq-dev/attachments/20140121/b739d87e/attachment.htm>