1500 lines of new tests written, 20 bugs found, codebase sans tests is 1700 lines (2200 with the aliases.) The first 800 lines of tests uncovered 6 bugs, the next 700 found 14.[1] Apparently practice does make you better at writing tests.
I'm still around a thousand test lines short of done, so at the current rate of one bug per 50 lines of tests I might expect to find 20 more bugs. Which'd put prelude.ml's pre-testing bugs/kSLOC at 24 bugs per thousand lines of source code (bug every 42 lines.) Prelude.ml contains around 700 top-level function definitions, giving a ratio of one bug every 17.5 functions.
Furthermore, I might expect the current set of tests to contain 36 bugs (1500 * 24/1000) and the finished tests to contain 60 bugs. A bug in a test would be omitting some relevant test case or expecting wrong behaviour.
A few questions arise: How to find more defects with less effort? How to assert the correctness of the tests? How to write code with fewer defects?
[1] My counting methodology was: Misbehaviour is a bug. Changing code and tests to change expected behaviour counts as a bug. Solving multiple bugs with a single search&replace counts as one bug. Changing performance characteristics of a function (e.g. making a function tail recursive) counts as a bug.
art with code
2009-01-04
Subscribe to:
Post Comments (Atom)
Blog Archive
-
▼
2009
(84)
-
▼
January
(15)
- Two things to make Ubuntu 8.10 less nerve-wrecking
- Performance debugging with R
- Programming books that I've liked
- Revised quickcheck.ml syntax
- Low-boilerplate testing in OCaml
- QuickCheck in make_suite.rb
- PreString tests done
- PreArray testing stats
- Prelude.ml: first pass of array tests complete
- Multi-threaded qsort in C
- Test generation / measuring code
- And even more testing
- Current testing stats
- Random thought on automated testing
- Prelude.ml testing
-
▼
January
(15)
2 comments:
Mutation testing can be useful.
Jester for Java: http://jester.sourceforge.net/
Heckle for Ruby: http://ruby.sadi.st/Heckle.html
Thank you for the links, mutation testing sounds like a good idea. I couldn't find an OCaml version but it should be doable to write one with CamlP4.
A fun (if cpu-burning) idea might be to use a code mutator to drive a test generator to search for a minimal specification of a given function. Come think of it, the mutator is a test generator targeting the tests.
Post a Comment