Monday 11th May 2015 6.05pm

Mon, May 11, 2015

As part of publicising my C++ Now 2015 talk next week, here is part 6 of 20 from its accompanying Handbook of Examples of Best Practice for C++ ¹¹⁄₁₄ (Boost) libraries:

6. QUALITY/SAFETY: Strongly consider running a per-commit pass of your unit tests under both valgrind and the runtime sanitisers

In Travis it is highly worth adding a special build job which runs your unit tests under:

valgrind memcheck (Linux only)
This detects illegal reads and writes, use of uninit values, use of unaddressable memory, illegal/double frees, and memory leaks. This tool is highly recommended, with its only downsides being a severe performance penalty (one can detect if running in valgrind inside your tests and treble timeouts. Look into the RUNNING_ON_VALGRIND macro in valgrind.h which by the way compiles just fine on MSVC too. You can also markup your code with valgrind instrumentation (also compatible with MSVC) and simply leave the instrumentation permanently in your binaries) and the fact it can't test Windows code.

Some will argue that their library is a pure constexpr metaprogramming library and does no memory allocation, and therefore running valgrind makes no sense for their library. Ah, but remember that valgrind isn't just testing your code, it is testing the code produced by the compiler. If you are doing cutting edge C++ 14 programming you may trigger code generation bugs in compilers past or future, or bugs in the STL caused by how your code uses it. A valgrind pass on your unit tests will catch bad code generation bugs, and potentially one day save you hours maybe days of frustrating debugging of weird segfaults!

Running your unit tests under valgrind is easy, simply prepend valgrind when calling your (preferably optimised though with debug info) test executable. You may find special compilation options will greatly improve the usefulness of error output, try -fno-omit-frame-pointer -fno-optimize-sibling-calls -fno-inline, though note disabling inlining may hide your bug.

Undefined behaviour sanitiser (GCC and clang only)
Turned on using -fsanitize=undefined, this detects when your code does undefined behaviour, and is sufficiently lightweight you should consider shipping release binaries with this permanently turned on along with stack smashing detection if using GCC 4.9 or later (-fstack-protector-strong). I personally have the ubsan always on for all builds of any code of mine capable of accepting untrusted input. At the time of writing, turning on the ubsan will prevent these things happening: use of misaligned pointer or reference, load of bool not 0 nor 1, out of bounds array indexing, bad casting, bad derived cast, bad cast of void* to type, bad or wrong vptr use, use of impossible enum value, divide by zero, bad function pointer call, use of null ptr, use of bytes not in object, exiting a value returning function without a return value, returning null from a function not allowed to return null, illegal shifts, signed integer overflow, reaching unreachable code, negative variable length array use.

As you can see, these tests make buffer overflow ROP chain exploits very hard, and therefore your code much, much harder to exploit from a security perspective. I think any library author whose library can accept untrusted input who doesn't always turn ubsan on is being irresponsible.

Thread sanitiser (GCC and clang only)
If your library is capable of threaded use or your unit testing creates threads, you definitely should soak execute your unit tests with the thread sanitiser (-fsanitize=thread) for a few hours per week which provides a good quality check of the correct use of the C11/C++11 atomic memory model e.g. are all your atomic acquires matched with atomic releases in the right order? Did you read a memory location which was written concurrently without an acquire-release serialisation lock? Sadly the tool can't detect use of memory fences which substantially reduces your flexibility when writing with atomics, so do bear that in mind.

Some may note I didn't recommend the address sanitiser (GCC and clang only). This is because you need to recompile your STL and libc with the address sanitiser to achieve perfect coverage, plus valgrind detects far more problems and valgrind detects bad code generated by the compiler and memory corruption by third party libraries. However if valgrind is just far too slow for your testing then employing the address sanitiser can be a useful substitute for valgrind for certain tests only. Note that the address sanitiser is perfect for untrusted input fuzz testing as it is much faster than valgrind, so I recommend the address sanitiser in the next section.

http://cppnow2015.sched.org/event/37beb4ec955c082f70729e4f6d1a1a05#.VUuMqvkUUuU