Saturday 9th May 2015 12.28am

Sat, May 9, 2015

As part of publicising my C++ Now 2015 talk next week, here is part 4 of 20 from its accompanying Handbook of Examples of Best Practice for C++ ¹¹⁄₁₄ (Boost) libraries:

4. QUALITY: Strongly consider using free CI per-commit testing, even if you have a private CI

Despite that Travis provides free of cost per-commit continuous integration testing for Linux and OS X, and that Appveyor provides the same for Microsoft Windows, there were still libraries in those reviewed which made use of neither and furthermore had no per-commit CI testing whatsoever.

I'll be blunt: not having per-commit CI testing is unacceptable in this modern day and age and is an excellent sign of a library author not committed to software quality. Especially when such CI services are free of cost, and it's purely your laziness that you haven't configured them yet.

So first things first, if your C++ ¹¹⁄₁₄ library is not using any form of per-commit testing yet, go add Travis and Appveyor right support now. Configuration is extremely easy if your project lives on Github, simply login into both services using your github account and enable your project. Next add a suitable .travis.yml and an .appveyor.yml file to the root of your project, and push the commit. Watch Travis and Appveyor build and run your CI suitable unit test suite, and report back on Github (especially if it's a pull request) whether the build and tests passed or failed. From now on when someone issues a pull request fixing a bug, you'll instantly know if that pull request compiles and passes all unit tests on Linux, OS X and Windows, and much more importantly, so will the pull request submitter and they usually will go fix problems themselves so you the maintainer never need find out a pull request is defective on some build target.

Example travis.yml's to clone from:

* https://github.com/krzysztof-jusiak/di/blob/cpp14/.travis.yml
* https://github.com/BoostGSoC13/boost.afio/blob/master/.travis.yml

Example appveyor.yml's to clone from:

* https://github.com/krzysztof-jusiak/di/blob/cpp14/.appveyor.yml
* https://github.com/BoostGSoC13/boost.afio/blob/master/appveyor.yml

Both Travis and Appveyor are excellent for getting an immediate 90% confidence signal that some commit did not break something. For a free service with little configuration effort that's fantastic. However if you want a 99% confidence signal you will need to spend a few months of your life configuring your own Jenkins CI installation probably best placed on its own dedicated server given the RAM you'll need (I suggest a cheap OVH dedicated server with at least 16Gb of RAM for about €15/month or US$20/month), most of that time will be learning how not to configure Jenkins as Jenkins is a black, black art indeed - but again great for being free of cost given the return on investment. Once mastered, Jenkins can do almost anything from per-commit testing to soak testing to input fuzz testing to automating a long list of tasks for you (e.g. diffing and synchronising two forks of a repo for you by bisecting commit histories against unit testing), but it will take many dedicated months to acquire the skills to configure a maintainable and stable Jenkins install.

Should you add Travis and Appveyor CI support if you already are using your own private Jenkins CI?

I think the answer is uncategorically yes. The reasons are these:

1. Having Travis + Appveyor badges (see https://raw.githubusercontent.com/krzysztof-jusiak/di/cpp14/README.md for example Markdown for badges) on your open source project is a universally recognised signal of attention to quality.

2. Other free tooling such as Coveralls.io have built in integration for github and travis. Hooking Jenkins into Coveralls isn't hard, but it "just works" with Travis instead and that's a similar pattern with most free tooling which consumes CI results.

3. Future tooling by Boost which dashboards Boost libraries and/or ranks libraries by a quality score will almost certainly automate on Travis and Appveyor being queryable by their RESTful APIs. In other words, placing your library in Github and adding Travis and Appveyor CI support has the highest chance of working immediately with any future Boost tooling with minimal additional effort by you.