Here at Percona we’ve spent a lot of time improving our development and testing practices. Why? Because constant innovation keeps us ahead and more productive. We want to work smarter, not harder. One of the tools we use is the Jenkins Continuous Integration server.
We use Jenkins pretty heavily to help with out development processes and workflow. The flexibility, modularity and plugin availability are great advantages to Jenkins, and it being free and open source software that we can freely modify and bug fix is immensely valuable (indeed, I now maintain the Jenkins Bazaar plugin).
We currently have:
- approximately 32 slaves performing builds
- 151 top level jobs (many of these are matrix jobs)
- 2,578 jobs (including all jobs in our matrix jobs)
- 29,124 builds (we auto-expire most builds, so we’ve done many, many more than this)
totalling 109GB of disk usage and a Jenkins master Java process that uses 4GB of resident memory (10GB virt for those keeping score at home).
Our jobs range from automated merging of approved code (and both regression and performance testing of the code before it gets merged to trunk), nightly performance QA runs, building binaries for release and parameterized builds to help developers test their code.
Our slaves are deployed using only about 800 lines of puppet manifests (along with a few templates, scripts etc) and we’ve found that using virt-manager/virsh and KVM to be the most reliable way of running slaves as virtual machines on our physical hardware.
I’ll be giving a talk at the Developer Automation and Continuous Integration miniconf at linux.conf.au 2013 in Canberra about how we got to this point, hopefully helping others improve their development process and automation.
