Technical update from my colleague Alister for how WordPress.com uses automated tests for build confidence, now running for on GitHub pull requests instead of after deployment to production. The tests and webhook “bridge” infrastructure are open source just like the Calypso source code itself.
At WordPress.com we strive to provide a consistent and reliable user experience as we merge and release hundreds of code changes each week.
We run automated unit and component tests for our Calypso user interface on every commit against every pull request (PR).
We also have 32 automated end-to-end (e2e) test scenarios that, until recently, we would only automatically run across our platform after merging and deploying to production. While these e2e scenarios have found regressions fairly quickly after deploying (the 32 scenarios execute in parallel in just 10 minutes), they don’t prevent us from merging and releasing regressions to our customer experience.
Introducing our Canaries
Earlier this year we decided to identify three of our 32 automated end-to-end test scenarios that would act as our “canaries”: a minimal subset of automated tests to quickly tell us if our most important flows are broken. These tests execute after a pull…
View original post 477 more words
The Sensible Leader 