Developer-Friendly Runbooks: A Guide

Fri, 24 Sep 2021 19:28:00 -0100

When things go wrong - and yes, they will go wrong - it’s extremely helpful to have easy access to a set of runbooks to guide the unfortunate engineer through the steps needed to mitigate the problem as swiftly as possible. In this post I’m going to describe the approach we use for this where I work, which we’ve found to work very well.

Team Nimbus and the Agents of Chaos

Sun, 29 Mar 2020 18:00:00 -0100

This blog post is about my team’s first ever Chaos Day - where we ran a series of experiments designed to test how our platform performed when we tried to disrupt it, or the workloads that run on it.

This entry was originally posted on Medium under my employer’s publication.

It was January 2020, and we had just gone through another Peak trading period - significant for a larger retailer. The Digital Platform had performed extremely well. There were no incidents, no last-minute panic scaling, and no fall-backs enabled — even though the number of services and overall complexity of the platform was significantly higher than this time last year. Not perhaps the backdrop to make a compelling case for running a series of complex operational test scenarios then? Well, we are not the sort to be resting on our laurels …

Operability on alexos.dev

Developer-Friendly Runbooks: A Guide

Team Nimbus and the Agents of Chaos