DevOps in New Zealand

Measure the thing you are trying to fix

DevOps proposals are unusually easy to sell on vocabulary and unusually hard to judge afterwards. Fix that by writing down where you are before anyone starts.

Four numbers, and they are worth collecting even roughly:

1. How often do you release? Daily, weekly, or whenever somebody is brave.
2. How long does a release take, from a developer finishing a change to it being live?
3. How often does a release cause a problem?
4. How long to recover when something breaks in production?

Ask every consultant what those numbers will be when they are finished. A proposal that cannot answer in those terms is selling tools. One that says "releases go from three hours with one person who knows how, to eight minutes any developer can run" is selling an outcome.

Start with the release

If you buy one thing, buy this. An automated pipeline that runs the tests, builds the application and deploys it, the same way every time, from a commit.

What it gets you:

• Nobody is remembering an eleven-step process at 7pm.
• Anyone on the team can release, so one person's leave is not a freeze.
• Releases become small and frequent, which is the single biggest reduction in risk available — small changes break less and are easier to undo.
• There is a record of what went out and when.

The knowledge concentration is the real business risk here. In a lot of New Zealand teams exactly one person can deploy, and everybody has quietly agreed not to think about it.

Demand a demonstrated rollback

Most teams can deploy. Far fewer have ever tested going back, and the discovery usually happens during an incident.

Ask each consultant to show you a rollback rather than describe one, and ask three follow-ups:

• How long does it take?
• Can anyone on the team do it, or only you?
• What happens to database changes? A schema migration that only runs forwards is the thing that makes rollbacks a fiction.

The database answer is the important one. It is why experienced teams write migrations in stages — add, backfill, switch, remove — so that yesterday's code still works against today's schema.

Monitoring: know before the customer rings

This is cheap, quick and consistently missing.

What a minimum looks like:

• An alert when the application is down, to a person, on a channel they will see at 2am.
• An alert when errors rise above normal, not only when everything stops.
• Logs collected somewhere searchable, rather than on a server somebody has to log into.
• A dashboard someone actually looks at.
• Certificate and domain expiry warnings, which cause a surprising share of outages.

Then decide who is on the other end. Alerts with no named recipient are decoration, and alerts that fire constantly get muted within a fortnight — so tune them until they are believable.

Environments should match, and be rebuildable

"It works on staging" is a sentence that costs money. It usually means staging has a different configuration, different data, or a setting somebody changed by hand in 2023 and forgot.

Infrastructure as code — Terraform, Bicep, CDK — puts the environment in files you own. That gives you environments that genuinely match, changes that can be reviewed, and the ability to rebuild after an incident rather than reconstruct from memory.

Ask for the repository as a deliverable. It is also what makes changing consultant possible.

Find out where the secrets are

Ask, as part of any review: where are the production passwords, API keys and certificates right now?

Common answers include a configuration file in the repository, a shared spreadsheet, an email from 2022, and one developer's laptop. All of them are worth fixing, and fixing them is usually quick.

Where they should be: a secrets manager, granted per environment, with a record of who accessed what and a way to rotate one without a deployment.

Buy the smallest useful thing

DevOps engagements can expand indefinitely. Containers lead to orchestration leads to service meshes, and a five-person team ends up operating infrastructure built for fifty.

For most New Zealand businesses the sensible stopping point is: automated pipeline, infrastructure as code, monitoring with real alerts, tested rollback, secrets managed properly. That is most of the benefit for a fraction of the complexity.

Be wary of a proposal that starts with Kubernetes. Ask what problem it solves that a simpler deployment would not — sometimes there is a good answer, and often the honest one is that it is what the consultant enjoys.

Comparing five quotes

Read for:

• Who asked how long a release takes today?
• Who committed to numbers for afterwards?
• Who demonstrated a rollback, including database changes?
• Who included monitoring and alerting rather than only pipelines?
• Who described what your team can run without them, and what documentation you get?

Describe how you release now, what broke most recently, who is the only person who can do something, and what you run on. Consultants who do this work will ask about your last bad release before they ask about your stack.