Deployment · By · 8 min read

A release is not finished when it is applied

Most deploy tooling ends at apply. A release worth trusting has a verify stage after it, a rollback that is a button, and an approval that is not one person's judgement.

The worst deploys I have been part of were not the ones that failed. A failed deploy is loud, and loud problems get fixed. The bad ones are the deploys that reported success, on every host, and left something broken that nobody noticed for two days.

That happens because most deployment tooling treats apply as the last step. Files are copied, a service restarts, the exit codes come back clean, and the pipeline goes green. Whether the thing that is now running is the thing you intended to ship is a separate question, and one that a lot of tools simply never ask.

Five stages, and apply is the fourth

The release summary lays out where a release actually is, and the shape of it is the argument.

A release summary card marked APPROVED for release LHA36, tagged github-9af749a666, for CI release 0.8.2-a.2 with 2 targets. A five-stage progress line runs Created at 04:49pm, Approve at 04:50pm, then Prepare, Apply and Verify still pending. Below are empty Prepare Time, Apply Time and TAT fields, and a Team control reading 2 of 4.
Created and Approve are done, the remaining three are not. Verify sits after Apply as its own stage rather than as something the pipeline assumes. The Team control on the right reads 2 of 4, and the three timing fields underneath fill in as the release moves.

Created, Approve, Prepare, Apply, Verify. The last one is the one I care about, because putting verification in the lifecycle rather than in a runbook changes what “done” means. A release that applied cleanly and has not been verified is not finished, and the interface says so by leaving a stage unfilled.

Prepare being separate from Apply is the other quietly useful bit. Staging the artefact onto the target and activating it are different operations with different failure modes, and collapsing them into one step is how you end up with a half-downloaded bundle applied to production.

The three fields underneath are prepare time, apply time and turnaround. Those are not vanity metrics. When someone asks why a change took three days, the useful answer is almost never about the apply, which took forty seconds. It is about the gap between created and approved, and that gap is only visible if something recorded both.

Approval is counted rather than collected

The control on the right of that card reads Team (2/4).

Approvals are tracked as a count against a group rather than as a single sign-off. I am not going to over-read the specific numbers, since this release already shows as approved at two of four, so the threshold is clearly not “all of them”. What the control establishes is the shape: approval on a release is a tally with a denominator, and the platform knows the difference between one person clicking yes and a group reaching whatever bar was set.

That distinction only ever matters in one situation, and it is the situation every approval process was built for: the person deploying is the person who wrote the change, and it is 18:00 on a Friday.

The bundle in this release is tagged with its commit, github-9af749a666, and labelled with its version, 0.8.2-a.2. Whatever gets approved is pinned to a specific artefact, so approving a release and shipping something else is not a thing that can quietly happen.

Mid-flight, there is a stop button

While a release is applying, it is not a black box you wait out.

A release row for LHA36, commit github-9af749a666 version 0.8.2-a.2, for customer LC5 in the LynxTrac QA Env, marked INSTALLING just now with a seven-step progress line showing five steps complete. A red ABORT button sits on the right.
The same release mid-apply. Five of seven steps done, and an ABORT sitting where you can reach it. Worth noticing that stopping is offered during the operation rather than only being possible afterwards by deploying something else on top.

The row shows progress step by step, and an ABORT next to it. That button exists for the ten minutes when you have just realised something is wrong and the alternative is watching it finish and then cleaning up after it.

Plenty of pipelines make you either wait or kill a CI job and hope the agents notice. Being able to stop a rollout from the row that is showing you the rollout is a small thing that is only ever needed under pressure.

Afterwards, rollback is a button

Then the same row, once it lands.

The same LHA36 release row now marked INSTALLED just now, with the progress line complete. Two buttons sit on the right: a green VERIFY and a red DOWNGRADE.
The same release after apply. VERIFY runs the check that closes the lifecycle. DOWNGRADE is the rollback, sitting on the row itself rather than in a document somebody has to find.

VERIFY and DOWNGRADE, on the row, immediately.

Rollback being a button rather than a procedure changes who can do it and how fast. Most rollback plans are a paragraph in a change ticket that describes, in general terms, what someone would do. They are written before the deploy by a person who is calm, and executed after it by a person who is not. The gap between those two states is where rollbacks go wrong.

A downgrade that is a control next to the release, operating on a release the system already has recorded with its version and commit, does not depend on anyone remembering what the previous version was.

I would still write the rollback plan. But there is a difference between a plan you follow and a plan you have to reconstruct.

Watching it land, per device

The tracking view carries a live console, and this is where the two-target release stops being an abstraction.

A live release console for LHA36. Timestamped lines interleave two hosts, app-node-04 running under one process id and a QA machine under another, each reporting Task 1 of CI triggered release 0.8.2-a.2. The final line reads Apply triggered by auto deployment service.
Two hosts applying the same release, interleaved by timestamp, each line carrying its process id. The last line names what started the whole thing: the auto deployment service, not a person.

Every line is stamped with a timestamp, a process id and its originating host, and the two targets interleave as they progress. When one host is slower than the other, you can see which one and how far behind.

The last line is my favourite detail in the whole recording: Apply triggered by auto deployment service. The release was not applied by someone clicking a button. It was applied by the platform, on the approval that was already recorded, and the console says so. Attribution for automated actions is one of those things you never think about until an auditor asks who deployed something overnight.

The tracking view holds more than the console: per-device stats, agent logs, task assignment, history, and a file comparison view. That last one I have only seen the tab for, so treat this as an observation rather than a description. Comparing what actually landed on disk against what the release intended is the obvious question to want answered after an apply, and it is the one a green tick never answers.

This does not replace your pipeline

The release in these screenshots was triggered by CI and carries its commit hash, which is exactly the right division of labour. Build and test are a pipeline problem. Getting a built artefact onto machines that belong to a customer, with an approval that survives an audit and a way back if it goes wrong, is a different problem and it does not stop at apply.

The thing I would take from it, if you are running deploys through something that ends at a green tick, is not that you need different tooling. It is worth asking what your equivalent of the verify stage is, and whether the answer is a step somebody performs or a step that is missing and merely assumed. We wrote about the wider shape of this in continuous deployment for IT operations, and the continuous deployments page covers how releases, approvals and rollbacks fit together here.

Related posts