Automation · By · 8 min read

Ad-hoc scripts still need a paper trail

SSH in, run the fix, close the terminal. It worked, and six weeks later nobody can say what happened. What a script library and a recorded run change about that.

The most common piece of automation in any infrastructure team is not in a repository. It is a command somebody ran once, from their own terminal, against a machine that was misbehaving. It worked. The terminal got closed. That is the entire lifecycle.

I am not going to pretend this is avoidable. Sometimes a box needs a thing done to it right now and the correct response is to do the thing. The problem is not that people run ad-hoc commands. It is that the run leaves nothing behind, so six weeks later, when the same box misbehaves in the same way, the knowledge that would fix it in thirty seconds exists only in one person’s shell history and possibly not even there.

Give the script somewhere to live

The first change is that the script stops being something you remember and becomes something that exists.

A remote script library showing 2 total, 0 pending, 0 approve and 2 approved. Two rows: System info with Powershell scripting tagged POWERSHELL running under powershell, and Fetching system info tagged NODE JS running under cmd. Both show a command preview, a 1 minute timeout, a created date and an APPROVED status.
Two scripts, each with the language it is written in, the terminal it runs under, a preview of the command, a timeout and an approval state. The counters along the top track how many are waiting on approval rather than how many exist.

Each entry carries its terminal type, so a PowerShell script and a Node script sit in the same library without either pretending to be the other. When you create one you choose whether it runs as an inline command or gets written out as a script file first, and which shell it runs under: Command, PowerShell, Git Bash or Bash.

The two columns I would not have thought to ask for are Timeout and Last running Time. A timeout on an ad-hoc command is the difference between a script that hangs and a script that fails, and those need very different responses at 2am. Last running Time tells you whether the thing in front of you is a script the team actually uses or one that somebody wrote in March and everyone has quietly stopped trusting.

Then there is the approval state, and the counters above the table that track it. A script that can execute arbitrary commands across a customer’s fleet is a fairly serious object to leave lying around, and the library treats it that way: written, then approved, then runnable.

A run is attached to a reason

The second change is what happens when you actually execute one.

The second step of the run wizard. A Details section with required Customer, an optional Ticket Ref, a required Title and a Description. Below, a Target Devices section with an Apply Parallelly toggle switched on, an Apply all devices toggle switched off, and a required Select devices field.
The form before anything is typed into it, which is the point: what it asks for. Customer and title are required, Ticket Ref is not, and Apply Parallelly is on by default. Nothing here was filled in during the recording, so this is the empty state rather than a worked example.

Customer is mandatory, which in a multi-tenant setup is the difference between a scoped action and an accident. Title is mandatory too, so every run has a human-readable name attached rather than being identifiable only by its command.

Ticket Ref is optional, and I think it is the most interesting field on the form. It costs nothing to fill in and it is the thing that connects a command that ran on a machine to the reason anyone wanted it to. Six weeks later, that field is the difference between “somebody ran a PowerShell script on this host in August” and a ticket explaining what was wrong and what this was meant to do about it.

Apply Parallelly decides whether the targets are hit at once or in sequence. Parallel is the default and is usually what you want for something read-only. Sequential is what you want when the script restarts a service and the machines behind it are meant to stay collectively available, which is exactly the kind of decision that gets made badly when the tool does not offer it.

Executing then asks you to confirm. A dialog reading “Are you sure you want to execute the remote script?” is not sophisticated, but it sits between a moment of inattention and a fleet-wide command, and I have never regretted one being there.

Runs are rows, and they have states

A remote script run history with two rows, LHA35 and LHA34, both for customer LC5 in the LynxTrac QA Env, created three minutes apart, both showing SCRIPT COMPLETED with relative timestamps of just now and one minute ago.
Two runs of the same script, minutes apart, each its own row with its own code. Before this state they read READY with an Execute button, then EXECUTING while the work is happening.

A run moves from READY to EXECUTING to SCRIPT COMPLETED, and each one gets its own code. Running the same script twice produces two records rather than overwriting one, which sounds obvious and is the thing shell history fundamentally cannot do.

While a run is executing there is a live console, in the same shape as the release console covered in a release is not finished when it is applied. For a one-minute script that counts for less than it does on a release, but the useful case is the one where a script that normally takes ten seconds is still going after two minutes and you would like to know which machine is holding it up.

The receipt

Afterwards, the run has a summary, and it is what justifies the whole apparatus.

A script execution summary marked SCRIPT COMPLETED. It shows Total Devices 1, Scripts 1, and 100 percent, with Completed 1, Failed 0 and In Progress 0. Below, a single device row with its hostname, address and MAC, expanded to show one task named Fetching system info marked Completed.
Devices attempted, scripts run, and the completed, failed and in-progress split. Underneath, the specific machine and the specific task, each with its own state. A run against forty devices reads the same way, with the failures separated out.

Devices attempted. Scripts run. Completed, failed and in progress as separate counts rather than one overall status. Then the individual machine, identified properly, and the named task underneath it with its own state.

The reason to care about the split is partial failure, which is the normal outcome of running anything across a real fleet. A script that ran on thirty-eight of forty machines is not a success and it is not a failure, and a tool that reports it as either is lying to you. Two machines were offline, or busy, or in a state the script did not anticipate, and the useful output is which two.

The run also carries per-device stats, the agent logs, the task list and a history. Agent logs is where you end up when a script failed for a reason the script itself did not report, which is most of them.

Three tools, three different problems

Policies, releases and scripts overlap enough to be confusing, so it is worth saying plainly which is for what.

A software policy is for a condition you want to hold indefinitely. It re-evaluates on an interval and corrects drift without anyone asking, which we wrote up in software policies: desired state, not one-off installs.

A release is for a versioned artefact going to machines under an approval, with a verify stage and a rollback.

A remote script is for the thing that does not fit either. A diagnostic you need the output of. A one-time cleanup. A fix you are still working out the shape of and may well turn into a policy once you do.

That third category is not going away, and pretending otherwise is how teams end up with an “automation strategy” that everybody bypasses at 2am. The realistic goal is not to eliminate the ad-hoc command. It is to make running one through the tool cheaper than running it through SSH, so the audit trail happens as a side effect of the path of least resistance rather than as a discipline anybody has to maintain.

There is a scheduling side to this as well, where scripts run on a cron or rate schedule, or off a trigger event, rather than because someone pressed a button. Each schedule keeps its own run history. That deserves its own post rather than a paragraph here.

The RMM overview covers how scripting sits alongside monitoring and patching, and RMM automation recipes has the practical version of what teams actually automate first.

LynxTrac is free forever for up to 2 servers, no card required. If you want to try it on real infrastructure instead of reading about it: app.lynxtrac.com.

Related posts