Eugene Petrenko

Install, Launch, Update: Shipping a CLI on Five Platforms — the devrig Approach

2026-08-06T00:00:00+00:00

TL;DR — This post is about an approach to making CLI tools installable and upgradable: one pasted line installs the tool on five platforms, and after that it keeps itself current. The recipe: generate the install script at build time with every URL and SHA-256 baked in, install versions content-addressed and side-by-side, let the binary own the launcher on your PATH, and define “update” as “run today’s install script” — so update logic never ages with old installs. Built for devrig, a JVM CLI where the runtime is bigger than the program, but most of it applies to any CLI you distribute yourself.

Writing a command-line tool on the JVM is a pleasant afternoon. Kotlin, a main, a few subcommands, ./gradlew installDist, done.

Shipping it to a stranger is the hard part. That stranger is on Windows on ARM, or on a Mac that has never had a JDK, or inside a container that turns out to be Alpine. They paste one line from your website into a terminal, and whatever happens next is your product’s first impression. In three weeks you ship an update, and every one of those machines has to get there without you.

We hit this wall with devrig, the CLI that bridges AI Agents to IntelliJ-family IDEs in MCP Steroid. When I wrote about devrig reaching Level 3 autonomy, the honest “cons” list opened with this:

JDK 25 is a hard requirement. devrig doesn’t bundle a JVM in 0.100; the host must provide Java 25.

Open question #3 in that same post was “Bundled-JRE devrig: ship a ~/.mcp-steroid/binaries/<os>-<cpu>-<sha>/ cache with a small wrapper, so the JDK-25 prerequisite stops being a prerequisite.” This post is what that turned into: not a bundled JRE, but a small distribution system, and the interesting part is how little of it lives inside the product.

The first generation of devrig (jonnyzzz/old-devrig-private, still public) was written in Go — a single static binary, nothing to provision — and its bootstrap script was still 182 lines of POSIX sh. Everything below exists because we moved to Kotlin and inherited the JVM’s one real distribution problem: the thing that runs your program is bigger than your program. If you read one section, read the updater that cannot age — the idea the rest exists to support.

One note on vintage, because this post mixes two. Quoted scripts and on-disk paths are the released devrig — you can curl them yourself and check every hash against my directory names. Where main has since moved on, I say so and name the issue. The gap between those two is where most of the interesting mistakes live.

The problem, stated properly
The architecture
The JDK data model: resolved live, PGP-verified, nothing hand-pinned
The generator: a template, a table, and a refusal to guess
The generated script: what a good curl | sh looks like
- The PowerShell twin, and why it is not a translation
Content-addressed, side-by-side installs
The handoff: the script does not write the launcher
The launcher, and self-healing on every start
Update: the updater that knows nothing
- And then the upgrade is: run the install script again
- The updater that cannot age
Distribution, and the release-ordering trap
Signing: implemented once, designed twice, not shipped yet
Testing scripts and generators
Where this was built and tested
What is not solved yet
Is this an open-source project on its own?

The problem, stated properly

The whole job in one sentence; every clause hides a trap.

Deliver a reusable JVM command-line tool to five platforms, with a runtime the user does not have, installed by one pasteable line, updatable without breaking a running copy, and reproducible enough that you can tell a security team exactly what got executed.

Concretely: macos-arm64, linux-x64, linux-arm64, windows-x64, windows-arm64. Two script languages (POSIX sh and PowerShell), two archive formats, two JDK vendors, one binary. Every obvious answer is wrong in a way you only discover on someone else’s laptop.

“Just require a JDK on PATH”

Where devrig 0.100 was, and the most defensible wrong answer: it cost us more support time than every other issue combined. JAVA_HOME points at a JDK 17 the user forgot about. The shell that launches an AI Agent has a different PATH than the shell the user tested in. And “please install Java 25 first” is a hostile first step for a tool whose pitch is removing setup steps.

“Publish to Homebrew, apt, winget, Chocolatey, Scoop”

Only wrong as a first channel. Five package managers means five review processes, five release lags and five sets of packaging conventions — a part-time job for a project releasing every few weeks — and the versions drift, so you end up debugging a bug that exists only in one downstream package. Not worthless: a package manager would hand us uninstall, proxy configuration, enterprise policy and inventory, four things this post later admits we have not solved. A desirable second channel; not a viable first one.

“`curl | sh` and figure it out at install time”

The natural install script asks the network what to install: the GitHub API for the latest release, the JDK vendor for the current build, resolve URLs, download. It demos well. Then: GitHub’s unauthenticated API rate-limits you to 60 requests per hour per IP, so a whole office behind one NAT fails at once. A vendor changes a URL shape and every historical install line breaks retroactively. Two users running the one-liner an hour apart get different versions with no way to tell. And you cannot answer “what exactly did that script download?” — the answer was decided at runtime, on their machine, by a server you do not control.

“Install over the previous version”

Overwrite the install directory and you have a window where the tree is half old and half new. If a copy is running — and for an MCP server bridging an AI Agent to an IDE, a copy is always running — you have just swapped its files underneath it. Rollback means re-downloading the old version, which you did not keep.

“Put all the update logic inside the binary”

The tempting one: devrig self-update checks a version endpoint, downloads, swaps, restarts. The problem is subtle and expensive: your update logic ages with the oldest installed copy. A machine running 0.99 updates using 0.99’s updater. If that updater had a bug, or predates the archive format you switched to, or hardcodes a URL you retired, that machine is stranded — the only code that could fix it is the code that is broken. A bootstrap problem discovered exactly once, in public. That failure mode shaped everything below.

The architecture

Our answer inverts the usual layering: the install script is a generated, self-contained artifact, not a thin wrapper around a smart product, and the product’s only job on the install path is to register its own launcher. Four ideas carry the design:

Resolve at build time, not install time. Every URL and SHA-256 for every platform is computed in CI and baked into the script, which detects your platform and your tools but never asks a server what to install.
Content-address everything. Each artifact unpacks into <kind>-<os>-<cpu>-<version>-<sha12>/, so two versions cannot collide: installs are side-by-side and re-installs free.
The binary owns its launcher. The script never writes the thing on your PATH; it hands off to devrig install devrig, and the binary re-checks and repairs that launcher on every start.
Update is install. Upgrading is running the current install script, so upgrade logic is always the newest code, never the oldest. When we later automated updates, the in-binary updater’s whole job stayed “download today’s script and exec it”.

Here is the whole system (click to zoom)

graph TB
    subgraph BUILD["① Build time — CI, per release"]
        VEND["Vendor metadata<br/>corretto.aws latest alias · Azul Metadata API<br/>GitHub Releases API"]
        MODEL[":installer-gen<br/>resolveAllJdks() → JdkModel<br/>url · sha256 · javaHome · archive"]
        PGP["PgpVerifier — BouncyCastle<br/>detached vendor signature vs<br/>PINNED fingerprint · gates every resolve"]
        GEN[":installer-gen · InstallerGenerator<br/>install.sh.tmpl / install.ps1.tmpl<br/>+ generation-time validation"]
        WGEN[":website-gen · WebsiteArtifacts<br/>reads VERSION + the GitHub release"]
        VEND --> MODEL
        PGP -.->|"fail-fast, incl. cache hits"| MODEL
        MODEL --> GEN
        GEN --> SCRIPTS["install.sh + install.ps1<br/>every URL + SHA-256 baked in"]
        WGEN --> VJSON["version.json<br/>updatePlugins.xml"]
    end

    subgraph DIST["② Distribution"]
        SITE["devrig.dev — GitHub Pages<br/>deployed from the 'website' branch<br/>(lags main until the release exists)"]
        REL["GitHub Release<br/>devrig-&lt;version&gt;-&lt;hash&gt;.zip"]
        CDN["Vendor CDNs, never mirrored<br/>corretto.aws · cdn.azul.com"]
        SCRIPTS --> SITE
        VJSON --> SITE
    end

    subgraph INSTALL["③ Install — one pasted line"]
        ONE["curl -fsSL https://devrig.dev/install.sh #124; sh<br/>irm https://devrig.dev/install.ps1 #124; iex"]
        PRE["preflight<br/>os/arch detect · musl reject<br/>downloader · hasher · extractor"]
        DL["download → SHA-256 verify → unpack verbatim"]
        CA["~/.mcp-steroid/binaries/<br/>&lt;kind&gt;-&lt;os&gt;-&lt;cpu&gt;-&lt;version&gt;-&lt;sha12&gt;/<br/>atomic promote · side-by-side"]
        HAND["devrig install devrig<br/>--install-script --jdk-home"]
        SITE --> ONE --> PRE --> DL --> CA --> HAND
        REL --> DL
        CDN --> DL
    end

    subgraph RUN["④ Every run"]
        SHIM["~/.mcp-steroid/bin/devrig · devrig.cmd<br/>pins DEVRIG_JAVA_HOME → exec install tree"]
        HEAL["ensureBinLauncher()<br/>repair if stale · never overwrite a newer one<br/>PATH symlink / HKCU PATH"]
        CHK["DevrigUpdateChecker + AutoUpdater<br/>GET version.json → promoted &gt; current?"]
        HAND --> SHIM
        SHIM --> HEAL
        SHIM --> CHK
    end

    VJSON -.->|"version-base"| CHK
    CHK -.->|"3–8 h tick: fetch today's script, exec it"| ONE

    style BUILD fill:#e1f5ff
    style DIST fill:#e8f5e9
    style INSTALL fill:#fff4e6
    style RUN fill:#f3e5f5

The diagram is dense by design — click it to open full-screen at readable size (Esc closes), or zoom the page in.

Now the parts, one at a time.

The JDK data model: resolved live, PGP-verified, nothing hand-pinned

:installer-gen is a build-tooling Gradle module with no IntelliJ dependencies. Its resolveAllJdks(cache, http) produces a JdkModel — one JdkArtifact per platform, every field computed from the live vendor source:

Amazon Corretto 25 for linux-x64, linux-arm64, macos-arm64, windows-x64 (CorrettoJdk.kt): HEAD the vendor’s latest alias, record where it redirects, pull the version out of the resolved URL.
Azul Zulu 25 for windows-arm64 (AzulJdk.kt) — the one platform Corretto does not publish — via the Azul Metadata API with latest=true&release_status=ga.

The major version is pinned in source; the patch level is not. “Latest 25” is resolved on every run, because a pinned 25.0.4.7.1 in source is a lie that rots quietly and the generator’s output is fully pinned anyway.

javaHome is the field that breaks naive implementations. Every vendor lays out its archive differently — amazon-corretto-25.jdk/Contents/Home on macOS, jdk25.0.4_7 on Windows, zulu25.36.15-ca-jdk25.0.4-win_aarch64 for Azul — so it is not guessed from a convention: the model streams the archive entries and takes the shallowest bin/java[.exe], which is also proof against archives nesting a jre/bin/java inside. Archives are downloaded and cached, never extracted.

Then the part that matters most:

Both vendors publish detached OpenPGP signatures. The signing key fingerprint is pinned in source and asserted before the signature is trusted — and the key itself is fetched live over HTTPS, so the pin is what defeats a compromised key endpoint.

PgpVerifier (BouncyCastle bcpg-jdk18on) checks Corretto’s <file>.sig and Azul’s signature-binary against constants that live in the source:

const val CORRETTO_KEY_FINGERPRINT = "6dc3636dae534049c8b94623a122542ab04f24e3"
const val AZUL_KEY_FINGERPRINT     = "27bc0c8cb3d81623f59bdadcb1998361219bd9c9"

It also insists the signature is a PGPSignature.BINARY_DOCUMENT, and on a mismatch refuses with “refusing to trust the signature (possible compromised key endpoint)”. It is fail-fast and runs on every resolve, including cache hits — the cache directory is shared, and a cache is not a trust boundary. The cache is built the same way: downloadWithEtag fails the build if the host exposes no ETag, and downloadVerifyingSha256 re-hashes on every return.

The generator: a template, a table, and a refusal to guess

InstallerGenerator does something unglamorous: it renders two text templates.

installer-gen/src/main/resources/templates/install.sh.tmpl
installer-gen/src/main/resources/templates/install.ps1.tmpl

The templates hold all the logic; the generator injects only the data — @@VERSION@@, @@DEVRIG_URL@@, @@DEVRIG_SHA256@@, @@DEVRIG_BINSUB@@, and the per-platform JDK table (@@PLATFORM_CASE_SH@@ as POSIX case arms, @@PLATFORM_TABLE_PS@@ as a PowerShell hashtable). Only the three POSIX platforms go into the sh table, only the two Windows ones into the PowerShell table; @@DEVRIG_BINSUB@@ is filled with a different value per template (bin/devrig vs bin/devrig.bat). If any @@PLACEHOLDER@@ survives rendering, generation fails.

What matters is what it refuses to do:

All five platforms or nothing. validateScriptTable requires exactly ALL_PLATFORMS; missing and extra platforms both fail the build.
Shell-safety validation on vendor-controlled strings. URLs and javaHome values are baked verbatim into single-quoted shell and PowerShell literals in a script people run via curl | sh, where a quote, backtick, $, backslash or control character would break out of the string:
```
private val SHELL_UNSAFE_CHARS = charArrayOf('\'', '"', '`', '$', '\\')

private fun requireShellSafe(label: String, value: String) =
    require(value.none { it in SHELL_UNSAFE_CHARS || it.isISOControl() }) {
        "$label contains a shell-unsafe character: '$value'"
    }
```
It runs on every URL, every javaHome, the devrig URL, both launcher subpaths and the version string. Today’s vendor values are clean; the check exists so a vendor URL change or a crafted archive entry name cannot produce an injectable installer, and generation time is the only place a human still sees the failure.
The launcher path inside the zip is discovered, not assumed. The obvious guess is devrig-<version>/bin/devrig, but the real top-level directory carries the build hash — devrig-0.101-40690055 — so devrigLaunchers() scans the actual archive entries for */bin/devrig and */bin/devrig.bat and asserts they exist. The devrig zip itself is resolved from the v<version> GitHub release, deliberately not latest, and the archive’s top directory must start with devrig-<version>-.
ASCII-only output. Both rendered scripts are scanned and rejected if any character is ≥ 128. PowerShell 5.1 misreads BOM-less UTF-8, and legacy Windows consoles corrupt Unicode. A tidy → in a log message is not worth a broken install on Windows 10

And one seam that matters more than it looks:

fun writeInstallerScripts(outDir: Path, table: Map<String, JdkScriptEntry>,
                          devrig: DevrigEntry, version: String)

writeInstallerScripts is pure — no network, no vendor lookups — so tests can render a real installer from a synthetic model pointing at local fixtures. That is why end-to-end installer tests run in seconds instead of downloading 200 MB JDKs.

The generated script: what a good `curl | sh` looks like

The live one is https://devrig.dev/install.sh: 207 lines, starting with a note that it is generated. Every design choice in it exists because something broke.

The whole body is a function, invoked on the last line.

main() {
  ...
}

main "$@"

curl | sh streams into a shell that executes as it reads, so a connection dropped at 60 % executes 60 % of your installer — a half-configured machine with no error message. Wrapping everything in main() called at the very end makes a truncated transfer a no-op: the highest-value line in the file.

Platform detection, then a hard stop on musl. uname -s and uname -m detect and normalize the platform, and DEVRIG_OS / DEVRIG_CPU override both, which is how the tests drive every platform arm. MCP Steroid drives IntelliJ IDEs, which need glibc, so Alpine is not a platform we can serve: the script detects musl three ways — ldd --version output, /lib/ld-musl-x86_64.so.1, /lib/ld-musl-aarch64.so.1 — and fails before any download, with a message that says why.

Preflight, and a promise never to install packages.

have_any curl wget        || add_missing 'curl (or wget) - to download files'
have_any sha256sum shasum || add_missing 'sha256sum (or shasum) - to verify downloads'

If anything is missing, the script prints the complete list plus the apt-get / apk / dnf / brew lines — all four, unconditionally, because guessing a distro badly is worse than printing an extra line — and stops. It never installs system packages on your behalf: a tool that sudos during install has lost the argument about trust before it starts.

install.ps1 has no equivalent preflight and does not need one: Invoke-WebRequest, Get-FileHash and Expand-Archive are built in. It also has no main() wrapper — irm | iex parses the whole string before executing any of it, so truncation safety comes free. Two asymmetries that look like oversights until you ask why.

Content-addressed install, with an atomic promote.

ia_sha12=$(printf '%s' "$ia_sha" | cut -c1-12)
ia_target="$BINARIES_DIR/${ia_kind}-${key}-${VERSION}-${ia_sha12}"
if [ -d "$ia_target" ]; then
  log "already installed: ${ia_target##*/}"
  printf '%s\n' "$ia_target"; return 0
fi

Download to .tmp.$$.…, verify SHA-256, unpack into a staging directory, then mv the finished tree into its final content-addressed name, so a reader only ever sees a complete tree. promote_tree tolerates losing that move — two installers running concurrently is not theoretical, an agent swarm does it routinely — and since the target name comes from the archive’s hash, whoever won unpacked the same archive, so “someone beat me to it” is a success, not a conflict. There is no lock anywhere in this path; content addressing makes one unnecessary. The PowerShell peer needs one extra clause: unlike POSIX mv, Move-Item nests the source into an existing destination directory instead of failing, so it detects and cleans the nested result.

Everything goes to stderr, via one log() helper. devrig speaks MCP over stdio, and a single stray byte on stdout corrupts the protocol frame — the agent then reports an unexplained handshake failure. Main.kt takes the rule to its conclusion: stdout is redirected to stderr before arguments are parsed, and restored only for non-MCP commands. An invariant that has to hold in the shell, not just in the code.

The PowerShell twin, and why it is not a translation

install.ps1 is generated from the same model and mirrors the same pipeline, but its comments read like a field report. Two examples that were real, user-facing bugs:

$ProgressPreference (#274). PowerShell’s progress bar redraws on every byte Invoke-WebRequest reads and every entry Expand-Archive unpacks, turning a multi-hundred-MB download into a UI-bound crawl. The script silences it — and because irm | iex runs in the caller’s scope, it saves the previous value and restores it in a finally.
Architecture detection (#273). [RuntimeInformation]::OSArchitecture does not exist on the .NET Framework 4.6.x that ships with base Windows 10, and under Set-StrictMode -Version Latest a missing property is a hard abort: a user ran irm | iex and got “OSArchitecture cannot be found on this object”. The fix reads PROCESSOR_ARCHITEW6432 first (WoW64 sets PROCESSOR_ARCHITECTURE=x86 on 64-bit Windows but exposes the true arch there), then PROCESSOR_ARCHITECTURE, and only then falls back to the .NET property, inside a try.

It also pins [Net.ServicePointManager]::SecurityProtocol to TLS 1.2 explicitly, because PowerShell 5.1 does not necessarily negotiate it. Both scripts close the child’s stdin explicitly — < /dev/null in sh, $null | in PowerShell — because a subprocess that inherits the bootstrapping shell’s stdin and decides to prompt will hang forever.

Content-addressed, side-by-side installs

The on-disk layout does most of the work:

~/.mcp-steroid/
├── bin/
│   └── devrig                                    # the stable launcher on your PATH
└── binaries/
    ├── devrig-macos-arm64-0.101-a7c8cf0d16e2/    # <kind>-<os>-<cpu>-<version>-<sha12>
    │   └── devrig-0.101-40690055/bin/devrig      #   <- the zip's own top dir: version + git hash
    └── jdk-macos-arm64-0.101-614107ed76e9/
        └── amazon-corretto-25.jdk/Contents/Home

That a7c8cf0d16e2 is the first 12 hex characters of the SHA-256 of the downloaded archive — the same value that appears as DEVRIG_SHA256 in the install script. You can verify the claim from your own shell, which is the point.

The 0.101 in the JDK directory name is a bug, and we shipped it. Both artifacts got stamped with the devrig release version, so a Corretto 25.0.4 tree ended up in a directory labelled 0.101. It survived review because there is a story you can tell about it — “the directory records which release chose this JDK” — and that story is wrong: two consecutive releases pinning the same Corretto build download and store it twice, and the name lies about its contents. #362 fixed it; each artifact now carries its own vendor-native version, and the integration test pins a JDK version deliberately different from the devrig version so the two can never silently re-merge. A plausible rationalization is the most expensive kind of comment.

What the naming buys: the target directory existing is the “already installed” check, so re-running the one-liner on a current machine touches the network only for the script itself; 0.101 and 0.102 have different names, so a new install cannot touch an old one and a running process keeps its files, which makes rollback a path edit rather than a download; and the unpack is verbatim — no --strip-components, no rearranging, which is why javaHome has to ride along in the baked table. The weak spot is that the check is directory existence, not content: a tree a user edited, or one a half-finished rm -rf left behind, is trusted forever. A sentinel file recording the full hash would fix that cheaply and does not exist yet.

The honest cost: nothing garbage-collects old versions. Every upgrade leaves a full devrig tree and a full JDK behind — the 0.101 devrig zip alone is 237 MB, most of it the bundled IDE plugin. And the naming scheme is not parseable by splitting on -, because the version segment can itself contain a hyphen (0.0.0-test, in the tests), so the GC has a parsing problem before it even reaches a policy problem. The v7 deployment spec designs the sweep, with a keep-set of current plus most-recent-previous and per-directory locks; none of it is implemented. Retention is only safe once you know nothing points at a tree, and the whole point of content addressing was to not track that.

The handoff: the script does not write the launcher

The install script does not create ~/.mcp-steroid/bin/devrig, does not touch PATH, does not edit shell profiles. Its last real action is:

DEVRIG_JAVA_HOME="$jdk_home" "$launcher" install devrig \
    --install-script="$launcher" --jdk-home="$jdk_home" < /dev/null \
  || fail "'devrig install devrig' failed - the launcher could not register itself"

It runs the freshly-unpacked binary and asks it to register itself. Note the variable: the handoff runs under DEVRIG_JAVA_HOME, never by setting the user’s JAVA_HOME — process-scoped, so the user’s environment is untouched. (The 0.101 script still uses JAVA_HOME; the contract below fixed it, and there is a test pinning it.) The two flags are belt-and-braces: devrig accepts them and can equally derive both paths itself, which is what it does on every later start.

Why give up that responsibility? Because the shape of the launcher is product knowledge, and it changes. Adding a pathing JAR to work around Windows’ “input line is too long” limit changed what the launcher must contain. Had the install script owned it, every existing machine would carry a stale launcher written by whichever installer version happened to run there, and fixing it would mean asking users to re-run an installer. Because the binary owns it, the fix shipped with the binary and applied itself on the next start.

What the installer must not do is a written, locked document — docs/install-scripts-contract.md, titled “install devrig, register PATH — nothing else”. Two prohibitions: never auto-register devrig with an AI Agent, because that edits Claude’s / Codex’s / Gemini’s own config files, state outside ~/.mcp-steroid that belongs to the user; and never auto-install the MCP Steroid plugin into an IDE. Both stay explicit — devrig install <agent>, devrig install plugin. Instead the flow promotes them: devrig install devrig ends by listing the exact commands you may run next, and the scripts print no next-steps block of their own, so guidance cannot drift as the command set changes. Writing that down as a contract rather than leaving it as taste turns “should the installer wire up my agent?” into a closed question: an installer that silently rewires your tools turns a “download this CLI” decision into a “modify my agents” decision you never made.

The launcher, and self-healing on every start

~/.mcp-steroid/bin/devrig on my machine, in full:

#!/bin/sh
# devrig launcher — managed by the devrig binary. Writes nothing to stdout (MCP stdio channel).
# Pins the JDK devrig runs under via DEVRIG_JAVA_HOME (its supported runtime), then hands off to
# the install-tree devrig launcher.
DEVRIG_JAVA_HOME="/Users/jonnyzzz/.mcp-steroid/binaries/jdk-macos-arm64-0.101-614107ed76e9/amazon-corretto-25.jdk/Contents/Home"; export DEVRIG_JAVA_HOME
exec "/Users/jonnyzzz/.mcp-steroid/binaries/devrig-macos-arm64-0.101-a7c8cf0d16e2/devrig-0.101-40690055/bin/devrig" "$@"

Six lines, three of them comments — the entire user-facing surface of the installation. The JDK pin answers “which Java does it run under?”: not the user’s PATH, not their JAVA_HOME, but the runtime it shipped with. On main, after 0.101, the header gained two machine-parseable lines — # devrig launcher version: and # devrig launcher source: — for a reason we get to in a moment.

The Windows peer is ~/.mcp-steroid/bin/devrig.cmd, same shape:

@echo off
set "DEVRIG_JAVA_HOME=<jdk home>"
call "<install tree>\bin\devrig.bat" %*

Deliberately no PowerShell at launch — a unit test asserts the generated launcher does not contain the string. The first generation used a devrig.bat that trampolined into a devrig.ps1, and paying PowerShell startup on every agent call is the mistake that taught us not to. DEVRIG_JAVA_HOME works because the build patches Gradle’s own start-script templates in startScripts.doLast, injecting a JAVA_HOME override ahead of Gradle’s “Determine the Java command” marker, under a require(...) that fails the build if Gradle ever moves that marker.

BinLauncher.ensureBinLauncher() runs on every devrig start and makes sure, first, that the launcher exists and points at devrig’s own current install tree and the JDK it is currently running under — so it self-heals if it was deleted, or went stale because an upgrade landed a new tree; and second, that the launcher is reachable on PATH: on POSIX by symlinking it into the first PATH directory that is both writable and under $HOME (pure Java, no subprocess), on Windows by registering the bin directory in the HKCU user PATH, gated by a bin/.user-path-registered marker file. Windows needs a PowerShell helper for exactly one reason: pure Java cannot persist a user PATH entry, only check membership.

And here is the part that “rewrite on every start” gets wrong if you stop there. Two versions coexist by design, so the moment an older devrig starts — an agent still holding a stale registration, a session that outlived an upgrade — it would happily rewrite the launcher backwards, and the newest install would lose. #373 is the guard: the launcher carries its writer’s version in that header line, and shouldKeepNewerLauncher refuses to overwrite a launcher stamped strictly newer than the running build (SNAPSHOT counts as newest). A kept-newer launcher still gets its +x bit repaired. Self-healing without that check is not self-healing, it is a race — and it took shipping side-by-side installs to see it.

The rest is other people’s edge cases:

writeIfChanged compares against the intended content, normalizing CRLF/LF and a trailing newline, and returns early when they match — but still repairs a lost +x.
A real write stages to a sibling <name>.new<pid>, chmod +x before the move, then moves atomically. If that is blocked — Windows holding the launcher open — it renames the original to <name>.old<pid>, which NTFS permits on an open file, moves again, and deletes the old one; five attempts, 10 ms apart. Crash leftovers are never swept: a stray .old1234 is cheaper than a sweeper that could delete a live launcher.
DEVRIG_BIN_NO_AUTO_REGISTER is an undocumented opt-out, and self-registration defaults to off for SNAPSHOT builds so a local dev build never clobbers a real launcher; an explicit devrig install forces it regardless. The whole thing is best-effort — failing to heal the launcher must never take down devrig mcp.

Whatever you typed to launch it, you end up pointing at the one stable launcher. That is the invariant, and it is why agent registrations point at ~/.mcp-steroid/bin/devrig and survive every upgrade without being rewritten.

Update: the updater that knows nothing

:website-gen publishes a version.json next to the scripts. It is not a manifest:

{"version-base":"0.101"}

One field — what version is promoted right now. DevrigUpdateChecker fetches it with a cache-busting query and 10-second timeouts, and when a newer version exists it writes to stderr and pushes an MCP notifications/message so the agent sees it too. Any failure returns null and is swallowed at debug level: an update check must never be able to break a run. The same endpoint serves the in-IDE plugin’s checker, which adds an ?intellij-version=<build> query so the answer can be build-aware.

Comparing versions deserved its own type. The first version compared strings by prefix, which is how we got #360: our release tags and our CI tags were not mutually orderable, so either could look like an update to the other. DevrigVersion now implements Comparable with two product rules — a SNAPSHOT build sorts above anything promoted, and build metadata after the first - carries no precedence, so 0.86.0-a1b2c3d does not rank below 0.86.0 — and the decision is a plain isUpdateAvailable(current, promoted) = promoted > current, which also means a downgrade is not an update.

And then the upgrade is: run the install script again

Everything above is what makes that sentence sufficient —

the script is regenerated per release, so it always carries the current URLs and hashes;
installs are content-addressed, so a new version lands beside the old one instead of over it;
the binary owns the launcher, so the new binary repoints it on its first start;
agent registrations point at the stable launcher path, so nothing needs re-registering;
the script is idempotent, so running it when you are already current prints already installed and downloads nothing.

For 0.101 — the release you get if you run the one-liner today — that is the whole story: devrig tells you, you re-run the line. And then we shipped the obvious next step, and the shape of it is the reason this post exists.

The updater that cannot age

AutoUpdater is merged on main and not in 0.101 — install today and you get the notify-only behaviour above; the next release turns this on. It does genuinely self-update, and here is the interesting part of it:

val scriptUrl = if (isWin) "https://devrig.dev/install.ps1" else "https://devrig.dev/install.sh"

That is the entire update algorithm: download today’s install script, run it, supervise it. There is no archive handling in the binary, no URL construction, no hash table, no unpack logic, no launcher rewriting — none of the artifact knowledge that goes stale. The parts that rot fastest live in a file fetched fresh from the website seconds before it runs.

The title oversells slightly, so let me undersell it: the binary still hardcodes the two script URLs, the OS branch, the process-spawning and timeout behaviour, the marker protocol, and the version semantics. A real aging surface — just a small, slow-moving one, and none of it is the part that changes when a vendor moves a URL or we switch an archive format.

Which answers the objection from the top of this post. “Put the updater inside the binary” fails because your update logic ages with the oldest installed copy. The escape is not to refuse to self-update, it is to make the in-binary part contain no logic to age. A machine that sat on 0.99 for a year does not upgrade using 0.99’s ideas about archive formats or vendor URLs; it upgrades using today’s, because all 0.99 knows how to do is fetch and exec.

What is around that one line is the part I underestimated:

Scheduling — and only where it is safe. The active updater runs only in devrig mcp sessions: it ticks, then sleeps Random.nextLong(180, 481).minutes — 3 to 8 hours, jittered so a fleet does not stampede the CDN — and repeats forever. A short one-shot command keeps the passive notice and never spawns an installer. Nobody’s devrig --version should turn into a 200 MB download.
Multi-process coordination, because on this machine six agents routinely each hold a devrig mcp. UpdateCoordination writes an update-<pid>-version-<v> marker under ~/.mcp-steroid/update/, then re-checks after announcing and yields if a lower pid is also in progress. Lowest pid wins; losers exit silently. An updated-<v> marker means “already installed here” and turns the tick into a restart notice, once per process.
Supervision. The installer is spawned detached with a one-hour timeout and its own per-pid log file. On Windows it prefers powershell.exe by absolute System32 path, falling back to whatever powershell or pwsh is on PATH.
No retry cap, deliberately. Too many transient root causes exist, and the goal is to keep users current, so every failure retries on the next 3–8 hour tick, forever. Diagnosis lives in stderr and the log files rather than in a state machine.
Never downgrade, and the marker GC is bounded below min(current, promoted) so a session running newer than the promoted version — what a rollback looks like — cannot delete records older sessions still need.
Three ways off. SNAPSHOT builds skip the whole thing, and either DEVRIG_NO_AUTO_UPDATE or the launcher-write opt-out DEVRIG_BIN_NO_AUTO_REGISTER disables it.

Two things I would still flag. version.json is unsigned, and it just stopped being advisory: that one field now decides whether a script is downloaded and executed. And nothing pins a session to a version — the update repoints the shared launcher, so the next thing an agent launches is the new binary, mid-task. “The fleet finishes on the version it started with” is a property we describe and do not yet enforce.

Distribution, and the release-ordering trap

The scripts and version.json are static files on GitHub Pages at devrig.dev — one canonical host, with the older mcp-steroid.jonnyzzz.com 307-redirecting to it rather than serving a second copy, and the code (both update checkers, both install scripts) pointing at the canonical name. :website-gen’s generateWebsite task depends on :installer-gen:generateInstaller and writes version.json, updatePlugins.xml, install.sh and install.ps1 into website/build/generated-static, which Hugo folds into the site root. Nothing generated is committed. The binaries live on the GitHub Release; the JDKs are never mirrored — users download them straight from the vendor.

The non-obvious component here is a branch. The site does not deploy from main:

Deploy from the long-lived website branch, NOT main. website tracks main but can deliberately lag it so website changes that depend on an UNRELEASED devrig binary — e.g. a new install.sh / install.ps1 contract — stay off the live site until a matching GitHub release exists.

The install script’s content depends on a GitHub release that does not exist until you publish it, and the release notes point at an install script that must not go live early. Deploy from main and there is a window — minutes or days — where devrig.dev/install.sh references a release tag that 404s, and every new user in that window gets a broken install with no explanation. A separate deploy branch, fast-forwarded after the release is published, closes it — as long as nobody advances the branch by hand out of order, which is convention plus a release-readiness check, not a permission. The Pages workflow also rebuilds on release: published, because generating updatePlugins.xml requires querying the release for its asset URL.

The ordering constraint is worth stating as a sequence, because getting it wrong is how you serve a 404:

Build the artifacts and publish the GitHub release for v<version>. Nothing about the website has moved yet, so the live site still describes the previous release, correctly.
Generate — :installer-gen:generateInstaller resolves the devrig zip from that release, which is why the release has to exist first, and resolves the JDKs live from the vendors.
Fast-forward website to main. This is the deploy: Hugo folds in the generated install.sh, install.ps1 and version.json, and the one-liner starts pointing at the new release in the same push that starts announcing it.
Only now does a devrig on someone’s laptop see a new version-base and tell its user.

In between steps 1 and 3, main can carry an install script for a release that does not exist yet, and no user is exposed to it.

Signing: implemented once, designed twice, not shipped yet

The current chain of trust, hop by hop, with no softening:

You paste a URL. HTTPS to devrig.dev (GitHub Pages, behind a CDN). Unauthenticated beyond TLS.
The script runs. Its contents are whatever that host served. Not signed. Ask for install.sh.asc today and you get a 404.
Artifacts download. Each is checked against a SHA-256 baked into the script at generation time. Strong — but only as strong as hop 2, since whoever can change the script can change the hashes.
The JDK hashes were earned. At generation time the vendor JDK archives were OpenPGP-verified against fingerprints pinned in source, fail-fast, on every resolve including cache hits. This part is real — with one honest asterisk: it protects the release engineer, not the user. None of that evidence travels to the machine, so a user cannot re-check it.
The devrig zip itself has no signature at all. Its only credential is a hash written by the same unsigned script. The runtime we did not build gets stronger provenance than the binary we did, which is a funny thing to discover about your own pipeline.
version.json is fetched on every run, and it is not signed. Until recently the worst a tampered one could do was fake an update notice. Now it decides whether a script is downloaded and executed. That one unsigned field is the weakest thing in this post, and it got weaker the day auto-update shipped.

So the gap is hop 2, the classic curl | sh circularity: verifying a script before executing it requires something already trusted on the machine.

We solved this before, in the project’s own prehistory. The Go generation fetched https://devrig.dev/download/latest.json with a .sig and verified it natively against two embedded SSH public keys — one ed25519, one RSA-4096 — using golang.org/x/crypto/ssh, no ssh-keygen subprocess; the release side had ssh-sign.sh and sync-release.sh. Its bootstrap script also re-verified the binary’s SHA-512 on every run, not just after download.

The current design, written up as v7 of docs/devrig-deployment-spec.md and marked “design ready for implementation”, resolves the circularity rather than ignoring it: a version.properties manifest, a version.properties.signatures file holding two ed25519 SSH signatures, and an allowed_signers file in OpenSSH format pinning both public keys — two keys, two operators, two machines, both of which must verify, so one compromised laptop is not enough. Verification is Java-only: the shell wrapper fetches and stores the signatures and never checks them, and the installed binary verifies on upgrade. That is the key move — the initial install is trust-on-first-use over HTTPS, and every subsequent update is verified by code already installed and already trusted, so you stop trying to bootstrap trust from a pipe. The manifest also carries a SHA for each wrapper script, so an upgrade can verify the launchers too, and a weekly CI job HEADs every URL in the manifest and spot-checks the SHAs, so a dead vendor URL fails a build rather than a user’s install.

None of that ships today. The honest one-line summary is HTTPS, plus generation-time vendor PGP, plus install-time SHA-256 — with the bootstrap hop unauthenticated, the promoted-version field unsigned, and a worked-out design for closing both. If you are evaluating this pattern for your own tool, that is the sentence to weigh.

Testing scripts and generators

Shell scripts and code generators are both famously untested, usually with the same excuse: “you’d need a real machine”. You need a container, and a seam.

The seam is writeInstallerScripts(outDir, table, devrig, version) — network-free, wrapping a renderInstallerScripts that touches neither network nor disk. A test hands it a synthetic five-platform table pointing at a local HTTP server and gets a real installer out. Every lane below rests on it.

Hermetic unit tests (./gradlew :installer-gen:test) — no network at all; the URL-routing fakes hard-error on anything they were not given:

Test	What it pins down
`InstallerGeneratorTest`	24 tests: placeholders all resolved, exactly-five-platform validation, ASCII-only, unsafe version strings, `resolveDevrig` version-mismatch
`JdkArtifactsTest`	Vendor parsing, the shallowest-`bin/java` scan with a nested `jre` decoy, and “expected 4 Corretto + 1 Azul” as a hard assertion
`PgpVerifierTest`	Tampered data, key-not-in-keyring, wrong pinned fingerprint, fingerprint matching that ignores case and spaces — against keys generated in-test
`CacheTest`	Hash-mismatch rejection, re-verification on a cache hit with a corrupted cache file, and “must not download when caching can’t be validated” for the missing-ETag case

Two are worth stealing as a technique. The $ProgressPreference test does not assert the line is present: it asserts it appears exactly once, at an index before both Invoke-WebRequest -Uri and Expand-Archive -Path, and is absent from install.sh. The #273 test requires any RuntimeInformation access to sit inside a preceding try {. Both encode the property that mattered, not the diff that fixed it. Fingerprints are injectable into resolveAllJdks for the same reason — so the PGP tests can pin a generated key.

Docker integration lanes (InstallerBootstrapTest and its PowerShell peer, in a separate installerIntegrationTest source set gated so a plain ./gradlew check compiles but never runs them):

“end-to-end on ubuntu (glibc)” — a fake devrig zip and fake JDK tarball served by an nginx side-car started first, so the test learns its bridge IP and bakes those URLs plus the fixtures’ real SHA-256s through the seam. In a digest-pinned Ubuntu image it then checks both downloads, SHA verified, trees at devrig-linux-x64-0.0.0-test-<sha12>, bin/java executable, the delegation carrying the exact --install-script= / --jdk-home= values, and no agent registration — then a second run asserting already installed: and zero downloads.
“refuses musl (alpine)” — the real script in alpine:3.21: non-zero exit, the message explains musl, and the output never contains “downloading”, which proves the ordering rather than the wording.
the pwsh lane — mcr.microsoft.com/powershell:lts-7.4-ubuntu-22.04, because Windows-native Docker is not something this CI provisions. DEVRIG_OS / DEVRIG_CPU stay unset so the real auto-detect branch runs, and the template anticipates it: a bin/java stub is accepted where bin/java.exe would be.

The best line in the suite is a constant: INSTALLER_HOME_DIR = "/home/tester one". HOME with a space in it. Every quoting bug in a shell installer dies there, permanently, for free.

The native Windows lane. A pwsh-on-Linux test cannot catch a PowerShell 5.1 / .NET Framework bug, which is exactly what #273 was. So InstallerPs1ExecutionTest runs the installer for real on windows-latest under both powershell.exe (5.1) and pwsh (7.x), serving fixtures from an in-process com.sun.net.httpserver.HttpServer (Invoke-WebRequest rejects file://), and asserts by negative: the output must not contain "The property 'OSArchitecture' cannot be found". InstallerScriptTest adds a cheap syntax gate — the PowerShell AST parser under 5.1, sh -n on Linux. Both lanes run on every branch push, with TeamCity carrying the same suites internally.

resolveAllJdks leans on two vendor APIs that can change under you, so it is covered three ways: hermetically over synthetic archives; by the fail-fast PGP check on every resolve, so a vendor serving something unexpected fails the build and not a user’s install; and by generateInstaller running for real in the release pipeline. Tests that intentionally hit public vendor feeds are tagged live-network and excluded from the default test task.

So: the scripts’ logic is tested hermetically, their behaviour in real shells in real containers, their content at generation time, and the one thing you cannot fake — Windows’ own PowerShell — gets a dedicated runner.

Where this was built and tested

Everything above lives in jonnyzzz/mcp-steroid. The vendor endpoints, regexes and API field names this post deliberately does not transcribe are all in these files:

The JDK model — JdkModel.kt, CorrettoJdk.kt, AzulJdk.kt, PgpVerifier.kt, Cache.kt.
The generator — InstallerGenerator.kt and the two templates, install.sh.tmpl / install.ps1.tmpl. The templates are the installer; the generator is 300 lines of validation around them.
The CLI side — BinLauncher.kt, InstallCommand.kt, DevrigUpdateChecker.kt, AutoUpdater.kt. In devrig-common: HomePaths.kt (the layout, pinned by tests and deliberately not configurable) and UpdateCoordination.kt (the marker protocol).
The contracts — docs/install-scripts-contract.md (locked) and docs/updates-check/devrig-auto-update.md.
The tests — InstallerBootstrapTest.kt is the one to read. The Windows lanes live in test-integration-agent-launch.
The design doc — docs/devrig-deployment-spec.md, v7, including the signing and GC work that is designed but unshipped. Its iteration history runs v1 → v7.
The prehistory — jonnyzzz/old-devrig-private, the Go generation: a per-project .devrig home, SHA-512 content addressing, re-verification on every run, and a latest.json update feed signed under two embedded SSH keys we have not yet re-adopted.

Almost none of the sharp edges above were found by design review. The Windows work exists because people ran irm | iex and got an error message instead of a tool, then told us. The two open items on the JDK model — folding the resolved URL into the cache key, tightening Azul’s latest selection — came out of a quorum review of that PR, not from the author. Distribution code is exactly the kind of thing where a second and third reader pay for themselves, because the failure modes live on machines you do not own. One more warning from the old generation: IsUpdateAvailable() there returned info.Version == thisVersion, the inverse of what its own comment described, so it announced an update precisely when you were current. Update paths need tests more than they need cleverness.

What is not solved yet

The list I would want to read if I were evaluating this for my own project.

version.json is unsigned, and it now triggers code execution. Top of the list. Everything else here is inconvenience; this one is a trust gap that auto-update promoted from theoretical to live.
The bootstrap hop is unauthenticated, as laid out above. Design exists; implementation does not.
No session pinning. An update repoints the shared launcher under a running fleet.
No garbage collection, and no uninstall. Old trees accumulate forever; devrig gc does not exist. And between the launcher, a PATH symlink, an HKCU PATH entry, marker files, agent registrations and a few hundred MB per version, there is no supported way to remove any of it.
No rollback command, and no health-checked activation, despite the design assuming rollbacks happen — the auto-updater’s GC bound explicitly handles “a session running newer than the promoted version, post-rollback”. A release that downloads and unpacks cleanly but fails to start still becomes the launcher’s target, and rolling back means hand-editing two paths in ~/.mcp-steroid/bin/devrig.
The signing design stops at the happy path. Two keys and an allowed_signers file is the easy half. Rotation, revocation, recovery after a compromise, anti-replay for a signed-but-stale manifest, and a migration story for clients installed before a verifier exists are all unwritten. So is macOS notarization and Windows code-signing for the artifacts themselves.
No proxy or custom-CA story. The one that bothers us most: the person we keep describing — corporate Windows box, no admin rights — is exactly the person behind a TLS-inspecting proxy. Neither script is documented as honouring HTTPS_PROXY or an enterprise root CA. curl and Invoke-WebRequest pick some of that up from the environment by accident; “by accident” is not a support answer.
No download retry or resume. A transfer that dies at 80 % is a failed install and a re-run from zero, on a 200 MB JDK. The auto-updater’s answer is “try again in 3–8 hours”, which is fine for it and no help to a human watching a progress line.
Test gaps we know about. requireShellSafe — showcased above as security-critical — has no test of its own. Nothing exercises the wget fallback. And windows-latest is Windows Server 2025 with a modern .NET, so the machine that actually broke in #273 — base Windows 10 with .NET Framework 4.6.x — is still not in CI.
Vendor drift is only caught at release time. The weekly “HEAD every URL and re-check the SHAs” job is in the design, not in .github/workflows, so a vendor pulling a build is discovered by the next release or by a user. And because generation resolves live and the output is not committed, there is no retained record proving which coordinates a given release actually served — awkward for a post that opened by promising you could tell a security team exactly what got executed.
One user per machine. ~/.mcp-steroid is per-$HOME and explicitly not configurable — the plugin↔devrig marker contract pins the location. A shared CI box gets one installation.
Two vendors, five platforms. No Temurin, no GraalVM, no Intel macOS, no musl. All deliberate, and all a real “sorry” to real people.

Is this an open-source project on its own?

I think so, and I keep coming back to it because the reusable part is genuinely small. Strip out everything MCP Steroid–specific and what is left is:

A JDK data model resolved live from vendor sources and PGP-verified against pinned fingerprints (~three files plus a cache).
A generator that renders two script templates from that model, with generation-time validation (one file plus the templates).
A convention: content-addressed side-by-side installs under a fixed home, with the binary owning the launcher, and “update” defined as “run the current install script”.
A test harness: a pure render seam, an nginx side-car, three container lanes plus one native Windows runner, and a HOME with a space in it.

That is a plausible toolkit — call it ship-your-jvm-cli. Not a library yet: the binary-side half (launcher registration, update supervision, version policy, multi-process coordination) is still welded to this product, and a library has to offer those as APIs rather than as a description. And what makes the generator half work is mostly conventions and negative constraints rather than code — the rules: resolve at build time, verify vendor signatures at generation time, never let the script own the launcher, make update mean install, and put everything the script prints on stderr.

Becoming a real project would mean pluggable artifact coordinates rather than “devrig on jonnyzzz/mcp-steroid releases”; a configurable home, which conflicts with the marker contract that made it fixed in the first place — a design argument, not a refactor; Temurin and GraalVM alongside Corretto and Zulu; and a finished signing story.

If you have shipped a JVM CLI to strangers, I would like to compare notes — particularly on two things. First, signing a curl | sh bootstrap: our designed answer is trust-on-first-use plus two-key ed25519 verification for every update thereafter, and since it is designed rather than shipped, this is a genuine question and not a humblebrag — has anyone done better without shipping a package manager? Second, retention: how do you decide an old install tree is safe to delete, when the whole point of content addressing was to never track what points at it?

Open an issue on jonnyzzz/mcp-steroid, or reply on LinkedIn or Twitter — and if ship-your-jvm-cli is something you would actually use rather than nod at, say so, because that is the only signal that would make us extract it

Personal Assistant without OpenClaw or Hermes: Just One Prompt

2026-07-19T00:00:00+00:00

I wanted a personal assistant for quite some time. I can message from my phone. Something absolutely simple and what I can understand and fine-tune to my needs. Not a chatbot — an assistant: “watch that Slack DM thread and ping me when they reply”, “write down our AWS account conventions”, “build me a tool that inspects my iTerm2 tabs”. Requests arrive on Slack and Telegram at random times, some finish in a minute, some take forty, and we want the acknowledgment to land now and the result when it is ready — without message two waiting behind message one.

The naive approach — one AI Agent in one chat loop that does everything itself — falls apart on exactly that schedule. The Agent picks up a long task, its context fills with the work, the instructions dilute, and the next message queues behind a forty-minute errand. Worse: when the session eventually compacts or restarts, half-done work simply evaporates, and nobody tells you.

So the rule at the top of our operating contract is a role split, not a feature: “You assist and manage — you never do the work yourself.” The interactive session routes, tracks, follows up, and reports. Every actual task runs in a fresh, disposable, headless sub-agent process. The whole thing was scaffolded, smoke-tested, run through its first real tasks, hit its first production incident, and shipped its first self-authored fix in a single afternoon — 2026-07-17, 13:29 to about 15:50. Now it’s running on the second computer.

And here is the part I want to be upfront about, because the title promises it: the founding spec was one message pasted into an interactive Claude Code session. Not a product install, not a config wizard — a prompt. Lightly redacted, the worst typos cleaned, the rest left as I typed it the first time:

THE PROMPT — the message that started the whole system

you are my personal assistant, your goal is to setup and prepare the /workflows for that.
Do the following steps to set this up:
  - create a repository under gh jonnyzzz jonnyzzz-personal, private one
  - configure a workflows to regularly monitor and handle all the requests via the Slack's me-only
    private channel, it will be our communication protocol
  - configure and setup telegram communication channel, create the scripts to poll for messages
    and store all messages on the disk and handle them, use uv python, once done, ask me
    for credentials, 
  - only accept messages from my user accounts
  - credentials are never committed to the repository, only stay locally.

So the flow is the following -- you poll and process these two input channels. For each
request -- you MUST start the new sub-agent process, use run-agent.sh for that -- once the
process is completed -- process and send followup to the channel from which you receive
the input.

You should create the self-improvement loop -- start every 3 hours new sub-agent which
3x quorum processes and analyzes everything what was done and self improves. Running agentic
loops is necessary -- your task to manage that flow.

Commit and push all the changes to your flows to this new gh repo, update the repo regularly
and re-read from it too.

I expect you to script the most of the tasks to simplify the processes, effectiveness, and
outcomes. Setup hooks in this repository for yourself to always remember your goals, you are
assisting, managing, and never doing.

Everything below — the wrapper scripts, the hooks, the state files, the quorum, the iTerm2 supervisor — grew from these lines plus the follow-up conversation. Even this post draft did. The sub-agent mechanics trace back to the run-agent.sh approach the prompt names. I am not pretending this replaces OpenClaw or Hermes; the point is narrower and, I think, more useful: this class of assistant now costs one prompt and one agent session, and at that price you should build one before you adopt one.

Discipline for an AI Agent should live in the architecture — process boundaries, files on disk, git — not in a paragraph of prompt the model is supposed to remember forty minutes into a session.

This post is the build log, and then a comparison — as fair as I can make it — against the two big names in the personal-agent space right now: OpenClaw and Hermes Agent.

The system started from one only prompt above and evolved to a really powerful and supportive buddy, that learns from what you do, and how it works. Below, is that it thinks of itself.

With this post I share how my AI Agents evolved into a personal assistant, given the right context, and the best tools. This appers in-line with my goals and program: Agentic AI Experience & Tools. Find me on LinkedIn or Twitter, and if an assistant like this would fit your own workflow, get in touch — I am happy to help you shape one.

One Rule at the Top, Enforced Structurally

The orchestrator is an interactive Claude Code session with a CLAUDE.md contract that is auto-loaded every session. Line 5 is the prime directive quoted above, and it is enforced three ways, because hoping the model remembers is not a mechanism:

Process boundaries. Every request becomes a fresh claude -p headless run via a wrapper script run-agent.sh — one request, one process, no shared context. The orchestrator is forbidden from editing even its own flow files (scripts, prompts, hooks, docs) inline; flow changes also go through a sub-agent.
Hooks. Claude Code hooks re-print the directive at every session start and inject a one-line reminder before every single prompt: “You assist and manage, never do.” The session-start hook also appends live state pulled from the state files with jq — active watches, agents still running — so a fresh session wakes up already knowing “agent X is in flight, watch Y is armed”.
An inverted frame for workers. The sub-agent prompt frame has to explicitly flip the rule: “you ARE the worker here, so do the task yourself — the never-do-inline rule applies to the orchestrator, not you.” Sub-agents run in the same directory and inherit the same CLAUDE.md; without that line the never-do rule would recurse forever. The best line in the whole repo

There is exactly one carve-out, and it exists for an architectural reason we will get to: requests that are purely channel communication — send or read a Slack or Telegram message on my behalf — are handled by the orchestrator directly. Slack is reachable only from the orchestrator session, so there is no worker to delegate a Slack send to, and Telegram sends are code-restricted to my own chat either way. Even the carve-out leaves a trail: every direct reply is recorded through the state script, so the audit surface stays complete.

Two Channels In, One Owner

Slack side: a private channel, reached through the session’s Slack MCP tools, where only messages from my user id count as requests; new-message detection is a plain timestamp watermark in a state file.

Telegram side: a dedicated bot with a hard owner allowlist — enforced inside the CLI code, not in the prompt. The poller surfaces only allowlisted senders as work; send refuses non-allowlisted chat ids outright; strangers land in a log on stderr and are never replied to. Even bootstrapping is stranger-proof: with an empty allowlist, the poller logs every unknown sender and warns on stderr, a seen command aggregates them into candidate ids, the flow posts those ids to Slack for me to confirm, and only then does the allowlist get filled — the bot never replies to an unconfirmed id, not even once.

The Telegram CLI is a single-file Python script with PEP-723 inline metadata, run via uv, talking to the Bot API with httpx. Its signature feature came from a direct directive at 14:47 on day one: polling CLIs must run-and-wait, exiting after a 15-second idle bucket.

Any other chats: Just the same way as we implemented Slack and Telegram – are one prompt away from the availability.

Bucket Mode: Near-Realtime From a Blocking Poll

There is no daemon, no webhook, no queue broker anywhere in this system. The “runtime” is one interactive session running a repeating inbox cycle. Slack ingress is the boring half — an MCP channel read against the timestamp watermark, once per tick. The Telegram side is where the design lives: the event source is a blocking CLI with carefully chosen exit semantics:

poll --first-wait 240 --idle 15 --max-wait 280

wait up to 240s for the FIRST message       # nothing arrived: quiet exit 0, tick ends
on arrival, print it immediately            # unbuffered JSON line per message, as it lands
then switch to idle mode: exit after 15s    # a burst of messages becomes ONE bucket of work
never run past 280s total                   # hard cap, the poll can never block forever

Each underlying getUpdates long-poll timeout is clamped to the remaining budget with a hard ~50-second cap, and a sub-second tail is slept out rather than re-polled — a deliberate courtesy, per the comment in the CLI itself, instead of hammering the API with near-zero-timeout calls. The result: each loop tick holds a roughly four-minute listen, so response latency is near-instant for the cost of one cheap blocking call. All updates — authorized or not — are appended to a jsonl transcript before the offset advances, so nothing is consumed invisibly.

I like this trade a lot. OpenClaw and Hermes both answer “how do messages get in” with a resident gateway process. We answer it with a budget: latency comes from long-polling math, not infrastructure.

One Request, One Process

Dispatch is a small shell wrapper around claude -p, https://run-agent.sh. For every request it creates a run directory with three artifacts — prompt.md, result.json, stderr.log — appends a start record {id, channel, started, dir, pid, status:"running"} to an append-only agents.jsonl, runs the sub-agent with JSON output, then appends a done record with the exit code. Every agent lifecycle event is an append-only record; every run leaves a full paper trail. A discipline rule sits on top: never hand-append JSON to state files — always go through the state script.

The hardening matters more than the happy path: the wrapper traps EXIT, TERM, and INT, and writes an exit:143, "aborted: dispatcher killed" done-record if it dies before the normal one is written — guarded by a flag so it never double-writes. An id can never stick in “running” forever just because the wrapper was killed — the trap covers every catchable death, and the reaper we will meet below closes the one it cannot, a straight kill -9. That trap was not designed up front. It was shipped in response to an incident.

The First Incident, and the Fix, Two Hours Later

At 13:45 on day one — four minutes after the first real task landed — the first concurrent dispatch of two sub-agents went wrong. The dispatcher combined a shell & with the tool-level background dispatch it should have used, and one wrapper died mid-run. The work was recovered and the lessons were logged.

At 15:50 the same day, the system’s first self-improvement cycle shipped the systemic fix in one commit: the pid tracking and abort trap above, a state script for scripted bookkeeping, a deliver-first inbox flow with an explicit failure path and a reaper, and codified delegation rules. The rule that came out of it landed as literally step 7 of the day-one inbox prompt: dispatch with the tool’s background mode — “NEVER a shell &“. (It sits at step 9 today — the loop has since quorum-grown two more steps above it.)

Incident → three-agent post-mortem → quorum-approved fix → committed process change, all machine-executed on day one. Fix shipped same afternoon as the incident. That arc, more than any feature, is why I trust the design.

Deliver First, Fail Loudly, Reap the Dead

The inbox cycle runs every ~5 minutes and its ordering is opinionated:

pull the repo                # re-read your own instructions before acting
deliver finished work FIRST  # before polling for anything new
   success: read result.json, reply short, outcome first
   failure: send a notice + the tail of stderr.log — never silence
reap: entries "running" >30 min AND whose dispatcher pid fails kill -0
   → close as exit:143, note "reaped: dispatcher/agent dead", report each on its channel
re-check watches             # e.g. "tell me when that thread gets a reply"; expire with a notice after 24h
poll Telegram (bucket), 
read Slack
dispatch each new request    # ack "On it — task <id>", background, never shell &
catch late finishers, 
push flow changes
nothing new → end quietly, no channel noise

Step 1 means a changed instruction file takes effect mid-cycle. Step 2 means a finished task is reported before new work is taken. The failure path means a crashed sub-agent produces a notice with its stderr tail instead of vanishing. The reaper uses kill -0 as a liveness check so a genuinely long-running agent is left alone, while a dead dispatcher’s orphaned record gets closed and reported. Between the abort trap, the failure notice, and the reaper, there is no path where a dispatched request just disappears

The listing above is already a historical document, by the way: by the end of day two the loop had quorum-grown to twelve steps — an early re-arm and a self-improvement catch-up — plus a standing rule that the cycle must never end in a terminal state. The ten steps you see are day one’s version; the upgrades came from the process described next.

One pre-dispatch window remains, and it belongs on the record: the Telegram offset advances at poll time, so a message consumed moments before a session death lives only in the jsonl transcript, which nothing re-reads as work. Slack does not share the hole — its watermark is written back only after processing. A future quorum cycle owes us a fix.

A Quorum That Rewrites Its Own Process

“Self-improving agent” is usually a euphemism for “the model edits its own prompt, and we hope”. Ours is a pipeline with an evidence phase, independence, a threshold, and verification.

Every 3 hours: gather read-only evidence — git log --stat since the last cycle, both state jsonl files, the newest agent run directories, the previous cycle report. Then spawn three independent analysis sub-agents in parallel over the same evidence — in-session Agent-tool tasks this time, not headless dispatches — each with a distinct lens:

reliability — what failed, hung, was retried, or silently dropped;
effectiveness — routing speed, reply usefulness, loop closure;
simplicity — what to script, simplify, or delete; where the orchestrator did work inline.

Each returns proposals as {file, change, rationale}. The decision rule, as first shipped: accept only if at least 2 of 3 lenses independently support the same file and intent. Rejected proposals are written to a rejection log with reasons — the disagreement is an artifact, not a discard. Accepted changes are applied by exactly one applier sub-agent, dispatched through the same headless wrapper as any other task (the orchestrator never edits flows inline, even here), then mechanically verified — bash -n on every changed shell script, executables stay executable, the Telegram CLI still authenticates, the settings file still parses with jq — then pushed, with a cycle report to Slack.

Cycle #1’s real numbers: 18 proposals, 4 bundles accepted, 8 rejected and logged, one commit. Yes, 4 + 8 ≠ 18 — an accepted “bundle” groups several overlapping proposals into one coherent change, which is exactly what a 2-of-3 agreement on file and intent produces. That commit was the incident fix.

Then the quorum rewrote its own constitution. The strict overlap rule — accept only what two lenses independently author — turned out to be a filter so fine it soon produced a cycle with zero accepts, so at 22:14 on day one a cycle replaced it, through its own machinery: three analysts still propose, the proposals are deduped, and three independent voters now approve or reject each one, accept at 2-of-3 votes. One cycle later the scheduler got the same treatment: after one overnight run analyzed a window where nothing had happened, the gate became “3 hours elapsed AND new activity since the last cycle”, empty windows skipped and logged. Seven cycles ran in the first two days, and the process that runs them is itself the thing they keep amending

The Repo Is the Source of Truth

The closing rule of the contract: “If the flows in this repo and your memory disagree, the repo wins — re-read it.” Behavior is data on disk, not weights-in-context. Prompts are re-read every cycle and the repo is re-pulled before acting, so a git push is a live behavior deployment — including from another machine. The sync script is deliberately forgiving (pull --rebase --autostash, continue offline, retry push next cycle) because the loop must never wedge on git.

My favorite small artifact of this discipline: our AWS notes document that a local config file literally spells a profile name Sanbox — typo and all — instruct agents to use the typo as-is, mark fixing it as a pending manual action for me, and forbid agents from editing that file themselves. Reality documented over reality “fixed” out from under the human.

Secrets follow the same locality principle. Credentials live only in a gitignored directory with tight file modes; state and logs are gitignored too — only flow logic is committed. A pre-commit hook blocks staged secrets three ways: anything under the secrets directory (only its .example templates and README are exempt), any .env file, and any added diff line matching credential shapes. A third layer rides in every sub-agent frame — a standing “never commit or print tokens” line — because defense in depth assumes any single layer will eventually miss.

The sharpest boundary is capability asymmetry: the Slack connection exists only in the orchestrator session via MCP — there is deliberately no Slack CLI or token in the repo, so a compromised or confused sub-agent cannot reach Slack tokens at all. Telegram is softer, and I want to be precise about how much softer: the CLI’s send path is code-restricted to my own chat, so through our tooling the worst case is a message to me. But the raw bot token does live in the local secrets directory, sub-agents are permitted to read credentials, and a rogue or confused one could call the Bot API directly, bypassing the CLI entirely. The allowlist is a guarantee about the tool, not a bound on the token. That asymmetry is also why channel sends are the one thing the orchestrator does directly.

The Assistant That Babysits Other AI Agents

“Build me a tool that inspects my iTerm2 tabs” was on the wish list in the opening paragraph, and it grew into my favorite capability of the whole system. This machine usually runs eight to eleven iTerm2 consoles with other AI Agents inside — Claude Code sessions, Codex CLI sessions — each grinding through its own long task. Somebody has to notice when one of them stalls on an approval prompt or runs its context to 100%. Now the assistant does.

The read path is one AppleScript pass, strictly read-only. The status subcommand walks every window, tab, and session without activating or focusing anything, and prints for each: a header (session id, name, tty, window/tab position, plus iTerm2’s processing / at-prompt flags when exposed), the last twelve non-blank visible lines, and a first-pass status guess — WORKING, WAITING, IDLE/DONE, or BLOCKED, with a [100% context] flag when that shows on screen. The classifier is grep over screen tails, nothing smarter: spinner glyphs mean WORKING, “waiting for” means WAITING, a shell prompt means IDLE/DONE, a [y/n] or a “Goal blocked” means BLOCKED. BLOCKED is checked first because — the comment in the script says it plainly — a missed approval prompt is the costliest miss. The sub-agent running the survey is briefed to treat the guesses as first-pass only, re-read each tail, and report most-actionable-first. “What are my terminals doing” is now a routine Telegram request — the reply ledger shows three fleet summaries inside one day.

Watching is safe; writing into another agent’s console is not. So the write path got the opposite treatment: it exists, and it is deliberately hard to use. The hard rule in CLAUDE.md says console writes go ONLY through the guarded send subcommand, never raw osascript, and send refuses in layers:

it resolves the target by session id AND demands the expected tty on the command line — if the session’s actual tty differs, it refuses: writing into the wrong console is the nightmare scenario;
it demands --expect, a literal substring that must be on the target screen right now — a TOCTOU guard: if the prompt scrolled away or changed between survey and send, it refuses;
keystrokes come from a safe vocabulary — y, n, enter, a single digit; arbitrary text is an explicit --raw opt-in;
every write lands in an audit log with the exact bytes sent and content hashes of the screen before and after.

On top of the tool, the approval flow adds policy: approve ONCE with the least-privilege key, never “always allow” or “don’t ask again”, one approval per request, abort untouched if the screen shows anything but the expected prompt

The base inspector was one of day one’s first two dispatches; the survey and the guarded writer came out of quorum cycle #4 on day two — the guardrails themselves were proposed, 2-of-3 voted, and shipped by the same three-lens process as every other flow change.

And on day two the thing earned its keep. The afternoon survey flagged window 3: a Codex agent double-stuck, asking approval to kill its own hung Gradle test worker and showing “Goal blocked” right above. The status report flagged it for me instead of acting on its own; the unstuck ran only after I said go. The audit log then records three writes in sixty-six seconds: a y against the expected text “confirmed stuck Gradle test worker” — approve the kill, once; /goal resume plus a newline against “Goal blocked”; and the discovery of the day — in the Codex TUI, Enter does not submit while the agent is mid-turn. The typed command just sat there on screen. So the third audited keystroke is a single Tab, which is how Codex queues input for when the current turn ends. Nobody documents these things. That Tab went in through the --raw escape hatch, under audit — and by the next quorum cycle it had graduated to a first-class key in the safe vocabulary, next to a new “Agent TUI quirks” note in the repo recording which console queues on Tab and which submits on Enter. Your assistant learns these things one guarded keystroke at a time, and then it writes them down.

An AI Agent supervising other AI Agents is only as trustworthy as the narrowest write path you give it.

Let me be precise about what this is: best-effort supervision of peer agents, not a control plane. The consoles expose no API — the assistant reads the same screen text I would, and the classifications are regex guesses a sub-agent double-checks before reporting. What makes me comfortable anyway is the thesis this post keeps repeating: every intervention is id-and-tty-verified, precondition-checked, least-privilege, and audited. The discipline lives in the architecture, not in the model’s good intentions.

The last quorum cycle of day two made this a standing service: the assistant now keeps a notify-only watch on the other consoles and pings me when one of them stalls — paired with a hard rule in the contract that automated watches never write to a console. Helping the agents in the other terminals is the job; pressing their keys stays mine.

The Field: OpenClaw and Hermes

Now the comparison — because the two projects everyone will name in reply occupy the same niche with nearly opposite choices.

First, identities, honestly hedged. OpenClaw here means the self-hosted personal-agent runtime at github.com/openclaw/openclaw — released November 2025 as Warelay and renamed through CLAWDIS/Clawdbot/Moltbot to OpenClaw, per Wikipedia. The GitHub API gave it 383,244 stars on 2026-07-17, while Wikipedia’s March 2026 snapshot said ~247k — treat any star count as a photo of a moving train. Not to be confused with pjasicek/OpenClaw, a C++ remake of the 1997 game Captain Claw, no relevance to AI Agents. Hermes here means Nous Research’s Hermes Agent (216,301 stars by the same API call the same day) — not the Hermes LLM weight series from the same lab, and not Meta’s Hermes JS engine. Turing Post reported Hermes overtaking OpenClaw on OpenRouter daily tokens in June 2026 — 224B against 186B per day — though I have not verified OpenRouter’s underlying numbers independently. Both projects move fast — several defaults below are as-of-mid-July-2026 and worth re-checking against their docs.

Axis	jonnyzzz-personal	OpenClaw	Hermes Agent
Runtime	interactive session + loop	Gateway daemon	gateway service
Ingress	MCP read + bucket poll	persistent channel sockets	push + long-poll adapters
Channels	2, one owner	20+	20+
Session model	fresh process per request	DMs share main session	persistent per session key
Memory	git repo + jsonl	Markdown + daily notes	MEMORY.md + FTS5 SQLite
Self-improvement	3-lens quorum, 3-hourly	Skill Workshop, human-gated	per-turn background review
Isolation	capability asymmetry	sandbox off by default	six exec backends
Model coupling	Claude Code only	model-agnostic	model-agnostic
Behavior deploy	`git push`	workspace + skill file edits	file writes + skills

Ingress: a daemon versus a budget. OpenClaw’s Gateway is “the single source of truth for sessions, routing, and channel connections” — one always-on process holding persistent connections to twenty-plus platforms, exposing a WebSocket API on a localhost port. Hermes runs the same shape — one resident gateway process with 20+ adapters. Neither is quite push end to end, to be fair: in both projects the Telegram channel defaults to long polling with webhooks as the opt-in mode. But the shape holds — a resident daemon owns the connections. That buys instant delivery and breadth we simply do not have. It also buys daemon problems: OpenClaw’s docs note events are not replayed after a gap and pending sub-agent announcements are lost on Gateway restart. Our blocking bucket poll delivers a burst as one unit of work, resumes after any restart (the watermark and offset are files), and costs no infrastructure while idle. Two channels are enough for one owner.

Sessions and trust granularity. This is the deepest philosophical split. In OpenClaw, by default all DMs collapse into the agent’s main session — the docs themselves say “true isolation requires one agent per person” — and runs are serialized per session lane, which as I read it means a shared session can head-of-line block. Hermes resolves a session key and carries persistent conversation history across turns — genuine continuity, which our stateless workers lack. We chose the opposite: one request, one process, no carried context, and the orchestrator’s jsonl files as the only shared truth. Continuity lives in files we can read; isolation is the default, not the upgrade.

Self-improvement governance. All three systems admit the same thing — an agent will spot improvements to itself — and gate it differently. OpenClaw’s Skill Workshop is a human-review proposal queue: the agent drafts, a human approves (the viral “self-improving” framing overstates the default — though nothing I read in the docs blocks an agent with write access from editing skill files directly); behavior there deploys as edits to the agent’s workspace files and skills. Hermes runs a background review after each turn — optionally on a cheaper sidecar model — that writes memory and skills autonomously; a write_approval flag can stage writes for human consent, but it ships off by default, so out of the box the agent writes freely. Ours is machine-gated rather than human-gated: three independent lenses over shared evidence, a 2-of-3 quorum on file+intent, a rejection log, one applier, and mechanical verification before push. Slower than Hermes, more autonomous than OpenClaw — and every change lands as a reviewable, revertable git diff. Hermes’s self-writes are mostly markdown too — MEMORY.md, SKILL.md, though a skill can carry helper scripts — and staged writes can be diff-reviewed before landing; what I found no mention of in its docs is version history after they land — a file on disk, but no git log behind it.

Memory and source of truth. OpenClaw is refreshingly blunt: “the model only remembers what gets saved to disk” — curated MEMORY.md, daily notes, optional vector search. Hermes goes further: bounded, agent-curated core memory (over-limit writes error out, forcing consolidation) plus FTS5 full-text search over every past session in SQLite. Both beat us at recall — we cannot search old conversations at all, only grep the repo and the jsonl trail. What we have instead is that our entire behavior surface is version-controlled: no hidden state in a database, every behavior change has a hash, an author process, and a revert path.

Isolation defaults. OpenClaw ships with sandboxing off — tools execute on the host unless you opt into Docker with hardening flags — and that ecosystem’s attack surface is not hypothetical: Cisco’s AI threat researchers documented a malicious community skill, its ranking inflated to #1 on the public skills registry (molthub then, ClawHub today), silently exfiltrating data via curl, prompt injection included, and Wikipedia catalogs the wider fallout. Hermes offers six execution backends from local to Docker to serverless; the same docs describe command approval and allowlists on top. Our answer is smaller but structural: sub-agents are fresh processes with full run artifacts, the Slack connection simply does not exist outside the orchestrator, and the Telegram CLI refuses every chat id but mine — a tool-level check, not a hard bound, as the secrets section admits. And to be honest on the same axis we just judged OpenClaw on: our sub-agents run headless with permission prompts bypassed by default (--permission-mode bypassPermissions, overridable per run to acceptEdits), executing on the host. Unattended runs need it, but it means our isolation comes from process freshness and capability absence, not from a sandbox — we do not ship one either.

To be fair about what this comparison is: OpenClaw and Hermes are products — installable, multi-channel, multi-user-adjacent, with skill registries and communities. Ours is one owner’s operations repo that happens to embody the opposite bets. If you want WhatsApp, twenty channels, and a heartbeat that thinks every 30 minutes by default — an hour under OAuth-style auth — run theirs. If you want to know exactly why your assistant did what it did last Tuesday, git log is a hard act to beat.

The compliance argument. There is one more consequence of the size gap, and for anyone working inside a company it may be the decisive one. Everything we added on top of Claude Code is the prompt folded at the top of this post, roughly one thousand lines of generated scripts, and a git history where every behavior change is a commit. A corporate security or compliance review can actually read that — one sitting, line by line. The review still inherits the vendor layer underneath, to be clear — Claude Code itself, the Slack MCP server, and the unsandboxed bypassPermissions execution admitted above — but an AI Coding tool is a thing your company has likely already assessed or is assessing anyway; the marginal surface this assistant adds is one prompt and a thousand lines. Now put OpenClaw or Hermes through the same review: a full runtime, twenty-plus channel adapters, a community skills registry with a demonstrated malicious-skill incident, and a release cadence that outruns any approval process. I am not saying they cannot pass review — I am saying the prompt-sized system is reviewable in a way a platform is not. And building it yourself carries a bonus no product ships: you watched every piece of it land as a diff.

What I’d Steal, and What Stays

From Hermes I would steal searchable session history — FTS5 over past conversations is the feature I miss most, and jsonl grep is a poor substitute. From OpenClaw, the heartbeat contract: an explicit act-or-stay-silent protocol for proactive ticks is cleaner than our watches mechanism. Some of our own edges are unsolved too: the orchestrator is a single interactive session on a single machine, seven quorum cycles in two days are still a small sample to generalize from, and parts of the day-one code probably contain subtle bugs we have not hit yet.

What stays, without hesitation: assist/manage/never-do as enforced architecture, deliver-first ordering with a loud failure path, and the repo as the only source of truth. The constraint made the design better — the same sentence closed a section of the Tart desktop post three days ago, and it earned its repeat

Show Me Yours

If you run OpenClaw or Hermes daily, I want to hear where my comparison is unfair — both projects move fast, and some defaults above will already be stale by the time you read this. And if you have built your own orchestrator on Claude Code — especially if it survived its own first incident — show me the post-mortem. Find me on LinkedIn or Twitter, and if an assistant like this would fit your own workflow, get in touch — I am happy to help you shape one. Or skip the middleman: unfold the prompt at the top, swap in your own repo name, and paste it into a fresh agent session. That is the whole install — the rest is a conversation.

Related reading: how we audit what agents do to git in Inside the Git Hooks for AI Agents, and how we gave a Linux Agent eyes and hands on macOS in A Real macOS Desktop for a Linux Agent.

A Real macOS Desktop for a Linux Agent: Driving Tart VMs over SSH with Agent Skills

2026-07-16T00:00:00+00:00

The AI Agent that took this screenshot has never touched a Mac. It is running on Linux, it reached an Apple Silicon Mac over SSH, booted a Tart VM there, and captured a real macOS Aqua session — Finder, Dock, wallpaper and all.

In the previous post we booted a full IntelliJ IDEA inside a Docker container — Xvfb for the display, fluxbox to keep the windows still, ffmpeg recording, live video streamed into a browser tab (watch it on YouTube). I mentioned that Docker limits us to Linux, and that “solutions like tart to run virtual macOS” were left uncovered. That was the honest gap: if the thing you need your Agent to test is macOS itself — Gatekeeper, the TCC consent prompts, Aqua window focus — a Linux container will never show it to you.

I experimented with capturing a demo video with an AI Agent on macOS. We processed all the experience, and condensed it into this post — the macOS half of the story. This time the deliverable is not a Kotlin test harness — it is a set of Agent Skills: plain-markdown instructions plus a small bin/tart-remote orchestrator that any AI Agent on Linux can pick up to boot, see, and drive a real macOS desktop over SSH. It is open source in jonnyzzz/tart-skills.

The problem: a macOS GUI, and no Mac for the Agent

Tart runs macOS and Linux virtual machines on Apple Silicon using Apple’s Virtualization.framework (tart.run). It is excellent — but it only runs on an Apple Silicon Mac. In my setup, most of the agentic payload is running on Linux, and I decided to explore the approach of connecting a Linux machine to a Mac VM, over SSH. The Agent side is not picky — we tested from Linux, and the same commands work from a Docker container or from another Mac. The one hard requirement is the VM host: that is always an Apple Silicon Mac. With this post I want to show what is possible.

There is a second problem, softer but just as real. The lessons from the Docker post lived inside one project’s harness. This time we wanted the knowledge to travel — between Claude Code, Codex, Gemini, and whatever comes next. Markdown skills plus a thin CLI move between Agents in a way a test module never will.

The two SSH hops

The whole design is one clean seam. The Agent never talks to Tart directly; it runs ssh mac 'tart-remote …', and tart-remote owns everything Mac- and VM-specific.

graph LR
    Agent["Linux AI Agent<br/>(the skills)"]
    Mac["Mac host, Apple Silicon<br/>bin/tart-remote + tart<br/>boots + supervises the VM"]
    VM["macOS GUI VM<br/>Aqua session:<br/>IntelliJ, Finder…"]

    Agent -->|ssh| Mac
    Mac -->|"ssh mux<br/>screencapture / cliclick"| VM
    Mac -.->|"stdout: PNG / MP4 / IP"| Agent

    style Agent fill:#e1f5ff
    style Mac fill:#fff4e6
    style VM fill:#f3e5f5

A Tart VM is a long-lived foreground process — a naive ssh mac 'tart run …' boots a VM that dies the moment the SSH command returns. So vm-up boots it detached (nohup … </dev/null & disown) and the VM keeps its GUI session alive after the Agent disconnects. The inner hop is multiplexed with SSH ControlMaster / ControlPersist, which — besides being faster — dodges macOS sshd’s “Too many authentication failures” lockout that a busy Agent otherwise trips within seconds.

Six skills sit on top of that orchestrator:

Skill	What the Agent uses it for
`tart-remote-setup`	The Linux→Mac hop: install `tart-remote`, verify Tart is there
`tart-vm-manage`	Boot (detached) / provision / status / stop / delete the VM
`tart-vm-screenshot`	Capture the screen as a PNG — the Agent’s eyes
`tart-vm-intellij`	Launch IntelliJ IDEA and drive it with click / type
`tart-vm-video`	Record the GUI as video
`tart-vm-cache`	Share one IDE copy across every VM — faster, less disk

A full run reads the way you’d hope (abbreviated — $MAC is your Mac, $V a task-unique TART_VM=…, and cache-setup runs once first — let it finish before provision, or the VM falls back to downloading its own IntelliJ; the repo has the exact quick start):

ssh "$MAC" "$V ~/bin/tart-remote vm-up"         # boot a macOS VM, detached
ssh "$MAC" "$V ~/bin/tart-remote provision"     # cliclick, ffmpeg, IntelliJ, TCC grant
ssh "$MAC" "$V ~/bin/tart-remote screenshot -" > desktop.png
ssh "$MAC" "$V ~/bin/tart-remote start-ide"     # launch IntelliJ into the GUI session
ssh "$MAC" "$V ~/bin/tart-remote click 800 450" # drive it
ssh "$MAC" "$V ~/bin/tart-remote record 15 -" > clip.mov
ssh "$MAC" "$V ~/bin/tart-remote vm-gc"         # clean up after yourself

By default vm-up boots the macos-tahoe-base image (macOS 26) — set TART_IMAGE to reuse an image you already have pulled, the first pull is tens of gigabytes.

Two design rules carried the whole thing. stdout is data, stderr is logs — so screenshot - and record N - stream raw bytes you redirect straight into a file, while every log line goes to stderr. And state is read straight from tart list / tart ip, so there is no bookkeeping to get out of sync.

The gotchas, learned the hard way

Most of this project was distilled from a prior video-production effort that drove Tart VMs the hard way — hand-typed ssh commands, VMs that died the moment a terminal closed, and a run nobody could replay. A few lessons cost a real debugging session in devrig each, and they are the interesting part.

The keychain that blocks your VM. On a headless macOS 15+ host, booting a macOS guest failed with a security error, Code=-9, “Failed to create new HostKey”. Nothing in that message says the word “keychain”, and yet that is the cause: Virtualization.framework needs the host’s login.keychain to be unlocked, and a headless SSH session leaves it locked (tart FAQ, openai/tart#1146). A Linux guest boots fine; only macOS trips it. The fix is one line in vm-up — security unlock-keychain — gated behind a TART_KEYCHAIN_PW you only set on a trusted host.

The Agent grants its own screen-recording consent. Without a Screen Recording grant in TCC.db, screencapture over SSH returns a black frame — and on macOS 15+ the consent dialog that would fix it can never appear in a headless session. Which client needs the grant has drifted across macOS versions: historically /usr/libexec/sshd-keygen-wrapper, and — since Sequoia picked up OpenSSH 9.8’s sshd split — sometimes com.apple.sshd-session. Recent Cirrus Labs base images pre-bake the wrapper grant — SIP is off there, which is what makes TCC.db writable at all, not what disables the checks — so a fresh VM often captures fine before any provisioning; provisioning writes the com.apple.sshd-session grant on top to cover both. Then macOS 15+ still periodically pops a “requesting to bypass the system private window picker” dialog. Captures keep working, but the box sits in the frame — so the Agent notices it in its own screenshot and clicks it away. An Agent approving a macOS privacy prompt about the Agent is my favorite moment in the whole project

A stranded ⌘ that opened the logout dialog. cliclick’s kp: accepts only named keys — no letters — so the natural “⌘A to clear the field” as kd:cmd kp:a ku:cmd errors on a after pressing Cmd down and never releases it. Cmd stays held, the next keystrokes become shortcuts, and one of our test runs cheerfully opened the macOS “quit everything and log out” dialog. bin/tart-remote now releases cmd,ctrl,alt,shift after every key call, even when cliclick returns an error, so a typo can’t strand a modifier. For a letter with a modifier you use t: inside the combo — kd:cmd t:a ku:cmd.

No Homebrew required. The remote Mac we validated against had no Xcode Command Line Tools and no sshpass. So Tart installs brew-less from its release tarball into ~/bin, and the inner hop authenticates through OpenSSH’s SSH_ASKPASS (SSH_ASKPASS_REQUIRE=force, which needs no tty) when sshpass is absent. The only hard dependencies on the Mac are tart and a stock ssh. The constraint made the design better.

One IDE, shared across every VM

Downloading a full IntelliJ into every VM is slow and wastes tens of gigabytes. Instead, cache-setup downloads IntelliJ IDEA Community Edition once into a folder on the Mac host, vm-up mounts that folder read-only into every VM, and the IDE runs straight from the mount — no per-VM download, no per-VM copy. We watched a fresh VM, with nothing in /Applications, launch IntelliJ straight off the shared read-only mount. For managing IDE versions we point at devrig, the little CLI that downloads and starts JetBrains IDE backends for an Agent.

Because the Mac is shared between many AI Agent tasks at once, the skills bake in etiquette: a unique VM name per task, tart-remote ls to see the neighbors, and an always-clean-up-with-vm-gc rule even when the task failed. Two Agents each booted their own VM concurrently while we tested, and neither stepped on the other.

What we actually get

Validated end-to-end, over both SSH hops, against a remote Apple Silicon Mac with no Homebrew on it: clone → keychain-unlocked detached boot → provision → a real desktop PNG → IntelliJ launched — and the Agent walked the first-run dialogs itself: the screen-recording consent, the User Agreement checkbox plus Continue, Data Sharing, the local-network prompt — all the way to the Welcome screen → a valid 1600×900 QuickTime video. The screenshots in this post are those captures, streamed straight through both hops to the Agent.

There are edges we have not solved yet. Sharing devrig’s downloads across VMs is not wired up — devrig wants a writable home, and the shared cache is read-only. The periodic private-window-picker dialog still appears and still needs a click. TART_KEYCHAIN_PW puts a password on a command line, so it is for trusted hosts only. And mind the coordinates: the guest display is HiDPI, so screenshots carry 2× the pixels of the logical resolution your clicks use — provision pins the mode with displayplacer (a macOS 26 guest ignores the VM display setting otherwise), but the divide-by-two stays yours.

The same thesis as the Docker post, one level up: an AI Agent needs eyes and hands on every operating system its users run — not just the one that happens to fit in a container

Try it, point it at a Mac

Clone jonnyzzz/tart-skills, turn on Remote Login on any Apple Silicon Mac you can reach, and let your Agent take its first macOS screenshot. If you built the Docker variant from the last post, this is the macOS half of the same idea. Then show me what your Agent drove — tell me on LinkedIn or Twitter, or open an issue on the repo.

A Real IntelliJ IDEA Inside Docker: Integration Tests with X Server, Window Manager, and Live Video

2026-07-05T00:00:00+00:00

The demo: an integration test boots IntelliJ IDEA inside a Docker container, streams its live screen into the browser tab on the right, and a Claude agent drives the IDE debugger — all of it the setup described below. Watch it on YouTube.

The question I keep asking myself while writing my IntelliJ plugins – is there a way to write an integration test that proves the plugin actually works inside the IDE. You need tests 10x more thanks to AI Agents and agentic loops, and since I integrate deeply with the IDE, I need that even more. The integration tests are also the source of the Agentic traces I use to self-improve MCP Steroid.

In this post, I share insights to the setup of full integration tests with IntelliJ-based IDEs, running with GUI, inside Docker container.

It started about 8 years ago, when we first introduced performance tests into the IntelliJ platform — such tests are now visible under the IntelliJ Community. We’ve been using these tests to stabilize IntelliJ Shared Indexes. This time, I provide the fully reusable way to run integration tests, now in Docker and with full GUI and tools.

We walk through the essentials: the Docker image, the X server setup, how we keep IntelliJ manageable in there, and how the video reaches your browser. Enough details that you (better your AI Agent) can re-implement the approach for your own GUI-under-test — it is not IntelliJ-specific at all. All the code is open source in the test-integration module, and the tech powers the demo videos we are preparing for the project.

This continues the MCP Steroid series — the original post explains why AI Agents get the full IntelliJ runtime, and the devrig post covers the autonomous IDE lifecycle. Today is about the test infrastructure underneath.

Why not headless, why not mocks

First of all, you can find the testing framework in the IntelliJ SDK, that is great and enough for most of the cases. Here we discuss more high-level integration tests, which are as close to the end-user IDE as possible, with modal dialogs, and production files layout, production threading, deadlocks, and more.

The interesting bugs live in the seams: a modal dialog that blocks the whole IDE, the trust prompt that appears only on first open of a project, indexing that makes every PSI call return incomplete results or fail. MCP Steroid server executes Kotlin code inside the live IDE process — such the integration test has to prove that works while the IDE is doing all its usual IDE things.

Headless IntelliJ exists, but it is a different product in practice. No window subsystem means no dialogs, no focus, no screenshots — and screenshots are an MCP Steroid feature (steroid_take_screenshot). If the agent can see the IDE in production, the test environment must render actual pixels, and the IDE behaviour should be tested appropriately. The headless IDE — the same flavor the test framework uses — runs a different threading model that is much more convenient for testing. For the reference, Remote Development functions slightly differently, we are not covering it. I want to be certain my plugin and my features are behaving in the GUI version of the IDE, just like the end-users do.

The requirement list settled to: real IDE, real display, reproducible on any machine with Docker, and observable — when a 15-minute test goes sideways, “read 40MB of idea.log” is a bad debugging story. We wanted to just look at the screen, and I confirm, an AI Agent can do that too!

In the MCP Steroid we combine approaches – there are fast and lightweight tests with the IntelliJ Platform SDK, and there are heavy docker-based integration tests to cover the most heavy end-to-end flows. We are discussing the second below.

Why Docker?

Since I want to stay as close as possible to end-user scenarios – a clean machine is required. We run MCP Steroid together with AI Agents such as Claude Code, Codex, and Gemini. I do not want my local machine setup to interfere with the actual integration test. That is where Docker comes in – every time, we run in the same scripted, clean environment.

Docker comes with the price – I can only integration-test my project with Linux. There are solutions like tart to run virtual macOS and other solutions on Windows, which I do not cover so far.

The image: heavy base, thin IDE layer

The Docker setup is two layers. A heavy ide-base image holds everything that changes rarely and caches well; a thin per-IDE layer adds an IDE distribution on top. The base starts from a small Ubuntu image, installs the X stack, and ends by dropping to a non-root user that does nothing (full Dockerfile):

FROM ubuntu:24.04

# X server, window manager, capture tools, fonts, and the JetBrains Runtime's native X deps
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
        xvfb \
        x11-utils \
        fluxbox \
        fonts-dejavu \
        fonts-jetbrains-mono \
        ffmpeg \
        scrot \
        xdotool \
        xclip \
        xterm \
        libxi6 libxrender1 libxtst6 libxext6 libx11-6 \
        libfreetype6 libfontconfig1 \
        curl git unzip rsync procps ca-certificates \
        # … a few more utilities trimmed for the post
    && rm -rf /var/lib/apt/lists/*

# … plus six Temurin JDKs (8, 11, 17, 21, 24, 25) and Node.js 20, and the AI Agent CLIs
# (Claude Code, Codex, Gemini) behind a daily cache-bust ARG so they stay fresh

# a non-root user that owns the dotfiles and caches. ubuntu:24.04 already ships a
# default user at uid 1000, so free the slot before claiming it for `agent`.
RUN userdel -r ubuntu 2>/dev/null || true && useradd -m -u 1000 -s /bin/bash agent

USER agent
WORKDIR /home/agent
CMD ["sleep", "infinity"]

Every package earns its place:

xvfb — the virtual X server, our display
fluxbox — a tiny window manager; without one, IntelliJ windows have no position, no focus, no stacking
x11-utils — gives us xdpyinfo for the display readiness probe
xdotool — window geometry and raw input injection from scripts
scrot + ffmpeg — screenshots and screen recording
fonts-jetbrains-mono, fonts-dejavu — an IDE without fonts renders as squares, not much to look at
the libx* set — the JetBrains Runtime is a JVM with AWT; these are its native X client dependencies

The six JDKs let the IDE detect system JDKs and let tests exercise multi-version SDK selection; the daily cache-bust keeps the fast-moving agent CLIs fresh while the expensive apt layer rebuilds rarely. The same way you may need node, python, and other dev dependencies.

The last three lines matter more than they look. The container’s main process is literally sleep infinity — the container is an empty, running room. Xvfb, the window manager, ffmpeg, the IDE itself — every real process is started later, via docker exec, from the test harness. This is the single most useful design decision in the whole setup: the harness owns the exact startup order, sees every process’s output separately, can restart any piece, and cleanup is LIFO through one stack of dispose actions. A CMD that tries to boot everything in a shell script gives you none of that.

The IDE layer is almost embarrassingly small (full Dockerfile), and potentially cheap enough to run via docker exec or devrig:

ARG BASE_IMAGE=mcp-steroid-ide-base-test
FROM ${BASE_IMAGE}
USER root
COPY ide.tar.gz /tmp/ide.tar.gz
RUN mkdir -p /opt/idea && chmod a+rwx /opt && \
    tar -xzf /tmp/ide.tar.gz -C /opt/idea --strip-components=1 && \
    rm /tmp/ide.tar.gz
USER agent
ENV PATH="/opt/idea/bin:/usr/local/bin:/usr/bin:$PATH"
CMD ["sleep", "infinity"]

The ide.tar.gz is downloaded by the build on the host — the same resolver that powers devrig backend download — and hard-linked into a hash-keyed build context, so Docker’s BuildKit reuses the ~1.5 GB layer between runs instead of re-snapshotting it. PyCharm, GoLand, WebStorm, Android Studio, and CLion get identical thin layers, only the archive and launcher name differ; Rider additionally installs the .NET SDK.

Starting the container

The harness launches the container with plain docker run — no Testcontainers library right now. I would recommend the Testcontainers library, since everything we do is similar and should be supported there as well. We are considering moving to it later in the project too.

Here is the run command:

docker run -d --rm --init \
    --add-host=host.docker.internal:host-gateway \
    -v <run-dir>:/mcp-run-dir:rw \
    -v <repo-cache>:/repo-cache:ro \
    -v <m2-cache>:/home/agent/.m2:rw \
    -v <gradle-cache>:/home/agent/.gradle:rw \
    -p 6754 -p 8765 -p 5005 \
    <image-id>

The container command comes from the image’s CMD — that sleep infinity again. The Maven and Gradle cache mounts persist dependency downloads between runs. Each -p publishes the container port to a random free host port; the harness reads the mapping back and prints it, so parallel test runs never fight over ports.

Three published ports carry everything interesting:

Port	Purpose
6754	MCP Steroid HTTP server inside the IDE — the test’s API
8765	video streaming dashboard — the part for humans
5005	JDWP — attach a debugger to the containerized IDE JVM

That last one is worth a pause: the IDE inside the container runs with -agentlib:jdwp=...,server=y,suspend=n, so when a test misbehaves you attach IntelliJ-on-the-host to IntelliJ-in-the-container and step through the plugin code live. Debugging an IDE with an IDE — it never stops being funny.

Two small flags carry weight. --init gives us a proper PID 1 that reaps zombie processes — with dozens of docker exec sessions spawning Xvfb, ffmpeg, rsync loops and an entire JVM, orphans are guaranteed. And we do not pass --user: the image bakes a non-root agent user (uid 1000 by default) that owns its dotfiles and caches; overriding the uid from outside breaks fluxbox and Maven in creative ways. The host run directory is made world-writable instead. Not elegant, works everywhere.

The X server: Xvfb, a readiness probe, and fluxbox

Everything display-related lives in a small driver before anything else runs. The startup is two commands and one wait loop (the display driver):

## Start the server
Xvfb :99 -screen 0 3840x2160x24 -ac

## Wait for it in the other session; poll until xdpyinfo succeeds
for i in $(seq 1 150); do 
   xdpyinfo -display $DISPLAY >/dev/null 2>&1 && exit 0; sleep 0.1; 
done; exit 1

Xvfb :99 -screen 0 3840x2160x24 -ac — a 4K virtual screen, 24-bit color, access control off (we are alone inside the container). The readiness probe matters: Xvfb takes a moment to create its socket, and an IDE launched a hair too early dies with a cryptic Can't connect to X11 display. Polling xdpyinfo until it succeeds costs at most 15 seconds and removed a whole class of flaky starts. From here on, the harness stamps DISPLAY=:99 onto every docker exec it issues, and that is the entire “installation” of the display

Then the window manager. You might think you can skip it — the X server renders fine without one. You can not. Without a WM, windows get no focus management, popups draw at 0,0, and IntelliJ’s own window activation logic quietly breaks. fluxbox is small, and is fully configured by writing three tiny dotfiles before launch (the window-manager setup):

## optional wallpaper — point it at your own image, or drop this line
printf 'background: fullscreen\nbackground.pixmap: /usr/share/images/mcp-steroid-wallpaper.jpg\n' > /home/agent/.fluxbox/overlay
printf 'session.screen0.toolbar.visible: false\n' > /home/agent/.fluxbox/init
printf '
    [app] (name=jetbrains-idea)
      [Decorations]  {NONE}
    [end]
    [app] (name=xterm)
      [Decorations]  {NONE}
    [end]
' > /home/agent/.fluxbox/apps

## Start it
fluxbox

After that, xdotool windowsize / windowmove pins the layout: the IDE takes the left two thirds of the 4K screen, and an xterm on the right third tails the test’s own status output. When you watch the video stream, that xterm is the running commentary of what the harness is doing to the IDE at that moment.

Managing IntelliJ inside the container

Getting IntelliJ to launch in Docker is easy. Getting it to reach a working editor with zero human clicks is where the real knowledge accumulated, one failure at a time. The launch itself is one line — the launcher script from the unpacked distribution, pointed at the project:

## make sure DISPLAY variable is set
DISPLAY=:99 /opt/idea/bin/idea /home/agent/project-home

That one line only survives because the IDE is fully prepared on disk before it runs. JDK pinning happens in the factory as the project is deployed (intelliJ-factory.kt); the rest is startIde() (intelliJ.kt), in this order:

Pin the project and Gradle JDK in the project’s own .idea/misc.xml and .idea/gradle.xml, so import picks the right JVM.
Create the folders — config, system, logs, plugins, and the project dir, so the IDE never races to create them.
Write the consent files — the config-dir files and the JVM userPrefs that pre-accept EULA and privacy policy (more on these below).
Write trusted-paths.xml — mark the project path (and, for tests, /) as trusted, so the Trust Project dialog never appears.
Write the .vmoptions — the full set of JVM flags (below).
Write jdk.table.xml — pre-populate the JDK table for IntelliJ IDEA-family IDEs, so project-open Gradle sync resolves the JDK instead of stalling ~8 minutes on an unknown SDK.

Two of these carry most of the weight: the VM options and the consent files. Let’s look at each.

The VM options

Everything that keeps that launcher alive unattended is in the generated .vmoptions file. Its location is special — <idea-home>.vmoptions, so for us /opt/idea.vmoptions. Here is the full set we generate:

-Xmx6g
-Xms1g

## Debugger is enabled, should we need it — it listens inside the Docker container
-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005

## Redirect the IDE folders to explicit paths — enough to sandbox the IDE, even without Docker
-Didea.config.path=/mcp-run-dir/intellij/ide-config
-Didea.system.path=/home/agent/ide-system
-Didea.log.path=/mcp-run-dir/intellij/ide-log
-Didea.plugins.path=/home/agent/ide-plugins

## MCP Steroid plugin — bind on all interfaces so Docker can map the port out
-Dmcp.steroid.server.host=0.0.0.0
-Dmcp.steroid.server.port=6754
-Dmcp.steroid.dialog.killer.enabled=true
-Dmcp.steroid.updates.enabled=false
-Dmcp.steroid.analytics.enabled=false
-Dmcp.steroid.idea.description.enabled=false
-Dmcp.steroid.storage.path=/mcp-run-dir/intellij/mcp-steroid

## Suppress the AI promo window (it fetches a remote URL on first run and, in Docker,
## times out after 480s — an 8-minute startup deadlock). Also cut the network timeout.
-Dllm.show.ai.promotion.window.on.start=false
-Didea.connection.timeout=3000

## License server for commercial IDEs (Rider, etc.)
-DJETBRAINS_LICENSE_SERVER=https://your-intellij-linense-server-or-ide-services

## Skip EULA, consent dialogs, trust prompts, and onboarding
-Didea.trust.disabled=true
-Djb.consents.confirmation.enabled=false
-Djb.privacy.policy.text=<!--999.999-->
-Djb.privacy.policy.ai.assistant.text=<!--999.999-->
-Dmarketplace.eula.reviewed.and.accepted=true
-Dwriterside.eula.reviewed.and.accepted=true
-Didea.initially.ask.config=never
-Dide.newUsersOnboarding=false
-Dnosplash=true

## Belt-and-suspenders vs the SDK-download consent modal that blocks JPS builds
-Dunknown.sdk=false
-Dunknown.sdk.auto=false
-Dunknown.sdk.modal.jps=false

## Never talk to the outside world about updates or telemetry
-Didea.suppress.statistics.report=true
-Didea.local.statistics.without.report=true
-Dfeature.usage.event.log.send.on.ide.close=false
-Dide.enable.notification.trace.data.sharing=false
-Didea.updates.url=http://127.0.0.1
-Dide.no.platform.update=true
-Dide.browser.disabled=true

Each of those trust/consent/onboarding flags is necessary to avoid a modal dialog on the first start. The good news – it works now, and it is a solvable grind. My favorite: the AI promo window that appears on a fresh IDE once made startup deadlock for a full eight minutes inside Docker before we found llm.show.ai.promotion.window.on.start=false. The unknown.sdk trio is a close second — a background build task deciding it wants to download a JDK, and asking for consent with a modal, on a machine with no human. A dialog nobody can click is a distributed systems problem now. And because every IDE release can add a new first-run popup, the plugin ships an active dialog killer as the last line of defense.

Config, logs, and system dirs are split deliberately: config and logs live on the bind-mounted run directory, so after a failed CI run you get the full idea.log and the exact IDE config as artifacts. The system dir (caches, indexes) stays on the container filesystem, where it is fast and disposable. The MCP Steroid plugin ZIP is unpacked into the plugins dir before launch, and a floating license server is configured via -DJETBRAINS_LICENSE_SERVER so commercial IDEs like Rider skip the license dialog too — ours points at an internal JetBrains server, you would bring your own here.

VM options cover most of the first-run popups, but not all. A few consents live in files the IDE reads before any option is applied, so we write those ourselves before launch. IntelliJ records EULA and privacy-policy acceptance both in the config dir and in the JVM user preferences under the home directory; we pre-write them with a version far in the future (999.999) so the IDE always thinks the latest terms are already accepted:

# inside the IDE config dir
options/other.xml
early-access-registry.txt
options/AIOnboardingPromoWindowAdvisor.xml

# inside the agent home (JVM userPrefs + consent options)
.java/.userPrefs/jetbrains/prefs.xml
.java/.userPrefs/jetbrains/privacy_policy/prefs.xml
.config/JetBrains/consentOptions/accepted

The user-prefs files carry the future version — accepted_version=999.999, privacy_policy_accepted_version=999.999, eua_accepted_version=999.999, and so on — while the config-dir files switch off onboarding and the AI-promo window, and consentOptions/accepted records the usage-statistics choice. The exact list (there are a couple more, including an encoded Java-prefs path) is in IdeStartupConfig.kt. It is not glamorous, but it is the difference between a clean start and an IDE waiting on a dialog nobody will ever click.

With all of that written to disk, we finally run the launcher — and now we wait.

Waiting for IntelliJ to be ready

Since MCP Steroid is already a tool to access IDE internals, we use the tool itself to monitor IDE readiness, for that we pass the following steps:

poll the MCP Steroid port until it answers, then wait for the IDE window
register the JDKs in the IDE (essential for IntelliJ IDEA-based tools)
set the project SDK
trigger the Maven/Gradle import and wait for it to actually finish
wait for indexing and background progress to settle
resolve any remaining unknown SDKs

The first step is a plain poll — curl the MCP port until it answers, while failing fast if the IDE process dies or the log shows a startup failure (mcp-steroid.kt). Every step after that is a Kotlin snippet we send through the steroid_execute_code tool, which runs it inside the live IDE. Setting the project SDK, for example, is just the IntelliJ API (mcp-steroid-jdks.kt):

// runs inside the live IDE via steroid_execute_code
val sdk = ProjectJdkTable.getInstance()
    .getSdksOfType(JavaSdk.getInstance())
    .first { it.name == "21" || it.name == "temurin-21" || it.name == "corretto-21" }
edtWriteAction { JavaSdkUtil.applyJdkToProject(project, sdk) }

Triggering the import and, crucially, waiting for it to finish is the same idea — subscribe to ProjectDataImportListener (Gradle) or MavenImportListener (Maven), fire the refresh, and await the finished event instead of a blind sleep. The last step waits for Observation.awaitConfiguration to drain, then for the IDE to reach smart mode with zero running background progress indicators, held for several quiet rounds — indexing arrives in waves as libraries and their sources download, so a one-shot check returns too early. It also fails fast if progress freezes, so a stuck import does not hang to the timeout. The import and settle logic is in mcp-steroid-import.kt.

The end-user AI/Dev experience in the MCP Steroid project

An integration test must be designed to start from the IDE as a usual test.

I’ve seen many integration tests that were hard to start locally and only ever ran on CI. That makes them unique, and maintained only by their authors. With Agentic AI and coding agents we can’t afford that price – each test run must be as cheap and as fast as possible, because our AI Agents will call such tests many times.

From an AI Agent’s (or a developer’s) point of view, the whole machinery above collapses into one call:

@Test
@Timeout(value = 15, unit = TimeUnit.MINUTES)
fun `container starts and IDE becomes ready`() = runWithCloseableStack { lifetime ->
    IntelliJContainer.create(lifetime, IntelliJContainerOpts(consoleTitle = "ide-container"))
}

IntelliJContainer.create (intelliJ-factory.kt) builds the image, starts the container, brings up Xvfb → fluxbox → video → screenshots → xterm, deploys the plugin and the test project, launches the IDE, and blocks in waitForMcpReady(). That gets the IDE and the MCP server up; a test that needs a fully imported project calls waitForProjectReady() next — the JDK/import/indexing steps from above. The lifetime stack unwinds all of it in reverse order — ffmpeg gets its SIGINT before the container goes away, so the video file always has a valid trailer. After that, tests speak plain MCP over HTTP: steroid_execute_code runs Kotlin inside the live IDE, steroid_list_windows reports which dialogs are showing and whether indexing is in progress, steroid_take_screenshot returns what the agent sees.

Sending the live video to a browser tab

Now the fun part. Running an integration test with GUI version of an IDE. Are you curious to see what’s inside?

Screenshots came first — a scrot loop drops one PNG per second into the mounted volume, and that alone made failures debuggable. But flipping through PNG files is archaeology. During long agent runs we wanted to watch, live, like over a colleague’s shoulder.

The obvious answer is VNC, and we deliberately skipped it. VNC means a VNC server in the container, a client on every developer machine, one more port, one more password. Our real requirement was much weaker: one-directional, view-only, zero-install. Something a browser tab can show.

So the pipeline is: ffmpeg grabs the X display into a growing fragmented MP4 file, and a tiny Node.js server streams that file into a <video> tag as it grows. No VNC, no WebRTC, no websockets — HTTP chunked transfer, invented last century, works great.

The capture side (the capture + streaming driver) — 24 fps, and a keyframe every second (-g 24), which is load-bearing, as we will see:

ffmpeg -nostdin -y \
    -f x11grab -video_size 3840x2160 -framerate 24 -i :99 \
    -c:v libx264 -preset ultrafast -tune zerolatency -crf 28 \
    -pix_fmt yuv420p -r 24 \
    -g 24 -keyint_min 24 \
    -x264-params keyint=24:min-keyint=24:scenecut=0:rc-lookahead=0 \
    -movflags frag_keyframe+empty_moov+default_base_moof \
    -frag_duration 1000000 \
    -flush_packets 1 -fflags +flush_packets \
    /tmp/video/recording.mp4

Three details here cost us real debugging days, so read them as a checklist:

-movflags frag_keyframe+empty_moov turns the output into a fragmented MP4. A normal MP4 writes its index (the moov atom) at the end, when encoding finishes — useless for a file you want to play while it is being written. Fragmented MP4 is self-describing from the first fragment.
The keyframe interval must be forced down. x264’s default is one keyframe per ~250 frames, and each fragment starts on a keyframe — so the browser would sit on a black screen 20–50 seconds waiting for the first fragment to complete. keyint=24 (one second) makes playback start almost immediately. We set both the generic -g option and x264’s own keyint params, so the encoder and the muxer agree on one-second fragments.
ffmpeg writes to a container-local path, not the mounted volume. Docker Desktop’s virtiofs does not flush file data to the host while the writing process keeps the file open — the host sees a zero-byte file for the entire recording. So ffmpeg writes to /tmp inside the container, and a one-second rsync --inplace loop mirrors the growing file to the mounted volume. Closed files (like the scrot PNGs) flush fine; it is specifically the always-open file that bites. The streaming server below reads the same container-local file — the rsync copy exists only so the host keeps the recording as an artifact.

The serving side is a few hundred lines of Node.js baked into the image, more than half of it the dashboard page (video-server.js). The tail-follow core is under a hundred lines: keep the HTTP response open, remember a byte position, and every ~100 ms send whatever the file grew by (slightly simplified here):

// GET /video.mp4 — stream the growing file with chunked transfer encoding
function serveVideo(req, res) {
    res.writeHead(200, { 'Content-Type': 'video/mp4', 'Connection': 'keep-alive' });
    let position = 0;
    const scheduleNext = (ms) => setTimeout(sendChunk, ms);

    function sendChunk() {
        if (res.destroyed) return;
        const stat = getVideoStat();
        if (!stat) { scheduleNext(100); return; }

        if (stat.size > position) {
            const stream = fs.createReadStream(VIDEO_FILE, { start: position, end: stat.size - 1 });
            stream.on('data', (chunk) => { if (!res.destroyed) res.write(chunk); });
            stream.on('end', () => { position = stat.size; scheduleNext(75); });
        } else {
            scheduleNext(100);
        }
    }
    sendChunk();
}

GET / serves a small dashboard page — a full-viewport <video autoplay muted playsinline> element pointed at /video.mp4, the run ID in the header, and a status dot polling GET /status every 300 ms. When the harness shuts the server down, the page notices — repeated /status failures flip the dot to “stopped” and the tab closes itself. On the host side, the harness waits for /status to answer and runs open http://localhost:<mapped-8765>/ — on macOS the tab appears by itself the moment the stream is alive. On other platforms you follow the VIDEO_DASHBOARD= line from session-info.txt in the run directory, it is printed to the test output too:

waitFor(35_000L, "Video streaming server ready") {
    curl("$dashboardUrl/status").httpCode == 200
}
if (isMacOs) ProcessBuilder("open", dashboardUrl).start()

One browser caveat we learned the embarrassing way while recording the demo: Safari refused to play the growing fragmented MP4 — it waited for the first frame forever, Chrome handled the same stream fine. If you build this, test in Chrome first and save yourself an hour of blaming the encoder.

The latency of the whole chain — X11 frame to browser pixel — comes to a couple of seconds in our runs. For “is the agent stuck on a dialog or actually indexing”, that is more than enough.

The same pipeline, on camera

The demo video at the top of this post is this exact infrastructure, filmed. It runs the pipeline one level deeper: a macOS VM (via Tart) runs IntelliJ, which runs the integration test from the gutter, which boots the Docker container with the inner IDE, and Chrome pops up with the live dashboard streaming that inner IDE while a Claude agent drives its debugger. An IDE in a container inside a VM, watched from a browser tab — and every layer is exactly the code above.

Producing a readable video needed only three small overrides on top of the stock pipeline: the inner display drops from 4K to 1280×720 (so IDE fonts map ~1:1 onto video pixels), CRF goes from 28 to 20, and the x264 preset from ultrafast to veryfast. The demo checkout carries a small local patch that reads these from environment variables — candidates to land on main as proper knobs, not there yet. Upstream CI still runs the defaults: same code path exercised daily, slightly nicer pixels for the camera.

Re-implementing this for your own GUI tests

Nothing above is IntelliJ-specific. The recipe, condensed:

Base image: Debian/Ubuntu + xvfb, a small WM (fluxbox), x11-utils, xdotool, scrot, ffmpeg, fonts, and your app’s native GUI libraries. End it with CMD ["sleep", "infinity"].
Own the startup order from the outside. Start Xvfb via docker exec, poll xdpyinfo until the display answers, then the WM, then capture, then your application. Register cleanups in reverse.
Disarm every first-run dialog your application can show. Hunt them down one by one; each is a hang in CI. Budget real time for this — it took us the longest.
Health-check with fail-fast branches: process alive, log clean, endpoint answering. Never poll for success alone.
Screenshots first (scrot in a loop — trivial and already saves you), video second: ffmpeg x11grab → fragmented MP4 with 1-second keyframes → tail-follow HTTP server → <video> tag. Write to a container-local path and rsync to the volume if you are on Docker Desktop.
Publish three ports: your app’s API, the video dashboard, and a debugger.

We drive all of this from Kotlin, but it is nothing a shell script can’t do. The whole sequence, wired together — the two background loops (scrot, rsync) are the ones the sections above only described:

set -euo pipefail
cid=$(docker run -d --rm --init -p 8765 my-gui-image)   # the "sleep infinity" room
x() { docker exec -e DISPLAY=:99 "$cid" "$@"; }          # every command carries DISPLAY
x mkdir -p /tmp/video /tmp/shots

x Xvfb :99 -screen 0 1280x720x24 -ac &                   # 1. virtual display
x bash -c 'for i in $(seq 1 150); do xdpyinfo -display :99 >/dev/null 2>&1 && exit 0; sleep 0.1; done; exit 1'
x fluxbox &                                              # 2. window manager
x xclock &                                               # 3. your GUI app (stand-in here)
x bash -c 'while :; do scrot /tmp/shots/$(date +%s).png; sleep 1; done' &   # screenshots
x ffmpeg -f x11grab -video_size 1280x720 -framerate 24 -i :99 \
    -c:v libx264 -preset ultrafast -tune zerolatency -crf 28 -pix_fmt yuv420p \
    -g 24 -keyint_min 24 -x264-params keyint=24:min-keyint=24:scenecut=0:rc-lookahead=0 \
    -movflags frag_keyframe+empty_moov+default_base_moof -frag_duration 1000000 \
    -flush_packets 1 -fflags +flush_packets /tmp/video/recording.mp4 &   # 4. capture
x bash -c 'while :; do rsync --inplace /tmp/video/recording.mp4 /mcp-run-dir/; sleep 1; done' &
x node /usr/local/bin/video-server.js /tmp/video/recording.mp4 8765 &   # 5. stream
open "http://localhost:$(docker port "$cid" 8765/tcp | cut -d: -f2)/"        # 6. watch

The parts I would warn you about, honestly: the vmoptions dialog-hunting never truly ends — every IDE release can add a new first-run popup, and we find out when CI hangs. The 4K Xvfb display makes ffmpeg eat a steady chunk of CPU; on weak CI workers you may want 1080p. And the whole docker exec orchestration is more code than a docker-compose file — the flexibility is worth it for us, but it is a real trade-off.

Try it, steal it, improve it

Everything in this post is in the open jonnyzzz/mcp-steroid repository — the base and agent Dockerfiles, the X server driver, the video pipeline, and the video-server.js you can lift wholesale into your own project. To see it live:

./gradlew :test-integration:test --tests '*IntelliJContainerTest.container starts and IDE becomes ready'

On macOS a browser tab opens by itself on a live IDE booting inside a container; elsewhere, follow the VIDEO_DASHBOARD= URL from session-info.txt in the run directory.

If you build a variant of this for your own GUI testing — a different app, a different streaming trick, or you got WebRTC working where we settled for chunked HTTP — I genuinely want to hear about it. Tell me on LinkedIn or Twitter, or open an issue on the repo. And if the demo video is what brought you here: that is this same pipeline, just pointed at a camera.

Inside the Git Hooks: Tagging Every AI Agent Commit (Part 3)

2026-06-26T00:00:00+00:00

This is the third part of the series. Part 1 laid out the approach — keep the keys away from the agent, a git proxy as the hard boundary, git hooks as the lighter alternative. Part 2 was the build of the proxy. This one is the build of the hooks: the in-container layer you reach for when you own the image but not the network path.

The job is the same two questions — which commits did the agent produce, and did it touch only what we allowed? — answered from inside the sandbox instead of on the wire. A reminder from Part 1: this is best-effort observability, not a security boundary. A sufficiently determined agent in its own sandbox can defeat any of it. We build it anyway, because for the overwhelmingly common case where the agent is just doing its work, it gives you clean provenance for almost nothing — and the proxy is still there as the floor.

The plan: stamp a session-id trailer on every commit as it is written, keep that tag alive across amend/rebase/squash, and on push capture a restorable copy of what is going out. Four hooks do it — prepare-commit-msg, commit-msg, post-rewrite, pre-push — wired in through core.hooksPath. Let me walk the load-bearing parts.

Wiring: Hijack Every Hook Without Breaking the Repo’s Own

Part 1 covered the install: bake the hooks into the image root-owned and read-only, and point core.hooksPath at them in the system config so every repository the agent touches — /workspace and any clone it makes later — runs them. But core.hooksPath is greedy: it replaces $GIT_DIR/hooks entirely. The moment you set it, a project’s own .git/hooks/pre-commit stops firing. That is a regression you do not want to ship to developers who rely on their own hooks.

So the first thing each of our hooks does is run the repository’s own hook of the same name, honor its exit code, and only then do our work:

run_local_hook(name, args, stdin):
    # resolve the MAIN repo's .git even from a linked worktree, and as an absolute path
    git_dir = git rev-parse --path-format=absolute --git-common-dir
    local   = "$git_dir/hooks/$name"
    if is_executable(local):
        exec local with (args, stdin)   # its exit code becomes ours — honor a rejection
    # else: nothing to chain, carry on

Two traps hide in those four lines. First, recursion: if you resolve the hook path through core.hooksPath you point straight back at yourself and loop forever — so resolve the literal $GIT_DIR/hooks, never the configured path. Second, worktrees: a linked worktree has its own private gitdir, but the conventional hooks live in the main repo’s .git. --git-common-dir resolves to the main .git from anywhere, so a hook running inside a worktree still finds the project’s real hooks.

For hooks that receive data on stdin (pre-push, post-rewrite), there is a third trap: you need that stdin twice — once for your own logic, once to replay to the local hook — and command substitution silently strips trailing newlines and mangles blank lines. So buffer stdin to a temp file and replay the exact bytes:

stdin_buf = mktemp(); read all of stdin into stdin_buf
... parse stdin_buf for our own work ...
run_local_hook(name, args, stdin=stdin_buf)   # byte-for-byte, not via $(...)

One more constraint shaped the whole library: it runs on the macOS dev host and the Linux agent image, so it is written for bash 3.2 — no associative arrays, no namerefs. Portability is a feature when your hooks have to behave identically in two very different places

Tagging: One Trailer, Re-Collected Every Time

The tag itself is a git trailer — a structured Key: value line at the foot of the commit message, the same machinery as Signed-off-by:

Agent-Session: 2026-06-20T11:04:18Z-7f3a9c

The naive approach — append that line to the message — falls apart the first time someone squashes three commits or rebases a branch. Trailers get duplicated, buried mid-message, or dropped. The robust approach is to treat the trailer block as something you recompute on every message, not something you append once:

apply_tag(msgfile, sid):
    existing = all "Agent-Session:" values already in the message
    from_urls = session ids parsed out of any orchestrator URLs pasted in the body
    all = dedup(existing + from_urls + [sid])        # order-preserving, drop blanks
    body = message with the raw Agent-Session lines AND those URLs stripped out
    # re-emit ONE trailing trailer block, deduped, via the real tool
    git interpret-trailers --if-exists addIfDifferent --if-missing add \
        (one --trailer "Agent-Session: <v>" per v in all) <<< body  > msgfile

Three things matter here. We let git interpret-trailers place the block — it knows where the trailer section ends and how to keep it well-formed, which hand-rolled string-appending never gets right. We re-collect existing values before re-emitting, so a squash of commits from different sessions preserves every distinct session id and dedups identical ones — the audit trail survives history rewriting. And we also harvest ids out of any session URL the agent pasted into the message body, so a commit references its origin even when the trailer was never added by hand.

This runs in prepare-commit-msg, the primary tagging point. It fires for commit, amend, merge, cherry-pick and every rebase reword/squash/fixup path — and, as Part 1 noted, it is the one commit-time hook that git commit --no-verify does not suppress, which is exactly why the tag lives there.

# prepare-commit-msg
run_local_hook("prepare-commit-msg", argv, stdin)  || exit with its code
apply_tag(msgfile, resolve_sid())   || warn "tagging failed (continuing)"
exit 0

commit-msg is a thin backstop: prepare-commit-msg already covers every path, so this one only acts if the final message somehow has no tag — a guard against a future regression, never the normal route. It never blocks the commit

Where does the session id come from? From a fixed file the environment writes once per session — covered in Part 1. The resolver reads the first line of that file and sanitizes it into a trailer-safe token: keep the : so an ISO-8601 timestamp survives, drop / so it can never break a path you later build from the id. No file, or a blank one, means “skip tagging” — the hooks never block on a missing id.

Provenance Across Rewrites

When the agent amends or rebases, commit hashes change, and a tag-by-hash audit trail would lose the thread. post-rewrite closes that gap: git hands it <old-sha> <new-sha> pairs on stdin (space-separated — not tab, a detail that has bitten many a hook), and we append them to a per-session provenance log before delegating to the repo’s own hook:

# post-rewrite  (stdin: "<old> <new>" per line)
buffer stdin
for "<old> <new>" in stdin:
    append "<timestamp>  <mode>  <old>  <new>"  to  $ARTIFACTS/_provenance/<sid>.log
run_local_hook("post-rewrite", argv, stdin=buffer)

Now an old → new chain is recoverable even after the local history has been reshaped, and that log later travels with the push capture

Enforcement on Push: Re-Tag, or Bail Out

The commit hooks are easy to bypass — --no-verify, a tool that writes commits directly, an agent in a hurry. pre-push is the net that catches untagged commits before they leave the box. For each pushed branch it works out which commits are actually new, and if any of them lack the tag, it rewrites them and aborts the push so the corrected commits go out on the retry — what you might call “amend on push”.

The first subtlety is bounding the range. git hands pre-push a line per ref — <local-ref> <local-sha> <remote-ref> <remote-sha> — and the remote-sha is the authoritative “what the server already has” boundary. Except when it is all-zero, which means a brand-new branch: there is no remote tip, and if you naively take “everything reachable from local-sha” you will try to re-tag and capture the entire history of the repo. So for a new branch, fall back to the merge-base against the remote’s existing tracking tips; if even that cannot be found, do not guess — skip:

push_base(local_sha, remote_sha, remote):
    if remote_sha is not all-zero:  return remote_sha          # normal update
    tips = object names of refs/remotes/<remote>/*
    if no tips:  return ""                                     # cannot bound — caller must skip
    return git merge-base local_sha <tips...>                  # new branch: nearest known ancestor

(A small landmine worth flagging: git merge-base --remotes is not a thing. You have to enumerate the tracking-tip object names and pass them explicitly.)

With a base in hand, the enforcement pass is straightforward — only rewrite real refs/heads/* updates (skip deletions, and skip tags, since rewriting a tag ref would clobber it):

# pre-push enforcement
need_retry = 0
for "<localref> <localsha> _ <remotesha>" in stdin:
    skip if deletion, or localref not under refs/heads/
    base = push_base(localsha, remotesha, remote)   ; skip if empty
    if any commit in base..localsha has no Agent-Session trailer and we have a sid:
        retag_range(base, localref, localsha, sid)   # checkout-free, see below
        need_retry = 1
if need_retry: exit 1   # "re-tagged commits; please re-run your push"

Checking a commit for the trailer is a one-liner thanks to git’s own formatter — git show -s --format='%(trailers:key=Agent-Session,valueonly)' prints the value or nothing.

Re-Tagging Without a Checkout

Rewriting commits inside a hook is where most implementations reach for git rebase — and inherit all of rebase’s hazards: it checks out, it fights a dirty working tree, it can drop empty commits, and a failure leaves rebase state lying around. We avoid all of it with git commit-tree, which builds a commit object from a tree with no working-tree involvement at all:

retag_range(boundary, tipref, tip, sid):
    map = {}                                  # old sha -> new sha
    for c in  git rev-list --reverse --topo-order  boundary..tip:     # parents before children
        tree     = c^{tree}
        newparents = [ map[p] or p  for p in parents(c) ]             # remap to rewritten parents
        msg      = apply_tag(message_of(c), sid)
        newc     = commit-tree tree (-p each newparent) -F msg \
                       with c's original author/committer name+email+DATE preserved
        map[c]   = newc
    git update-ref tipref  map[tip]           # move the branch to the rewritten tip

Two details make it correct. Topological order, parents first, with a parent-remap table: when you rewrite a commit its children must be re-pointed at the new parent, or you fork history. And you must preserve the original author and committer identity and dates (GIT_AUTHOR_DATE, GIT_COMMITTER_DATE, and the name/email pairs) — otherwise the “re-tag” silently rewrites timestamps and authorship, which is both wrong and alarming in a blame view. Because nothing is ever checked out, a dirty tree is fine and an empty commit is fine; there is no rebase state to clean up if a step fails. This is the same “rewrite the whole pushed suffix in topological order” idea Part 2 mentioned for the proxy — here is the actual mechanism.

This only makes sense for a real branch ref. A delete, a detached-HEAD push, or a raw-SHA push has no branch to move, so the hook skips the rewrite there rather than do something surprising.

Capture on Push: a Restorable Copy, Never Blocking

After enforcement, a second pass captures what is going out — and unlike enforcement, capture never blocks a push. If any of it fails, the developer’s push still succeeds; the failure is logged and that is all. Each piece is bounded to the push range (base..local-sha), and if the range is empty or cannot be bounded, the whole capture is skipped rather than writing a degenerate artifact or dumping full history.

# pre-push capture (per pushed ref, in a subshell so a failure is contained)
base  = push_base(...)            ; skip if empty
range = base..localsha
skip if  git rev-list --count range  == 0          # nothing new

git bundle create   bundle.bundle  localref --not base                 # restorable clone source
git rev-list --objects range | git pack-objects --no-thin --stdout  > pack.pack
git format-patch -o patches/  range                                    # human-readable, per commit
git rev-list --merges range  > merges.txt                              # format-patch omits merges
copy the provenance log + write metadata.json

A few choices worth calling out:

The non-thin pack. Here is the contrast with Part 2 that I like. The proxy could not pass a flag to the agent’s git — it had to advertise the no-thin capability and let the client comply. In a hook, you are the client, so you just say git pack-objects --no-thin and get a self-contained pack you can re-index anywhere. Same goal, opposite side of the wire.
Stream with --stdout. Letting pack-objects write its own file drops it in the repo and then rename()s it into place — which fails with EXDEV when your artifacts directory is on a different mount than the repo (it usually is). Streaming to stdout and redirecting sidesteps the cross-mount rename entirely.
A bundle and patches and a pack. They serve different masters: the bundle is a one-file restorable clone source, the patches are what a human actually reads in review, and the pack is the raw objects. Cheap to produce, and you will be glad of all three the day you need to reconstruct a branch the agent deleted.
metadata.json built with a JSON tool, not string interpolation — branch names and URLs contain characters that turn hand-built JSON into a parsing bug. Let the tool escape them.

Everything lands under one auto-uploaded artifacts directory, and capture directories are named flat (slash in a branch name becomes a dash) so a branch like feature/x does not blow up the path.

The Limits, Again — Briefly

Everything in Part 1’s “hooks can be broken” still holds and I will not repeat it in full: core.hooksPath can be overridden from a per-user config or -c on the command line, git push --no-verify skips pre-push, and an agent with sudo can simply delete the hook directory. None of that is a defect — it is the definition of an in-sandbox layer. The hooks make the honest, common case observable and cheap; when you need a guarantee, that is the proxy’s job, because the proxy is the one control the agent cannot reach.

There is also a softer failure mode the hooks lean into deliberately: they warn rather than block. Every hook writes its warnings both to stderr and to a log in the artifacts tree, so a tagging or capture hiccup is visible after the fact without ever having stopped the developer’s commit or push. For observability, a tool that breaks the thing it observes is worse than no tool.

Where This Sits in the Series

Two builds, one goal. The proxy is the wall outside the sandbox that the agent cannot route around; these hooks are the diligent assistant inside it. Run the proxy when you control the network, run the hooks when you control the image, and run both when you can — the hooks enrich the common case while the proxy guarantees the floor.

And both feed the same thing: a complete, attributed record of what every agent did to the code. That record is not just an audit trail — it is the raw material for measuring a fleet of agents and making the next run better, which is the larger system I am building now and will write about separately.

If you implement a version of this — a smarter tagging scheme, a cheaper capture, a hook trick I missed — I would love to hear it on LinkedIn or Twitter. And if your team is wiring AI Agents into real infrastructure and wants a hand on the boundary and the audit trail, that is exactly the kind of problem I enjoy — reach out and let’s scope it together.

Git Documentation Used in This Post

githooks — prepare-commit-msg, commit-msg, post-rewrite, pre-push
git-config — core.hooksPath
git-interpret-trailers — recomputing the trailer block
git-commit-tree — checkout-free commit rewriting
git-rev-list and git-rev-parse — ranges, --git-common-dir
git-bundle, git-pack-objects, git-format-patch — the capture artifacts

Inside the Git Proxy: Capturing What an AI Agent Pushed (Part 2)

2026-06-22T00:00:00+00:00

This is the second half of What Did the Agent Just Push?. Part 1 made the case for a git proxy as the hard boundary, set up the one rule that makes it work — the agent never holds the real keys — and showed the enforcement half: rejecting any head outside the allow-list before a byte reaches the server. Enforcement bounds the blast radius, and it is the part you can ship first.

This part is the capture half: how the proxy actually gets the traffic, reads what the agent pushed, and keeps it recoverable. It is the deep end — protocol gymnastics, packfile internals, and the side-effects of sitting in the middle of git’s wire protocol.

Recall the shape of a push from Part 1: a ref-advertisement GET /info/refs?service=git-receive-pack, then a POST /git-receive-pack carrying pkt-line commands followed by a PACK. The proxy already parsed those commands and located the pack — everything below works on that pack.

Make HTTPS the Only Way In

From my standpoint, the SSH is the best transport one should use with Git. On the other hand, Git supports multiple transports, and the tricks we are covering here are similar for all of them.

In our case, we decided to use HTTPS with token authentication, that requires much less additional setup than SSH. We are not using git:// protocol in production, as you may assume. An agent that can open an SSH connection to the server, but we specify no access keys. The git proxy server URL is explained to the agent in prompt files. Agent has no token to access the original Git server via HTTPS.

For HTTPS you have two ways in. Either the agent’s https_proxy points at you and you MITM the TLS tunnel with your own CA installed in the sandbox’s trust store, or you publish a dedicated hostname the agent’s remotes point at directly. If you MITM, guard one thing: a client opens the tunnel with CONNECT host:443 and then sends an inner Host: header, and the two must match — reject a mismatch with 421 Misdirected Request, or an agent can CONNECT an allowed host and then ask for a different one, slipping past your per-host policy.

Once the bytes are yours, classify by an exact whitelist, not a prefix match. Git’s HTTP surface is small and each endpoint has a fixed method, query shape and content type:

Request	Operation	Notes
`GET …/info/refs?service=git-upload-pack`	read	smart fetch/clone discovery
`GET …/info/refs?service=git-receive-pack`	write	push discovery — classified early
`POST …/git-upload-pack`	read	smart fetch
`POST …/git-receive-pack`	write	smart push
`GET …/info/refs` (no query), `…/HEAD`, `…/objects/…`	read	dumb protocol, read-only by design
anything else	deny	reject reserved git paths with 400

Two things fall out of this. A push is a write the moment you see its discovery GET, not just at the POST — so the discovery request is the natural place to influence the push (we use it below). And under this whitelist the dumb protocol is read-only: the one write path you allow is smart receive-pack, which is the one place branch policy bites. (Git also has a legacy WebDAV-style HTTP push, git-http-push, that writes outside receive-pack — so deny DAV methods and those paths too, or a write could sneak in behind your back). One endpoint deserves a hard no: the dumb objects/info/alternates (and http-alternates) can point the client at object stores outside your proxy, so reject it — otherwise an allow-listed clone can quietly source objects from somewhere you don’t audit.

Capture: log every commit that was pushed

The proxy already knows which refs moved — that was enforcement, in Part 1. Provenance needs the commits. We want to read the pack and log each commit’s hash, author and subject — ideally store the pack too, for offline recovery.

There is one obstacle. By default, git is allowed to send a thin pack: a pack whose objects may be deltas against base objects that are not in the pack, because the receiver is assumed to already have them (gitformat-pack). A thin pack is efficient on the wire and miserable to parse in isolation — you don’t have the bases. (It is possible to request missing objects via the HTTP dumb protocol from the main Git server, I would not go this way).

The fix is a capability the protocol already defines. The server can advertise no-thin, and per the spec “A client MUST NOT send a thin pack if the server advertises the no-thin capability” (gitprotocol-capabilities). So the proxy rewrites the ref advertisement on its way back to the agent, injecting no-thin into the capability list. The agent’s git then sends a self-contained pack the proxy can parse on its own.

on GET /info/refs?service=git-receive-pack:
    strip_header(request, "Accept-Encoding")        # advert comes back uncompressed so we can rewrite it
    advert = fetch_upstream_advertisement()
    advert = inject_capability(advert, "no-thin")   # rewrites the first ref line's pkt-line and its
                                                    #   4-byte length prefix; mind the pkt-line limit
                                                    #   (65520 total / 65516 of payload)
    return advert
    # if no-thin won't fit, you have two choices: fail CLOSED (reject the push) if you need a hard audit
    # guarantee, or fall back to storing the raw thin pack and resolving it later against the upstream
    # objects — but do not pretend you walked it; a thin pack is not self-contained

Two honest caveats here, because the details bite:

no-thin does not turn off delta compression. It only guarantees that every delta’s base is inside the same pack. Git will still delta-encode a commit against another commit in the pack. So your pack walker must resolve in-pack commit deltas — but only the commit objects, which are few and small. Trees and blobs you can inflate-and-discard. Memory stays bounded by the commit set, not by the pack size. It means we can unpack the PACK file and read the commits, trees and the file BLOBs without looking for additional objects in the parent repository.
Self-contained packs are bigger than thin ones, and the gap is not subtle. In one test, injecting no-thin flipped a real push from a 490-byte thin pack (a ref-delta against a base the server already had) to a 5,697-byte self-contained one — and it overrode the client’s hardcoded --thin with zero cooperation from the agent. You can reproduce the effect yourself: push one small commit and compare the POST body size with and without no-thin in the advertisement (GIT_TRACE_PACKET=1 makes the framing visible). For a handful of commits that is a fine trade; for a 2 GB monorepo migration it is not. Know which one you are running.
BLOGs are not DIFFS. Git stores the whole file content in BLOGs, which are in the commit. BLOBs do not show what was changed, it’s the *actual content of the changed file**. Git packs BLOBs with delta compression on the PACK file. Should you want to find the actual change – you still need to access the parent commit objects -> all tree objects they reference -> and the BLOBs of the previous revisions. There can be multiple base objects in the merge-commits (aka commits with multiple parents).

Capture should happen after the server accepts the push, gated on the upstream result. The receive-pack response carries a report-status: unpack ok plus a per-ref ok <ref> or ng <ref> <reason>. A push can partially succeed — unpack ok and then ng on one ref — so don’t log a commit as “pushed” until the server says ok for the ref that carries it.

capture(response, pack):                           # runs only after the server accepted the push
    status = parse_report_status(response)         # truth is in the response body, NOT the HTTP code:
    if status.unpack != "ok": return               #   "unpack ok" can still be followed by "ng <ref> ..."
    accepted = [r for r in status.refs if r.result == "ok"]   # log ONLY the refs the server actually took
    if not accepted: return                        #   (a 200 can carry per-ref "ng" — never trust the 200)
    store(pack, key = f"{session_id}/{new_head}.pack")        # spool the pack to storage; no second copy
    for commit in walk_commits(pack, reachable_from=accepted):   # bases are in-pack thanks to no-thin, so
        log(session_id, commit.hash, commit.author, commit.subject)  # resolve commit deltas; drop blobs

Run the capture off to the side — a detached, best-effort task. If logging fails, the developer’s push must still succeed. Observability that breaks the thing it observes is worse than no observability

A Pack Is Snapshots, Not Diffs

There is a deeper limit worth stating plainly, because it surprises people and it decides what you can actually recover. Git stores snapshots, not diffs. A blob is the whole file, never a change to it; the deltas you see inside a packfile (thin or not) are a storage-and-transport trick — one object encoded against another to save bytes — and not the semantic change the agent made. When you resolve the pack you get back whole files: the complete new version of everything that was touched, never a tidy “here is what changed”. So the pack, by itself, tells you the new state, not the edit.

To learn what changed you have to diff a commit against its parent — and that is where the pack runs out. For commits inside the pushed range you are fine: a commit and its parent are both new, both in the pack, so you can diff them from the pack alone. The gap is the boundary — the first new commit’s parent is the old head, and the old head’s tree and the old versions of the changed files are precisely the objects the pack omits, because the server already had them. A non-thin pack is self-contained for its new objects, but it is silent about the past. (A thin pack looks tempting here, since its wire bytes encode the changed blob as a delta against the parent’s version — but that base blob still is not in the pack, so you cannot apply it without fetching the parent anyway). And you cannot cheat by faking an empty advertisement to make the client upload all of history: receive-pack does a compare-and-swap on each command’s old object id and rejects a mismatch, and rewriting that old id back would break a signed push.

This has a blunt consequence, and it is the real reason the next section exists. If the same file was touched across several commits, you can reconstruct the chain of new snapshots from the pack — but turning that into a readable history of changes needs the parent at every boundary. Lose those parent objects to garbage collection and the diff is gone for good, even though you faithfully archived every pushed pack. So capturing the agent’s pushes is necessary but not sufficient: you also have to keep the parents reachable. That is why you back up heads in git, not just archive packs — pin the heads so the old commits, trees and blobs survive long enough to tell you what actually changed.

Keep the objects alive: backup marker refs

There is a subtle way to lose the very data you just captured. The agent pushes to a branch, you record it, the agent (or a cleanup job) deletes that branch, and now those objects are unreferenced on the server and eligible for garbage collection. Your .pack in object storage survives, but the server no longer has them.

The cheap fix is to pin each captured head with a backup marker ref — a second, ref-only push that points a marker like refs/heads/_captures/session-<id>/<branch>-<shortsha> at the head the agent just pushed. Because the head’s objects are already on the server, this push transfers no objects at all: it sends the canonical empty pack (a 32-byte pack with zero objects for SHA-1 repositories). It is fast, and idempotent — on a name collision you resolve the existing marker and confirm it points at exactly new_head before treating it as success (the short SHA in the name is a convenience, not a proof of identity) — and it keeps the commits reachable and GC-safe until your retention sweep removes them.

after a successful capture of <new_head> on <branch>:
    marker = f"refs/heads/_captures/session-{session_id}/{branch}-{short(new_head)}"
    # a create command is "<zero-id> <new_head> <marker>"; the objects are already upstream, so this
    # push carries the empty pack. It uses the proxy's OWN credential and skips the allow-list — the
    # proxy writes these markers, the agent never does
    receive_pack_push(zero_id, new_head, marker)
    # on a collision, resolve the existing marker and verify it == new_head, then treat as success

Fire it as a separate, isolated request — not a second command piggybacked into the agent’s own push. That distinction is load-bearing. If you inject your marker command into the client’s receive-pack body and the client used git push --atomic, then a failure of your command (a name collision, a slow host) atomically fails the agent’s real push. You would have turned your insurance policy into the thing that drops the developer’s work. A standalone ref push, sequenced after the agent’s push reports ok, cannot couple to it, and it leaves the agent’s signed certificate untouched. Give it its own generous timeout (real git hosts can be slow) and never loop it back through the forward proxy. It is independent and best-effort — if it fails, you log it and move on.

One honest limitation, though, and it is worth saying out loud: signed pushes can take the marker away from you entirely. If the upstream requires a valid push certificate on the capture namespace, you are blocked both ways — you cannot add the marker to the agent’s own push (that alters its signed command list and breaks the signature), and a separate marker push would itself need a certificate, which the proxy cannot produce without a signing identity of its own. So under mandatory signed-push you have two options: give the proxy its own service key that the host trusts for refs/heads/_captures/* and let it sign its marker pushes — a different signature than the agent’s, made by the proxy as itself — or drop the server-side marker altogether and lean on the .pack you already captured into your own storage, which needs no ref on the host at all. The marker is insurance; when the host won’t let you buy it, your own captured pack is the policy that still pays out. You can workaround that by adding the secondary Git repository, that is synchronized with the main repository, it will be used to keep all the captured packs and markers, and without certificates.

Two more things make the markers pay off. First, they are exactly what saves you from a force-push or a delete: the marker pins new_head, so even if the agent later force-pushes its branch backward or deletes it, the objects stay reachable through the marker and survive git gc --prune=now. Drop the marker in your retention sweep and they become collectable again. Second, the marker is what makes recovery cheap: a warm repository that already has old_head fetches only the old_head..new_head range, and git log old..new plus git show reconstructs exactly the commits and patches that were pushed — force-push included, since A..B means “reachable from B but not A”. And this is where the snapshot/diff problem from earlier gets solved almost for free: because every push gets its own marker, the marker you wrote for the previous push pins what is now the parent of this one. The chain of markers keeps each boundary’s old side reachable, so the parent objects you need to compute a real diff are still there — you backed up the heads as you went, exactly as that section argued. One caveat from the real world: a custom ref namespace like refs/captures/* is not portable — some hosts honor only refs/heads/* and refs/tags/* and silently reject the rest, so a lightweight tag is often the most portable pin. Tags auto-follow into clones, though, so remember to clean them up after recovery

Side-Effects to Watch

A proxy that parses pushes is a man-in-the-middle by design, and a few of its side-effects bite in production rather than in the demo:

You have to buffer the command prefix before you can forward. The ref commands sit in front of the pack, so policy can’t run until you have read and held that much. You do not have to hold the whole pack in memory, though — once policy passes, stream or tee the pack to disk or object storage as it flows upstream. Either way, apply a size cap: an unbounded push, accidental or malicious, is a memory- and disk-exhaustion vector. This is buffering you are adding to the path; bound it deliberately.
Object ids are not always 40 hex characters. Most parsers quietly assume SHA-1. A repository using the SHA-256 object format has 64-character ids, a different empty-pack size, and will be rejected as malformed by SHA-1-only code. Either handle both hash lengths or document the limitation loudly — a silent “malformed push” on a SHA-256 repo is a miserable thing to debug (gitformat-pack).
Signed pushes sign the heads, not the pack — so read them, never edit them. git push --signed wraps the ref-update command list in a push certificate and GPG-signs it. The signature covers the affected heads — each old-id new-id refname update — plus the pusher identity, the destination URL and a server-issued nonce. It does not cover the pack: each new-id is a SHA, so signing the heads already certifies the objects by hash (gitprotocol-pack). The tempting conclusion is “then a proxy can never handle a signed push.” Not quite. The proxy never needs to touch that list — it reads it and decides: forward the certificate byte-for-byte when every head is allowed, reject the whole push when one is not. Enforcement, capture and markers all survive, because none of them edit the signed bytes. What signed push removes are the edit-the-list shortcuts — remap an old id, inject an extra ref, strip a disallowed one — each of which stops the signature verifying. Two consequences are worth pinning down:
- Filtering is all-or-nothing. You cannot quietly drop one disallowed ref from a multi-ref push; you forward it whole or reject it whole. Reject is the safe default, so the boundary still holds — you just lose the ability to partially allow a signed push.
- Relay the nonce untouched. The nonce lives in the advertisement; when you inject no-thin there, keep the push-cert nonce intact, or the client’s certificate will not verify upstream.
This is also the deeper reason capture and markers stay strictly out of band: they must never appear in the signed command list, or they would invalidate the very push they are trying to record.
HTTP success is not git success. Worth repeating because it is the easiest one to get wrong: a 200 from the server’s git backend does not mean the objects were stored. The truth is in the report-status body. Gate every capture on it.

None of these are reasons not to build the proxy. They are the difference between one that works on your laptop and one you can leave running in front of a fleet.

Where This Leaves You

That is the whole proxy: intercept HTTPS, force a parseable pack, gate capture on the server’s real answer, pin the heads so nothing you captured gets collected, and respect the protocol’s sharp edges — signed, atomic, partial-success. None of it is exotic; it is mostly careful reading of the Git wire-protocol docs linked below. The payoff is the thing Part 1 was after: you know exactly what every agent pushed, you can recover it, and only the heads you allowed ever change upstream

One more reason to learn these internals now rather than wait: the substrate is being rebuilt. The agent-first forges and next-generation version-control systems landing in 2026 — Cursor’s Origin and Epic’s Lore, both of which I touch on in Part 1 — still move objects, still pack and delta them, still have to tell you what an agent changed. The transport and the storage may shift under you, but pkt-lines, packs, thin-vs-self-contained, and “snapshots, not diffs” are the vocabulary you carry across all of it.

If you came straight here for the mechanics, the why — credential isolation, the lighter git-hooks alternative, and how to choose between them — is in Part 1. And the build of that lighter alternative — the in-container hooks that tag every commit and capture each push — is Part 3.

And if your team is building this for real and wants a second pair of hands on the proxy or the capture pipeline, this is exactly the kind of problem I enjoy. I am happy to consult and help you build it — reach out on LinkedIn and let’s scope it together.

Given all the information – my implementation we log all the pushes intercepted to a log file. We assume the Git server will keep all objects (in our case it does) and run pinning job outside of the main flow.

Git Documentation Used in This Post

The proxy is mostly careful reading of the public Git wire-protocol and format docs:

gitprotocol-http — the smart HTTP transport
gitprotocol-common — pkt-line framing and limits
gitprotocol-capabilities — no-thin, report-status
gitprotocol-pack — the receive-pack exchange, status report, push certificate
gitformat-pack — pack and thin-pack format, object hashing

For the deeper design notes, the Git project keeps a set of technical documents under Documentation/technical in the source tree — the protocol and format docs themselves were promoted to the top-level gitprotocol-* and gitformat-pack man pages linked above.

What Did the Agent Just Push? Auditing Git for AI Agents

2026-06-20T00:00:00+00:00

You hand a repository to an autonomous AI Agent and let it work. It clones, branches, commits, pushes, maybe opens a pull request. A few minutes later it tells you it is done. Two questions stay on the table, and neither has a comfortable answer:

Which commits did it actually produce? Not the summary it wrote for you — the real objects that landed on the server.
Are you sure it only touched the branches you allowed? One confused agent force-pushing main is a bad afternoon.

I have been running fleets of agents on the same codebases for a while now (see Orchestrating AI Fleets and the Git Fork Pattern), and the more agents you run, the more these two questions turn from curiosity into operational risk. This post is about answering them at the layer where the truth lives: git itself.

This post is the approach: why this is a git problem, the one rule that makes everything else possible, and the two mechanisms — a git proxy and git hooks — at a level you can reason about and act on. The deepest wire-protocol mechanics of the proxy are involved enough that they get their own follow-up; here we cover the shape and the decisions, with enough pseudo-code that your AI Agent can start implementing it in your stack.

Why This Is a Git Problem, Not a Policy Document

The naive answer is “just tell the agent which branches it may touch.” That is a policy. Policies are suggestions to a process that has a shell, a network, and a strong urge to finish its task. Agents are smart — give one a constraint phrased as a request and it may find a creative way around it, not out of bad intent but because working around obstacles is literally what we trained it to do.

So we do not rely on asking nicely. We rely on two properties:

Blast radius — only the heads we selected can change on the real server, enforced somewhere the agent cannot route around.
Provenance — every commit the agent created is captured and attributed, so we can recover, review, and audit it after the fact. This will be a great source for iterative learning later.

There are two ways to get there. The first is a git proxy that sits outside the agent’s sandbox — the hard boundary. The second is git hooks baked into the sandbox image — a lighter, best-effort layer for when you own the image but not the network path. We use the proxy as the real wall and treat hooks as extra telemetry. Here is why.

The One Rule Underneath Everything: Don’t Give Agents the Real Keys

Before any proxy or hook, there is a decision that makes the rest possible.

The agent never holds the credentials to the real git server.

Inside the sandbox, the agent gets a short-lived, scoped token that authenticates it to our endpoint — not to GitHub or Space, not to your internal git host. We inject that token the boring way: a git credential helper or a URL rewrite, never the real key on disk. The real push credential lives in the proxy, which the agent cannot read. When a push is allowed, the proxy authenticates upstream with its own service credential.

This is the property that survives a clever agent. A sandbox is a permission boundary, and inside its own sandbox an agent may have enough rights to do almost anything — read its environment, rewrite files, disable the tooling we installed. If the real key were anywhere in that environment, a determined agent could find it and talk to the Git server directly, and every control we built would be decoration. Because the key is not there, the worst an escaped agent can reach is the proxy — and the proxy enforces policy and holds the key. The blast radius is bounded by where the credential lives, not by how well-behaved the agent is

I want to note – the pattern how we are going to limit the agent push/pull operations is not standard, and potentially is hard-to-impossible to implement out of the box with GitHub, Space or any third-party Git host. Agentic AI opens yet another direction for us – software is now very cheap to build, given you know how.

Below we explain enough ideas to implement it. Git supports multiple protocols, including HTTP, SSH, ~~and Git over TCP~~. We decided to use HTTP below, but the same technique can be applied to other protocols as well, mainly to SSH.

Solution 1: The Git Proxy — the Hard Boundary

A git push over HTTP is a surprisingly inspectable conversation. If you put a proxy on the path between the agent and the server, you can read the whole exchange, decide whether to allow it, and capture everything that flows through — without the agent being able to tell the difference, and without it being able to go around you, because it has nowhere else to go

What a push actually looks like on the wire

Git’s “smart HTTP” protocol does a push in two HTTP requests (gitprotocol-http):

Ref advertisement — GET /info/refs?service=git-receive-pack. After a # service=git-receive-pack header line and a flush, the server lists every ref and its current object id, and on the first ref line it appends a NUL byte followed by the capabilities it supports.
The push itself — POST /git-receive-pack. The body is a sequence of commands (“update refs/heads/x from <old> to <new>”) followed by a PACK — the binary blob of objects being transferred.

The advertisement, the command list and the status report are all framed in pkt-lines: a 4-byte hex length prefix in front of each chunk, with a special 0000 “flush” packet as a separator (gitprotocol-common). The PACK payload itself is raw bytes that follow the command section. That framing is the whole reason a proxy can parse the stream cheaply — you always know where the next record starts.

Enforcement: reject the heads you didn’t allow

This is the easy and most valuable half. The ref-update commands sit in plain pkt-lines at the front of the POST body, before the pack. Parse them, compare each target ref against your allow-list, and refuse the ones that don’t match — before a single byte reaches the real server.

on POST /git-receive-pack(body):
    # read pkt-lines until the 0000 flush that ends the
    # command list; the pack begins right after it.
    # --signed wraps the commands in a push-cert ...
    # push-cert-end block (then push-options): parse the
    # commands from inside it and forward the cert
    # byte-for-byte — the signature covers it
    commands, pack = split_at_pack(body)

    for (old, new, ref) in parse_commands(commands):

        # e.g. only refs/heads/agent/* may change
        if not allowed(ref):                      
            return reject(ref, "ref not in allow-list")

        if is_delete(new) and not allow_deletes(ref):
            return reject(ref, "deletes not permitted")

    # only now, with our real credential
    response = forward_upstream(body)

    # the audit trail — the capture step, in a follow-up
    capture(response, pack)

Because the proxy holds the credential and the agent does not, this is a wall, not a request. An agent that dislikes the wall cannot push around it — its token only opens the proxy.

That is the enforcement half — the part that bounds the blast radius, and the part you can ship first. The other half is capture: getting the traffic to the proxy in the first place, reading the pushed pack, logging every commit, and keeping the objects alive for recovery. It turns out to be a meaty topic on its own — MITM and endpoint classification, thin packs and the no-thin trick, why a pack gives you whole files and not diffs, backup marker refs, and the side-effects of sitting in the middle of a signed, atomic protocol — so I’ll give the build its own follow-up post. If you only ever do one thing, do enforcement; if you want the audit trail too, that follow-up is where it gets built.

Solution 2: Git Hooks — the Lighter, In-Container Alternative

The proxy needs you to control the network path. Sometimes you don’t — you control the image the agent runs in, but the egress is whatever it is. In that case git hooks get you most of the provenance, with none of the wire-protocol work.

The idea: stamp every commit with a session id as it is created, and capture artifacts on push.

prepare-commit-msg is the primary tagging point. It fires for commit, amend, merge, cherry-pick and the rebase reword/squash/fixup paths — and, crucially, it is not suppressed by git commit --no-verify (unlike pre-commit and commit-msg, which are). That is exactly why it is the primary tag: the one commit-time hook the agent can’t wave away. Have it append a trailer like Agent-Session: <id> to the message. A trailer is a structured Key: value line at the end of the message, the same machinery as Signed-off-by, so it survives rebases and you can grep for it later. Re-collect existing values and re-emit one block, so a squash preserves distinct sessions and dedups identical ones.
commit-msg is a thin validator — a second chance to confirm the trailer is present. Like the others, it only observes and never blocks the commit.
post-rewrite records old→new commit mapping after amend/rebase, so provenance follows the rewrite. (Mind the detail: its stdin is space-separated, not tab.)
pre-push is the enforcement-and-capture net. If it finds an untagged commit, it rewrites the whole pushed suffix in topological order with a checkout-free git commit-tree — no rebase, so no dirty-tree or empty-commit hazards — re-parenting each descendant onto its rewritten ancestor (and keeping every parent of a merge with repeated -p), then moves the branch and aborts the push so the corrected commits go out on the retry. That auto-rewrite only makes sense when the pushed <local-ref> is a real refs/heads/* branch; for a delete, a detached-HEAD push or a raw-SHA push there is no branch to move, so abort with instructions instead. On a clean push it captures — a git bundle (validate it with git bundle verify), per-commit patches, and a small metadata.json. Be honest about what this is, though: pre-push fires before the remote answers, so these are attempted-push artifacts — they record what the agent tried to send, not what the server finally accepted. For “what actually landed”, you need the proxy or a server-side hook.

To make the hooks apply to every repository — /workspace and any clone the agent makes later — wire them through core.hooksPath rather than copying files into each .git/hooks. Bake the hooks into the image root-owned and read-only, and set core.hooksPath in the root-owned /etc/gitconfig.

# Dockerfile (sketch)
COPY --chmod=555 --chown=root:root githooks/ /opt/githooks/
RUN git config --system core.hooksPath /opt/githooks
# resolve one canonical Agent-Session id in the entrypoint and write it to a fixed, well-known FILE
# (say /.session_id) that the hooks read — NOT an env var. A file is harder for the agent to unset or
# forge than the environment, and it survives sub-shells and `git` wrappers. Pre-create it owner-writable
# in the image (the entrypoint runs non-root); a missing or blank file just means "skip tagging" — the
# hooks still never block.

Why a file and not the obvious environment variable: the agent shares that environment. It can unset the variable, override it, or spawn git with a scrubbed env — the same class of bypass we just saw with GIT_CONFIG_*. A fixed path the entrypoint writes once, before the agent runs, is a smaller target. It is still inside the sandbox, so still defeat-able by a determined agent — but it removes the easy footgun, and that is the right bar for a best-effort layer.

Hooks Can Be Broken by the Agent or an IDE — and That’s Expected

Here is the part most write-ups skip, and the reason hooks are the second solution, not the first.

A hook runs inside the agent’s sandbox, which means it lives in a place where the agent often has enough permissions to defeat it. The honest list of holes:

Config precedence runs the wrong way for you. It is tempting to think /etc/gitconfig is authoritative because it is “system”. It is the opposite — system config is the lowest precedence, and a per-user ~/.gitconfig overrides it (git-config, SCOPES). So an agent that can write its own global config can point core.hooksPath somewhere empty. Root-owned read-only files raise the bar, but precedence is not on your side.
And precedence is only the easy bypass. git -c core.hooksPath=… on the command line, a repo-local .git/config, or the GIT_CONFIG_COUNT / GIT_CONFIG_NOSYSTEM environment variables all override the system file too. A git wrapper that scrubs GIT_CONFIG_* and refuses -c core.hooksPath= would close these, but now you are maintaining a wrapper — and you are still inside the sandbox.
git push --no-verify skips pre-push entirely, unless you also wrap the git binary.
A sufficiently privileged agent can just remove the hooks. In a dedicated sandbox the agent may well have sudo or own the filesystem. If it decides to delete /opt/githooks, it can.

We do not treat any of this as a defect. Hooks are best-effort observability, not a security boundary. They give you clean provenance for the overwhelmingly common case where the agent is doing ordinary work, and they cost almost nothing. The moment you need a guarantee rather than a strong default, you are back to the proxy — because the proxy is the one control the agent cannot reach, and the credential isolation behind it is the one fact the agent cannot wish away.

That is the whole reason for the ordering. Proxy first, hooks as a helpful layer on top.

The Two Solutions, Side by Side

Property	Git Proxy (Solution 1)	Git Hooks (Solution 2)
Where it runs	Outside the sandbox, on the path	Inside the sandbox image
Can the agent bypass it?	No — it holds the keys, agent can’t	Yes — `--no-verify`, sudo, config
Enforces blast radius	Yes, before bytes reach the server	Best-effort (`pre-push` only)
Captures every commit	Yes (raw pack if `no-thin` skipped)	Attempts only, until a hook is off
Needs control of the network	Yes	No
Needs control of the image	No	Yes
Right mental model	A wall	A diligent assistant
Access possible outside of the session	Controlled by the proxy	Controlled by the Git Hosting

If you can run only one, run the proxy. If you can run both, the hooks make the common case richer while the proxy guarantees the floor.

The long answer actually is to design and use the combination of both. For example, I believe that we still need to keep track of all changes that were created by the AI Agent; it will help in the future us better understand what happened, how quality and other project metrics were affected. It will definitely help us to improve the agent’s performance and the project’s quality. This information will be necessary to measure the impact of the agent on the project.

We are missing Solution 3. The pipeline to mark, backup, and analyze all the AI-made changes at scale. I’m building it now.

Meanwhile, Git Itself Is Being Rebuilt for Agents

It is worth zooming out, because the ground under all of this is moving. In June 2026 two projects landed that take the same problem somewhere very different. Cursor Origin (overview) is a git forge built for agents rather than humans — its whole premise is that when dozens of background agents clone, branch and push against one repository at once, git’s human-scale assumptions become the bottleneck. And Lore, Epic Games’ new open-source version control system, rebuilds the substrate — content-addressed, local-first, and equally at home with code and the multi-gigabyte binaries that games and film drag around.

Two different layers, one signal: version control is being rethought for a world where most commits are not typed by a person. That does not retire anything in this post. Most teams still run plain git against GitHub, GitLab or an internal host today — and even an agent-first forge has to answer the same two questions this whole write-up is about: which commits did the agent produce, and did it touch only what we allowed? The boundary and the audit trail travel with you whatever the substrate; the proxy and the hooks are how you get them on the git you already have.

Conclusion: AI Automation Is Not Just Running the Jobs

It is easy to think the hard part of agent automation is getting the agent to do the work. It is not. The agent does the work happily. The hard part is everything around the work:

Bound the blast radius. Assume the agent is capable and occasionally wrong, keep the real credentials out of its reach, and put enforcement somewhere it cannot route around. A boundary the agent can edit is a preference, not a boundary.
Collect the data. Every commit, every push, attributed to a session and stored. Not because you expect disaster, but because that log is the raw material for the next iteration — which prompts produce which diffs, where agents go off the rails, what a good run looks like versus a bad one. You cannot improve a system you cannot see, and an agent that pushes into a black box teaches you nothing.

That second point is the one our team keeps coming back to. The capture is not just an audit trail; it is a feedback loop. The same data that lets you recover a lost branch lets you measure your fleet and tune it — the kind of self-improvement loop I wrote about in the AI Agent roadmap research and the multi-agent orchestration work. Give agents room to run, take away the keys to the things they shouldn’t touch, and write down everything they do. Then read it back and make the next run better.

The mechanics of the proxy — how it intercepts the traffic, forces a parseable pack, logs every commit, and pins the objects so nothing gets garbage-collected — are involved enough to deserve their own follow-up post: Inside the Git Proxy: Capturing What an AI Agent Pushed (Part 2). And the lighter, in-container alternative — the git hooks that stamp a session id on every commit and capture each push from inside the sandbox — gets its own build in Inside the Git Hooks: Tagging Every AI Agent Commit (Part 3).

If you have built something similar — a different boundary, a smarter capture, a hook trick I missed — I would love to hear it. Find me on LinkedIn or Twitter, and tell me what your agents tried to get away with.

And if your team is wiring AI Agents into real infrastructure and wants a second pair of hands on the isolation and audit layer — the proxy, the credential boundary, the capture pipeline — this is exactly the kind of problem I enjoy. I am happy to consult and help you build it: from a quick architecture review to designing the whole thing with your team. Reach out on LinkedIn and let’s scope it together.

Git Documentation Used in This Post

Everything above leans on the public Git documentation rather than any one implementation:

gitprotocol-http — the smart HTTP transport
gitprotocol-common — pkt-line framing
gitprotocol-capabilities — capabilities on the advertisement
githooks and git-config — hooks and core.hooksPath
git-interpret-trailers, git-bundle, git-commit-tree, git-index-pack — the hook toolbox

The wire-protocol and packfile internals — report-status, the pack format, push certificates — are covered in the follow-up post.

devrig: MCP Steroid Goes Level 3 — Autonomous IDE Setup for AI Agents

2026-06-02T00:00:00+00:00

If you have ever wired an AI Agent to an IntelliJ-family IDE through MCP, you know the routine: start the IDE, install the plugin, write down the port number, paste an HTTP URL into your agent’s config, restart the agent, hope you guessed which project the agent is supposed to see. Two IDEs open? Pick one. PyCharm and IntelliJ both running? Hope the port-number lottery favors you.

The new devrig CLI in MCP Steroid 0.100 kills that ritual. One stdio command — devrig mcp — and the agent gets a deterministic view of every IDE, every project, every window, no matter how many are open. If the right IDE is not running, devrig downloads it and starts it. And the agent’s MCP config? devrig install claude writes it for you — and from then on devrig mcp connects to every running IDE that has the MCP Steroid plugin installed.

This is what I call Level 3 of MCP Steroid: the agent is no longer the customer of a manually configured environment. The environment configures itself for the agent.

The headline number: the e2e test in DevrigManagedBackendAgentE2ETest.kt provisions an IntelliJ Community via devrig, opens Keycloak, and asks Claude to find PSI usages of org.keycloak.models.UserModel over its ReferencesSearch. The verified count is 4,633, in one prompt.

I want to show what it does, why it matters, where the trade-offs are, and — most importantly — ask what you would have it do next.

The three levels

I have been organizing the work in my head as three levels of autonomy. They line up roughly with Swarmia’s five levels of coding-agent autonomy, but applied to the agent ⇄ IDE boundary:

Level 1 — Manual. The user starts the IDE, installs the plugin, copies the port into the agent’s MCP config. Every step is human. This is where most teams still live.
Level 2 — Auto-discovery. The agent’s MCP client finds whichever IDE happens to be running, reads its open projects, routes calls correctly. The user still has to launch the IDE.
Level 3 — Fully autonomous lifecycle. devrig install writes the agent’s MCP config. devrig backend download/start brings the IDE into existence. Open project, route calls, stop cleanly. The human is out of the loop.

devrig is the first thing in the MCP Steroid family that delivers Level 3 end-to-end.

A quick map of where we were

The original MCP Steroid post argued that AI Agents need the full IntelliJ runtime, not a curated list of tool functions. The MCP standard defines two transports: stdio and Streamable HTTP. JetBrains’ own bundled MCP Server picks HTTP — it listens on a port and the agent dials in. Stdio still dominates IDE use cases though (~67 % of deployments in 2026 per a recent adoption survey), and for good reason: it maps 1:1 to “launch a subprocess; talk to it over pipes”. No ports, no DNS, no firewall, no “which IDE answered?”.

For one user, one machine, one IDE, the HTTP model is fine. For a real agentic-coding workstation it is a small disaster:

Real-world setup	What breaks
IntelliJ + PyCharm + Rider open simultaneously	Each grabs a different port; agent picks one
Two windows of the same product	Same port, two projects, ambiguous routing
Project on a remote dev machine	Manual SSH tunnel, manual port-forward
First-time setup on a clean laptop	Install IDE → install plugin → restart → …
Background batch job needing an IDE	“Please leave the IDE open” — for hours

This is the swamp devrig is built to drain. The product website calls it “one bridge, every IDE” — one devrig mcp process connects your AI Agent to all the IntelliJ-family IDEs running on your machine at once, across projects, and can download and start more on demand.

What `devrig` actually is

Architecturally, devrig is the smallest possible thing the agent talks to: a Kotlin-based command-line program that speaks MCP over stdio on its front and routes everything else to whatever IntelliJ-family IDE makes sense. The same binary doubles as an admin CLI you can run by hand.

One scope note, because it is a common question: today devrig is a suite that bridges the agent to IDEs running the MCP Steroid plugin — that is what gives the agent the full IntelliJ runtime. It is not a front-end for the IntelliJ-native MCP Server (JetBrains’ own bundled one). devrig already detects those IDEs so it can offer to fix them, but it does not route the agent through them yet. Speaking to the IntelliJ-native MCP Server directly — so an agent reaches a stock IDE with no plugin installed — is planned for a future release.

The current subcommand surface, straight from devrig --help:

The agent only ever sees devrig mcp. Everything else is for humans or CLI integrations, or for swarm scripts (run-agent.sh and friends) orchestrating fleets. devrig ships without a bundled runtime and is compiled for Java 25: provide a JDK 25 on PATH, via DEVRIG_JAVA_HOME. Amazon Corretto 25 works; so does any other JDK 25.

We are going to automatically fetch the required runtime JDK in the next versions.

`devrig install` — the new entry point

For most users, the very first command they run is the one that wires devrig into their coding agent. It landed in 0.100 to close issue #64:

Three flavors, all delegating to the agent’s own CLI under the hood so the resulting config is exactly the one the agent itself expects:

devrig install claude → claude mcp add --scope user mcp-steroid -- devrig mcp
devrig install codex → codex mcp add mcp-steroid -- devrig mcp
devrig install gemini → gemini mcp add --type stdio --scope user --trust mcp-steroid devrig mcp

These three syntaxes were the single biggest source of setup failures while scripting the demo. Three CLIs, two of which have the -- argv-forwarding gotcha, one with a different flag spelling each release. The bug-of-the-week was forgetting the mcp subcommand and registering devrig with nothing after it — at which point the agent tried to run plain devrig, got the help banner on stdout, and reported “MCP server initialize failed” with no explanation. The devrig install subcommand makes that class of failure unrepresentable. (The subcommand was originally spelled mpc; mcp is now the canonical name and mpc stays as a hidden alias so older registrations keep working.) Re-running devrig install is a safe, idempotent upsert: it finds every prior devrig/mcp-steroid entry and consolidates them into one.

It also records JAVA_HOME explicitly so the registered MCP command is reproducible — the JDK the user actually has on their host is the one the agent will use, not whatever the user’s shell JAVA_HOME resolves to next month.

Discovery, the unglamorous backbone

Pop a few JetBrains IDEs open and run devrig backend. On my workstation right now, with one real IntelliJ open and one previously-downloaded managed Community sitting on disk:

Two paths feed that list:

Marker files — every IDE with MCP Steroid installed writes a schema-versioned JSON file at ~/.mcp-steroid/markers/<pid>.mcp-steroid carrying its pid, MCP URL, IDE identity, plugin version, bearer token, and a per-JVM bootHash. devrig reads them and subscribes to each IDE’s NDJSON projects-stream endpoint for a live snapshot of open projects.
Active port scan of 127.0.0.1:63342..63361 (the IntelliJ built-in Netty server, which picks the first free port from 63342) and :64342..64361 (JetBrains’ own bundled MCP server). This catches IDEs running without MCP Steroid so we can offer to fix them.

devrig backend lists managed backends too — IDEs devrig itself downloaded under ~/.mcp-steroid/backends/ — even when they are not currently running. That’s the second row in the screenshot. Discovery, install state, and lifecycle state share one consolidated view, and every row carries the plugin version so an out-of-date MCP Steroid is visible at a glance.

Names are not the same string the IDE uses internally. The devrig naming spec defines an exposed form: slug(originalName)-projectHash, where projectHash is a SHA-256-derived base62 string (8 chars) over canonical inputs. Base62 was a deliberate switch from base64url — the earlier scheme produced exposed names like mcp-steroid-gumou--- whenever the encoder spilled trailing - / _ characters. Identifier-safe alphabet, always 8 chars, stable across IDE restarts.

Routing across many IDEs, many projects

When two IDEs both have a project called myapp open, what does the agent see? Names alone are ambiguous, and the agent will not be impressed by “tried to open myapp, got the wrong one”.

Same scheme as above — <slug>-<projectHash> — applied per project per IDE pid. Two myapps become two distinct exposed names. Windows route the same way (used by screenshots and window-targeted actions).

There is one subtlety worth knowing, and it is where the first follow-up to 0.100 landed. In 0.100, when the agent called steroid_open_project, devrig guessed the target IDE — any running managed backend first (the agent’s own sandbox), the newest discovered IDE otherwise. That guess is fine until two IDEs could each plausibly answer. So issue #87 made the choice explicit: steroid_list_projects now self-describes a backends[] list, each entry carrying an opaque, routable backend_name, and steroid_open_project now requires that backend_name on the devrig surface to say exactly which IDE opens the project. It is a devrig-only parameter — a direct one-IDE connection does not expose it, and ignores it if sent — and an unknown value comes back as a self-correcting error listing the routable ones. The guidance the agent gets: prefer the backend that already has the same repository — or a sibling git worktree of it — open. One catch worth repeating in the prompt: those ids are not stable across IDE restarts, so the rule is re-read steroid_list_projects, don’t cache backend_name.

Auto-downloading and auto-starting a managed IDE

The biggest jump from Level 2 to Level 3. devrig backend download (no id) lists what is available, and declares which IDEs are compatible with the plugin version that ships in this build of devrig:

A few interesting things here:

Community editions come from the real GitHub releases of idea-community / pycharm-community, not from the marketing-version JetBrains feed — the JetBrains products API doesn’t surface true IC/PC release artifacts in a stable enough way. devrig resolves them directly from the open-source repos.
Android Studio comes from the Quail canary feed — there is no data.services.jetbrains.com row for it, but Quail publishes a canonical SHA-256 per archive.
Plugin compatibility is gated at download time. Every row carries the IDE’s build baseline (e.g. 261). The bundled MCP Steroid plugin’s since-build / until-build from plugin.xml is read at startup, and IDEs whose baseline falls outside the supported range are marked incompatible — devrig backend download refuses to fetch them rather than letting the agent land in an IDE that silently won’t load the plugin. Primary baseline today: 2026.1 (261), with 2026.2 EAP (262) as secondary verification. 2025.3 (253) is dropped.
Archive caching is hash-qualified. Community and Ultimate archives that happen to share a filename cannot collide; both unpacked trees carry a SHA-derived prefix.

$ devrig backend download idea-ultimate
… resolving IDE 2026.1.2 from data.services.jetbrains.com …
… downloading idea-ultimate-2026.1.2 (≈ 1.2 GB) …
id:       idea-ultimate-2026.1.2
state:    installed
install:  /Users/jonnyzzz/.mcp-steroid/backends/idea-ultimate-2026.1.2
cache:    /Users/jonnyzzz/.mcp-steroid/caches/idea-ultimate-2026.1.2

$ devrig backend start idea-ultimate
pid:    67890
log:    /Users/jonnyzzz/.mcp-steroid/caches/idea-ultimate-2026.1.2/logs/managed.log
config: /Users/jonnyzzz/.mcp-steroid/caches/idea-ultimate-2026.1.2/config

The started IDE is headless from the user’s perspective — it can stay in the background as long as the agent needs it, and devrig backend stop reaps it cleanly. A global lock serialises download / start / stop so two concurrent agents can’t trash the same on-disk install. Managed backends emit anonymous PostHog beacons (devrig_session_initialized, devrig_exec_code, start / stop, errors) so I can tell whether agents are actually using the lifecycle or working around it.

The whole flow, end to end

The sequence above is exactly what the end-to-end test does, and it is the cleanest demonstration of Level 3 I have. In one prompt, with no MCP config and no IDE running up front: devrig install claude, then devrig backend download idea-community, devrig backend start idea-community, and then Claude opens Keycloak and runs JavaPsiFacade.findClass("org.keycloak.models.UserModel") → ReferencesSearch.search(psiClass).findAll().size. The return value is real:

4,633 PSI usages of org.keycloak.models.UserModel — in one prompt.

It comes straight from the e2e test (DevrigManagedBackendAgentE2ETest.kt). One honesty note: the IDE archive and Keycloak’s Maven indices are pre-warmed before the run, so the wall-clock stays short — but that only changes the timing. The agent’s actual work — provision the IDE, open the project, run the PSI search — is byte-identical whether the caches are warm or cold.

What 0.100 changed beyond `devrig`

The 0.100 release notes (GitHub release v0.100) cover the full delta, but the agent-facing items worth naming alongside devrig:

Typed modal mode for steroid_execute_code. The earlier mix of dialog-killer and allow-modal booleans is replaced with one explicit parameter: smart_non_modal (default — close stray dialogs, verify non-modal, commit + refresh, wait for smart mode), non_modal (assert only, no mid-run policy), or unleashed (intentional modal-dialog workflows).
The tool surface narrowed from 10 → 8. steroid_action_discovery and steroid_apply_patch are gone; the same workflows live as recipes (mcp-steroid://ide/action-discovery, mcp-steroid://ide/apply-patch with the in-script applyPatch { } DSL) reached only through steroid_fetch_resource. The IDE-conditional context the resource carries is the reason for not advertising recipes via the generic MCP resources/list.
steroid_input switched to window_id (from steroid_list_windows / steroid_take_screenshot). Breaking change to that tool’s input — worth flagging if you have scripts pinning the old screenshot_execution_id.
The devrig↔plugin wire contract is pinned, additive-only, with contract tests covering the full steroid_execute_code parameter set (issue #23).
Token-efficient tabular output. printCsv / printToon script-context helpers let steroid_execute_code return compact tables without token bloat (#34 / #35).
Human review mode is gone in full — the review.* registry keys, the mcp.steroid.review.mode system property, the storage helpers, the docs and website pages. The product is focused on agent-driven IDE execution rather than a human-gated review step.

Pros — what changes for the agent

One transport, one server. devrig mcp is the only thing the agent’s MCP config has to know. No ports, no URLs, no per-machine drift. Stdio is the dominant IDE transport for a reason.
The install is a single command. devrig install claude|codex|gemini writes the agent config the way the agent itself expects. The three-CLI argv-forwarding tax is paid once, in devrig, not on every user’s machine.
Multi-IDE, multi-window, multi-project — out of the box. Hash-suffixed routing makes “which one?” disambiguation explicit; base62 hashes are identifier-safe.
Compatibility gating. If the bundled plugin can’t load in a given IDE build, that IDE is flagged at download time. Fewer silent dead-ends.
First run is one command, not a setup checklist. No “go install the IDE first” step in the agent’s bring-up guide — devrig backend download is the install step (a real download the first time, cached after).
Background mode is normal. Spinning up an IDE for a 4-hour batch job stops being weird; it is just start + stop. The Docker test matrix exercises that path daily.
Inspectable. devrig backend --json and devrig project --json give swarm scripts a deterministic snapshot to make decisions on. Managed backends report their lifecycle via PostHog beacons.

Cons — and I want you to read these honestly

JDK 25 is a hard requirement. devrig doesn’t bundle a JVM in 0.100; the host must provide Java 25. A bundled-runtime story is on the roadmap (the deployment spec exists), not in the box yet.
Port-range assumption. The active scanner pings exactly 63342..63361 and 64342..64361. Anything outside those ranges is invisible. Right default in 2026; still a default.
Managed-IDE operations are one-at-a-time. A global lock prevents concurrent download / start / stop. Two agents on the same machine launching different IDEs in parallel will serialise. Probably the right call; worth knowing.
Detection lag. A crashed IDE stays in the routing table for ~2 seconds until the next poll.
The trust dialog can still block a fully-autonomous open. On 2026.1.2 the project-trust auto-grant regressed (issue #65): a freshly opened project can stall on the Trust dialog — the one human gate that quietly defeats the “human out of the loop” pitch until someone clicks it. Embarrassing for a Level 3 story, and it belongs near the top of the fix list.
Multi-IDE labeling is still settling. With several IDEs open, steroid_list_projects / steroid_list_windows could mislabel which backend a project belongs to (issue #89). The explicit backend_name model above points the right way, but making the labels themselves reliable across many IDEs is the actual fix, and that part is still in flight.
Plugin install on user-launched IDEs still has a human seam. devrig backend provision prints instructions — it does not yet drop the plugin file into the user’s main IDE config automatically. Closing that seam is on the list.
Single-user assumption. ~/.mcp-steroid/ is per user. Multi-tenant CI servers will need a story; today they get one home directory.
macOS / Linux first. Windows is supported (the devrig install PowerShell path is real), but most production hours so far live on macOS arm64.
2025.3 is dropped. If you are on 253, the plugin will refuse to load and devrig will refuse to download the matching managed backend. Upgrade to 2026.1.

Open questions — what should it do next?

This is the part I really wrote this post for. 0.100 is shipped, a fast follow-up added explicit backend_name routing, and the foundation works. Now I want to choose what to build next based on what you actually need.

A few directions on the whiteboard — please tell me which ones matter, and what I am missing:

Remote IDEs as a first-class concept. devrig already routes between local IDEs. Same abstraction wants to extend to a JetBrains Gateway / SSH host: tell devrig “use the IDE on dev-box-7” and it sets up the channel. Niche, or the next obvious step?
Fully autonomous plugin install into a user-launched IDE. Drop the plugin file in the right directory, restart the IDE, verify the marker appeared, report back. Worth it, or do most users want a manual confirmation step?
Bundled-JRE devrig. Ship a ~/.mcp-steroid/binaries/<os>-<cpu>-<sha>/ cache with a small wrapper, so the JDK-25 prerequisite stops being a prerequisite. The deployment spec is written.
Fleet mode. One devrig server, N parallel agents, one user — each agent gets its own managed IDE window without trampling the others. The lock today serialises; fleet mode would need a different concurrency story.
devrig doctor. Self-diagnostic command: “your plugin is out of date in IDE 2, port 63344 is taken by a different process, your JDK is missing on Linux/arm64, your claude MCP entry points at a stale path.”
Sandboxed IDE profiles. Spawn an IDE with a throwaway config dir for one task, then delete. Reproducible runs without polluting the user’s main IDE setup.
CI-friendly mode. Headless start optimised for GitHub Actions / TeamCity where the “human in the loop” assumption no longer holds.
Plugin auto-update. When devrig itself updates and ships a newer plugin, push it into every managed IDE on the machine.
Auth and policy. The per-IDE bearer token is already on the marker file. The next question is who is allowed to do what — does that matter to teams, or is it overkill outside enterprise?
More agent CLIs. Cursor, Aider, Continue, IntelliJ AI Assistant chat — devrig install today knows three. Adding a fourth is mcpAddStdioArgs plus an enum entry. Which one would actually unblock you?

Reply on LinkedIn, Twitter, or open an issue on jonnyzzz/mcp-steroid — and the more concrete the workflow you describe, the more directly I can shape this around it. Issue #64 tracked the auto-install ask before it landed; the same template works for anything on the list above.

Where this is going, and where it is

I have been writing about Agentic Experience and Tools as the discipline of building products with the AI Agent as the user, not as a feature bolted onto a human-facing UI. MCP Steroid is that bet, and it has a deliberate arc:

The plugin came first: hand the agent the full IntelliJ runtime instead of a curated tool menu. devrig is the next move — wrap that capability in one stdio bridge so the agent reaches every MCP Steroid–enabled IDE on the machine, and can bring a managed one into existence on its own. The step after that is remote: the same protocol pointed at IDEs on other hosts, and eventually fleets of them, which is probably where the commercial product starts to make sense.

So where are we, honestly? Level 3 has worked end to end — the Keycloak run is not a mock, and the backend_name routing that landed right after 0.100 makes “which IDE?” an explicit, answerable question. The foundation is real. It is also still an independent experiment with sharp edges: the 2026.1.2 trust-dialog regression can still stall an autonomous open, the JDK-25 prerequisite is unbundled, plugin promotion into a user’s own IDE still has a manual step, and ~/.mcp-steroid/ assumes a single user. Remote and fleet mode are designed, not shipped.

The honest summary: the agent can now stop asking a human for help just to get started against a real IDE — and that, I think, is where this kind of tooling gets interesting. What it should reach for next I genuinely do not know yet; that is what the open questions above are for. Tell me which one is yours.

From HTTP MCP to stdio: Lessons Connecting an AI Agent to a Desktop IDE

2026-05-20T00:00:00+00:00

If you want to connect an AI Agent to a locally running desktop application — in my case the IntelliJ family — there is one obvious idea. Run an MCP server inside the application. Bind it to a port. Point the agent at that port as an HTTP MCP server. Done.

It is the obvious idea because it is the correct first idea. I built exactly that, shipped it as part of my independent experiment, and used it every day. It kept breaking in real workstation setups — including my own. This is the story of why, and why the answer turned out to be the least glamorous transport in the spec: stdio, behind a small CLI that does the routing.

The obvious approach: an HTTP MCP server inside the IDE

The MCP standard defines two transports: stdio and Streamable HTTP (the older HTTP+SSE transport was deprecated in the 2025-03-26 spec). For a desktop app the HTTP route looks natural — the IDE is already a long-running statefull process, so let it open a socket and answer MCP calls.

So I implemented the HTTP layer and reached for the Ktor MCP integration in the Kotlin SDK. And immediately hit the first snag (state January 2026): at the time, that integration only spoke the SSE transport, not the newer Streamable HTTP the latest spec wanted. I was stuck on a spec-version mismatch before I had connected a single agent. (The Kotlin SDK has since grown mcpStreamableHttp() helpers; back then it had not.)

The thing that saved me here was not clever code. It was integration tests for the MCP layer. It was using Docker image, where an agent (claude, codex, gemini) is started and connected to the MCP server running in the test. It asks an agent to look at the MCP tools and call them.

Every time I or an AI Agent re-implemented the transport, a test would go red and tell me I had broken the contract. I re-did that layer more than once, and the test suite caught a regression every single time. If you take one portable lesson from this whole post, take that one: an agent-facing protocol is exactly the kind of surface that deserves real integration tests, because the consumer – an LLM – will not file a bug report unless you ask it explicitly (which I often do)

Good integration-test coverage of the MCP layer paid for itself many times over.

Where HTTP-against-a-desktop-app falls apart

The naive model assumes one user, one machine, one IDE. That is not what a real agentic-coding workstation looks like. Here is the pile of complications, roughly in the order they bit me.

A dynamic port at startup, with no limit on instances. Nothing stops you from running several instances of the same IDE at once — distinct processes, each launched with its own config directory. A fixed port is therefore impossible — the first IDE would grab it and every other instance would have to fail to bind. So each IDE has to pick its port dynamically: start from a configurable base and increment until one is free. This is, not coincidentally, exactly how the IntelliJ built-in web server behaves (it starts from 63342). It works. But it turns the port into a moving target.

Several IDEs at once, by design. IntelliJ is not a single app — JetBrains ships a whole family of IDEs (IDEA, PyCharm, GoLand, WebStorm, Rider, and more). One can have two or three of them open at once; power users run more than one instance — distinct processes with their own config directories — for different projects. Some use EAP versions in a row with release builds. Each such IntelliJ-based IDE process is its own MCP server on its own base-plus-increment port. (Several project windows in one instance are a different axis — one port, several projects.) Now I have a set of endpoints to discover and route between, not one.

Start-order dependence. Because the ports are assigned first-come-first-served from the base, the port a given IDE gets depends on the order you launched everything in. To make the agent talk to a specific IDE, you have to start that IDE first so it grabs the predictable port. That is a terrible thing to ask a human to remember, and an impossible thing to ask an agent to reason about. That is not manageable to maintain each time the port changes.

Agents can’t disambiguate identically-named MCP servers. This was the quiet killer. You cannot simply register five “mcp-steroid” servers with an agent and expect it to choose wisely — to the agent they look like the same server, and it does not understand that they are different IDEs. In practice, I ended up registering exactly one server, on the first port in the range — and then back to the start-order ritual to make sure the right IDE owned that port.

Nested IDEs. Sometimes an IDE is launched from another IDE. Now the set of processes the MCP server must cover is not even stable for the duration of a session.

Agent up, IDE down. The one that hurt the most in daily use. The agent runs for hours; the IDE does not. I shut an IDE down to update it, or it crashed, or I just forgot to open it — and the agent was instantly dead in the water, frequently unable to reconnect even after the IDE came back. An HTTP client dialing a port that no longer answers has nowhere to go. The MCP Steroid implementation specifically handles unknown sessions to keep an agent working after an IDE is restarted.

Real-world setup	What HTTP-in-the-IDE breaks
Two instances of the same IDE	Can’t share a port; base+increment; which one answers?
IDEA + PyCharm + GoLand open at once	Several servers, several ports, agent picks one
“Use this IDE” intent	Depends on launch order — start it first or lose
Agent registered with N identical servers	Agent can’t disambiguate; only one ends up usable
IDE shut down to update while the agent runs	Agent dead, often can’t reconnect

None of these is fatal on its own. I worked around every one of them and kept using the tool. But the workarounds stack into a fragile pile, and the pile is the point: HTTP-against-a-desktop-app is fighting the environment instead of fitting it.

The pivot: the app doesn’t care about the transport

Here is the schema that unlocked everything.

The IDE side does not care whether the bytes arrive over HTTP or stdio. MCP is MCP. The transport is a detail at the edge.

Instead of forcing the application to host the network endpoint, move all the routing and connection-handling to a small program the agent launches directly — and let that talk to the IDEs however it likes.

That program is a CLI. The agent speaks MCP to it over stdio: launch a subprocess, talk over pipes. No agent-facing ports, no DNS, no firewall, no “which IDE answered?”. And because the CLI owns the routing, it can do all the things the IDE-hosted server could not: discover every running IDE, pick the right one, start one if none exists, and survive an IDE restarting underneath it.

In MCP Steroid this CLI is devrig. The agent’s entire configuration becomes one stdio command; devrig is the coordinator that connects to all your IDEs, regardless of how many are running. The mechanics of what devrig discovers, downloads, and routes — and the screenshots — are the subject of a follow-up, the Level 3 post; this post is about why it has to exist.

        Coding Agent
            │  MCP over stdio  (one subprocess, no ports)
            ▼
        devrig  ── coordinator / router ──┐
            │            │            │   │
            ▼            ▼            ▼   ▼
        IDEA #1      PyCharm       GoLand   (start one on demand if none)

Why stdio + a coordinator is actually more powerful

Moving to stdio is not just “the same thing without ports.” It changes what is possible.

Sessionless routing, because the state lives on disk. MCP Steroid is a stateful MCP — its commands change the IDE’s state (files, refactorings, run configs, etc.). That sounds like it would need sticky sessions, but it is the opposite: because the durable state is in the process, there is no MCP session to pin. Any command can be routed to any backend capable of executing it. That is what makes multi-IDE-per-project realistic — GoLand handling the Go module, an IntelliJ handling the Java service, both serving the same checkout. The one catch I am still sharpening: the project key has to encode the IDE type, or the agent can’t reliably tell the GoLand-myapp from the IDEA-myapp and route to the right one. The data is there; making it obvious to the agent is the open challenge.

Restart resilience. The stdio coordinator can let connections come and go. The “agent up, IDE down” dead end stops being a dead end — devrig can reconnect, or start an IDE, instead of leaving the agent staring at a closed socket. It can also provision the IDE configurations if neccessary for the project.

Provision on demand. Once the agent talks to a coordinator instead of a fixed socket, “no IDE is running” stops being a dead end — the coordinator can bring one into existence for the task rather than asking a human to “please leave the IDE open for the next four hours.” How that will work once devrig ships it is the story for the follow-up Level 3 post; the point here is that the architecture is what makes it possible at all.

A path to remote. Once the agent talks to a coordinator instead of a fixed local socket, the particular local IDE stops being the center of the architecture. The same protocol extends to remote IDEs — and eventually to fleets of them. That is the part I am most interested in commercially, and the reason devrig itself is the thing worth promoting, not any single IDE behind it.

Routing to a project: `project_name` and discovery

A coordinator only helps if the agent can say which project it means without knowing anything about ports, process ids, or how many IDEs are open. So the most important thing the coordinator does is turn “all the IDEs on this machine” into one flat, addressable list of projects. Two halves make that work: discovery and a single addressing key, project_name.

Discovery is the coordinator’s live picture of the world. It keeps an up-to-date view of the IDEs running on the machine and the projects each one currently has open, as one map — every open project, across every running IDE, in a single table. When an IDE opens or closes a project, the map updates; the agent never has to be told.

project_name is the only thing the agent has to care about. The coordinator exposes each discovered project under a stable name and hands the agent that list. Every project-targeting tool takes exactly one project argument — project_name — and nothing else about where the project lives. The agent reads the list, picks a name, and calls the tool.

The routing itself is then almost boring, which is the point. On each call the coordinator looks the project_name up in its discovery map, finds the IDE that actually has that project open, and forwards the call there — translating the exposed name back to whatever that IDE calls the project internally. If two IDEs happen to have a project with the same name open, the coordinator makes the exposed project_name unique, so the agent can still aim at exactly one. The agent addresses a name; the coordinator does the plumbing.

This is the concrete shape of the sessionless routing above: the key is the project, not a session, so any call for a given project_name simply lands wherever that project lives. That’s the whole trick behind “one agent, many IDEs” — the agent holds a list of names, and discovery keeps that list honest.

MCP, CLI, or skill? There is no single answer

One more thing the journey clarified, and I want to leave it as an open question rather than a verdict. There is a real design-space discussion hiding under all of this: should a capability be exposed as an MCP tool, a CLI, or an agent skill — or some combination, skill+CLI, skill+MCP?

These are at least three distinct axes in a multidimensional feature space, not a ranked list. Some combinations are clearly more powerful for specific scenarios, and I do not believe there is one correct point. The honest design move is to assume you will want several of them and factor your architecture so that adding another surface is cheap. Choosing stdio + a CLI coordinator was partly a bet on exactly that flexibility: the CLI is a substrate the MCP, the skills, and the human-facing commands can all sit on top of.

Give devrig a try. And let me know what MCP+CLI+Skills equation looks like for you.

What I would tell my earlier self

Start with the obvious transport, but treat it as a hypothesis. HTTP-in-the-IDE was the right thing to try first and the right thing to abandon once the environment voted against it.
Match the transport to the environment, not to the protocol’s defaults. A desktop app full of swappable, restartable, multi-instance processes wants a subprocess-and-pipes model, not a server-and-port one.
Push routing to the edge you control. The IDE can’t own discovery sanely; a CLI the agent launches can.
Stateful-on-disk means sessionless. That inversion is what makes many-backends routing simple.
Test the agent-facing layer like a public API. Your most demanding consumer can’t describe what you broke.

This is experimental, in-progress work — devrig is the go-to direction now, but I am still learning where it bends. It has rough edges I am not hiding: a runtime prerequisite you have to satisfy by hand, single-user assumptions, and lifecycle operations that serialize. Rather than restate the exact version numbers here (they will move release to release), the prerequisites, the trade-offs I am least sure about, and an explicit list of open questions are what the follow-up Level 3 post will cover — head there when you want to actually run it.

If you have wired an agent to a desktop app and hit a different wall than these — or solved one of them a better way — I want to hear it. Reply on LinkedIn or Twitter, or open an issue on jonnyzzz/mcp-steroid. The more concrete the workstation you describe, the more directly it shapes what devrig does next.

130 Release-Roadmap Issues, 818 Agent Runs: A Mid-Iteration Reality Check

2026-04-28T00:00:00+00:00

Every product team running toward a release has the same artifact: a roadmap. Not the whole backlog — the umbrella epic for the next release, with a hundred-plus sub-issues hanging off it. Most are stubs: a title, a description, maybe some comments. No implementation plan. No code-level analysis. No effort estimate. No dependency map.

A senior engineer researching one such issue typically spends 2–4 hours: reading the codebase, understanding the architecture, scoping the change, writing it up as an actionable spec. For a release roadmap of 130+ issues against a multi-million-line codebase, that’s weeks of analyst work before anyone writes a single line of code.

We ran an AI agent swarm instead — three different agent CLIs, Claude, Codex, and Gemini, orchestrated together. Four calendar days. 818 agent runs. ~$576 total cost. Every roadmap issue researched. Then, while the release was still in flight, we ran a second pass to check the research against what the team was actually doing — explicitly not a “did it ship” scorecard, because nothing was expected to be done yet. Here’s how it works, what it produced, and the honest verdict.

It’s a release roadmap, and there is a process

This was not “point a swarm at a backlog and see what happens.” It was a release roadmap, and the swarm followed a defined process, orchestrated by a root agent that does as little work as possible and delegates everything:

Collect. Pull every unresolved sub-issue of the release umbrella epic from the issue tracker (through a small tracker CLI we built). Pick the product-requirements (PRD-prefixed) issues and the engineering issues, and reconstruct the tree of linked sub-issues for each root — the dependency graph matters as much as the issues themselves.
Research. Run each issue through the multi-round pipeline below.
Review by quorum. When a root issue is done, or whenever a decision is unclear, spin up three review agents and take the quorum verdict — not one model’s opinion.
Loop. Keep going in a /loop until the roadmap is covered, applying the same principle recursively: the orchestrator delegates, the sub-agents do the work.

The whole thing runs on run-agent.sh and the RLM methodology — one task per agent, fresh context, markdown files carrying state between them.

The Pipeline: R1 → R2 → R3

Each issue goes through three rounds of agent work. Two issues run in parallel at all times, with the shell orchestration managing dispatch, detecting completion, skipping already-researched issues, and retrying agents that time out.

Round 1 (Claude) reads the issue text, explores the codebase using MCP Steroid, and produces a first-draft research document: problem framing, relevant source files with class and method references, and 2–3 solution options. Each option includes a complexity rating (Low / Med / High), effort estimate (S / M / L / XL), key changes, test strategy, and a trade-offs table.

Round 2 (Codex) validates every file reference from Round 1 against the actual codebase. It finds exact line numbers, checks git history for recent changes to the relevant files, and flags edge cases Round 1 missed. Every file reference gets a validation status: confirmed, renamed, or not found. This is the mechanical correctness pass.

Round 3 (Claude) synthesises Round 1 + Round 2, resolves conflicts, and writes the final recommendation: a concrete implementation plan with validated code references, a test strategy, and follow-up work.

Claude for R1/R3 and Codex for R2 reflects what each model does well in practice. R2 is largely mechanical — check that this file exists, find this class, confirm this method — and Codex handles that efficiently. R1 and R3 require reasoning across options and writing a coherent recommendation, where Claude produces better outputs. Gemini was in the swarm too, mostly in the review quorum (more on its small share below).

The Tools

Three things make this pipeline work at scale.

run-agent.sh is the orchestration script that launches Claude, Codex, and Gemini in isolated sub-processes. Each run gets its own timestamped directory: exact prompt, full stdout/stderr capture, PID tracking, exit codes. Full traceability for every agent execution. The shell orchestration loops over TASK files, detects completion by checking output files, skips already-researched issues, and retries timeouts. See the full description here.

MCP Steroid gives agents programmatic access to the full codebase: PSI navigation, class resolution, call graph tracing, symbol search. Without it, agents produce vague references like “look in the parser module” or “check the rendering pipeline”. With it, they produce specific class names, method signatures, and line ranges — navigating the actual PSI tree and resolving inheritance chains. This is the difference between a summary and a useful research document.

TASK* files are structured task definitions written once and consumed by many agents (they carry no extension — the orchestration matches TASK, TASK-2, and friends by name). The shell script loops over them and dispatches agent runs accordingly. The format is simple and consistent enough that agents parse it without extra tooling. Here are the actual task files that drove the run (generalized — internal identifiers removed):

TASK — collect the roadmap and orchestrate

Your main goal is to collect all the relevant issues from the release-roadmap umbrella epic
in our issue tracker (query: all unresolved sub-tasks of the release epic).

Use our small issue-tracker CLI to read the tracker. If it lacks a feature or fails, start
run-agent.sh with a task to fix it. Equip that agent with THE_PROMPT_v5.md from
https://run-agent.sh to research, fix, review, deploy, and build the fix to the CLI, so you
can move on. Repeat that process as many times as you need.

Pick all tasks with the PRD* prefix and the engineering tasks for the product without the
prefix. For each task create a <task-id>.md document with discussion. Each such task has a
tree of linked tasks, which you need to include in the document. I want the graph of linked
sub-tasks for each root task.

Do that work in parallel using https://jonnyzzz.com/RLM.md. Our main goal is descriptive
write-ups for all selected tasks, and the related/linked issue tree.

Once a selected root issue is done, start 3 agents with run-agent.sh to review the work and
do the fixes.

Repeat that work in a /loop until the work is complete. You are the root agent; your main
goal is to orchestrate the research, do as little actual work as possible, and delegate
everything to run-agent.sh, as suggested in RLM.md and THE_PROMPT_v5.md. Apply it recursively.

When not clear, start 3x run-agents and make the quorum decisions.

TASK-2 — the research group of five

Now create a research group of 5 agents running on the target codebase, each via
run-agent.sh. Our goal is to research each task as deeply as possible and provide 3 potential
ways of solving it. Use MCP Steroid to work on the assignment. Since there are far too many
tasks, prepare all the prompts in the prompts/ folder and a run script that calls all agents
on all tasks. For each task, 3 runs: claude -> codex -> claude. Each run includes the same
run-agent.sh and THE_PROMPT_v5.md, and refers to MCP Steroid's AGENTS.md as the usage guide.
Write all the outcomes to the issue-id.md file respectively. Make each agent use run-agent.sh
to research different parts of the monorepo. Your goal is to delegate and manage the process;
set up a /loop so we keep running and don't stop. Work one issue at a time. For issues with
linked issues, prefer the leaf issues first; create a dedicated file like
issue-id-leaf-another-issue-id.md and place details there. Include a high-level overview in
the initial issue.

TASK-3 — close the gap on the stub issues

## Goal
Research all remaining .md files in this directory that have no proposed solutions yet
(53 standalone stubs with only tracker metadata, plus 2 leaf files missing solutions).

For each issue:
1. Check tracker status via our issue-tracker CLI — if already resolved/closed, append
   "Status: Resolved" and skip deep research.
2. Check for sufficient information — if the description is primarily marketing/strategy
   language with no concrete technical requirements (no user-facing behaviour, no API
   contract, no acceptance criteria), append "Status: Insufficient information — PRD lacks
   technical detail" and list what is missing. Do NOT fabricate solutions for thin PRDs.
3. If still open and detailed enough, run the full R1 -> R2 -> R3 pipeline
   (claude -> codex -> claude) via run-agent.sh, writing solutions into the issue's .md file.

Each agent fetches the latest issue description from the tracker before researching. Apply
THE_PROMPT_v5.md from https://run-agent.sh and use MCP Steroid to research the monorepo. Work
2 issues in parallel. The process may already be running — check existing .md sizes before
launching agents (skip if a file already has solution sections).

TASK-4 — the mid-iteration snapshot

Your goal now is to run a snapshot review of the investigation. Time has passed since the
original research, and the release is still in flight — so we should be able to see what has
moved, how the team approached features, and which approaches were selected, without expecting
anything to be finished.

For each suggested ticket, run a batch of run-agent.sh agents to review whether and how it was
addressed. Learn from each decision and compare it against the options we proposed.

Use RLM.md, run-agent.sh, and any other tools to assess all the tickets. For more context and
approaches, look at the other TASK*.md files in this folder; they contain the necessary
information and approaches you should re-use.

The Numbers

Here’s what the full run produced:

Metric	Value
Issues analyzed	130+
Agent runs total	818
Codex runs	322 (~39%)
Claude runs	318 (~39%)
Gemini runs	12 (~1%)
Other / unattributed (orchestration, review, retries)	166 (~20%)
Total token consumption	~155–200M tokens
Total cost (API prices)	~$576 (~$315 Codex, ~$259 Claude)
Cost per fully-researched issue	~$5 (~$576 / 120)
Calendar days	4 (overnight gaps)
Issues with full 3-round research	120
Total output	52,162 lines
Average output per issue	~435 lines
Issues found already implemented	9
Issues flagged as directly implementable	~39%

A few of those numbers deserve commentary.

Three agents, very uneven shares. Codex and Claude split the pipeline almost evenly. Gemini landed at ~1% (12 runs) — not because of model quality, but because it kept tripping over filesystem sandbox path restrictions in this setup and we fell back to the other two rather than fight it mid-run. Worth knowing if you’re planning a three-CLI swarm: the orchestration is easy; the per-CLI sandbox and auth quirks are what actually cost you time.

~$5 per issue. ~$576 across 120 fully-researched issues — the cost of 2–4 hours of senior engineer analysis, done in roughly 90 minutes of wall-clock time per issue (two pipelines in parallel). At any reasonable engineer billing rate, the economics aren’t close.

9 issues already implemented. During research, agents cross-referenced issue descriptions against the current codebase and found nine cases where the requested functionality already existed — implemented at some point without closing the ticket. Catching this early avoids days of duplicated work.

120 of 130+ got full 3-round research. The initial four-day swarm covered 65 issues; a later pass researched the 55 remaining stubs — 8h49m of continuous execution, two parallel workers, no API failures — bringing the total to 120. (Peak concurrency was only two workers, with a long overnight idle gap; there’s easy throughput left on the table.)

What the Output Looks Like

This is not summarization. It’s the analyst work a senior engineer does before writing code. A typical fully-researched issue document contains:

A problem framing section: what the issue requests, what the codebase does instead, where the gap is
File:class:method references with line numbers — confirmed by R2 against the actual repository
A git history note on recent changes to those files — flagging active areas where merge risk is higher
Two or three solution options, each with complexity, effort, key changes, test strategy, and a trade-offs table
A validation section from Round 2: which references were confirmed, which were corrected, which edge cases were added
A final recommendation from Round 3 with rationale, implementation sequence, and follow-up work

When the output is good, a team lead reads it in 15–20 minutes and makes a prioritization decision with full context. The shift is real: from “I need half a day to understand this issue before I can scope it” to “I need to review what the agent found and decide which option makes sense.”

A Mid-Iteration Snapshot, Not a Postmortem

Here is the part that’s easy to get wrong. After the research, while the release was still in flight, we ran a second pass against the same roadmap — same orchestration, same MCP Steroid, same run-agent.sh — asking a different question. Not “what should we implement,” but “what is the team actually doing, and how does it compare to what we proposed?”

This is a mid-iteration snapshot, and the framing matters: at the time of the check, nothing was expected to be finished. The release hadn’t shipped. So this is not a scorecard of “did the research come true.” It’s a directional read — a chance to see, on the issues where work had started, whether the research pointed the right way.

The snapshot was map-reduce: one agent per issue (refetch status, scan git history for related commits, compare proposal vs. current direction), then a thematic reduce across about eight subsystem groupings, then a final reduce into a single lessons document. Same RLM principle — one task per agent, fresh context, markdown carries state — with two reduce stages because the input was too large for one synthesis pass.

Snapshot outcome (138 items, incl. parent/leaf)	Share
Work landed — matched the proposal	5%
Work landed — diverged from the proposal	33%
In progress	23%
Still open — proposal still valid	27%
Abandoned / obsolete	11%
Insufficient PRD — never actionable	1%

Read against a finished release, “5% matched” would look brutal. Read as a mid-cycle snapshot, it’s the wrong number to fixate on — because of what “diverged” means for the other 33%. In most diverged cases the agent identified the right root cause or dependency, but maintainers chose a different implementation layer, scope, or product shape. The research was directionally right, even if the patch landed elsewhere — in an adjacent caller, an integration layer, or a shared platform surface rather than where it pointed.

That reframes what the pipeline is for. The output is not a patch location to apply as-is. It’s a structured comparison of options plus an identification of root cause and dependencies. The hit rate is the wrong metric; the value is in the framing and the dependency map.

What We Learned

Some solutions aren’t viable — even after three review rounds

This is the uncomfortable one. A non-trivial share of recommendations were simply not good, and three rounds of review didn’t save them. The agents over-technicalized product questions (treating “should we keep this feature” as an engineering task), confidently prescribed a local patch where the real decision was about ownership or release sequencing, overcomplicated easy tasks, and oversimplified hard ones. Three passes of a confident model reviewing another confident model can converge on a polished, plausible, wrong answer. More rounds did not fix wrong framing.

Agents reject their own options — then keep them anyway

The clearest tell came from a reviewer’s note. In one document, the agent had generated three solution options and explicitly marked the third “should not be implemented” — and then left it in the final write-up. The agent’s own judgment said no, and the option still shipped into the document, adding clutter a human had to read past. An output set that never drops the ideas its own author rejected isn’t a research report; it’s a transcript. The fix is mechanical: prune options the agent flags as non-viable before synthesis, instead of dutifully carrying every branch forward.

We needed an adaptive number of iterations, not a fixed three

Three rounds was the wrong constant in both directions. For small, well-groomed issues with an obvious code seam, R1 alone was enough — R2 and R3 just reworded it. For cross-module, cross-repo, or product-gated issues, three rounds weren’t nearly enough to converge, and a fixed cap let unresolved disagreement get written up as a confident recommendation. The right design is an adaptive iteration count: keep looping review-and-revise on a task until it converges — until a fresh reviewer adds no new finding — and stop early when it already has. A product-gated task should short-circuit to “ask the owner,” not burn three deep-research passes on a moving foundation.

MCP Steroid is the force multiplier

The gap between agent outputs with and without full codebase access is large. Without it, agents default to generic descriptions: “update the parser”, “check the rendering pipeline”. With MCP Steroid, they navigate the actual PSI tree, resolve inheritance chains, and trace call graphs to find where a change propagates. That’s not a small specificity bump — it’s the difference between a useful document and a vague one. One pattern to fix: agents default to one file per MCP call; a small script that reads 5–10 related files per call would cut round-trips and speed up R1 noticeably.

The message bus was useful enough to graduate

Between rounds, findings flowed through a single append-only markdown file — MESSAGE-BUS.md. Each agent wrote what it learned; later agents and the reducers read it back. It was the most reusable artifact of the whole run, and it has since graduated into the agent-orchestration framework we use elsewhere as a first-class, append-only, typed protocol — messages carry a type (FACT, DECISION, QUESTION, ANSWER, PROGRESS, ERROR, BLOCKER, COMPLETE), an id, and a back-reference, so a swarm can coordinate without a shared database. The honest caveat: because parent and leaf tickets often referenced the same root problem, the bus filled with duplicates — so any reducer has to dedupe by the underlying issue before drawing conclusions, or it double-counts.

Codex buffers all output

Codex streams its progress to stderr and writes nothing to its stdout file until the run completes. A monitor that watches the stdout file size will conclude a running Codex agent is stuck. It isn’t — it’s thinking. Check the process PID (or stderr), not the stdout file size. We spent more time than we’d like to admit investigating “stalled” agents that were running normally. It’s already improved in the recent version of run-agent.sh where we run agents with a verbose mode, NDJSON outout, and better monitoring of file sizes.

Cost is not the constraint — human review is

At ~$5 per fully-researched issue, cost doesn’t register for any team with a real roadmap. The constraint is human review bandwidth: the pipeline produced ~435 lines per issue, 52,000+ lines total, all waiting for judgment. Designing the output format for fast human review matters far more than shaving a dollar off the per-issue cost.

The Human Verdict, Honestly

A senior product lead reviewed the output and gave a blunt read, which I’ll paraphrase because it’s the most useful thing in this whole post. Handing this research — with its implementation advice — straight to junior, mid, or even early-senior engineers is dangerous, because the agents:

describe even complete nonsense very convincingly, instead of saying “I don’t have enough information, I give up”;
charge ahead researching a technical solution without thinking about the product side;
overcomplicate easy things and oversimplify hard ones.

His conclusion: this kind of mass research is something you show a product lead plus a tech lead for a fast review — people with enough context to validate or dismiss it quickly, and who, frankly, would have gotten there without the research anyway. The genuine upside: for small, well-groomed tasks where the product direction is settled, the output is exactly the spec an implementation pipeline can run with.

He also flagged the small, human things the agents missed and the ones they nailed: handle externally-reported issues more cautiously (have a human confirm before acting, and check whether the product already does the requested thing); and diagrams and flow charts are genuinely helpful — the kind a senior draws by hand to think, and the kind the agents should produce more of.

The honest position: agent research is a tool for senior judgment to operate on, not a substitute for it.

What’s Next

The changes the snapshot points to — none of which remove the senior human from the loop:

Product-state precheck before deep research — assignee, fix version, recent human comments, decision state — so product-gated issues short-circuit instead of getting three rounds.
Adaptive iteration count — loop review-and-revise until convergence, not a fixed three.
Prune agent-rejected options before synthesis.
Mandatory alternate-layer enumeration — at least one plausible alternate implementation surface per recommendation, with the rationale for the recommended one.
External-repo scanning for build/CI/release issues, which often land in adjacent repos.
Dedupe by underlying issue before any reduce.

The bulk-implementation idea survives the snapshot. For the subset of issues where the product direction is settled, the paths still hold, and the change is self-contained, the same pipeline can keep going — R4 implements from the R3 spec, R5 reviews and validates tests, R6 does a final integration check — closing the loop from issue stub to pull request, without the over-confident handoffs the first round would have produced.

The pipeline is built on tools I’ve written about before: run-agent.sh for agent execution, MCP Steroid for codebase access, and the RLM methodology for keeping context manageable. What this experiment adds is evidence that they work together at scale — not for a handful of issues, but for an entire release roadmap — and an honest account of where the output needs a senior human in the loop.

Questions about the pipeline or running research swarms at scale? Find me on LinkedIn or X.

Eugene Petrenko

Install, Launch, Update: Shipping a CLI on Five Platforms — the devrig Approach

The problem, stated properly

“Just require a JDK on PATH”

“Publish to Homebrew, apt, winget, Chocolatey, Scoop”

“curl | sh and figure it out at install time”

“Install over the previous version”

“Put all the update logic inside the binary”

The architecture

The JDK data model: resolved live, PGP-verified, nothing hand-pinned

The generator: a template, a table, and a refusal to guess

The generated script: what a good curl | sh looks like

The PowerShell twin, and why it is not a translation

Content-addressed, side-by-side installs

The handoff: the script does not write the launcher

The launcher, and self-healing on every start

Update: the updater that knows nothing

And then the upgrade is: run the install script again

The updater that cannot age

Distribution, and the release-ordering trap

Signing: implemented once, designed twice, not shipped yet

Testing scripts and generators

Where this was built and tested

What is not solved yet

Is this an open-source project on its own?

Personal Assistant without OpenClaw or Hermes: Just One Prompt

One Rule at the Top, Enforced Structurally

Two Channels In, One Owner

Bucket Mode: Near-Realtime From a Blocking Poll

One Request, One Process

The First Incident, and the Fix, Two Hours Later

Deliver First, Fail Loudly, Reap the Dead

A Quorum That Rewrites Its Own Process

The Repo Is the Source of Truth

The Assistant That Babysits Other AI Agents

The Field: OpenClaw and Hermes

What I’d Steal, and What Stays

Show Me Yours

A Real macOS Desktop for a Linux Agent: Driving Tart VMs over SSH with Agent Skills

The problem: a macOS GUI, and no Mac for the Agent

The two SSH hops

The gotchas, learned the hard way

One IDE, shared across every VM

What we actually get

Try it, point it at a Mac

A Real IntelliJ IDEA Inside Docker: Integration Tests with X Server, Window Manager, and Live Video

Why not headless, why not mocks

Why Docker?

The image: heavy base, thin IDE layer

Starting the container

The X server: Xvfb, a readiness probe, and fluxbox

Managing IntelliJ inside the container

The VM options

The consent files

Waiting for IntelliJ to be ready

The end-user AI/Dev experience in the MCP Steroid project

Sending the live video to a browser tab

The same pipeline, on camera

Re-implementing this for your own GUI tests

Try it, steal it, improve it

Inside the Git Hooks: Tagging Every AI Agent Commit (Part 3)

Wiring: Hijack Every Hook Without Breaking the Repo’s Own

Tagging: One Trailer, Re-Collected Every Time

Provenance Across Rewrites

Enforcement on Push: Re-Tag, or Bail Out

Re-Tagging Without a Checkout

Capture on Push: a Restorable Copy, Never Blocking

The Limits, Again — Briefly

Where This Sits in the Series

Git Documentation Used in This Post

Inside the Git Proxy: Capturing What an AI Agent Pushed (Part 2)

Make HTTPS the Only Way In

Capture: log every commit that was pushed

A Pack Is Snapshots, Not Diffs

Keep the objects alive: backup marker refs

Side-Effects to Watch

Where This Leaves You

Git Documentation Used in This Post

What Did the Agent Just Push? Auditing Git for AI Agents

Why This Is a Git Problem, Not a Policy Document

“`curl | sh` and figure it out at install time”

The generated script: what a good `curl | sh` looks like

What `devrig` actually is

`devrig install` — the new entry point

What 0.100 changed beyond `devrig`

Routing to a project: `project_name` and discovery