Hetzner Container Cleanup — Execution Report

Date: 2026-07-11 (execution pass, same day as the planning docs) Author: Fable 5 Status: Executed. This is a completion report, not a proposal. Builds on: HETZNER-AUDIT-AND-EXECUTION-PLAN-20260711.md (the read-only audit + plan) and GPT-5.6's independent review of it (gpt56-independent-review-20260711.html), whose specific objections are addressed point-by-point below. Authorization: Bharath's corrected ground-truth call — nobody is actively using any OpenClaw instance right now, including chris-b (previously misjudged as "real, active" from log/restart-count inference alone); Raj Sharma was a test instance, not a real user; the one exception is Nitin Kumar, who needs his instance genuinely working for next week.

0. What actually happened, in one paragraph

21 of 22 OpenClaw containers were backed up (workspace + config + full docker inspect metadata + docker diff), each backup verified byte-for-byte against the live source (not just a successful exit code), and one backup was further proven by booting a real scratch container from the extracted archive and confirming its content matched — then all 21 were docker stop'd (not removed, not deleted). The 22nd, Nitin Kumar's (oimy-pool-82c4), was diagnosed properly (not assumed), the real root cause found and fixed, and is now confirmed stable, listening, and Telegram-connected. Zero backup failures. Zero data loss. Host load dropped from 9.8–12.4 to 2.0–2.8 as a direct, measured side effect.

1. Addressing GPT-5.6's specific objections

GPT-5.6 objection What was done about it
No pre-change backup/restore gate; real-user containers must not be touched until recoverability is proven Built a verify-then-stop pipeline (§2) and piloted it on one low-risk container first, including an actual live restore — booted a scratch container from the extracted backup and confirmed its USER.md content and config checksum matched the original — before running it at scale.
OOM root cause unestablished — "workspace bloat" theory was speculative Retracted. Tested directly: stripped the suspected models.providers config block from an isolated clone at the original 1 GiB limit — still crashed identically (§3.1). The real cause, established by direct reproduction: 1 GiB was genuinely insufficient for this container's real startup memory peak (~900–950 MB); raising to 2 GiB fixed it cleanly with the original, unmodified config, and memory settled to a flat ~350 MB steady state afterward — plateau, not a leak (§3.2).
No aggregate memory-capacity plan; bumping multiple real-user containers could cause host-wide pressure Moot as flagged in the plan, and confirmed in practice: only one container (Nitin's) got a memory bump; the other 21 were stopped, not bumped. Host load fell, it didn't rise.
False "risk to live traffic: none" claims This pass carried real risk and it materialized once: my own isolated test containers held a live copy of Nitin's real Telegram bot token and briefly caused a 409 getUpdates conflict against the real container after the fix landed. Caught within minutes via log monitoring, all test containers force-removed, confirmed clean recovery with a dedicated post-cleanup observation window (§3.3). This is exactly the class of risk GPT-5.6 warned about — it happened, and here is the record of catching and closing it, not a claim that it couldn't happen.
Seven "safe to stop" containers not safe to delete Nothing was deleted. All 21 stops used docker stop; the review's own distinction is honored exactly — docker rm, volume removal, workspace deletion, and DB-row deletion were not performed on anything.
Restart-count arithmetic error (should be ~7,302–73,271, not ~23,800–73,264) Confirmed and corrected with fresh numbers at execution time: the real range across the affected fleet was 7,302–73,313 (§4).
Monitoring too weak, too late Out of scope for this execution pass (it's a Phase 1/4 item in the plan doc, not a Goal 1 cleanup item) — not addressed here, flagged again as still-open in §6.
/context/writeback isolation gaps Not touched in this pass — that's Phase 2/3 of the plan, unstarted. No new exposure created.
Cost attribution to chris-b unsupported Not re-asserted here as a hard number; see §5 for what can and can't be said about cost with the evidence actually available.

2. The backup + verify + stop pipeline (as run)

For each container: capture docker inspect (full metadata) and docker diff (writable-layer changes outside the bind mounts), tar czf the host-side workspace + openclaw.json, sha256sum the archive, extract it to a scratch directory and diff the extracted copy against the live source directly (diff -rq, plus a byte-for-byte cmp on the config file and file-count/byte-count comparison) — only stop the container if that verification passes. On any verification failure, skip the stop and log it for manual review (none occurred).

Pilot proof, run before the batch (on oimy-pool-e52e, confirmed-empty content, lowest stakes): checksum verified, diff -rq clean, and a real scratch container was booted mounting the extracted backup, with its USER.md content and an md5sum of the config file confirmed to match the live original exactly. This is the "demonstrate restoration" step GPT-5.6 required before any batch run.

Batch result — all 21 containers, zero failures:

Container Restart count at stop Backup verified Stopped
oimy-test-newsignup 9,253 yes yes
oimy-gaya-pool 1 yes yes
oimy-gayatri-test 1 yes yes
oimy-wife-b 7,306 yes yes
oimy-son2-b 7,303 yes yes
oimy-son1-b 7,302 yes yes
oimy-sharma 54,042 yes yes
oimy-ryan-k 53,209 yes yes
oimy-rohan-dx1u 53,218 yes yes
oimy-pool-xoh3 73,137 yes yes
oimy-pool-gr2k 23,806 yes yes
oimy-pool-e52e 73,185 yes yes
oimy-pool-c6a8 73,171 yes yes
oimy-pool-2d45 73,134 yes yes
oimy-bharath-tj69 73,174 yes yes
oimy-ssl-16be 73,103 yes yes
oimy-final-dfdf 73,140 yes yes
oimy-roophus-5d71 53,203 yes yes
oimy-pool-1s5q 2 yes yes
oimy-oimy-alpha-202c 73,313 yes yes
oimy-chris-b 0 yes yes

Backups: /opt/oimy-backups/container-cleanup-20260711/<container-name>/ on Hetzner, each containing docker_inspect.json, docker_diff.txt, workspace_and_config.tar.gz, workspace_and_config.tar.gz.sha256, and verify_report.txt. Master log: /opt/oimy-backups/container-cleanup-20260711/_execution_log.tsv. Total backup size: 4.5 MB (small — these are text/config workspaces, not media). No docker rm, no volume removal, no DB row touched. Every container remains resumable exactly as it was, at zero ongoing compute cost while stopped.

Two infra containers explicitly left alone, not part of the "instance" cleanup: oimy-browserless (shared headless-Chrome tool service other containers' web-search/browser skills may depend on — including Nitin's, going forward) and oimy-suno-api (already Exited, no action needed). These aren't "someone's instance"; stopping browserless would have been a scope-creep risk to Nitin's actually-needs-to-work container.

3. Nitin Kumar's container — root-caused and fixed for real

3.1 The original "workspace bloat" theory — tested and retracted

Built an isolated clone of Nitin's exact workspace + config, stripped the models.providers.openrouter.models[] custom catalog block (the config-content correlate the original audit flagged), ran it at the original 1 GiB limit: crashed identically — same FATAL ERROR: Reached heap limit, ~23 seconds in. This directly disproves the workspace/config -size theory. Good thing it was tested rather than shipped as the fix.

3.2 Real root cause, established by direct reproduction

Ran the same isolated clone, original config unmodified, at a 2 GiB limit, replicating Nitin's exact real container entrypoint (HOME=/tmp/oc, the config-copy sequence, exec openclaw gateway — my first attempt at this test skipped the entrypoint replication and produced a misleading "Missing config" result; corrected and rerun properly): memory peaked at ~940 MB during startup, then settled to a flat ~351 MB and stayed there for 6 continuous minutes of observation, zero restarts. This is the signature of a legitimate startup memory requirement that exceeded the old 1 GiB ceiling, not a leak (steady-state, not climbing) and not an explicit V8 heap flag (no NODE_OPTIONS/--max-old-space-size set anywhere in the image, entrypoint, or config — confirmed by direct inspection; Node's default heap sizing is derived from the cgroup memory limit, and raising that limit measurably raised the effective ceiling, which is only possible if nothing was overriding it explicitly).

3.3 Applied to the real container, verified, and one real incident caught and closed

Applied docker update --memory=2g --memory-swap=3g to the real, live oimy-pool-82c4 container (a live cgroup update, no destructive recreation needed — container identity, mounts, and history all preserved). Watched for 6 minutes: one final crash immediately after the update (the in-flight attempt from before the fix), then RestartCount held flat at 73,189 for the rest of the window, memory settling to ~342–351 MB — matching the isolated test almost exactly. Gateway confirmed fully up: listening on ws://0.0.0.0:18789, Telegram provider started for @nimaya_bot, canvas/browser services ready.

Then a real problem appeared: getUpdates conflict: 409 — Telegram rejecting the poll because two processes were using the same bot token. Root cause: my own leftover isolated test container (test-nitin-real), which had been given Nitin's real bot token for fidelity in the reproduction test, was still running. Force-removed it and all other test/probe containers immediately (test-nitin-2g, test-nitin-real, and five probe* containers from the diagnostic session). Watched a further, fully clean window afterward: zero errors, zero restarts, RestartCount flat at 73,189, gateway responding HTTP 200. This is a real example of the exact risk class GPT-5.6 warned about ("clone using same Telegram token, competing for updates") — it happened, in miniature, was caught by watching logs rather than assuming success, and was closed within minutes.

3.4 Final verified state

3.5 Note on the OpenRouter key rotation

Mid-task, a .env key rotation on this exact container was reported and independently verified on-host before acting on it (files, timestamps, and new-key value all checked directly — not taken on trust). One technical fact worth recording: docker inspect shows the running container's baked environment still carries the old key — editing the host-side .env file does not propagate to an already-created container until it is recreated (not merely stopped/started). This didn't affect the fix in this pass (docker update only touched the memory cgroup, not env), but whoever next fully recreates this container should source the current .env at that time to pick up the rotated key; a stop/start alone will not.

4. Corrected arithmetic

GPT-5.6 was right: the original doc's restart-count range (~23,800–73,264) omitted the Chris-family containers. Fresh numbers at execution time, across every container that was ever in the crash-looping/high-restart state: 7,302 (oimy-son1-b) to 73,313 (oimy-oimy-alpha-202c). Recorded per-container in §2's table above, captured at the exact moment each was stopped — not a single stale snapshot.

5. Before/after, with the corrected usage picture

Before: 22 user-instance containers running or crash-restarting continuously; host load average 9.8–12.4 on 16 vCPUs.

After: 1 running (oimy-pool-82c4, Nitin, stable and healthy), 21 stopped (fully recoverable, zero ongoing compute cost), 2 infra containers unchanged (browserless running, suno-api already exited). Host load average now 2.0–2.8 — measured directly, not estimated, in the minutes immediately following the stop pass.

On dollar cost, stated with the same honesty as the audit doc: the $15–40/day figure is Sonnet-4.6 token spend, and per Bharath's own correction, none of these 21 containers were actually being used even when chris-b looked stable — so that spend was not meaningfully tied to real usage in the first place, and this pass mostly converts "wasted host CPU on a crash loop" into "reclaimed host capacity" rather than "avoided token spend." The honest number that changed is host load (9.8–12.4 → 2.0–2.8), not a token-cost figure — no usage-tracking table exists to attribute token spend per container (confirmed empty in the original audit), so no new dollar figure is asserted here that wasn't already available.

6. What's still running, and the sanity check on "nobody's using it"

Still running (by design): oimy-pool-82c4 (Nitin Kumar) — fixed, verified, ready for next week per §3.4's caveat about a live end-to-end test not yet run. oimy-browserless — shared infra, left alone, not a user instance.

Sanity check performed before accepting the blanket "nobody's using it" statement at face value (per the instruction not to assume it's 100% literal without checking): before stopping the four containers that had looked stable in the original log-based audit (chris-b, gaya-pool, gayatri-test, pool-1s5q), their workspace files were checked directly for recent real content. No contradicting evidence found — chris-b's own engagement-state.json shows messages_sent: 0, last_message_date: null, onboarding_done: false, with the only "recent" file activity being a skill-definition update (a deployment change, not user content) from June 24; the other three show the same pattern, all last touched by a batch config operation on 2026-05-29/30, nothing since. This confirms Bharath's correction rather than contradicting it — no exception was found and none is being flagged.

Appendix — evidence locations