Hetzner Cost Audit + Execution Plan (Stage 1: Planning Only)

Date: 2026-07-11 (same-day follow-up) Author: Fable 5 Status: Planning artifact for Bharath's review — no containers, services, or files were stopped, killed, removed, or modified in producing this document. Everything below is read-only investigation (docker inspect, docker logs, psql SELECT queries against the Neon DB, filesystem reads) plus a written plan for a future execution pass. Builds on: HEAVY-HERMES-ARCHITECTURE-20260710.md, HEAVY-HERMES-NOUS-PORTAL-ARCHITECTURE-20260711.md, HERMES-VS-OIMY-ENGINE-RELIABILITY-DEPLOYMENT-20260711.md.

Executed: Goal 1 (the container cleanup) was executed the same day, after Bharath's ground-truth correction and GPT-5.6's independent review — see HETZNER-CLEANUP-EXECUTION-REPORT-20260711.md (https://cdn.oimyai.com/docs/hetzner-cleanup-execution-report-20260711.html) for the actual backup/verify/stop results and Nitin Kumar's OOM root-cause and fix.

0. TL;DR — read this first, it changes the shape of Goal 1

The audit turned up something bigger than the "real vs. stale container" question it was asked to answer: 18 of the 22 non-infrastructure OpenClaw containers on the Hetzner box are in a continuous OOM crash-restart loop, most of them since the fleet's last redeploy on 2026-05-27 — roughly six weeks. Restart counts range from ~23,800 to ~73,264 (docker inspect --format '{{.RestartCount}}'), consistent with a crash roughly every 50–70 seconds, nonstop, since creation. This includes containers belonging to real, named, signed-up users — Nitin Kumar, Matti, Raj Sharma, Ryan Kosiba, Rohan Richard, Swarooph Nirmal, Aj, and Bharath's own dogfooding instance (bharath-tj69). Only 4 containers are actually stable (chris-b, gaya-pool, gayatri-test, oimy-pool-1s5q) — and only chris-b has real production usage. This means most of these real users have most likely never had a working OpenClaw experience this whole time, and nobody noticed.

This changes Goal 1's framing: the question isn't primarily "which containers are safe to stop to save money" — a container that OOMs at boot before ever reaching a model call generates ~$0 in token cost regardless of whether its owner is real or a test account. The real finding is a product-breaking reliability bug, hiding in plain sight, that has to be fixed before "real vs. stale" can even be answered with confidence — because right now almost nothing in the fleet is in an observable, working state to judge activity from.

Both goals below are answered in full. Goal 1 gives the classification asked for, corrected by this finding. Goal 2 gives the phased plan, with the crash-loop fix pulled forward as the first, most urgent step (not originally on Bharath's list, but a hazard that has to be disclosed and dealt with before anything else on that list has stable ground to stand on).

GOAL 1 — Hetzner/OpenClaw container audit (read-only)

1.1 How this was done

Three independent sources of truth, cross-checked against each other rather than trusted individually: 1. docker ps -a / docker inspect on Hetzner — ground truth for what's actually running, its restart count, memory limit, and current resource usage. 2. The Neon Postgres DB (instances, users, families, message_logs, session_logs, usage_tracking — connection found in /opt/oimy/management/server.js) — the system of record the management API (port 18800) is built on. This is where real user identity (name, email) and provisioning history live. 3. Container workspace filesystems (/opt/oimy/containers/<name>/workspace/USER.md, memory/, family/) — bind-mounted per container, the actual conversation/profile content.

Important negative finding, worth stating up front: message_logs, session_logs, and usage_tracking — the tables that should hold per-container conversation activity and cost — are completely empty (0 rows) for every single instance, including chris-b, the one container with confirmed real, sustained usage. This logging pipeline exists in the schema and is referenced by foreign keys but was never actually wired up to write from the OpenClaw containers. This DB cannot answer "did this container have real activity" — only the container's own workspace files and Docker-level signals can. Recommend fixing this logging gap regardless of the Hermes/OpenClaw decision — it's the kind of visibility gap that let the crash-loop finding above go unnoticed for six weeks.

1.2 The Chen family, calibrated

Bharath's example: "the Chen family is a test family so those instances can be killed." Found in the DB: three instances, oimy-pool-09tu (Emma Chen, emma.chen.demo@oimytest.com), oimy-pool-dcaf (Sarah Chen), oimy-pool-ctgm (David Chen) — all @oimytest.com demo emails, all DB status stopped. But there is no live Docker container and no workspace directory on disk for any of the three — they were already fully removed in some earlier, undocumented cleanup pass, with only the DB row surviving as a stopped record.

This is useful as calibration in exactly the way Bharath intended (@oimytest.com / .demo@ email pattern, obviously synthetic family surname repeated across three "members" = test), but it also surfaces a real precedent worth flagging: a destructive cleanup already happened here once, with no audit trail — no note of when, who, or whether data was extracted first. This is the exact failure mode Bharath is trying to avoid this time by asking for planning-before-execution, and the Chen precedent is quiet evidence it's a reasonable thing to be careful about, not a hypothetical risk.

1.3 The crash-loop finding — root cause and confidence

Confirmed via docker logs on multiple affected containers: they fail at startup with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory (Node/V8 OOM), before reaching any application logic — no cron output, no tool calls, nothing in the logs but the boot attempt and the crash. docker inspect shows the affected containers capped at --memory 1073741824 (exactly 1 GiB), a limit set uniformly across the fleet (provision-instance.sh). The host itself has 30 GB RAM with 19 GB free — this is a per-container cgroup limit, not real host memory pressure.

The correlation that points at a likely cause (stated as a hypothesis with real evidence behind it, not a certainty): the 4 stable containers have small workspaces (gaya-pool 128 KB, chris-b 844 KB and no memory cap at all) while a crash-looping container checked for comparison (sharma) has a larger workspace (1.1 MB) and a larger openclaw.json (1,653 bytes vs. 481–537 bytes for the stable ones). The most likely explanation: OpenClaw's skill/workspace footprint has grown since the 1 GiB limit was set (73+ skills per BACKGROUND.md), and most instances now need more than 1 GiB just to boot — a few small/minimal instances still happen to fit. This is a plausible, evidence-backed hypothesis, not a confirmed root cause — confirming it properly (e.g., temporarily raising one crash-looping container's memory limit and watching it boot clean) is an execution-phase step, explicitly not done here since it would mean touching a live container.

1.4 Full classification

1.5 Rollup counts

• Real, active, verifiably working: 2 (chris-b, gaya-pool) + 2 more stable-but-lower-confidence (gayatri-test, pool-1s5q) = 4 containers
• Real, identifiable user, currently broken (crash-looping or down) — repair before any stop/keep decision: 11 containers (bharath-tj69, sharma, ryan-k, rohan-dx1u, pool-xoh3, pool-gr2k, roophus-5d71, pool-82c4, wife-b, son1-b, son2-b)
• Confirmed test/unclaimed, broken, empty content, safe to stop once formally verified: 7 containers (pool-e52e, pool-c6a8, pool-2d45, ssl-16be, final-dfdf, oimy-alpha-202c, test-newsignup)
• Already fully gone (Chen family + 24 others): no action possible or needed; DB hygiene only
• DB/reality mismatch (oimy-alpha-8352): DB hygiene only

1.6 The cost question, answered honestly

The Hetzner box is a single fixed-cost VM hosting everything (OpenClaw containers, the Hermes gateways, oimy-engine, Postgres access, the management API) — there is no per-container hosting bill to trim; stopping containers doesn't reduce the server bill. The $15–40/day figure is Claude Sonnet 4.6 token spend, and given that a container OOMs before ever reaching a model call, that spend is very likely concentrated almost entirely in chris-b (the one container with sustained real usage) — stopping the 7 confirmed-test containers will not meaningfully move that number, because they were not generating token spend in their current broken state regardless.

The real cost being paid right now is host contention, not dollars: 18 containers restart-crashing roughly every minute, each briefly spiking CPU during boot-then-OOM (docker stats showed ~50% of a core each while cycling), on a 16-vCPU box currently showing load average 9.8–12.4. That's a material fraction of the box's compute being spent on nothing but crash-and-restart, on the same host that also runs oimy-engine's live Grok-4.3 traffic and the Hermes gateways. Fixing the crash loop is worth more than any cost-cleanup number — it very plausibly improves the latency/stability of the systems Bharath already cares about being reliable, for free, as a side effect.

GOAL 2 — Execution plan (design only, not executed)

2.1 Phase 0 — Diagnose and fix the crash-loop fleet (NEW — pulled forward, blocking)

Not on Bharath's original list, but everything else here depends on trustworthy containers to test against, and it directly serves his own item 1 ("cycles running out of control").

• Pick one low-risk crash-looping container (recommend oimy-pool-e52e — confirmed empty, zero real data at risk) and, on a copy/clone, test raising the memory limit (e.g., to 2–4 GiB) to confirm the OOM hypothesis (§1.3) before touching any container tied to a real user.
• Once confirmed, apply the fix to provision-instance.sh's default and to each affected real-user container in a controlled order, verifying clean boot (no restart-count increment over a 10-minute observation window) before moving to the next.
• Priority order for repair, by stakes: oimy-pool-82c4 (Nitin, real captured onboarding data) and bharath-tj69 (Bharath's own) first; then sharma/ryan-k/ rohan-dx1u/roophus-5d71/pool-xoh3; then Chris's family (wife-b/son1-b/son2-b, currently fully down — restarting these needs Chris's/Bharath's sign-off since they're someone else's real family instances); pool-gr2k last (owner's continued interest unconfirmed).
• Add restart-count alerting (a simple periodic check against the existing management/host layer) so a repeat of "broken for six weeks, unnoticed" cannot happen silently again, regardless of which framework runs the fleet later.
• Risk to live traffic: none. This phase only touches already-broken containers; chris-b and the other 3 stable containers are explicitly out of scope.
• Effort: ~1 day for the diagnosis + a controlled repair pass across 11 real-user containers.

2.2 Phase 1 — Architecture audit: failure points, leaks, braking points (Bharath's item 1)

Beyond the crash-loop finding, direct answers to what was asked:

• Unbounded retries/circuit breakers in api.py: a targeted grep found no obvious while True/unbounded-retry pattern, but this was not an exhaustive code review — a real Phase 1 deliverable should be a line-by-line pass over the bridge (_kimi_jobs), the companion-call retry paths (Ollama/OpenRouter/DeepSeek fallback chain), and the Hermes send-retry logic once heavy Hermes exists. Not done here because it needs focused reading time, not a spot-check.
• Leaks: checked disk (53% used, 138 GB free), the main DB (oimy.db, 23 MB — small, healthy), and log directories (largest is 3.6 MB of Hermes logs) — no leak found on this box today. Worth a recurring check, not a current fire.
• Braking-point reliability, checked directly:
• DISABLE_BRIDGE — confirmed live in api.py (os.environ.get("DISABLE_BRIDGE"), two call sites), does what it says.
• The 5-minute watchdog cron (oimy_engine_eval_watchdog.sh) — confirmed active and installed; its known past gap (narrow driver-name allowlist) was already found and fixed once (2026-07-09 per project memory) — worth a fresh read to confirm it still covers every current eval/test driver name, since that class of gap has recurred before.
• reset_to_production_env.sh — confirmed exists, was used successfully during the actual 2026-07-03 incident it was built for.
• The honest overall read: the brakes that exist work when triggered. The actual gap was never "the brakes are unreliable" — it was that dangerous operations (mutating a live systemd unit's env for an eval, an entire container fleet crash-looping) had no brakes wired to them at all, and nothing was watching. Phase 1's real deliverable is closing the "nothing was watching" gap (alerting) more than re-testing brakes that already work.
• Effort: ~1–2 days for the full code read + wiring basic alerting (container health, restart counts, config-drift) beyond what already exists.

2.3 Phase 2 — Build /context + /context/writeback for real

This has been designed twice (2026-07-10 doc §3.3, reaffirmed 2026-07-11) and never built. Concrete steps, unchanged from the existing design, restated as an execution checklist: 1. On oimy-engine-dev (port 8090) only — implement POST /context per the 2026-07-10 doc's §3.3 spec: thin wrapper over EntityExtractor.get_context_summary, _build_profile_context, anchor_retrieval top-K, h30.update_constraint_ledger / get_constraint_ledger, Honcho insight, topic tracker — reusing the existing /chat JWT check. 2. Implement POST /context/writeback — routes a Hermes-composed turn through the same post-response paths /chat already uses (entity extraction, deep-profile store, H3.0 update, Honcho append), tagged source="hermes-heavy". 3. Add the session_meta field (per the reliability doc §2) so Hermes' own session/turn state can become authoritative for Hermes-routed traffic, once Phase 3 exists to consume it. 4. Smoke-test both endpoints directly with curl before any Hermes hook depends on them. 5. Risk to live traffic: none — dev engine only, /chat and prod untouched. - Effort: ~1 day, matches the original P2 estimate; this is the most schedule-predictable phase in the whole plan since the design work is already done twice over.

2.4 Phase 3 — Stand up Hermes-heavy for real

Per the recommendations already made in the two prior docs, now sequenced as build steps: 1. Docker migration first (per the Portal doc §4.3): dev profile → soak → live profile, pinned ghcr.io/nousresearch/hermes-agent:0.18.2, pipx kept as rollback until a clean week passes. Drop the hand-patched vendor fix (superseded upstream, confirmed in source). 2. New oimy-heavy profile, third Telegram bot token, model.default: x-ai/grok-4.3 initially (prove the harness on a strong model before the finetune, per the existing plan). 3. The pre_llm_call hook (fetch_grounding.sh) wired to Phase 2's /context; verify the hook mechanism live under the Docker-supervised gateway (this was flagged as unverified in the original design — first real test of it happens here). 4. The compiled SOUL.md (soul.d/ + build_soul.py) — the ~18 K module set already scoped in the 2026-07-10 doc. 5. A note on the existing provision-hermes-family.sh (found on Hetzner, dated 2026-06-09): a partial, single-server (non-containerized, non-/context-aware) Hermes profile provisioning script already exists as an earlier draft. It predates this design and should be treated as a reference/starting point to fold into the real provisioning flow, not a thing to run as-is. 6. Gemma-4 finetune serving: per the Portal doc §5.2 — this remains the real prerequisite gap (1.35 tok/s measured on Hetzner CPU is unusable). Stand up the Mac-mini-tunnel path for the family test, or Modal L4 for anything past that, before this phase can extend past the Grok-4.3 proving run. 7. Risk to live traffic: none in steps 1–4 (new profile, new bot token, dev engine only). Step 6 has zero interaction with prod by construction (separate serving endpoint). - Effort: ~3–4 days across steps 1–4 (matches the original P0–P3 estimate); step 6's timeline depends on which serving option Bharath picks and is not on this plan's critical path for proving the harness itself.

2.5 Phase 4 — Harden the full pipeline, both sides, to "well-oiled machine"

Explicitly not either/or, per Bharath's own framing — both api.py and the Hermes-heavy path need to be genuinely solid: - api.py side: close out Phase 1's code-review findings; keep the data/knowledge layer (entity facts, deep profile, anchors, ledger) as the thing this hardening protects, per the reliability doc's §3 recommendation — this is real product IP and deserves the same investment regardless of which layer composes the final reply. - Hermes side: once Phase 3's dev soak is clean, apply the same Docker-supervised, pinned-version pattern to the shared-GPU-pool serving layer (§5.2/§5.3 of the Portal doc) — decoupling per-user Hermes containers (cheap, CPU-only) from a small shared model pool, not one GPU per user. - Both sides: the alerting gap found in Phase 0/1 (nothing watches restart counts or config drift) should cover both oimy-engine and any new Hermes containers from day one — this is the concrete, buildable version of "well-oiled machine," not a vague aspiration. - Effort: ongoing; treat as the standing bar every subsequent phase is held to, not a fixed-duration task.

2.6 Phase 5 — Real testing

Two distinct kinds, both requested: 1. Simulated families against real spun-up Hetzner instances (not eval-script-against-API) — reuse the loop-phase0 longitudinal family-simulator design already built for the engine, but point it at real heavy-Hermes instances once Phase 3 is soaked, so the test exercises the actual deployment shape (container boot, session persistence, the hook, the writeback loop) rather than just the model+prompt. 2. A real configured instance for Bharath personally — the existing oimy-dev (@OimySuperB_bot) is the natural candidate since it's already his test bot; either promote it to the new oimy-heavy profile once proven, or provision a dedicated one, per his call. Must demonstrate proactive messaging, follow-ups, and real multi-turn engagement — not a single-turn smoke test — before being called "working." - Risk to live traffic: none — both are new/dev-scoped by construction. - Effort: ~2–3 days to wire the simulator against a real instance; the personal-test soak period is Bharath's own pace, not a fixed engineering estimate.

2.7 Phase 6 — iMessage via Photon

Status checked directly today: oimy-photon-imessage.service is healthy and has been running continuously for over a month (active (running) since 2026-06-09, 62.6 MB memory, no errors in the available journal window) — the 2026-07-11 deprioritization from earlier the same day evidently didn't mean it was broken, just parked. Per this new instruction, it's back in scope. Since it's already stable and already running, folding it into the same Hermes gateway as another platform (per the reliability doc §3's "bring everything into it" answer) is additive, not a rebuild — sequence this after Phase 3's Hermes-heavy pattern is proven on Telegram, so Photon integration inherits a working pattern rather than being a second simultaneous experiment. - Risk to live traffic: low — Photon is currently stable and isolated; integration work should happen on a branch/profile that doesn't touch the currently-running service until ready to cut over. - Effort: ~1–2 days, contingent on Phase 3 being done first.

2.8 Sequencing summary

Phase 0 (crash-loop fix, ~1 day)  ──┐
Phase 1 (audit + brakes, ~1-2 days)─┼──► Phase 2 (/context + writeback, ~1 day)
                                     │         │
                                     │         ▼
                                     │    Phase 3 (Hermes-heavy stood up, ~3-4 days)
                                     │         │
                                     ▼         ▼
                              Phase 4 (hardening, ongoing) ◄── runs alongside 3 & 5
                                              │
                                              ▼
                                    Phase 5 (real testing, ~2-3 days)
                                              │
                                              ▼
                                    Phase 6 (Photon/iMessage, ~1-2 days)

Where live production traffic is at risk, stated plainly: nowhere in Phases 0–3 or 5–6 as scoped — every step targets dev engine, new profiles, new bot tokens, or already-broken containers. The one moment real risk appears is whenever Bharath decides to cut real traffic over from today's oimy-engine.service (Grok 4.3, live Telegram bots) to the Hermes-heavy path — that's a Phase 4/5 boundary decision, not an engineering step with a fixed date, and should get its own explicit go/no-go per the rollback plan already specified in the Portal doc's §7.

Appendix — facts verified for this document (2026-07-11)

• Container inventory: docker ps -a on Hetzner, 24 total (22 OpenClaw user instances + oimy-browserless + oimy-suno-api infra).
• Restart counts: docker inspect --format '{{.RestartCount}}' for all 22 user-instance containers (full table in §1.4). Memory limits: docker inspect --format '{{.HostConfig.Memory}}' — 1,073,741,824 bytes (1 GiB) for all crash-looping containers, 0 (unlimited) for chris-b/browserless.
• Crash signature: docker logs on oimy-sharma and oimy-pool-xoh3 — repeated FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory, no application-level log lines before the crash.
• Host resources: free -h (30 GB total, 19 GB available), nproc (16), uptime (load avg 9.77/11.54/12.44), df -h / (53% used, 138 GB free).
• Neon Postgres: connection string read from /opt/oimy/management/server.js:874; 45 tables; instances/users/families joined for identity mapping; message_logs/session_logs/ usage_tracking confirmed empty (0 rows) via SELECT count(*) across all instances.
• Chen family: emma.chen.demo@oimytest.com / sarah.chen.demo@oimytest.com / david.chen.demo@oimytest.com, instances oimy-pool-09tu/oimy-pool-dcaf/ oimy-pool-ctgm, DB status stopped, confirmed absent from both docker ps -a and /opt/oimy/containers/ on disk.
• Workspace content: USER.md read directly for every container with a filesystem workspace; empty-template vs. filled-in distinguished by direct content read, not filename pattern alone (per the coordinator's explicit instruction not to classify by name only).
• oimy-photon-imessage.service: systemctl status — active since 2026-06-09, stable, 62.6 MB memory, no errors in the available journal.
• provision-hermes-family.sh: found at /opt/oimy/scripts/, dated 2026-06-09 — a partial, pre-existing, non-containerized Hermes provisioning draft, read in full.
• api.py braking points: DISABLE_BRIDGE (2 call sites), _kimi_jobs/held_turns (bridge staleness guard from the 2026-07-10 fix), oimy_engine_eval_watchdog.sh (5-min cron, confirmed installed), reset_to_production_env.sh (confirmed present) — all read directly, not inferred from memory.