Hermes vs. api.py: The Reliability Question, and Two Deployment Shapes

Date: 2026-07-11 (follow-up, same day) Author: Fable 5 Status: Recommended Builds on: HEAVY-HERMES-ARCHITECTURE-20260710.md and HEAVY-HERMES-NOUS-PORTAL-ARCHITECTURE-20260711.md — this doc does not replace either; it answers a bigger question those docs didn't: given Wednesday night's bug list, should OiMy lean on Hermes much harder — including data and memory, not just the chat channel — and should it be deployed self-contained-on-Mac or on GPU cloud modeled on OpenClaw? Follow-up: the concrete audit + phased execution plan requested after this doc — including a major live-fleet reliability finding on the OpenClaw side — is in HETZNER-AUDIT-AND-EXECUTION-PLAN-20260711.md (https://cdn.oimyai.com/docs/hetzner-audit-and-execution-plan-20260711.html).

This doc revises one part of the prior recommendation (§2 below) based on direct evidence, and holds firm on another part (§3) — both for stated reasons, not by default.

0. TL;DR

1. The Wednesday bug list is not good evidence that api.py's product logic is unreliable. Read closely, it's three things: (a) one real architectural debt item — no first-class session concept, four subsystems independently inventing overlapping time windows; (b) two concurrency bugs from a hand-rolled ThreadingHTTPServer handler (confirmed at api.py:6610 — real threads, shared state, needed threading.local()); (c) two process failures — an eval script that flips live production's systemd env and a watchdog whose driver-name allowlist didn't include a new script name. (b) and (c) are not a statement about whether custom code is inherently unreliable; they're the specific, ordinary failure modes of a fast-moving small team running experiments directly against a box serving real users, without version control (now fixed — verified git status clean, main tracking origin/main as of today) or environment isolation between eval and prod.

2. The one real architectural item — session/turn-boundary bookkeeping — is exactly the kind of undifferentiated infrastructure a mature agent harness is built to own. This is the honest, narrow case for leaning on Hermes harder, and I'm making it: let Hermes' own session store become the authoritative turn/session boundary for Hermes-routed traffic, and retire the engine's bespoke version (_session_start_index(), the rolling topic cache, thread TTL) rather than keep maintaining two competing implementations.

3. Do not move data and memory into Hermes. Entity facts, the deep-profile pipeline, coaching-anchor retrieval, the constraint ledger, H3.0 — none of Wednesday's bugs were in this layer, this layer is OiMy's actual differentiation and months of tuning, and hermes-agent is currently changing at a rate (~1,720 commits / 998 PRs / 370+ contributors between 0.17.0 and 0.18.0; ~667 more commits in the ten days since) where deep coupling to its internals is a worse reliability bet than the status quo, not a better one — confirmed firsthand: the Telegram platform module itself relocated (gateway/platforms/telegram.py → plugins/platforms/telegram/adapter.py) between the two versions this project has touched. The existing /context bridge design (engine owns data, Hermes composes and fetches per turn) is the correct shape precisely because it only touches Hermes' stable, documented hook surface, never its internals.

4. "A well-tested harness" needs a caveat neither framework currently earns on its own. Live evidence found today: one of the 22 OpenClaw production containers (oimy-test-newsignup) has been crash-looping every ~66 seconds — 9,184 restarts since July 4, unnoticed for a full week. OpenClaw is the "proven, works well" comparator in Bharath's framing, and it has an ops failure of exactly the silent, unmonitored kind Wednesday's api.py fixes were chasing. The lesson isn't "OpenClaw is bad" — it's that framework maturity and operational discipline are different variables, and the Wednesday exercise conflated them. A framework swap doesn't buy monitoring, alerting, or config isolation; those have to be built regardless of which harness composes replies.

5. Shape A (Mac-bundled) and Shape B (GPU cloud, OpenClaw-pattern) are not really competing architectures — they're two different bottom layers under the same provisioning pattern. Recommendation: build Shape B as the default going forward (replacing OpenClaw's role over time), correcting one implicit assumption in how it was proposed (§5.2 — share a small GPU-serving pool across many lightweight per-user Hermes containers, don't dedicate a GPU per user). Offer Shape A as a real, deliberate product tier for privacy-maximalist households — not the default path — because of an honest, unavoidable availability gap (§4).

1. What Wednesday's bug list actually shows, read as a set rather than a headline count

Net read: one architectural debt item genuinely argues for leaning on a mature harness's session layer. Two items are ordinary concurrency bugs, fixed the ordinary way (and now covered by 21 new unit tests). Two items are process failures that a framework swap does nothing about — they need environment isolation (a separate eval sandbox, not env-var flips against the live service) and a watchdog that fails safe on unrecognized processes instead of silently "correcting" them. Recommend fixing the process items this week, independent of any Hermes decision — they're cheap, they're the most direct fix for "our own tooling flipped prod to eval config without anyone knowing," and they'll still be a risk even after a full Hermes migration if the same practice continues.

One correction to the record: /opt/oimy-engine is now git-tracked (git status clean, main up to date with origin/main, verified today) — the standing ask from the overnight log has been resolved since Wednesday. Worth noting because it removes one line item from "how unreliable is our own practice," not because it changes anything above.

2. The revision: let Hermes own session/turn state for Hermes-routed traffic

The prior Heavy Hermes design (2026-07-10) already has Hermes maintaining its own session persistence (sessions/, FTS-indexed state.db, context_window: 20) and already says "Hermes owns conversational flow... keep it" (§3.5). What's new here, in direct response to the session-boundary bug: make that ownership real by deletion, not just addition.

• For any user routed through heavy Hermes, the engine should stop computing its own session boundary (_session_start_index(), the rolling topic-summary cache, the 300s thread TTL) and instead treat Hermes' session/turn context as ground truth for "what turn is this, what's the current session." The /context endpoint's [CURRENT TOPIC] / [OPEN THREADS] blocks become read-only knowledge-layer additions layered onto a Hermes-supplied turn boundary, not a competing computation of the boundary itself. Concretely: /context accepts a session_meta field the hook already has (Hermes knows its own session id / turn count / gap-since-last-message) and uses it instead of recomputing an independent guess.
• This is a narrow, well-scoped change — it touches the one class of bug that's actually infrastructure-shaped, and nothing else. It does not touch entity facts, profile inference, anchors, or the ledger.
• Sequencing: this is a Phase 2 refinement (on top of the prior doc's P2//context build), not a prerequisite — build /context first per the existing plan, then land this once heavy Hermes has real traffic to validate against.
• Honest limit: this only helps Hermes-routed traffic. Direct /chat callers (web app, Rork) still need the engine's own session-boundary logic — the fix that shipped Wednesday (assistant-turn restoration, real timestamp-based gap detection) stays as the source of truth for those paths. Don't delete it; scope the Hermes-side change to Hermes-side traffic only.

3. The data/memory question, answered directly: no

"Bring data and memory into Hermes" would mean one of two things, and both are worse than what exists:

(a) Reimplement entity-fact extraction, deep-profile inference, anchor retrieval, and the constraint ledger as Hermes memory/skills. This means porting real, tested Python (anchor retrieval's threshold/skill-boost/demotion logic, the tiered profile-context assembly, the Opus-driven deep-inference pipeline) into Hermes' plugin surface — which reorganizes between minor versions (confirmed: the Telegram platform module moved between 0.15.2 and 0.18.2) and changed ~1,720 commits' worth between two recent releases. Every dependency this project has on Hermes internals is a dependency on a surface that is not stable release-to-release. The stable surface is the documented pre_llm_call hook contract ({"context": "..."}) and plain HTTP — both unchanged across the versions checked. Build against that; don't build against Hermes' guts.

(b) Keep a second, parallel copy of the data (e.g., mirrored into Honcho or Hermes' memories/ plugin) so Hermes can "own" it directly. This reintroduces the exact contradiction problem the prior doc already ruled out (§3.5 of the 2026-07-10 doc): two long-term memories about the same family drift and disagree ("didn't you say Emma was 7?"). It also forks the single-sourcing the gold-standard dataset pipeline depends on — the training pipeline's assemble_contexts.py renders the same blocks /context serves; a second data copy breaks that contract silently.

The correct scope stands from the prior doc, and Wednesday's bugs don't move it: engine owns data and memory, permanently, regardless of which harness composes the reply or which model generates it. This isn't inertia — it's that the specific problem being solved (unreliability) was never located in this layer, and the specific cost of moving it (coupling to a fast-churning internal API surface) is real and measured, not hypothetical.

4. Shape A: fully self-contained in the Mac app — evaluated honestly

The existing OIMY-MAC-APP-PLAN.md already scopes bundling Ollama + Gemma 4 12B QAT + api.py + Whisper + TTS + a Cloudflare tunnel into one .app, ~1 week for Phase 1. Substituting Hermes for api.py as the reasoning/orchestration layer inside that bundle is architecturally plausible — Hermes is a Python package, same bundling mechanism — with one real added cost: hermes-agent's dependency footprint is materially heavier than api.py's (openai, httpx[socks], croniter, jinja2, pydantic, cryptography, PyJWT[crypto], ruamel.yaml, prompt_toolkit, rich, tenacity, vs. api.py's flask/requests/ pyjwt/httpx) and it's designed to run as a supervised, long-lived gateway process (its own hook/skill/cron subsystems), not as a simple bundled subprocess — the Docker-image supervision this doc's companion piece recommends for server deployment doesn't exist inside a .app bundle. Non-blocking, but a real Phase-1 scope increase, not a drop-in swap.

The honest problem, addressed directly as asked: the product's core moment — a parent messages Oi about a meltdown, wants an answer now — requires the Mac to be on, awake, connected, and the tunnel alive, with no supervision, alerting, or failover if any of that fails. The Mac-mini-as-always-on-hub framing genuinely helps (a dedicated always-on machine, not a laptop that sleeps on lid-close) but does not remove the gap: power outages, ISP outages, silent OS-update reboots, a wedged cloudflared process, or the app simply crashing all produce silent unavailability with nobody watching — precisely the failure mode this whole exercise is trying to eliminate, just moved from "our server, unsupervised process" to "a family's kitchen counter, unsupervised process, with the family itself as the only possible responder." For "the kid is having a meltdown, message Oi," this is a worse availability story than today's Hetzner deployment, not a better one, however private it is.

Recommendation for Shape A: build it, but position it as an opt-in, privacy-maximalist product tier — not the default distribution path and not the answer to the reliability question. It's a genuinely good offering for households that want zero cloud dependency and are willing to own their own uptime (which may include Bharath's own family, if that's the tradeoff he wants for himself) — market it as that, honestly, rather than as the general-availability path.

5. Shape B: GPU cloud, modeled on OpenClaw's proven pattern — evaluated, with one correction

5.1 What's actually proven today

app.oimyai.com runs 19–22 Docker containers (live count today: 19, one crash-looping) on the Hetzner box, each an OpenClaw gateway on port 18789 (mapped to a 19000-range external port), a separate proxy VM running Traefik for SSL/SNI routing, and a management API on port 18800 (confirmed live and auth-gated) that provisions containers via SSH. This genuinely is a proven per-user/pool isolation pattern at OiMy's current real scale, and it is the right skeleton to reuse — the management layer, the per-user container boundary, and the SSH-based provisioning don't need to be reinvented.

5.2 The correction: don't dedicate a GPU per user

As proposed, Shape B reads as "swap OpenClaw+Sonnet-4.6 for Hermes+Gemma4-finetune, per container, per user." Taken literally that inverts the cost model in the wrong direction: today's containers are cheap to run and expensive per-message (Sonnet 4.6 metered by token, $15–40/day aggregate across 22 containers, near-zero idle cost). A literal one-Ollama-per- container model, if each container needs GPU-backed inference to hit usable latency (confirmed today: 1.35 tok/s measured on Hetzner CPU — unusable), means paying for GPU capacity per container whether or not that user is actively chatting — the opposite of today's pay-per-message economics, and probably far more expensive than $15–40/day at any real user count.

The right shape decouples two things that don't need to scale together: - Per-user isolation (identity, session, channel, JWT, rate limits) — many cheap, CPU-only Hermes gateway containers, one per user/pool, exactly like today's OpenClaw containers. This is where the proven pattern applies directly. - Model compute (the Gemma 4 finetune actually generating text) — a small, shared pool of GPU-backed serving endpoints (Modal L4 autoscaling, or a fixed dedicated box) that all Hermes containers call via base_url, the same way this project's own eval infrastructure already shares one Ollama endpoint across many callers via a tunnel. Ollama/vLLM handle concurrent requests against one loaded model; the pool sizes with concurrent active conversations, not total provisioned users — the same shape as the current per-token billing implicitly already has, just metered in GPU-seconds instead of tokens.

This is also, not incidentally, exactly what the /context-bridge architecture already assumes at the engine layer (one engine, many Hermes profiles) — Shape B is that same decoupling applied one layer down, to model serving.

5.3 Recommendation

Build Shape B as the default going forward, replacing OpenClaw's role for OiMy's production/hosted product over time — reusing the existing management API and per-user container pattern, fronting a shared Modal-L4 (or equivalent) serving pool for the finetune, gated on the same serving-readiness work already flagged as the real blocker in the companion doc (§5.2 there). This gets: OpenClaw's proven isolation pattern, Hermes' Docker-native supervision at the per-instance level, real availability (no dependency on any family's own hardware), and a cost model that scales with usage rather than headcount.

6. Honcho — self-hosting confirmed, lowers migration friction either way

Flagged as an open question in the companion doc (the one third party holding derived real-user insight). Confirmed today: Honcho is AGPL-3.0, self-hosts via Docker Compose (FastAPI + Postgres/pgvector + Redis + an LLM for background reasoning), and a maintained community project (elkimek/honcho-self-hosted) packages this specifically for Hermes Agent integration with no code changes required on the Hermes side. OpenClaw also has native Honcho support. This is good news independent of everything above: whichever shape wins, self-hosting Honcho (closing the one remaining real data-custody gap) is low-friction and not blocked on any other decision in this doc. Worth scheduling regardless.

7. Direct answers to the two framing questions

"Is this reliability problem best solved by a framework swap, or by hardening api.py?" Neither, as posed. It's solved by (a) fixing the two process gaps this week (isolated eval environments, a watchdog that fails safe) — framework-independent, cheap, real; (b) letting Hermes own session/turn bookkeeping for Hermes-routed traffic, because that's the one bug class that's genuinely infrastructure-shaped and Hermes has already solved it — this is the Hermes lean-in, scoped to where the evidence points; (c) continuing to hardened and own the data/knowledge layer in the engine, because none of the instability lived there and it's the actual product. "Unstable" is not a permanent property of api.py's product logic — it's a symptom of thin process discipline plus one real, now-fixed design gap, the same shakeout any actively-developed system goes through (Hermes' own 0.17→0.18 changelog — "resolved 100% of P0/P1 backlog, ~700 items" — is the identical pattern one layer up, just with 370 contributors finding the bugs instead of one overnight session).

"Bring data and memory into Hermes, not just the chat channel?" No — held firm, for reasons specific to this week's new evidence (§3), not by default repetition of the prior answer.

Appendix — facts verified for this document (2026-07-11)

• oimy-engine.service: ThreadingHTTPServer(("0.0.0.0", port), OiMyHandler) — /opt/oimy-engine/api.py:6610 (confirms the concurrency-bug root cause class).
• /opt/oimy-engine git status: clean, main up to date with origin/main — the overnight log's standing ask has been resolved since Wednesday.
• OpenClaw production: docker ps on Hetzner, 2026-07-11 — 19 running containers (one, oimy-test-newsignup, crash-looping: docker inspect shows RestartCount=9184, container created 2026-07-04 18:34 UTC, log line repeating every ~66s: Missing config. Run \openclaw setup` or set gateway.mode=local). Management API confirmed live and auth-gated atlocalhost:18800({"error":"Unauthorized"}`).
• /home/exedev/oimy/oimy-skill-engine/BACKGROUND.md: source for OpenClaw's cost ($15–40/day, Claude Sonnet 4.6 via OpenRouter, 22 containers), the proxy-VM/Traefik split, and the original rationale for building api.py as a cheap/on-device-capable successor.
• OiMy-Platform/docs/OIMY-MAC-APP-PLAN.md: Phase 1 scope (~1 week), full bundled stack, dependency list for api.py's current bundled requirements (flask, requests, pyjwt, httpx) — compared against hermes-agent's METADATA dependency list (checked in the companion doc's research) to derive the packaging-cost delta in §4.
• hermes-agent velocity: GitHub release notes for v0.18.0 ("The Judgment Release," 2026-07-01) — since v0.17.0: ~1,720 commits, 998 merged PRs, 2,215 files changed, ~251,000 insertions, 949 issues closed, 370+ contributors; since v0.18.0, ~667 more commits across ~990 files as of v0.18.1/0.18.2 (2026-07-07). Module relocation (gateway/platforms/telegram.py → plugins/platforms/telegram/adapter.py) independently confirmed by direct source inspection in the companion doc.
• Honcho self-hosting: elkimek/honcho-self-hosted (GitHub) — Docker Compose stack (Honcho API + Postgres/pgvector + Redis + worker LLM), documented Hermes Agent integration requiring no code changes; AGPL-3.0 upstream (plastic-labs/honcho).