Service postmortem · 27
How one game error exposed a chain of service regressions
On August 30, a game with unsupported komi first triggered a KataGo parameter error. The same investigation then uncovered an oversized stream completion, a registration runtime mismatch, repeated Token creation, and old-client sync failures. This is the timeline, measured impact, recovery evidence, and unfinished work.
The first visible failure arrived at 20:47
At 20:47 CST on August 30, 2026, a Web full-game review sent komi that was neither an integer nor a half-integer to Modal. KataGo rejected it and the job failed after zero milliseconds of GPU time. The error page preserved a query identifier and search budget, but it did not explain early enough that the imported game's komi was unsupported.
The invalid-komi request at 20:47 begins the timeline; its retries and the second provider failure remain separate tracks. Thirty failures were two retry chains
Database records from 20:47 to 21:13 show five invalid-komi attempts from one user and one game, plus 25 oversized-response attempts from a second user. Modal completed 43 jobs in the same period. Automatic or manual retries amplified two faults; this was not 30 affected users, nor was the entire analysis service continuously unavailable.
Impact is recounted by users, games, and successful jobs so 30 attempts are not presented as 30 affected people. Invalid komi should stop before a provider call
KataGo accepts integer or half-integer komi from -400 to 400. The old path carried an invalid value to the provider and surfaced an internal error. Web, REST, and streaming entry points now share one validation contract and return an explicit HTTP 400 before billing or GPU dispatch. The service does not silently round, because changing komi changes the game being analyzed.
Validation now sits before billing and GPU dispatch, preventing unsupported values from crossing the Worker boundary. The terminal event repeated the whole stream
The second fault was in the Modal stream. Intermediate events had already delivered each result, then the completion event attached the full result array again. A long full-game review exceeded the provider's response-size limit. Completion now carries only a small summary such as usage, while results remain streamed. The deployed production App returned one result and complete usage in a 64-visits verification query.
Results still arrive incrementally while the complete event shrinks from duplicated payload to a small usage receipt. The review found three independent compatibility regressions
Registration was not caused by invalid komi: its password-derivation count exceeded the Cloudflare Worker runtime limit, so execution stopped before an account row existed. Client sync also used a redirect mode unsupported at the Edge, while some older clients lacked a stable device slot and created another API Token on reconnect. These faults had no direct causal link to the game, but all exposed a missing server-to-old-client release contract.
Registration, redirect handling, and Token rotation are separate branches found in one review, not one shared root cause. Recovery covered the primary and fallback paths
Compatibility fixes reached Cloudflare at 15:54 on August 31. At 16:15, the REST fallback began consuming the RunPod stream unconditionally and rebuilding results when terminal output was absent; Modal Fast was redeployed afterward. At 16:18, a production RPC on an NVIDIA L4 returned one result and 70 visits. A new successful registration row after the Worker release also confirmed that account creation was writing again.
Cloudflare, Modal, and RunPod were verified separately; a deployment receipt was not treated as production-behavior proof. Future releases need one compatibility matrix
The release gate will cover Web, current and older native clients, REST, WSS, registration, automatic Token rotation, Modal streaming, and RunPod fallback, with response-size tests for long games. Registration also needs privacy-safe events before database insertion so “no account record” no longer gets confused with “no request.” This release lacked new frontend source maps; service behavior is unaffected, but symbolication is delayed, so release-credential checks join the follow-up work.
The new release contract places clients, protocols, providers, fallback, and observability in one preflight matrix.
Repairs now in place
- Komi validation precedes billing and provider dispatch
- Modal completion no longer duplicates the result set
- RunPod fallback always consumes and reconstructs streamed output
- Registration parameters fit the Cloudflare Worker runtime
- One device rotates its Token automatically; the cap moved from 5 to 20
- Edge redirects and legacy IGS byte streams remain compatible
- A lost WebGL context falls back to a visible board instead of a blank surface