Model archaeology · 20
Does a KataGo answer have a shelf life?
Change the network, engine, search budget, or symmetry and the first candidate can move. This article shows how to read the full analysis conditions before deciding whether an old result still holds, needs a note, or should be retired.
An answer is a run configuration, not one coordinate
A KataGo result depends on the network, engine version, rules, komi, search budget, symmetry, and randomization. Saving only “Q10 recommended” discards most of what another researcher needs to reproduce it.
One board shows two sets of analysis conditions side by side; more than the coordinate has changed. Separate raw policy from the searched first candidate
Official clarification says search does not rewrite raw policy, while order=0 is the search-ranked result. They may favor different regions; when reading an old result, distinguish a change in the network's first instinct from a change after search.
Line A is the network's first instinct; line B is search order=0. They answer different questions. An architecture change reprices every visit
KataGo 1.17 describes Transformer networks as stronger per visit and often faster than prior convolutional nets. Equal visits across architectures therefore need not mean equal compute or decision quality.
Equal visits are not equal computation; record architecture and elapsed cost together. Network Elo stays inside its training run
The public archive retains historical checkpoints but warns that approximate Elo is not comparable across training runs and carries roughly two-sigma uncertainty. We use dates and identities, not a spurious universal strength curve.
Historical checkpoints remain in separate training-run lanes instead of one false precision line. Symmetry is the cheapest useful stress test
After fixing nnRandomize, root symmetry, and visits, all eight board symmetries test whether a label depends on an arbitrary orientation. A split should be called search instability, not proof that an older model was wrong.
If eight equivalent orientations do not return to one candidate cluster, the answer is not yet a theorem. Use three expiry states instead of old versus new
An old answer may keep its coordinate but change rationale, change coordinate within a near-tie, or truly reverse the local conclusion. Those states call for retain, annotate, or retire—not a revolution headline after every upgrade.
Retain, annotate, and retire give editors more information than a binary model upgrade. Keep the analysis conditions beside the result
A complete record includes engine, network, rules, komi, maxVisits, nnRandomize, rootSymmetry, date, raw policy, and order=0. Rerunning those conditions after an upgrade reveals real answer drift instead of a changed software default.
A complete analysis record turns an old screenshot into a result that can be checked again.
Primary sources
Citation conditions for old AI analysis
- Engine and network are identifiable
- Rules and komi are saved
- Raw policy and order=0 are separate
- Randomization and symmetry are fixed
- Expiry has more than a new/old label