Model archaeology · 20
Does a KataGo answer have a shelf life?
Change the network, engine, search budget, or symmetry and the first candidate can move. Rather than inventing a historical rerun, this article defines a reproducible answer passport for deciding whether old analysis remains citable.
An answer is a run configuration, not one coordinate
A KataGo result depends on the network, engine version, rules, komi, search budget, symmetry, and randomization. Saving only “Q10 recommended” discards most of what another researcher needs to reproduce it.
One board is compared under two answer passports; the experiment changes, not merely the coordinate. Separate raw policy from the searched first candidate
Official clarification says search does not rewrite raw policy, while order=0 is the search-ranked result. They may favor different regions, so model drift and search drift must be reported separately.
Line A is the network's first instinct; line B is search order=0. They answer different questions. An architecture change reprices every visit
KataGo 1.17 describes Transformer networks as stronger per visit and often faster than prior convolutional nets. Equal visits across architectures therefore need not mean equal compute or decision quality.
Equal visits are not equal computation; record architecture and elapsed cost together. Network Elo stays inside its training run
The public archive retains historical checkpoints but warns that approximate Elo is not comparable across training runs and carries roughly two-sigma uncertainty. We use dates and identities, not a spurious universal strength curve.
Historical checkpoints remain in separate training-run lanes instead of one false precision line. Symmetry is the cheapest useful stress test
After fixing nnRandomize, root symmetry, and visits, all eight board symmetries test whether a label depends on an arbitrary orientation. A split should be called search instability, not proof that an older model was wrong.
If eight equivalent orientations do not return to one candidate cluster, the answer is not yet a theorem. Use three expiry states instead of old versus new
An old answer may keep its coordinate but change rationale, change coordinate within a near-tie, or truly reverse the local conclusion. Those states call for retain, annotate, or retire—not a revolution headline after every upgrade.
Retain, annotate, and retire give editors more information than a binary model upgrade. Publish an answer passport with every conclusion
At minimum store engine, network, rules, komi, maxVisits, nnRandomize, rootSymmetry, date, raw policy, and order=0. A future audit can then rerun the same passport and measure real drift.
The answer passport turns an old screenshot into a rerunnable research record.
Primary sources
Before citing old AI analysis
- Engine and network are identifiable
- Rules and komi are saved
- Raw policy and order=0 are separate
- Randomization and symmetry are fixed
- Expiry has more than a new/old label