Search-stability research brief · 15

Does giving KataGo thousands more visits really make the answer reliable

Five hundred professional positions, two strong models, and eight board symmetries show that more search usually reduces median drift but can expose a split first. Stable coordinates, stable values, and trustworthy teaching labels are different claims.

1. Visits are a search budget, not answer confidence

   Visits count tree-search allocations; they do not state that the first candidate has converged. Position complexity, candidate margins, thread scheduling, and the network prior all change what one budget means. Five hundred twelve visits therefore cannot be translated into one reliability percentage, and the count is not a confidence interval.

2. One quarter of top moves still change from 128 to 512 visits

   Across the frozen 500 professional positions, the b11 Transformer agrees on its top move at 128 and 512 visits in 74.8% of cases. Median root score drift is only 0.131 points, while the 90th percentile reaches 0.600. The typical position looks calm, but the tail can overturn teaching language; reporting only the median hides the labels with the highest risk.

3. A second strong model also reorders the candidates

   On a 100-position cross-teacher subset, b11 and a separate b10 Transformer agree on the top move at the same 512-visit budget in 78.0% of positions. Agreement does not prove truth, but disagreement shows that a claim depends on architecture, training, or search interaction. Dramatic high-loss, low-human-probability cases deserve this cross-check first.

4. Rotations and reflections reveal errors hidden by one orientation

   Ten anomaly-enriched positions were transformed through all eight D4 symmetries and mapped back to canonical coordinates. Only 60% had exact eight-way top-move agreement at 512 visits; 2,048 visits raised that to 80%. The transforms are repeated measurements of one position, not eight independent samples, and test whether learned approximate invariance supports a unique label.

5. More search can expose a split before resolving it

   A Hane Naoki audit position chose B17 in all eight orientations at 512 visits. At 2,048, four rotations chose B17 and four reflections chose B7; only at 8,192 did all eight converge on B7. Deeper search did not monotonically strengthen the original answer. It first showed that shallow unanimity was accidental, then found a new stable candidate.

6. A stable coordinate can still hide an unstable value

   Some positions keep one coordinate across budgets and orientations while their root score range remains above one point. For teaching, stable “where” does not imply stable “how much,” and the latter determines whether words such as “clearly” or “must” are justified. Preserve the candidate set, top coordinate, value range, and principal variation instead of one bestMove field.

7. Stopping rules should follow consequence, not one fixed number

   Routine review can stop when candidate set, coordinate, and score are stable. A label that drives intensive training, public criticism, or historical interpretation should escalate budget and add model or symmetry checks. If deeper search remains split, expose uncertainty or withdraw the label. Compute is best spent at boundaries where it can change the teaching action, not spread evenly across every move.