Empirical Go AI research · 13

The most interesting Go AI frontier is how it exposes good moves humans cannot see

Fifty professional games, 500 positions, 397,561 actual search visits, 10,000 whole-game bootstrap replicates, and targeted audits up to 8,192 visits across all eight board symmetries. The question is no longer only which move is best, but why people miss it, whether the label is stable, and what deserves to be taught.

1. SOTA is a set of measurement roles, not an eternal champion

   As of 24 August 2026, KataGo v1.18.0 is the latest public engine. This study uses the b11 768-channel transformer released with v1.17.x as the strong teacher, a b10 512-channel transformer for cross-checking, and b18 Human SL for rank- and era-conditioned human policy. The official network page warns that approximate Elo values are not necessarily comparable across bots or runs. SOTA here does not mean one weight file is permanently the world champion. The teacher estimates candidate value, the cross-model check exposes model dependence, and Human SL models what a person might play. Their roles differ, and none is an error-free oracle.

2. From public records to a frozen 500-position design

   The source is the public-domain AEB/CWI professional archive. We selected ten Japanese title games in each of five eras from 1950–1969 through 2016–2023, then deterministically sampled ten main-line middlegame positions per game. Japanese rules and SGF komi were preserved, so no position was chosen after seeing its result. The main teacher searched at 128 and 512 visits. Human SL returned 5d, 9d, era-matched, and 2023 policies for every position; a second strong model checked 100 positions. Every interval uses 10,000 game-cluster bootstrap replicates rather than pretending ten adjacent positions from one game are independent evidence.

3. The single top move is a poor teaching unit

   Under the era-matched human policy, the exact AI top move has a median probability of 22.4%, while the set within one point reaches 80.0%—a 57.6 percentage-point gap. The exact move is below 1% in 13.4% of positions, but the entire good set is below 5% in only 4.0%. This separates two cases: a player never entered the AI's high-value region, or the player did enter it but missed the coordinate promoted by search noise and a tiny value margin. The former may reveal a conceptual blind spot. The latter often should not be called a mistake at all.

4. When the good set is less visible, played-move loss tends to rise

   For 482 positions, the played move remained in the same 512-visit root search and therefore had a comparable point loss. The Spearman correlation between loss and good-set surprisal is 0.419, with a game-cluster 95% interval of 0.327–0.503. To remove the possibility that a few entire games were simply harder, we centered both ranks within each game. The within-game correlation remains 0.413, interval 0.309–0.508. This supports the discoverability hypothesis, but it is still an association rather than evidence that a particular lesson causes learning.

5. Modern policy covers slightly more, but this is not an era strength table

   On the same positions, proyear_2023 assigns the one-point good set 2.2 percentage points more probability on average than the era-matched profile, with a clustered interval of 1.6–2.9. Earlier eras show larger gaps; the 2005–2015 interval crosses zero. This measures movement in policy distributions, not how many points stronger modern professionals are. The sample is not globally random, game structure and joseki change with time, and Human SL is a conditional distribution learned from records rather than a direct strength assay.

6. Budget, model, and symmetry audits overturn compelling stories

   Top-move agreement between 128 and 512 visits is only 74.8%; a second strong model agrees on 78.0% of its 100-position subset. In a D4 audit of ten anomalous positions, eight reached exact eight-way agreement at 2,048 visits, while only 67.5% of matched orientations preserved the same move from 512 to 2,048. Targeted 8,192-visit analysis produced opposite outcomes. Hane Naoki's position finally converged eight ways on B7. Sakata Eio's old A4 blunder label was invalidated, and no candidate remained within one point in all eight orientations. A trustworthy system must be able to withdraw a label instead of generating confident prose around a shallow result.

7. Next-generation review needs an evidence chain, not isBestMove

   A lesson candidate should pass a budget gate, a cross-model gate, and a D4 symmetry gate before Human SL classifies it as common, marginal, or hidden for the target player. Stable and hidden positions deserve priority; stable and common positions need only confirmation; unstable positions should show uncertainty or suppress automatic explanation. The missing endpoint is causal: randomize review ordered by point loss against review ordered jointly by loss, discoverability, and stability, then test delayed recall and transfer to unfamiliar positions. Teachability becomes a product fact only if the latter changes decisions in the next game.