Explanation trust · 26
Does a Go-talking language model understand—or only sound as if it does?
LoGos mixes structured Go data, long reasoning, and reinforcement learning and reports professional-level move play. Yet fluent commentary can contain atomic claims unsupported by the board or engine. Validate claims, not style.
Separate move strength from explanation quality
A model can choose a strong coordinate while being wrong about thickness, ladders, or sente; fluent terminology does not prove strong play either. The abilities need independent benchmarks.
One response splits into a move track and a language track with separate scores. Structured board input beats prose alone
LoGos mixes expert Go data with general long reasoning and supplies rules-aware board tooling and KataGo-Bench. Its professional-level claim belongs to its reported tasks and evaluation setup.
Coordinate-only prose and explicit board state enter different validation paths. Encoded concepts are not automatically verbalized faithfully
ACL 2022 probes policy networks with 10,000 human-annotated games and nontrivially predicts concepts such as ko and atari from later layers. A signal in representation still needs a faithful language bridge.
A concept probe inside the network remains one layer away from natural language. Decompose commentary into atomic claims
“This attacks White while keeping sente” contains at least legality, target identity, forcingness, and future-value claims. Route each to rules, KataGo variations, or expert references to locate hallucination.
One fluent paragraph becomes four independently checkable claim cards. Use tool-augmented evaluation, not LLM self-grading
The 2026 ACT-Eval chess study finds many incorrect subclaims even in strong models without tools. Tools improve factuality, while coverage of expert concepts remains limited. Go needs the same claim-level audit design.
One overall score separates into factual correctness and concept coverage. Use local continuity to separate evidence from fluent wording
Switch A and B from the same historical position and verify the explanation move by move. If a shape, sequence, or function appears only in A but the same rationale is pasted onto B, that rationale is not supported by B's line. Win rate and score lead quantify local efficiency; they do not replace an explanation of the actual stones.
At moves 5–6, A stays in the center while B jumps to two corners; the same central-fight explanation would not be grounded for B. Publish an explanation contract
List legality, candidate rank, PV support, rule conditions, factual claims, pedagogical analogy, and uncertainty. Language organizes; board tools constrain; experts audit conceptual completeness.
The contract shows which evidence warrants every part of the explanation.
Models, data, and evaluation
Before publishing AI Go commentary
- Score move and explanation separately
- Validate coordinates with rules
- Split prose into atomic claims
- Do not overread a PV
- Separate factuality from concept coverage