TsumegoBench: How Good Are LLMs at Go?
I gave current LLMs simple life-and-death problems. More interesting than the ranking was how quickly I could build a benchmark of my own.
On July 21, 2026, Shin Jin-seo beat KataGo.
He won the series 2:1. Shin played Black with a two-stone handicap. KataGo played White with 0.5 komi.
That is remarkable. It does not mean humans are suddenly better at Go than machines again. KataGo gave Shin two stones and, more importantly, it is a specialized Go engine. Nobody should build a diagnosis of civilization from three games anyway.
The article still reached me at exactly the right time. I had spent roughly an afternoon building a small Go benchmark for Large Language Models with Codex. Language models such as ChatGPT. LLMs, for short.
In my daily work, I deal with generalists. They write text and code, analyze documents, and are sold with increasingly grand promises about their reasoning, meaning their ability to think logically. I care less about how impressive that sounds in a model announcement. I want to know whether a model can solve a concrete task reliably.
So I gave it one.
A benchmark here is simply a fixed set of tasks with expected answers. Every model gets the same test, then I can compare the results.
A small test instead of a grand intelligence debate
The basic rules of Go do not take long to explain: black and white stones sit on a grid.
To read an actual position, the model has to reconstruct a board from coordinates, identify connected groups and their liberties, find legal moves, and consider possible replies. That requires pattern matching, spatial reconstruction, and tactical reading.
Tsumego reduce this to a local problem. A group has to live or die. They are a bit like checkmate puzzles in chess, except with liberties, eyes, and stones taking away each other’s breathing room.
For TsumegoBench, I prepared 100 public problems from GoProblems. Then I quickly decided to think smaller again. The first version, my 1.0.0 mini benchmark, contains 20 problems ranging from 30 to 25 kyu.
Those are community ratings, not a scientifically calibrated difficulty scale. To me, this is the easy range, a test of the basics. Before throwing hard dan-level problems at models, I want to know whether they can get the simple stuff onto the board at all.
The benchmark checks only the first move.
The model receives the board size, the color to move, the objective, and lists containing the coordinates of every black and white stone. It gets no diagram, cannot call a Go engine, and receives no examples. I also give it no higher-level extra instructions, no custom system prompt. Every problem starts as a single isolated request.
The answer must contain a move and a short explanation as JSON, a machine-readable format. For now, only the move counts toward the score:
- Is the coordinate valid?
- Is the point empty and the move legal?
- Is the move in the list of accepted solutions?
The reference moves come from GoProblems together with the problems. The benchmark validates their coordinates, legality, and stored variations. A different legal move still counts as wrong if it is not stored as an accepted solution.
That is a real limitation. A hit shows only that the model returned an accepted first move; it says nothing about whether the model read the full sequence. Conversely, a move missing from the stored solutions may still be correct on the board.
The first table looks clearer than it is
By now, I have run eight models through the 20 problems once each.
| Model | Correct | Rate | Cost |
|---|---|---|---|
| Claude Fable 5 | 17/20 | 85% | $4.92 |
| Gemini 3.1 Pro Preview | 16/20 | 80% | $2.91 |
| Claude Opus 4.8 | 15/20 | 75% | $5.23 |
| Claude Opus 5 | 14/20 | 70% | $2.33 |
| GPT-5.6 Sol | 14/20 | 70% | $2.99 |
| DeepSeek V4 Pro | 7/20 | 35% | $0.48 |
| DeepSeek V4 Flash | 6/20 | 30% | $0.04 |
| Gemini 3.6 Flash | 5/20 | 25% | $0.21 |
Together, that is 160 answers, just under 1.39 million tokens, and about $19.11. Roughly speaking, tokens are the small chunks of text used to meter model usage and cost.
The complete runs, including results per problem, answers, costs, tokens, and latency, are live at tsumego.owlex.de.
Fable is ahead in these runs. That is about as much as I can responsibly say. With 20 problems, a single hit changes the score by five percentage points. There was also only one run per model.
I would like to test more models and repeat the runs. Unfortunately, all of this costs money, and so far nobody is paying me to throw Go problems at LLMs.
The problems did not behave as neatly as their kyu ratings might suggest either. Two were solved by all eight models. TB-1.0.0-015, also from the easy range, was solved by none of them.
The problem sits in the corner, and the stored solution is a corner move. Moves like that can be difficult for Go beginners too. I find it pretty funny that all eight tested LLMs failed at this corner.
There were also five invalid answers: four illegal moves and one response in the wrong format.
A good explanation does not save a wrong move
More interesting to me than Fable taking first place were the explanations.
The models wrote about liberties, eyes, attacking a group, and possible replies. Even for wrong or illegal moves, their explanations often sounded clean and carefully reasoned.
I can automatically reject broken JSON. I can do the same when a model tries to play on an occupied point. A plausible explanation attached to a wrong result is more uncomfortable. At first, it looks like quality. If I cannot check the task myself, I might never notice the mistake.
In the end, a stone has to go on a specific point. Whether it belongs there can be checked on the board.
More measured effort did not rescue the weaker runs either. DeepSeek V4 Pro used 335,897 tokens and took a little over five minutes per problem on average. It got seven correct. V4 Flash used 233,637 tokens, took just under two minutes per problem, and got six correct.
The more resource-intensive run produced one additional correct move. More tokens and a longer runtime did not automatically produce a better result in these runs.
I wanted a benchmark. After an afternoon, I had one.
I had GoProblems, Codex, and a concrete question, but no AI lab or benchmark department.
Of course it was not finished. The reference variations needed checking. Coordinates and legality needed tests. Codex could take some of that work off my hands. I still had to decide which answers count and which failures matter in my problem space.
My next model choice does not require a public benchmark suite. I can use cases from the actual task and aim the evaluation at the mistakes that would cause problems for me later.
I care about four things:
- Does the model solve my task correctly often enough?
- How much does that cost in my actual configuration?
- How long does it take?
- Can I detect errors automatically, or do I need an expert every time?
Fable had the most correct answers in these runs, but it cost more than Gemini and GPT. DeepSeek Flash was almost free and solved six problems. For a low-risk task with good automated checks, a cheap model may be perfectly adequate. If a convincingly phrased error becomes expensive later, the same calculation looks very different.
A general leaderboard does not account for the cost of my errors.
Please do not cosplay this as science
TsumegoBench has clear limitations. The models did not even run with identical settings. For DeepSeek and Gemini, the randomness setting known as temperature was set to 0. Claude and GPT received a high budget for internal reasoning. Other values stayed at each provider’s defaults. OpenRouter, the routing service handling my model requests, also did not send every run to the same technical provider.
There are only 20 problems and one run per model. The dataset is public and may have appeared in training data, in whole or in part. The coordinate-list input also tests a very particular kind of Go understanding. The results could look different with an image or an interactive board.
For my decision, that is enough: how often a model gets it right here, how convincing its mistakes look, and what the run costs me.