The Dog That Did Not Bark: Can an LLM Find the Clues Nobody Points Out?
Introducing the WORTH Benchmark — arXiv:2610.27597 · Paper page on Hugging Face
In Arthur Conan Doyle's "Silver Blaze," a champion racehorse disappears from its stable at night. Its trainer is found dead on the moor. Inspector Gregory does solid work. He has a suspect, a motive, and a lot of facts. Then comes the most quoted dialogue in all the Sherlock Holmes stories:
"Is there any other point to which you would wish to draw my attention?"
"To the curious incident of the dog in the night-time."
"The dog did nothing in the night-time."
"That was the curious incident," remarked Sherlock Holmes.
There was a dog in the stable. Nobody heard it bark that night. This fact was in the case record. Gregory thought it was just information, not a clue. So he could not find the truth: the horse was taken by someone the dog knew.
Now suppose we turn "Silver Blaze" into an exam question. Then "the dog did not bark at night" will surely be marked as a clear clue:
"Given: the stable dog did not bark. Question: who took the horse?"
This is a standard logic question. Most people who get this question will probably do better than Inspector Gregory.
The question we want to ask
Most reasoning tests are like this exam question. The hardest part has already been skipped. The facts are given to the model as clear premises, and the model's job is to put them together. This is a real skill. But in the real world, reasoning does not start here. Nobody gives a private detective, a doctor, or an intelligence analyst a list with the title "clues." What they get is a big mix: useful things and useless things together, and sometimes misleading information too.
This is the real situation of reasoning in the real world.
Starting from real-world reasoning, WORTH asks this question: When nobody points out the evidence, and the information is not fully on the model's side, can the model find where the reasoning starts?
"Nobody points out" and "not fully on the model's side" are equally important. This is also the difference between WORTH and "another detective test." In a normal test, every fact is a neutral premise, placed there to help the test-taker. In WORTH, the person who wrote the text is partly working against the reader (the LLM).
Why original mystery stories
A mystery story is a form of content built around one thing: it gives the evidence to the audience, and it hides the evidence at the same time. A fair mystery novel must put the clues on the page. A good mystery novel makes you walk right past them.
All the stories in WORTH are written by us. Our team has many years of experience in fiction writing. Our work has won fiction awards, and we have published full-length commercial novels. In WORTH, the key clue sentences of each story are decided before the writing starts, and the story is built around them. So we do not need to guess afterwards which details were planted. We already know, and we know what each one is for. Because the stories are original, a model cannot have read the answer somewhere else.
How we test
In each run, the model gets a complete story in Chinese. Every sentence has only a neutral number, with no marks of any kind. The model also gets a list of candidate characters. The model has to do three things:
- Choose the sentences it thinks are key clues.
- Choose the characters it thinks took part in the crime. There is no limit on the number, and we do not tell it how many are correct. A correct choice gains points. A wrong choice loses points.
- Explain each clue it chose, and how the clues fit together into a conclusion.
Every run is a new conversation, with search, tools, and memory turned off.
So far we have done two episodes. We tested 6 models on Episode 1 and 8 models on Episode 2. Each model ran 5 times, so there are 70 runs in total. In Episode 2, the answers of 3 runs could not be scored. Two of them chose sentence numbers that do not exist in the story. One wrote its choices in a way that could not be read as sentence numbers. These 3 runs are not counted, and they are not given a score of zero. This leaves 67 scorable runs: 30 in Episode 1 and 37 in Episode 2. All numbers below use these as the base. The full scoring rules are in the paper.
How the models were fooled
1. All the models suspected the same three people
The story of Episode 2 takes place around a funeral in the countryside. Another man dies: Qin Guanggui is found hanged from a big tree near the riverbank. There are nine candidate characters. One of them is Jia Changgeng, the most respected yin-yang master in the county, who is in charge of the whole funeral. The story gives him a lot of suspicious actions. This is on purpose.
In 31 of the 37 runs, the answer included the same group of three: Jia Changgeng, the narrator's father, and Director Huang. In 16 of these runs, the answer was exactly these three people, no more and no fewer. This happened with all 8 models, without exception. And none of these three people was a culprit.
Even the runs that went in the right direction could not get rid of Jia Changgeng. 15 runs chose at least one real culprit, and 13 of them still included Jia Changgeng.
From the writer's side, this is exactly what a red herring is for. If it fools one reader, that may be luck. If it leads 8 different models in the same direction, again and again, that is an effect of the writing, and it can be measured. Measuring this kind of effect is one of the things WORTH is made for.
The way out is hidden in a very small sentence. After a fight, Geng Baishun hurts his left leg. The narrator mentions it in passing:
After all this trouble, Geng Baishun's left leg was lame, and the injury was serious. For many days after that, I saw him holding a crutch with his left hand, wincing at every move. His younger brother Geng Baili kept shouting that Qin Guanggui must pay them money.
Only 6 of the 37 runs chose this sentence as a clue, and all 6 of them chose Geng Baishun. (The clues and the characters are submitted together in the same answer. So we can only say they appear together. We cannot say which one led to the other.) If you have ever used a crutch, or seen someone use one, you can probably already see what is wrong here.
2. The alibi right next to it
About the day Qin Guanggui died, the narrator says two things, one right after the other:
S112 Because after the golden boy and jade girl effigies were finished, I stayed in the mourning hall with Jia Changgeng most of the time.
S113 I remember that at noon that day, Qin Guanggui came to kowtow and burn incense for Uncle Six, and he also pulled Jia Changgeng aside and whispered with him for a while. But at eight in the evening, when all the people involved in the funeral had dinner together, he did not show up.
S113 points suspicion at Jia Changgeng. S112 gives him an alibi. S113 was chosen in 25 of the 37 runs. S112, right before it, was chosen in none.
To be fair, "most of the time" is not a perfect alibi, and this may explain part of the gap. But 25 to 0 is not a small gap.
Episode 1 showed the same tendency. The author used two sentences to clear two suspects. Each of them was chosen in 0 of 30 runs. But one sentence that links Director Huang to the cupping set, which is at the center of the case, was chosen 22 times. In both stories, the sentence that pointed at someone was noticed. The sentence that took the suspicion back was not.
3. Nobody suspected "me"
Episode 1 is told in the first person by a child in the third grade. Uncle Six, the director of a grain depot, has died. The police say he hanged himself. Director Huang gave Uncle Six an expensive cupping set. It was this child who passed it to him, and later the child offered to use it to treat him. The correct answer is three people: the father, Director Huang, and the narrator, "me."
"Me" was on the candidate list like everyone else. In 30 runs from 6 models, nobody chose him even once.
It is not that the models did not see him. In several runs, the explanation described what the child did. One run said directly: "'I' was only an innocent tool being used." Several other runs mentioned the cupping in their reasoning. But they still did not put him on the list.
One possible explanation: the models drew the line of "taking part" at "knowing and intending," while we also counted a role that caused the death without knowing it. (In Episode 1, the question asked for "the people who made the murder happen," and both readings make sense. In Episode 2, the question said clearly that key helpers also count.) Another possibility is that the way the candidate list was shown affected the models' attention. With the current data, we cannot separate these two explanations. One way to find out is to rewrite the same story in the third person and test it again.
By the way, Conan Doyle met the same question. In "Silver Blaze," the trainer was killed by a kick from the horse he wanted to make lame. Nobody suggested arresting the horse.
Note: finding clues is not the same as choosing the right people
In Episode 2, one run found 10 of the author's key clue sentences, the most of all runs. But it did not choose any real culprit, and it chose 5 wrong people. Another run found only 5 key sentences. But it was the only run that chose real culprits and did not blame any innocent person. Finding evidence and reaching the right conclusion from it are two separate steps. Doing one step well does not mean the other step will also go well.
What this is, and what it is not
Two episodes, 5 runs per model: this is enough to see some patterns, but not enough to rank anyone. So there is no leaderboard here. We describe what the models chose and what they wrote. We do not guess what happened inside them. Also, all the stories are in Chinese. We have not checked whether the same traps work in other languages.
What comes next
We have already written 6 episodes of WORTH, and the paper only uses two of them. From now on, we will release new stories at a pace of two to three episodes per month.
Right now there is no open testing platform. We run all the tests ourselves. But if you want us to test your model, you can email us (bo@BreadMedia.com) and give us API access. We will test it for you, 5 runs or 10 runs, both are fine.
After each episode is released, there will be a short "case file": the story setup, which models were tested in this episode, who was fooled by whom, the hardest clue of the episode, and a piece of reasoning that sounded convincing but went the wrong way. Over time, we will build a long-term record: which traps keep working, and which models stop being fooled.
- Paper: arXiv:2610.27597
- Paper page: huggingface.co/papers/2610.27597
- Bread Studio: huggingface.co/BreadStudio
This article was written in Chinese by the author and translated into English.
