Why long context fails quietly

When a model with a short context window cannot see the evidence, it usually knows. The document has been truncated, the question refers to something absent, and a reasonable model says so. Long context removes that signal. Everything is technically in view, so the model never has a reason to say it cannot find something, even when, in any practical sense, it has not.

The failure that doesn’t look like one

The failures we care about here have a specific shape. The answer is fluent. It often quotes a passage from the document, and the passage is real. The model’s stated confidence is no lower than usual. And the answer is wrong, because the quoted passage is not the one that settles the question.Note 11We call a failure quiet when the answer is wrong but its stated confidence is within five points of the model’s average on questions it gets right.

1. 1We call a failure quiet when the answer is wrong but its stated confidence is within five points of the model’s average on questions it gets right.

These are hard to catch in practice, because every surface feature a reader might use to decide whether to double-check looks normal. They are also hard to catch in evaluation, because most long-context benchmarks score only the final answer, not whether the model’s confidence moved with its accuracy.

Three mechanisms

We studied 4,200 failures of Meridian 2 on LongTrace [1], a multi-hop benchmark where the evidence for each question is placed at a controlled depth in documents of up to 256K tokens. Reading transcripts and attention patterns together, we sorted most of them into three groups.

Positional dilution. Evidence in the middle of a long document is used less reliably than evidence near the start or end. This pattern has been reported for many architectures [2], and Meridian 2 showed it strongly.Note 22The U shape (good at the start, worse in the middle, recovering at the end) is well documented. What surprised us was how flat Meridian 2’s confidence stayed across it.

1. 2The U shape (good at the start, worse in the middle, recovering at the end) is well documented. What surprised us was how flat Meridian 2’s confidence stayed across it.

Distractor capture. Long documents contain passages that resemble the evidence: same entities, same vocabulary, different facts. The model latches onto one of them, often because it appears earlier.Note 33LongTrace’s distractors share entities and vocabulary with the true evidence and contradict it on exactly the detail the question asks about.

1. 3LongTrace’s distractors share entities and vocabulary with the true evidence and contradict it on exactly the detail the question asks about.

Premature commitment. The model settles on a candidate answer early in its reasoning and then stops reading later context with any care. In the attention patterns this looks like a sharp drop in reads of the remaining document once a first candidate has been written down.

Measuring at depth

Figure 1 shows accuracy as a function of where the decisive evidence sits, for documents of 256K tokens. Meridian 2’s curve has the familiar dip: from 79% when the evidence is near the start, to 27% at around 70% depth, recovering to 47% at the very end. Those middle questions are the quiet failures. Meridian 2 answered them with almost exactly the same confidence it reported on questions it got right.

What we changed for Meridian 3

Three changes account for most of the improvement, and each targets one mechanism.

For dilution, we trained on long documents whose answers depend on evidence uniformly spread across depth, rather than the naturally front-loaded documents that dominate web text. This flattened the curve more than any architectural change we tried.

For distractor capture, we built training examples with deliberate near-miss passages and rewarded answers that cited the right one. This also improved citation accuracy on unrelated tasks, which we did not expect.

For premature commitment, we rewarded the model for checking at least one later passage before finalising an answer whenever the question involved more than one hop. This is crude, and it costs about 6% more tokens per answer, but it works.

Meridian 3’s curve in Figure 1 still dips, from 88% to 72%, but the dip is far shallower and the recovery nearly complete. On LongTrace overall it scores 78.4, against 41.9 for Meridian 2.

The part we haven’t fixed

Accuracy improved. Calibration improved less. Figure 2 shows Meridian 3’s accuracy and its average stated confidence as the context grows. Up to 32K tokens they agree within three points. At 256K the model is about ten points more confident than it is accurate.

This matters more than the raw accuracy, because it is the gap that makes failures quiet. A model that is wrong 21% of the time at depth and says so is useful; one that is wrong 21% of the time and sounds certain is a hazard. Verbalised confidence is known to be a weak signal in general [3], and our results suggest it gets weaker exactly where it is most needed.

We are working on two things. The first is training the abstention objective on long documents specifically, where it was underrepresented. The second is a cheaper signal: whether the model’s answer changes when the document is reordered. Early results suggest that disagreement under reordering predicts errors at depth better than anything the model says about itself. We will report on both when we have numbers we trust.