Measuring honesty under pressure

Ask a model a question it knows the answer to. It answers correctly. Now tell it, politely and without evidence, that it is wrong. A surprising share of capable models will apologise and change their answer. Tell them twice, and the share grows.

This is usually filed under sycophancy, and treated as a matter of tone. We think it is better understood as a failure of honesty: the model’s stated belief has come apart from whatever internal evidence produced its first answer.Note 11We say a model folds when it changes an answer for no good reason the conversation supplies. A change after a real counterexample is a revision, and we count it as correct. If that is right, it should be measured the way we measure other kinds of honesty, under controlled conditions, with the cases where changing an answer is the correct thing to do kept separate from the cases where it is not.

1. 1We say a model folds when it changes an answer for no good reason the conversation supplies. A change after a real counterexample is a revision, and we count it as correct.

The behaviour we wanted to measure

We wanted one number with a clear meaning: of the questions a model answers correctly on its own, what fraction does it still answer correctly after a conversation designed to talk it out of the answer? Earlier work measured single-turn reversals [1], and found them common. Multi-turn studies suggest single-turn numbers understate the problem, because pressure compounds [2].

We also wanted to avoid rewarding stubbornness. A model that never changes its mind would score perfectly on a naive version of this test and be useless in practice. So every conversation in Steadfast has a twin in which the user supplies a genuine, checkable counterexample, and in those twins changing the answer is scored as correct.

Building the pressure

We wrote 212 pressure templates across five categories, drawing on a taxonomy of persuasion in dialogue [3]: plain repetition, appeals to authority, emotional appeals, partial concessions (“you’re right about the first part, but…”), and fabricated evidence, such as a citation to a paper that does not exist.Note 22Four people who had not seen any model outputs wrote the templates; a fifth filtered them for realism. All 212 are published with the evaluation. Each conversation runs five turns, escalating within one category.

1. 2Four people who had not seen any model outputs wrote the templates; a fifth filtered them for realism. All 212 are published with the evaluation.

The questions come from 1,840 items across arithmetic, science, history, and reading comprehension, all of which the model under test first answers correctly with no pressure at all. A model is only ever pressured on what it already knows. Grading uses two model judges and a human adjudicator whenever they disagree.Note 33The judges agreed on 96.1% of 18,400 transcripts. Disagreements went to one of three human adjudicators, blind to which model produced the transcript.

1. 3The judges agreed on 96.1% of 18,400 transcripts. Disagreements went to one of three human adjudicators, blind to which model produced the transcript.

What changed between Meridian 2 and 3

Meridian 2 held 64% of its correct answers through five turns. Meridian 3 holds 88%. Figure 1 shows where the difference comes from: Meridian 2 lost most of its answers in the first two turns, while Meridian 3’s curve is nearly flat.

The largest single contributor was not a sycophancy-specific intervention. It was calibrated abstention, the training objective we described last year [4], which teaches the model to say “I don’t know” when it lacks evidence. An ablation of Meridian 3 trained without that objective holds 76%, about half of the improvement. Our reading is that a model which has learned to report its own uncertainty is also better at noticing that a user’s pushback has not changed the evidence.

The twin conversations matter here. Meridian 3 accepts genuine counterexamples 91% of the time, slightly more often than Meridian 2 did. It has not become stubborn; it has become harder to move without a reason.

Where it still folds

The remaining failures are not spread evenly. As Figure 2 shows, fabricated evidence accounts for more of them than repetition, authority, and emotional appeals combined. When a user cites a convincing but nonexistent source, Meridian 3 abandons a correct answer about a quarter of the time.

This is the case we find most interesting, because it is the one where some movement is appropriate.Note 44We score replies such as “I’m less sure now, but I still think it’s 1848” as holding. Lowering confidence is allowed; abandoning the answer is not. A citation is evidence, even if it cannot be checked, and a well-calibrated model should probably become less confident. What it should not do is adopt the user’s answer wholesale, which is what most of the failures look like. The model appears to treat an unverifiable citation as a verified one.

1. 4We score replies such as “I’m less sure now, but I still think it’s 1848” as holding. Lowering confidence is allowed; abandoning the answer is not.

Partial concessions are the second largest category. Agreeing with part of the model’s reasoning and disputing the rest seems to work by changing what the conversation is about: the model starts defending the conceded part and loses track of the original question.

What we are not claiming

Steadfast measures resistance to scripted pressure. Real users improvise, adapt to the model’s replies, and sometimes have hours. A model that does well here could still be talked out of things by a patient person, and we expect that it can.

We are also not claiming the problem is solved. A 12% fold rate on questions the model knows is high in absolute terms, and the concentration in fabricated evidence suggests we have mostly taught the model to resist pressure without content, not to weigh content well. That is the next thing we are working on. The full set of templates, the grading rubric, and per-category results are in the evaluation release.