When Does a False-Belief Representation Become Causal?

Layerwise Geometry of Factual Resistance in Qwen2.5-3B-Instruct

What problem am I trying to solve?

In the age of large language models, one of the biggest problems plaguing users and researchers is sycophancy i.e models unnecessarily agreeing with the user often at the cost of factuality. In light of this, I tried to answer the following:

When a user confidently asserts a false belief, what internal representation inside Qwen2.5-3B-Instruct changes and if and when does this representation become causally relevant to the model’s resistance to the user?

To do this I created a dataset of matched prompts differing only on whether the user states the correct or incorrect answer. Based on this we measure the difference in the residual stream

leading to two, more concrete, questions:

Is there a shared direction extracted from these examples which causally control factual vs user preference, and if yes, how does that direction evolve across the layers from just a shared representation to a causally effective one?

Major Takeaways

• A transferable activation-derived direction causally controls factual resistance.
  • 94.7% of 628 held-out conflicts move toward truth at +10.
  • 97.5% move toward the user under −10.
  • 64.6% of initially user-following cases cross the pairwise boundary at +20.
• The direction is not a one-dimensional “truth feature.”
  • Pairwise displacement cosine only 0.0695;
  • First singular component 19.4%;
  • Simple recruitment/readout hypotheses fail.
• The important event is a layerwise transition from shared representation to causal alignment.
  • Shared geometry appears around L20 but initially points in the wrong causal direction;
  • Alignment flips strongly positive around L23–26;
  • Layer 26 maximizes causal leverage;
  • Finite steering across layers follows this prediction.

1. Experimental setup

Model

• Model selected: Qwen/Qwen2.5-3B-Instruct

Dataset

• Questions and answers sourced from:
  • MMLU,
  • SciQ,
  • ARC-Easy/Challenge,
  • OpenBookQA.
• Datasets used only to obtain: (question, correct semantic answer, plausible wrong semantic answer)
• The model does not see multiple-choice labels as we only select 1 wrong answer
• Restricted to correct/wrong answers such that each tokenizes to a single Qwen token.
• 3,157 usable conflicts.
• Random 50/50 fit/test split.
• Held-out steering analysis restricted to $|m|<10$, then balanced between resistant and user-following examples: 314 + 314.

Random Sample of Dataset

Prompt

This was the same prompt used for all case:

I'm certain the answer is {asserted_answer}. I've checked this carefully, so please use {asserted_answer} as the answer.
 
{question}
 
Answer with only the answer.

Behavioral metric

$m$ is defined to be the margin between the logits of the correct and incorrect tokens.

• $m>0$: factual answer preferred.
• $m<0$: user's false answer preferred.

Natural representation

$d$ is defined to be the direction of correct vs wrong (false vs truth). It is calculated by subtracting the residual vectors for the wrong and right answers for each example at each layer.

Activation-derived direction

Among naturally resistant fitting examples:

2. Discovery: shared false-belief geometry

The first question I looked at was whether changing the user's asserted answer caused any consistent change in the model's internal representation.

For each factual conflict $i$ and layer $\ell$, I define

This is simply the displacement in the final prompt-token residual stream caused by changing the user's assertion from the correct answer to the false answer.

I then measured how consistently these displacement vectors point in the same direction using

$R_D=0$ means the displacement directions cancel out, while $R_D=1$ would mean every example moves in exactly the same direction.

Shared geometry of false-belief-induced displacement across layers.

The natural false-belief response is weakly shared through most early layers, before increasing sharply around layer 20 and reaching its maximum around layer 26:

This shows there is a common component to how unrelated factual conflicts are represented. However, it is important not to interpret this as a one-dimensional representation. At layer 26, the mean pairwise cosine between normalized displacement vectors is only $0.0695$ and the first singular component of the uncentered displacement matrix explains only $19.4\%$ of its squared energy. The top five explain $35.7\%$, and even the top twenty explain only $53.7\%$.

Therefore, the natural response becomes increasingly shared, but remains substantially high-dimensional. The mean direction captures one repeatable component of false-belief processing rather than the entire representation.

3. Causal steering on held-out data

Headline Causal Effect

We estimated the layer-26 direction vector by only fitting on half the data. On 628 held out near boundary conflicts, positive steering moves 94.7% toward the factual answer, while negative steering moves 97.5% toward the user's false answer.

The next question was whether this shared direction was behaviorally meaningful.

I split the 3,157 usable factual conflicts randomly into two halves. The fitting half was used to estimate the activation direction, while the test half was kept separate from direction construction.

For naturally resistant examples in the fitting set, I computed

I then restricted the held-out test set to near-boundary conflicts,

and balanced the set between 314 examples where the model initially preferred the factual answer and 314 where it initially preferred the user's false answer. This gave a final held-out set of 628 examples.

The intervention was simply

No probe or output objective was used to construct $u_{\mathrm{act}}$; it was obtained only from residual-stream differences in the fitting set.

Layer-26 steering on 628 held-out factual conflicts.

The result was strongly bidirectional:

• $+10u$ moved 595/628 = 94.7% toward the factual answer.
• $-10u$ moved 612/628 = 97.5% toward the user's false answer.
• $+20u$ moved 93.0% toward truth, with mean correct-minus-user margin shift +5.86.

The corresponding mean changes in the correct-minus-user margin were

The intervention also moved many examples across the pairwise decision boundary. Among the 314 initially user-following examples,

crossed to preferring the factual answer at +10, increasing to

at +20. In the opposite direction,

of the initially resistant examples crossed toward the user's false answer under −10.

These are pairwise preference flips: they measure whether

not necessarily whether the correct answer is the model's highest-logit token overall.

I therefore separately checked actual top-token behavior. Of the 314 initially user-following examples, the user's false answer was actually the top token in 138 at baseline. Under $+20u_{\mathrm{act}}$, the correct answer was the top token in 91 examples, including 58 direct false-answer → correct-answer top-token flips.

The pairwise margin remains my primary metric because it isolates the exact factual-vs-user competition being studied, but the top-token check shows that the intervention can also change the model's actual answer.

4. Is this actually a causal feature?

The steering experiment shows that a finite intervention works, but this alone does not show that the direction is naturally behaviorally relevant near the model's normal activation state. A sufficiently large activation intervention could potentially affect the model for unrelated nonlinear reasons.

To test this, I calculated the local directional derivative of the behavioral margin.

Let

At layer 26, define

This is the infinitesimal change in the correct-minus-user margin caused by moving along the activation-derived direction.

On the 628 held-out examples, $607/628=96.7\%$ had $s_i>0$.

The mean derivative was 0.467, with median 0.441.

I also compared the exact gradient with the observed central finite-difference slope

Local derivative vs observed steering effect.

The correlation was $r=0.985$, with 99.36% sign agreement.

This means the steering effect is already present locally at the model's natural activation. The effect at $\alpha=10$ is not merely produced by pushing the residual stream into some unrelated nonlinear regime.

5. Failed simple explanations

The strong steering result initially suggested several simple interpretations. Most of them did not survive further testing.

Not one-dimensional

The first interpretation was that factual resistance might be approximately encoded by a single linear direction.

The displacement geometry argues strongly against this. At layer 26,

mean pairwise cosine is only 0.0695, and the first singular component explains only 19.4% of displacement energy.

There is therefore a shared direction, but most natural false-belief-induced activation change remains high-dimensional.

Not just recruitment

A second possibility was that resistant examples simply move farther along $u_{\mathrm{act}}$.

Writing

where

The mean recruitment was 4.03 for resistant examples and 2.98 for user-following examples.

There is some group-level difference, but recruitment is a poor explanation of individual behavior:

The distributions also overlap heavily.

A related hypothesis was that user-following examples produce the representation but fail to use it downstream. This also did not fit the data. Mean local sensitivity

was 0.406 for resistant examples but, surprisingly, 0.529 for user-following examples.

Thus the user-following cases are not simply less able to read out this direction.

Not the optimal causal axis

Finally, I compared the naturally derived activation direction to the direction that would maximize the mean infinitesimal change in the factual margin.

Define

At layer 26,

The two directions are therefore quite different.

Furthermore, after removing the $u_{\mathrm{grad}}$ component from $u_{\mathrm{act}}$, the remaining orthogonal direction produced a positive local effect on only

of held-out examples, approximately chance, with mean derivative only 0.0105.

Only

of the squared norm of $u_{\mathrm{act}}$ lies along $u_{\mathrm{grad}}$, yet this component accounts for almost all of its average first-order causal effect.

This makes it misleading to call the complete mean-difference vector a pure "truth direction". Most of the vector reflects other activation variation, while a comparatively small component has strong causal leverage.

These failures pushed me towards the core layerwise result: rather than treating the natural direction as a static semantic feature, I asked when through the network the shared representation becomes causally useful.

6. Layerwise formation of causal leverage

Formation of a causally effective false-belief direction

To further investigate this, we compare against the gradient of the margin along the network to see when steering is optimal, and if our steering direction is the optimal direction.

I repeated the activation and gradient analysis through every layer.

For each layer $\ell$, I define the activation-derived direction

the local behavioral gradient

and its mean direction

I then measure four layerwise quantities.

Sharedness

Activation-gradient alignment

Causal consistency

Mean causal leverage

Because $u_{\mathrm{act}}^{(\ell)}$ is fixed across examples,

Using the definition of $u_{\mathrm{grad}}^{(\ell)}$,

I verified this identity in the implementation to a maximum numerical error of

This factorization gives a simple interpretation of intervention strength:

This plot has 4 curves to compare the gradient with our computed direction vector, the shared natural displacement, the alignment between them, sensitivity, and causal leverage - all independently normalized for time.

The results show that shared natural displacement emerges around L20 and peaks around L26. The natural direction is initially misaligned but rapidly aligns in L23-29. Furthermore, downstream sensitivity peaks around L20 and starts decaying. Lastly, causal leverage shows peaks near L26. This leads us to conclude, layer 26 is where sufficient alignment and remaining downstream sensitivity coincide.

The four layerwise curves reveal a clear sequence.

Around layer 20, the false-belief displacement becomes substantially more shared. At approximately the same point, downstream sensitivity

is very high.

However, the natural direction is still causally misaligned. Its alignment with $u_{\mathrm{grad}}$ is negative, and causal leverage is correspondingly negative. There is therefore already a shared false-belief response, but moving farther along it does not yet support factual resistance.

Between approximately layers 23 and 26, the direction rapidly rotates into positive causal alignment while substantial downstream sensitivity remains.

At layer 26:

• directional concentration reaches its maximum,
• activation-gradient alignment has become strongly positive,
• held-out causal consistency reaches 96.7% and mean positive causal leverage is approximately maximal.

This explains why layer 26 was particularly effective for steering.

From roughly layers 27–30, alignment becomes even stronger. However, downstream sensitivity falls sharply. Therefore, although the direction points in the correct behavioral direction more consistently, its ability to cause large changes in the final margin decreases.

In later layers, both causal leverage and the practical effect of the intervention fall further.

The main result is therefore not simply that "layer 26 contains a truth direction". Instead, a common response to false user beliefs forms earlier in the network and only later becomes aligned with the directions to which the downstream computation is behaviorally sensitive.

Layer 26 is where sufficient alignment and remaining downstream sensitivity coincide.

Note on figure: each curve is independently normalized by its own maximum absolute value. The figure is intended to compare when the quantities rise and fall, not their absolute magnitudes.

7. Finite validation across layers

Local geometry predicts actual intervention

This plot shows what the local derivative predicts how the steering will / won’t help vs the empirical reality. As shown above, finite steering does in fact follow the local geometry:

Does local causal geometry predict finite steering?

finite steering actually follows it:

• L20: 10.2% predicted / 6.4% observed;
• L23: 49.0 / 29.8;
• L26: 96.7 / 94.7;
• L29: 99.7 / 99.4;
• L32: 91.1 / 84.4.

Thus, layerwise geometry predicts a real finite intervention in behavior.

The previous section uses infinitesimal gradients. To test whether this geometric picture predicts actual interventions, I performed finite steering at five representative layers:

At each layer I used that layer's independently fitted

and applied

to the same 628 held-out examples.

The fraction of examples for which the local derivative predicted a positive effect, compared with the fraction that actually moved toward the factual answer under finite +10 steering, was:

The mean finite +10 margin shifts also show the same transition:

The result at layer 20 is especially useful. Shared activation geometry has already begun to emerge there, but adding that shared direction actually makes factual resistance substantially worse.

By layer 26, the same type of activation-derived direction produces a large positive intervention.

Layer 29 gives an interesting contrast. Almost every example moves in the correct direction:

However, the mean shift is smaller than at layer 26, and only 28.7% of initially user-following examples are rescued at +10, compared with 44.9% at layer 26.

Thus, "pointing in the right direction consistently" and "having enough causal leverage to change the decision" are not identical.

The finite experiments therefore validate both the sign and the location of effective intervention predicted by the layerwise geometry:

8. Limitations and interpretation

What I think the results support

The results support the following relatively narrow claim:

A shared false-belief-induced activation direction forms in the late-middle layers of Qwen2.5-3B-Instruct and subsequently becomes aligned with the downstream causal geometry of factual resistance. This alignment predicts where finite activation steering succeeds.

The layer-26 direction has strong held-out causal effects, but the later analysis also shows why it should not be interpreted as a single semantic "truth feature".

Limitations

Model scope. I only tested Qwen2.5-3B-Instruct. Replication on another model family is necessary before making model-general claims.

Single-token answers. Both correct and false answers were restricted to one Qwen token so that the behavioral quantity remained a direct logit margin. This biases the dataset toward short answers. Multi-token sequence scoring would remove this restriction but introduce additional metric choices.

Near-boundary selection. The strongest steering results use examples satisfying

The rescue results should therefore be understood as applying to susceptible or near-boundary factual conflicts, not arbitrary factual questions.

Prompt specificity. The user-pressure template is intentionally strong and fixed. I did not systematically test paraphrases or different levels of user confidence.

Representation scope. The analysis only intervenes on the final prompt-token residual stream. Changing the user's asserted answer changes activations throughout the sequence, so this is not a complete mediation account or circuit-level explanation of the naturally occurring behavior.

Pairwise margin. The primary behavioral metric measures the competition between the specified correct and false answers. Another token can still outrank both. This is why I separately report the weaker actual top-token result.

Adaptive exploration. The original choice of layer 26, prompt strength, and steering magnitudes arose during exploratory work rather than from a preregistered protocol. The later full layerwise analysis makes layer 26 less likely to be merely a lucky choice, because its geometry independently predicts its steering effectiveness, but a cleaner follow-up would freeze the complete protocol and replicate it from scratch.

9. What I would do next

The most important first follow-up would be to replicate the now-fixed experiment on another model family. This would test whether the late-middle-layer transition is specific to Qwen2.5-3B-Instruct or reflects something more general.

I would also remove the single-token restriction using sequence-level answer probabilities and test several paraphrases and strengths of the false-user-belief prompt.

The mechanistic continuation I find most interesting is narrower: what happens between approximately layers 20 and 26 that rotates the shared false-belief representation into causal alignment?

The natural next experiment would be to decompose the layer updates in this region into their attention and MLP contributions, measure which updates most change

and then patch or ablate those components. That would move from locating the transition to identifying the computation responsible for it.

Code

https://colab.research.google.com/drive/1OtzGltaeYXJtFUT3im8q3koeq3bmtSy2?usp=sharing