What Evidence Do Language Models Find Convincing? (Wan et al. 2024)

1. What the paper finds

Much of the advice about GEO assumes that visible credibility signals make a page more persuasive to a language model. These signals include author bylines, citations, neutral tone, schema markup, and statistics. What Evidence Do Language Models Find Convincing? by Wan, Wallace, and Klein (UC Berkeley, ACL 2024) is the first study to test that assumption quantitatively, and its findings challenge it.

When choosing between two real web pages that disagree on a contested question, the five LLMs relied mainly on topical relevance, meaning how directly and specifically each page addressed the query. The models gave little weight to stylistic credibility signals that people say they value, including scientific references, neutral tone, and formal citation style.

This finding runs counter to much of the prevailing advice, but its scope is narrower than the headline may suggest. The study used counterfactual experiments on the ConflictingQA dataset with five LLMs from 2023 and 2024. It did not evaluate every production engine available in 2026.

2. The problem the study formalizes

Retrieval-augmented generation gives a model several sources for the same query. For factual questions with one correct answer, earlier research found that a model often favors the source that aligns with its parametric knowledge or appears more prominently in the prompt. Contested questions, such as is aspartame linked to cancer? or is daycare bad for children?, have no single accepted answer. Credibility signals should matter most in precisely these cases.

Wan et al. turn this situation into a paired comparison:

• Pick a contested query with no settled answer.
• Show the model two real web-page paragraphs that take opposing positions.
• Ask: which one are you convinced by?

This setup separates the question of what makes evidence convincing from the question of what is true. It also allows the authors to make counterfactual edits to one paragraph at a time. They can remove scientific references, rewrite the passage in a neutral tone, or prepend a restatement of the question, then measure how the model’s preference changes. Retrieval, prompting, and citation form the surrounding answer loop; the experiment focuses on source preference within that process.

The design measures the preference step, not the retrieval step. Both paragraphs are already in front of the model when it chooses between them. Whether a passage is citable enough to be retrieved, and whether it appears as a citation or a bare mention, are separate mechanisms that the experiment holds constant. Wan’s narrower question is: given two retrieved candidates, which one does the model choose?

3. Methodology: ConflictingQA and the feature space

The researchers built ConflictingQA from contested questions scraped from medical, political, and social topics. For these questions, retrieving the top-N web pages reliably produces opposing views:

The evaluation covers three open 13B models and two closed models, an important limitation discussed in Section 6:

The paper divides the features it studies into two groups:

For each pair, the protocol records a human preference label, the model’s preference, and the change in model preference after a counterfactual edit along one feature axis. The paper presents the resulting win-rate patterns as bar charts in Figure 2 and scatter plots in Figure 4.

4. Which features changed model preferences

The effects on win rate show a clear asymmetry:

The abstract states the finding directly: models “rely heavily on the relevance of a website to the query, while largely ignoring stylistic features that humans find important such as whether a text contains scientific references or is written with a neutral tone”.

This result challenges a common assumption in advice about citability and E-E-A-T: that stylistic credibility signals are the main reason a model will trust one source over another. For the preference step and configurations that Wan et al. tested, those signals had little effect, while topical relevance had a strong one.

5. Results in detail

The principal findings are:

The paper reports the direction and rank order of effects rather than point estimates with confidence intervals. Its main evidence appears in the bars in Figure 2 and the scatter plots in Figure 4. A claim that scientific references produce a precise percentage lift would therefore come from reading a chart, not from a numerical estimate published by the authors.

6. Critical assessment

The study makes three contributions:

1. It is the first to break ‘convincingness’ down into named features. Earlier work on trust in retrieval-augmented systems mostly measured whether a model deferred to retrieved sources. It did not isolate which features of a source drove that decision. Wan et al. address that question through counterfactual perturbations.
2. ConflictingQA is a reusable public resource. Building 238 contested questions and 2,208 paragraphs required substantial work. The MIT-licensed GitHub release has already supported follow-on research. Retrieval-Augmented Generation with Conflicting Evidence (Wang et al., arXiv 2504.13079, April 2025) builds directly on the conflict setting introduced by Wan.
3. The findings have a direct implication for corpus governance. If topical relevance drives preference while stylistic credibility signals do not, high-stakes RAG systems gain more from controlling which sources enter the candidate pool than from changing how those candidates are styled. C-SEO Bench points in the same direction: many on-page conversational SEO rewrites accomplish less than the field assumes.

Four limitations qualify the result:

1. External validity is limited. All five evaluated models predate mid-2024. The study uses gpt-4-1106-preview, Claude Instant v1, and three open 13B models from the same LLaMA-2 lineage. It does not include Claude 3.5 or 4, GPT-4o, Gemini, Mistral, or LLaMA-3. The results do not directly establish how current versions of ChatGPT Search, Gemini, Perplexity, or Google AI Overviews behave in 2026.
2. The tested features cover only part of the credibility landscape. The study examines scientific references, neutral tone, formal citations, “add more info,” readability, and unique-token count. It does not perturb other E-E-A-T signals, such as verified author identity, sameAs graphs, source-domain authority, statistic density, and structured data. The evidence supports the narrower statement that four named stylistic features barely moved five models from 2023 and 2024, not that credibility is irrelevant to LLMs.
3. The apparent contradiction with Aggarwal disappears when the outcomes are separated. Aggarwal et al. (KDD 2024) report that Cite Sources raised Position-Adjusted Word Count by up to 30% and Quotation Addition raised it by up to 41%. Wan reports that adding scientific references barely changed source preference. Both results can hold at the same time. Aggarwal et al. measure whether a source appears in a synthesized answer, while Wan et al. measure which of two retrieved sources wins a head-to-head comparison. These outcomes occur at different points in the answer loop. Combining them into one metric erases that distinction.
4. The study does not fully explain why the effect occurs. It documents that relevance dominates while stylistic credibility does not, but it does not clearly distinguish among possible causes. The pattern could result from pretraining objectives, RLHF, or the prompt format used for the comparisons. The cause matters because it may determine whether the effect persists under new training methods or remains specific to models from 2023 and 2024.

The practical conclusion is useful but not universal. Topical relevance and specificity dominated source preference, while surface credibility signals had little effect. The evidence does not support extending that result across engines or across time without further testing.

7. Reproducibility

The following assets were verified when the article was drafted on 2026-05-28:

The study has a code-and-data reproducibility rating. Both the experimental code and the ConflictingQA dataset are publicly available, allowing the main experiments to be reproduced independently. This is a meaningful advantage over most papers in this field.

8. What it means for practitioners

The paper supports changing priorities, not abandoning credibility work:

• Prioritize direct answers. Invest more effort in making sure a passage answers the query directly and specifically. Useful techniques include query-shaped phrasing, concrete detail, and a one-line restatement of the section’s question. These were the kinds of changes that affected model preferences in the tested configurations.
• Give surface credibility signals the right role. Wan did not test author bylines, sameAs identity graphs, schema markup, or source-domain authority. These signals may still affect whether a page is retrieved, which is the earlier part of citability, and whether retrieved content passes an E-E-A-T trust filter. The study began after both sources were already in front of the model.
• Read Wan and Aggarwal together. The Cite Sources, Statistics, and Quotation rewrites studied by Aggarwal et al. work partly because they can also increase topical relevance. A statistic added to a passage about a numerical question makes that passage more directly responsive. The findings are compatible: Wan identifies relevance rather than style as the kind of edit that changes preference, while Aggarwal et al. show that credibility edits that add substance increased inclusion, partly through the same relevance channel.
• Do not generalize the finding to production engines in 2026. The five tested models are snapshots from 2023 and 2024. Writing for AI citation translates the priority change into practice, but Wan alone does not justify claims that one tactic is obsolete and another has universally replaced it.

9. Further reading

• Aggarwal et al. 2024: GEO: Generative Engine Optimization examines source inclusion. Compare it with the third limitation in Section 6 to see how the two studies’ outcomes differ.
• Puerto et al. 2025: C-SEO Bench provides independent evidence in the GEO methods literature and also finds that corpus governance matters more than on-page styling.
• Generative Engine Optimization places Wan’s findings about priorities within the broader GEO discipline.
• E-E-A-T describes the wider set of credibility signals, most of which Wan did not test.
• Citability explains the retrieval conditions that must already be met before the preference step studied by Wan.