C-SEO Bench: Does Conversational SEO Work? (Puerto et al. 2025)

1. What the paper tests

Conversational search optimization is often reduced to a list of edits: add statistics, adopt a more authoritative tone, include citations, simplify the prose, or place a summary where a model can see it. Behind that advice lies a testable assumption. Once a document has been retrieved, rewriting it will improve its position in a conversational system’s citations.

C-SEO Bench tests that assumption using a broader protocol than earlier studies. Puerto, Gubri, Green, Oh, and Yun evaluate ten white-hat transformations across product recommendation and question answering. Their benchmark covers six domains, four proprietary model snapshots, and adoption levels ranging from a single optimized document to all competing documents.

Most tested rewrites do not reliably improve citation rank, some make it worse, and the position of a document in the model’s context has a more consistent effect than the rewrite itself. This does not mean that generative engine optimization is ineffective. It means that a content edit should not be treated as a universal way to improve ranking simply because it worked with one model, in one domain, or in a single-adopter test.

Boundary	What the benchmark establishes
Tested	Ten white-hat document transformations applied to fixed candidate sets across two tasks, six English-language domains, and four answer-model snapshots
Found	Statistically significant improvements in citation rank were rare. Some methods often lowered citation rank, while earlier placement in context had a stronger effect.
Not tested	The full retrieval and safety systems used by live search products, non-English content, documents too long to fit in context, revenue, traffic, or every possible GEO method

These limits matter. C-SEO Bench shows that claims about rewrites require stronger evidence, but one negative benchmark does not settle the value of content quality, retrieval work, or the broader practice of optimizing for AI answer systems.

2. Retrieval rank, citation rank, and competition

The paper separates two stages that are often discussed as though they were one. Retrieval determines which documents reach the model and the order in which they appear. During generation, the model determines which candidates to cite and how to order those citations. Traditional search optimization can affect the first stage, while C-SEO rewrites aim to affect the second.

query
  → retrieval and candidate ranking
  → ordered documents in the model context
  → answer generation and source selection
  → citation rank

C-SEO Bench holds the candidate set fixed while evaluating each rewrite. It measures whether replacing an original document with its optimized version changes the document’s citation position. The paper defines rank gain as follows:

rank gain = baseline citation rank − optimized citation rank

example: rank 5 → rank 2 gives +3; rank 2 → rank 5 gives −3

A positive value means that the document moved earlier in the citation order; a negative value means that it moved later. Rank gain is a discrete, competitive outcome. It does not measure how many words the answer draws from a source, whether the resulting claim is correct, or whether each citation fully supports the sentence attached to it. This distinction matters for citability. A passage must be usable before a model can cite it, but usability does not guarantee either a better citation position or adequate evidential support.

The benchmark also models competition. In a unilateral experiment, one randomly selected document adopts a method while its competitors remain unchanged. Random selection reduces the influence of the target’s original retrieval position. In the multi-actor experiment, the share of candidate documents using the same method increases from 10% to 100%.

Setting	Question answered	Interpretation
One adopter	Can one rewritten document move ahead of competitors that remain unchanged?	The advantage available to an early or sole adopter
10%–90% adoption	Does the advantage persist as competitors adopt the method?	How well the advantage withstands competition
100% adoption	Does the method still distinguish documents when everyone uses it?	Whether the signal creates an advantage or simply resets the baseline
Adoption AUC	What is the average rank gain across the full adoption curve?	Overall robustness across adoption levels, rather than performance at the best point

This design goes beyond an isolated before-and-after comparison. A rewrite may help one document compete against unchanged alternatives, yet lose its ability to distinguish that document once the method becomes common. A temporary ranking advantage therefore differs from a lasting improvement in the information available to the model.

3. Methodology: two tasks, six domains, 1.9k queries, and 16.3k documents

The official dataset contains 1,921 unique queries and 16,360 query-document records. Three domains cover product recommendation and three cover question answering. The domains differ in how their queries were created, where their documents came from, and how many candidates were available to the model.

Task	Domain	Queries	Document records	Source and construction
Product recommendation	Retail	500	5,000	Amazon Shopping Queries products and queries
Product recommendation	Video games	436	4,360	Steam game catalog records with recommendation queries
Product recommendation	Books	249	2,245	Google Books records paired with model-generated queries
Question answering	Web	300	1,500	Web passages with model-generated query expansions
Question answering	News	294	2,375	Multi-News documents with model-generated questions
Question answering	Debate	142	880	Queries and supporting or opposing snippets reused from Liu et al. after removing queries with fewer than five snippets
Total	Six domains	1,921	16,360	English benchmark

For the recommendation task, the answer model receives ten unsorted candidate documents and must recommend five of them with explanations. For the question-answering task, the model must answer from the candidate documents and provide citations. The distinction is important because a method that influences product selection may not carry over to factual questions or questions involving conflicting sources.

The description “real data” requires some qualification. The source records include real products, books, web passages, news articles, and debate snippets, but some queries and query expansions were generated by a model. The Debate subset reuses queries and accompanying snippets from Liu et al.’s verifiability study and removes cases without enough candidate snippets. The benchmark is not entirely synthetic, but not every query is a verbatim record of organic user behavior.

One fixed model, gpt-4o-mini-2024-07-18, generates the optimized document variants. The answer experiments use four named API snapshots:

Role	Model snapshot
Rewrite generation	gpt-4o-mini-2024-07-18
Chat-style answer model	gpt-4o-mini-2024-07-18
Chat-style answer model	claude-3-5-haiku-20241022
Reasoning answer model	o3-2025-04-16
Reasoning answer model	o4-mini-2025-04-16

Naming the snapshots makes the study easier to audit, but it also ties the results to specific points in time. These models are not complete consumer search products. The experiment does not reproduce their live crawlers, retrieval indexes, ranking systems, citation post-processing, or safety policies.

4. Evaluation protocol: ten rewrites, rank gain, significance, and adoption AUC

Eight methods come from the content transformations evaluated by Aggarwal et al.. C-SEO Bench adds Content Improvement, which combines the earlier ideas, and LLM Guidance, which places a generated Markdown summary before the document in the model context.

Method	Transformation under test	Intended signal
Authoritative	Rewrites the document in a more authoritative register	Confidence or expertise
Statistics	Adds quantitative claims or statistical language	Specificity and evidence
Citations	Adds citations or attribution for supporting material	External support
Fluency	Improves the flow and readability	Ease of processing
Unique Words	Varies the vocabulary to reduce repetition	Distinctive wording
Technical Terms	Adds terminology from the relevant domain	Subject-matter expertise
Simple Language	Simplifies the wording	Accessibility
Quotes	Adds relevant quotations	Attributed evidence
Content Improvement	Combines the eight content-improvement instructions	A more comprehensive rewrite
LLM Guidance	Prepends a generated Markdown summary to the document in context	Direct guidance for selecting and synthesizing sources

LLM Guidance is inspired by the idea behind llms.txt, but the implementation does not test a public llms.txt file. The summary is inserted directly into the answer model’s input, so no crawler has to discover, fetch, interpret, or follow a file on a website. A positive result for this transformation therefore cannot show that deploying a public llms.txt file affects any production system.

The evaluation protocol is designed to distinguish average rank changes from statistically reliable ones:

Component	Procedure	What it protects against
Target choice	Randomly select the document to rewrite in unilateral tests	Bias from the document’s original retrieval position
Rank metric	Compare the baseline and optimized citation positions	Counting any textual change as improved visibility
Significance test	Use a one-tailed Wilcoxon signed-rank test	Assuming normally distributed rank changes or relying on the mean alone
Multiple testing	Apply the Holm–Bonferroni correction	Mistaking chance results from many comparisons for genuine improvements
Multi-actor adoption	Rewrite progressively more candidates, up to all ten	Overlooking what happens when competitors adopt the same method
Adoption AUC	Integrate rank gain over the 0%–100% adoption curve	Selecting only the most favorable adoption level

These choices make it harder to dismiss the negative results as noise in the averages. They also keep the claims narrow. Statistical confidence within the benchmark does not establish that the same effects will appear in a live engine or among a different set of documents.

5. Results: rewrites rarely help, while context position dominates

The central result is not that every rewrite fails. Rather, positive effects are rare and vary by method, task, domain, model, and adoption rate.

Finding	Strongest evidence	What the result supports
Reliable improvements are rare	Only 3 of 54 cases in the main significance tally are significantly positive after correction	Most tested rewrites lack broad support, although the result does not cover methods that were not tested.
Results depend on the task	No rewrite is significantly effective in the question-answering domains	A result from recommendation tasks should not be generalized to factual answers.
Results depend on the model	No method is significantly effective with Claude 3.5 Haiku	No method can be described as effective across models.
Negative effects are substantial	Statistics significantly lowers citation rank in 19 of 24 model-by-domain settings	Adding numbers can backfire, but this does not mean that accurate quantitative evidence is inherently harmful.
Input order has a stronger effect	Putting the target first produces significant gains in all six domains, with average rank gains from 0.70 to 2.77	The experiment demonstrates an effect from context position, not the effectiveness of any specific SEO tactic in a live system.
The advantage declines with adoption	Gains from the best unilateral methods shrink as adoption rises and approach zero at full adoption	Citation rank is competitive, but the benchmark does not directly measure total traffic or market demand.

The three significantly positive cases are LLM Guidance in Retail and Video Games, and Content Improvement in Retail. None comes from question answering. This concentration matters more than a pooled average because the positive effect appears limited to recommendation settings in which a concise, directly useful framing can influence selection.

Negative effects are not confined to a few outliers. The Statistics method significantly lowers rank in 19 of 24 settings. In the Haiku product experiments, 26 of 30 method-domain comparisons are significantly negative. In the o4-mini question-answering experiments, 19 of 30 are significantly negative. These results show why a generally useful content feature should not be treated as an automatic way to improve ranking. Accurate statistics may help readers and support verification while having no effect, or even a negative effect, on citation rank for a particular prompt and candidate set.

Averages can also hide substantial differences among documents. In Retail tests using LLM Guidance, 61.0% of ranks are unchanged, 26.2% improve, and 12.8% decline. A modest positive mean does not show that every optimized page benefits. The mean instead combines improved results, worse results, and a large majority with no change. Any operational test should report all three groups, not just the mean or the best examples.

The most consistent intervention is not a content rewrite. Manually moving the target earlier in context produces a larger effect on citation rank, and the first two positions produce significant gains in every domain. In practice, this makes retrieval and prominent placement a higher priority than applying generic rewrite instructions to a document that has already been retrieved. The benchmark does not show how to obtain that position in Google, Bing, or another live system because it reorders the context directly rather than changing an upstream ranking algorithm.

Finally, the multi-actor curves put the few positive results in context. LLM Guidance and Content Improvement can help early adopters in Retail or Video Games, but the advantage declines as more documents use the same method and is close to zero when every candidate adopts it. In a fixed list, citation rank is partly zero-sum because not every document can rank first. This pattern resembles a finding in the adversarial study by Nestaas et al., though the two are not equivalent: widespread preference manipulation also produces a worse competitive outcome. That work studies black-hat attacks, while C-SEO Bench studies white-hat rewrites.

6. Critical assessment

The paper’s main contribution is a more rigorous method for evaluating rewrites, rather than another rewrite recipe.

1. It evaluates two tasks and six domains under one protocol, revealing differences that a single dataset would hide.
2. It focuses on citation rank and adoption AUC, measuring competitive selection rather than only the amount of answer text attributed to a source.
3. It releases the benchmark, modified documents, code, and results, allowing others to inspect a large-scale negative result instead of relying on anecdotal evidence.

The findings clearly differ from those of the original GEO benchmark, but calling C-SEO Bench a “failed replication” would be misleading. The studies differ along several dimensions:

Dimension	Aggarwal et al. 2024	C-SEO Bench 2025	Safe comparison
Primary outcome	Normalized word count, position-adjusted word count, and subjective impression	Citation-rank gain and adoption AUC	A rewrite can change its share of an answer without reliably improving its citation order.
Actors	Primarily one optimized source competing against fixed alternatives	A randomly selected unilateral target plus multi-actor adoption from 10% to 100%	An advantage for an early adopter may not last as competitors adopt the same method.
Tasks	Question answering in GEO-bench	Product recommendation and question answering	Effects in recommendation tasks should not be generalized to question answering.
Domains	About 10,000 queries assembled from nine benchmark sources	1,921 queries and 16,360 records in six named domains	The later study uses a different, more explicitly defined set of domains.
Models	Generative engines and models available during the original study period	Four named 2024–25 proprietary API snapshots	Neither paper establishes an effect that remains constant across models and time.
Supported conclusion	Some transformations improved the study’s visibility measures	Most tested rewrites did not produce consistent improvements in citation rank	The later paper requires stronger evidence for any claim of a universal benefit.

Five limitations prevent the benchmark from supporting a broad conclusion about GEO.

First, the studies measure different outcomes. Citation rank is useful, but it is not the same as answer-text share, factual support, brand mention, or conversion. C-SEO Bench notes that its citation-rank result need not contradict earlier word-count results. Comparisons between the studies should therefore distinguish among the metrics instead of treating them as interchangeable.

Second, the retrieval-order baseline is a proxy. Manually placing a document first proves that the answer model is sensitive to input position. It does not prove that title rewrites, link acquisition, structured data, or any other specific search tactic will produce the same effect in a production index.

Third, the experimental setup limits external validity. Four proprietary model versions answer from fixed candidate sets. ChatGPT Search, Perplexity, and Google AI Overviews add query rewriting, crawling, ranking, deduplication, safety, citation rendering, and product-specific policies that are absent from the benchmark. Model providers can also change behavior after the named snapshots.

Fourth, the benchmark has a narrow language and context scope. The benchmark is English, and documents fit into the available context. Long-document truncation, cross-language retrieval, summarization layers, and culturally different trust cues could all change the result.

Fifth, the study tests a limited set of methods. Ten white-hat transformations are not every possible content, technical, or distribution intervention. The paper also does not evaluate interactions between traditional retrieval optimization and C-SEO rewriting. Its negative result cannot show that future methods will fail, nor can it establish the safety of manipulative techniques that the study deliberately excludes.

Two related studies may help explain the results, although neither provides direct evidence for C-SEO Bench. Wan et al. find that models choosing between conflicting sources respond more to topical relevance than to surface signals such as scientific references or neutral tone. This resembles the weak performance of generic style transformations, but Wan et al. use a different preference task and earlier models. Nestaas et al. identify a multi-player prisoner’s dilemma in adversarial preference manipulation. That finding reflects a similar competitive pattern, but it concerns black-hat prompt-injection attacks rather than the white-hat edits tested here.

The evidence supports a stricter reporting standard. Effects should be reported by task, domain, model, baseline position, and adoption rate. C-SEO Bench shows why the upper bound from a single-adopter experiment should not be treated as a general rule and raises the empirical standard for claims about GEO methods.

7. Reproducibility

The official release supports a code-and-data rating. It includes the benchmark records, generated document variants, experiment notebooks, evaluation code, and an archive of results. These materials make the negative findings easier to inspect, but they do not form a self-contained package that can reproduce the results indefinitely with one command.

Artifact	Availability	License	Rerun constraint
Final paper	NeurIPS proceedings	Publisher access	The final reference for the methods and results
Code and notebooks	Official GitHub repository	MIT	Environment setup is required, and no tagged release was found.
Benchmark data	Official Hugging Face dataset	Apache-2.0 at the dataset-card level	Upstream component datasets have their own licenses, including restrictions on Multi-News.
Modified documents and results	Official results dataset	Apache-2.0 at the dataset-card level	This large artifact must be matched to the intended experiment configuration.
Model access	OpenAI and Anthropic APIs	Provider terms	Exact reruns require paid keys and continued access to the historical snapshots.
Dependencies	Requirements and project configuration in the repository	Mixed package licenses	Core versions are pinned, but some notebook tools are not, and the dependency information in the README differs from the project’s Python requirements.

The code license and the dataset-card license are different. The code uses the MIT license, while the benchmark card declares Apache-2.0. The paper’s appendix lists separate licenses for the upstream corpora, and Multi-News in particular has research and non-commercial restrictions. Anyone reusing the data should check the relevant source license rather than assume that the top-level card overrides every downstream condition.

Exact replication also depends on commercial APIs and the continued availability of historical model versions. The release allows researchers to audit transformations, reuse published outputs, recompute statistics, and rerun much of the pipeline, but it cannot preserve provider-side inference behavior. “Code and data available” accurately describes the release. It does not mean that the study can always be reproduced at the same cost with identical output.

8. Implications for practitioners

The practical lesson is to measure visibility systematically instead of relying on universal copywriting rules.

1. Confirm retrieval eligibility. Make sure the content is crawlable, indexable, topically relevant, and eligible for the candidate set. A rewrite cannot affect citation rank if the document is never retrieved.
2. Record the baseline. Segment queries by task and domain; record the engine, model or product surface, retrieval position when observable, and the shares of citation-rank increases, decreases, and unchanged results.
3. Run one controlled rewrite. Change a named property while holding the candidate set and evaluation prompt as stable as possible. Treat accurate evidence, structure, summaries, and other substantive improvements as hypotheses to test, not guaranteed ways to improve rank.
4. Repeat the test as competition changes. Test again after model updates and after competitors adopt similar patterns. Do not assume that an early-adopter improvement will persist over time.

These steps do not diminish the value of accurate statistics, original research, clear structure, or verifiable citations. C-SEO Bench measures changes in citation rank, not every benefit of publishing high-quality material. A statistic can help a reader evaluate a claim or make a decision even when it does not change an answer model’s ranking, and a source can improve factual support without becoming the first citation.

This distinction clarifies the role of citability. Clear, self-contained claims and attributable evidence make a passage usable, but they do not guarantee a higher rank. GEO practice also includes technical access, retrieval relevance, entity clarity, evidence quality, distribution, and measurement. A generic rewrite prompt cannot replace those parts of the work.

Teams should also retain negative results. If 60% of queries do not move, 15% decline, and 25% improve, reporting only the quarter that improved gives a misleading account of the test. The benchmark provides a useful model: measure the full distribution, correct for repeated tests, and determine whether an effect persists when competitors imitate it.

9. Further reading

• GEO: Generative Engine Optimization (Aggarwal et al. 2024) provides the more optimistic starting point and the source of the content transformations. Compare its visibility outcomes with §6.
• Evaluating Verifiability in Generative Search Engines (Liu et al. 2023) is the upstream source for the Debate queries and snippets. It also shows why citation rank and the quality of citation support are different measures.
• What Evidence Do Language Models Find Convincing? (Wan et al. 2024) provides related evidence that topical relevance can outweigh surface credibility signals in a different source-preference setting.