Multimodal Signals

1. What multimodal signals are

Multimodal signals determine whether AI engines can read, ground, and cite non-text content, including images, video, audio, charts, and tables. Within Generative Engine Optimization, they apply citability and E-E-A-T principles to assets that cannot be understood from body copy alone.

For most current AI engines, multimodal reading works in two ways. Frontier multimodal models such as GPT-4V, Gemini, and Claude with vision can analyze an image when they receive the file (OpenAI GPT-4V system card · Gemini technical report). Web retrieval pipelines, however, still tend to pass text rather than pixels to the model. As a result, an asset’s alt text, caption, transcript, and schema markup usually carry more information into an AI answer than the asset itself.

Definition (GEO Wiki working definition): Multimodal signals are the machine-readable cues attached to non-text assets. They include a text channel (alt text, captions, transcripts, and surrounding prose), a structured-data channel (ImageObject, VideoObject, AudioObject, and Dataset), and a provenance channel (C2PA credentials, EXIF, and IPTC). Together, these cues determine whether an AI-generated answer can ground itself in the asset.

The four categories are images, video, audio, and a final group comprising charts, tables, and diagrams. Each depends on the same distinction between index-integrated AI, which can reuse an existing media index, and live-fetch AI, which reads a page at retrieval time.

2. Four asset types at a glance

Each asset type has a primary text channel, a corresponding structured-data channel, and a typical place where it appears in AI answers.

In every row, the primary text channel carries the information that an answer engine can most readily use. VideoObject makes this relationship explicit through its transcript field, which Schema.org defines as “the transcript of that object.” The two retrieval modes determine how an engine reaches that text.

3. How the two reading modes handle assets

Schema.org for AI distinguishes between index-integrated and live-fetch reading. The same distinction determines how AI engines handle non-text assets.

Index-integrated AI includes Google AI Overviews and Gemini when it uses Search. This mode reuses Google’s existing image and video index. Google has parsed image alt text for more than a decade, while YouTube auto-captions and uploaded transcripts use infrastructure that supported Google Video Search before AIO existed. AIO image carousels draw from Google Images, so multimodal extraction occurs during normal indexing rather than when the answer is generated. AIO therefore reuses Google’s existing quality systems instead of adding a separate AIO-specific layer (Google Search Central, “AI features and your website”).

Live-fetch AI includes ChatGPT search, Perplexity, and Claude with browsing. This mode reads HTML at retrieval time, including the alt attribute and surrounding prose. It does not run OCR on images at retrieval time, transcribe video, or process audio. Unless the text channel describes the asset, the retriever cannot access its content.

When a user uploads an image, video, or PDF directly to ChatGPT or Perplexity, those products can apply vision and OCR to the file. This user-upload workflow is distinct from web retrieval. For an image published on a web page, the retrieval pipeline generally receives its alt text, caption, and schema rather than its pixels. Model capability is therefore not the same as retrieval-pipeline capability.

Across both modes, the text channel remains the dominant signal for non-text assets. Index-integrated AI can use text that was indexed long before AI answers existed, while live-fetch AI usually has no pixel-processing stage in its retrieval pipeline. The text channel should therefore come first. Provenance and pixel-level capabilities provide additional support.

4. Image signals: alt text, captions, ImageObject, and provenance

Images have the longest history of machine-readable signals among the four asset types. Three signal layers work together.

Text channel: The alt attribute, a surrounding caption, and a descriptive heading near the image are available to every AI reading mode. Google’s image best practices state, “Google uses alt text along with computer vision algorithms and the contents of the page to understand the subject matter of the image” (Google Search Central, updated 2026-03-02). WCAG accessibility guidance and AI extraction practices overlap closely here because live-fetch AI sees much of the same text a screen reader needs. W3C Success Criterion 1.1.1 (Non-text Content) requires that “All non-text content that is presented to the user has a text alternative that serves the equivalent purpose” (W3C WAI).

Structured-data channel: ImageObject carries caption, contentUrl, license, creator, embeddedTextCaption, exifData, and representativeOfPage (schema.org/ImageObject). Product pages can use Product.image, while articles can use Article.image. Schema.org for AI explains how those fields fit into the broader markup system.

Provenance channel: C2PA content credentials provide cross-vendor cryptographic evidence of an image’s origin and edit history. EXIF camera metadata and IPTC photo credits identify the photographer, copyright holder, and source. In November 2025, the IPTC Photo Metadata Standard v2025.1 added an “AI System Used” property that identifies the model used to generate an image, with ChatGPT, DALL-E, and Google Gemini among the named examples (IPTC). These emerging provenance signals contribute to trust much like the authorship evidence described in E-E-A-T.

For live-fetch AI, only the image’s text channel is visible. The two most common high-volume uses are e-commerce product images, where alt text can identify the product and key variant, and editorial photography, where it can describe the scene, subject, and context. The same principles apply in other fields.

5. Video signals: transcripts, captions, VideoObject, and hosting

For video, hosting choice has the greatest effect on AI readability. A video uploaded to YouTube and the same video self-hosted without a transcript present very different information to an AI engine, regardless of their markup.

Text channel: A same-page transcript is the most effective multimodal improvement for most sites. SRT or VTT captions, the video description, and the title provide additional text. The transcript is the only form in which the spoken content enters the live-fetch retrieval pipeline.

Structured-data channel: VideoObject includes an explicit transcript field, which Schema.org defines as “the transcript of that object” (schema.org/VideoObject). Other important fields are description, thumbnailUrl, uploadDate, contentUrl, embedUrl, and duration in ISO 8601 format. AIO parses this structured data, while live-fetch systems can read it as page text, as described in Schema.org for AI.

Hosting effect: YouTube and Vimeo automatically generate captions and transcripts that Google’s index already consumes. Google’s video SEO best practices recommend, “Create a dedicated watch page for each video” (Google Search Central, updated 2025-12-18). For content analysis, a self-hosted video without a transcript is largely invisible in both reading modes because the page provides only a <video> element.

Captions burned into the video pixels are unavailable to a text-channel reader. Only file-based captions can be extracted as text. This is the video equivalent of placing body copy inside an image. For media-heavy sites, including publishers, education companies, and video-first businesses, hosting choice usually matters more than whether the markup is complete.

6. Audio signals and the limits of speakable schema

Speakable schema does not make content broadly “voice-AI-ready,” despite a common claim to the contrary.

Google’s documentation, last updated on 2025-12-10, says that speakable structured data remains in beta and serves “users in the U.S. that have Google Home devices set to English, and publishers that publish content in English” (Google Search Central, Speakable). These U.S.-only, English-only, and news-only restrictions have remained in place for years, and the December 2025 update did not remove them. The feature supports Google Assistant TTS playback; it is not a general signal that content is intended for AI.

For audio, the broadly useful signals are simpler:

• A transcript serves the same purpose as it does for video. Without one, a podcast’s spoken content never enters the text channel and remains invisible to live-fetch AI.
• Show notes and episode descriptions provide the summary that search engines and live-fetch AI read first.
• AudioObject carries transcript, contentUrl, caption, encodingFormat, and duration (schema.org/AudioObject). PodcastEpisode carries partOfSeries, episodeNumber, duration, datePublished, and an embedded audio object (schema.org/PodcastEpisode). The embedded AudioObject, rather than PodcastEpisode, carries the transcript.

Speakable supports TTS playback, while transcripts support general AI readability. They serve different purposes. Claims that speakable markup provides broad AI readability go beyond what the specification currently supports.

7. Charts, tables, and diagrams: publishing data as text

For analytical content, the underlying data matters more than the visual rendering because that data is often the evidence a reader or AI engine needs.

Extraction systems read HTML data tables, not charts rendered as pixels. A bar chart published as a PNG is unreadable to live-fetch AI, while the same chart accompanied by its underlying data table is fully readable. Two patterns address this limitation:

• Chart with a data-table fallback: Display the chart for readers, then include the underlying numbers in an HTML <table> or as plain text so AI systems can access the information.
• Caption with summary statistics: Add a paragraph that states the headline number, source, and time period in plain text. The paragraph can be cited even when the chart cannot be read.

<!-- Visual chart for readers -->
<img src="/charts/q1-revenue.png" alt="Quarterly revenue trend, Q1 2024 through Q1 2026, measured in millions of dollars">

<!-- Text fallback for AI retrieval -->
<figcaption>Q1 revenue grew from $12M (Q1 2024) to $19M (Q1 2026), a 58% increase.</figcaption>
<table>
  <thead><tr><th>Quarter</th><th>Revenue ($M)</th></tr></thead>
  <tbody>
    <tr><td>Q1 2024</td><td>12</td></tr>
    <tr><td>Q1 2025</td><td>15</td></tr>
    <tr><td>Q1 2026</td><td>19</td></tr>
  </tbody>
</table>

Schema.org provides Dataset (schema.org/Dataset) for full published datasets and Table semantics for tabular content. The essential requirement, however, is the HTML data table itself. Structured markup is an additional layer. Data tables are also among the most citable content formats described in Citability, so the same text that makes a chart readable can also support direct quotation in AI answers.

8. Trust and provenance for non-text assets

For non-text assets, E-E-A-T depends on evidence about who created the asset, where its content came from, and whether the underlying claims can be verified. ImageObject and VideoObject provide some of that evidence through the provenance fields described in Schema.org for AI.

Trust filters apply to non-text assets as well as prose. A large collection of AI-generated stock photos with no provenance, a video with a fabricated author byline, and a chart whose numbers have no source can all trigger the same AI-at-scale trust concerns described in AI Content Detection. The concern is the same even though the asset type differs.

Google DeepMind’s SynthID watermarking technology covers all four modalities: images, video, audio, and text. Its official page states, “The watermarks are embedded across Google’s generative AI consumer products, and are imperceptible to humans – but can be detected by SynthID’s technology” (Google DeepMind). The page does not publish specific detection-accuracy figures. SynthID therefore provides credible evidence that an effect may exist, but the effect remains unmeasured, consistent with the treatment of watermarking in AI Content Detection §6.

Image-provenance systems such as C2PA, SynthID, and IPTC’s AI-generation field are real and becoming more widely used. As of May 2026, however, no major AI engine has confirmed that these systems act as a citation gate. The evidence supports the direction of the effect, not its size.

9. What the evidence supports

Available evidence supports the overall mechanism, but it does not establish how much any specific markup choice changes citation rates. The same distinction between direction and effect size applies to the evidence discussed in Multilingual GEO §7 and Entity Recognition §6.

As of May 2026, no rigorous published benchmark measures the citation-rate lift from a specific multimodal markup choice, including alt-text quality, ImageObject completeness, transcript presence, or C2PA attestation. The evidence supports the direction of the effect across all four asset types, but not its magnitude. A precise claim such as “images increase AI citation rates by N%” exceeds the available evidence. Practitioners should follow the established direction of the evidence without assuming a specific, unverified effect size.

10. Common multimodal mistakes

These mistakes weaken citability or apply familiar practices in contexts where they do not work. Multilingual GEO §8 documents the same pattern across languages. Here, the errors come from confusing model capabilities with retrieval behavior or treating structured markup as a substitute for readable text.

The usual mistake is not simply missing the markup. It is treating multimodal retrieval as a vision problem when, in 2026, it remains primarily a text-channel problem.

11. Applying multimodal signals in GEO

Multimodal GEO applies citability and E-E-A-T to non-text assets. The distinction between index-integrated and live-fetch reading determines which signal matters most for each asset type.

Audit the text channel for every non-text asset before changing its markup or provenance. Most teams find that their biggest problems are missing video transcripts, missing alt text on content images, charts published as PNGs without underlying data, or AI-generated images published at scale without provenance. Address the text channel first, then add provenance and pixel-level capabilities when they add useful information.

For the term itself and its neighbors, see the GEO glossary.

References

Official sources and standards:

• Schema.org: ImageObject · VideoObject · AudioObject · PodcastEpisode · Dataset
• Google Search Central: Google Images best practices (updated 2026-03-02) · Video SEO best practices (updated 2025-12-18) · Speakable (SpeakableSpecification) structured data (updated 2025-12-10, still beta) · AI features and your website · Top ways to ensure your content performs well in Google’s AI experiences on Search
• W3C WAI: Understanding Success Criterion 1.1.1: Non-text Content
• IPTC: Photo Metadata Standard (v2025.1, 2025-11-27; added the “AI System Used” property)
• C2PA: Coalition for Content Provenance and Authenticity · Membership · Specifications
• Google DeepMind: SynthID (image, video, audio, and text watermarking)

Vendor and technical sources:

• OpenAI: GPT-4V(ision) system card (2023-09-25)
• Google DeepMind: Gemini: A Family of Highly Capable Multimodal Models (arXiv:2312.11805, 2023-12-19) · Introducing Gemini (2023-12-06), which says Google “designed Gemini to be natively multimodal, pre-trained from the start on different modalities”

Industry source:

• Goodwin, D. (2025-05-21). Google shares 8 ways to be successful with AI Search experiences. Search Engine Land.