# Answer engine optimisation: getting cited by AI assistants

> How AI assistants and AI search choose the pages they cite, what their makers have published about crawlers, and the page and technical changes worth making.

- Published: 2026-10-09; last updated 2026-11-13
- Author: the WeStack AI team
- Subject: Product engineering
- Address: https://westack.ai/blog/answer-engine-optimisation

> Answer engine optimisation (AEO) is the work of making your pages easy for AI assistants and AI search features to find, read and cite. Most of it is ordinary good publishing: let the right crawlers in, serve the content as HTML, answer the question early and back claims with sources. Google says its AI features need no special optimisation, and files such as llms.txt help less than their popularity suggests.

## How do AI assistants choose sources?

An assistant can know about you in two ways. The first is training data: text the model absorbed months or years ago, which it recalls without citing anything. The second is live retrieval: the assistant runs searches, fetches a handful of pages, reads them and writes an answer with links. Citations come almost entirely from the second route, and it is the one you can influence this quarter. It is the same retrieve-then-generate pattern used in [RAG and enterprise search](https://westack.ai/services/generative-ai), pointed at the public web.

Retrieval rarely uses the question as typed. Google's documentation describes query fan-out, in which the system issues several related searches across sub-topics and assembles the answer from the results. Take a company that sells payroll software to small firms. A buyer asks an assistant how to pay a contractor based in another EU country. The assistant may search separately for invoicing rules, tax treatment and payment methods, then cite one page for each. A long guide that mentions all three in passing loses to a page that answers one of them directly.

So a page gets cited when four things are true. It is in the index the assistant searches. It is retrieved for one of the sub-queries. The useful passage is easy to extract. The source looks trustworthy enough to quote.

It helps to be honest about what is known. The providers document which crawlers they run, how to allow or block them, and what makes a page eligible. None of them publishes how a page is chosen for a particular answer. Anything more specific than that, including much of what is sold as AEO or GEO expertise, is inference from experiments and should be labelled as such.

## What the providers have published

This is the documented part, checked at the time of writing.

| Provider | Crawlers and controls | What the documentation says |
|---|---|---|
| Google | Googlebot covers Search, including AI Overviews and AI Mode. Google-Extended is a separate control for training and for grounding in other Google products | Normal SEO applies. No special files, markup or rewriting are needed. Structured data should match the visible text |
| OpenAI | OAI-SearchBot for search, GPTBot for training, ChatGPT-User for fetches a user triggers | Sites that block OAI-SearchBot are not shown in ChatGPT search answers. Robots.txt rules may not apply to ChatGPT-User |
| Anthropic | Claude-SearchBot for search, ClaudeBot for training, Claude-User for fetches a user triggers | All three honour robots.txt. Blocking the search or user bots can reduce visibility in Claude's answers |
| Perplexity | PerplexityBot for its index, Perplexity-User for fetches a user triggers | PerplexityBot is not used to train foundation models. Perplexity-User generally ignores robots.txt because a person requested the page |
| Microsoft | Bingbot. Bing's index feeds Copilot | An AI Performance report in Bing Webmaster Tools shows citations and the queries used for grounding |

The sources are [Google's guide to generative AI features in Search](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide), [OpenAI's crawler documentation](https://developers.openai.com/api/docs/bots), [Anthropic's crawler article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-the-web), [Perplexity's crawler page](https://docs.perplexity.ai/guides/bots) and [Microsoft's AI Performance announcement](https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview).

Google's guide is blunt. From its side, optimising for generative AI search is still SEO, and it lists llms.txt files, splitting content into small chunks, rewriting for AI and manufactured brand mentions as things you can skip. Microsoft's advice is more specific: clear headings, tables and FAQ sections, claims backed by evidence, and fresh content.

Two practical points follow. Blocking a training crawler does not remove you from search answers, because the search crawlers are separate. And robots.txt does not govern every fetch, because user-triggered fetchers may not follow it.

## What to change on the page

Write for a reader in a hurry. That reader and an extraction system want the same things.

- **Answer first.** Put a direct answer of two or three sentences at the top of the page, before any background. The quoted block at the top of this article is that pattern.
- **Use headings people would type.** A heading phrased as the question, followed by the answer in the first sentence beneath it, is easy to match to a sub-query and easy to lift.
- **State facts with their sources.** Give the number, the date and a link to where it came from. A claim with a named primary source is safer to quote than one without.
- **Use tables for comparisons and numbered lists for steps.** They survive extraction better than a paragraph that holds six values.
- **Define your terms.** One plain sentence saying what a thing is can be quoted whole. A paragraph of context cannot.
- **Say something of your own.** Google's guide singles out first-hand experience and content that goes beyond common knowledge. A page that restates what ten others say gives an assistant no reason to pick it.
- **Date the page and keep it current.** Show when it was last reviewed, and update it when the facts change.

There is some research support. The paper that coined generative engine optimisation (Aggarwal and colleagues, KDD 2024) tested content changes on a benchmark of queries and reported that adding citations, quotations and statistics improved visibility in generated answers, by up to 40 per cent in the best case, with results that varied by domain. Treat that as a direction and not a forecast. It was a research benchmark, not today's products.

## What to change technically

**Serve the content as HTML.** Google says it processes JavaScript. We would not assume every AI fetcher does. If the answer only exists after client-side rendering, some systems will see an empty page. Server-rendered or static pages remove the doubt, which is one reason the framework choice in [Next.js vs Astro](https://westack.ai/blog/nextjs-vs-astro) matters for content sites.

**Be fast and plain.** Use semantic HTML, one clear main content area, canonical URLs and a sitemap. Keep content out from behind logins and interstitials. A fetcher working inside a time limit gives up on slow pages.

**Sort out robots.txt.** Decide on search crawlers and training crawlers separately.

```
# Search and answer crawlers: allow if you want to be cited
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# Training crawlers: a business decision
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
```

Whether to block training is a judgement call. Blocking protects content you sell. Allowing it may help future models know your brand, though nobody outside the labs can measure that.

**Check the layer in front of your site.** A content delivery network or bot protection service can block AI crawlers whatever your robots.txt says. Cloudflare, for example, changed its presets for new sites in September 2026: sites that say they carry advertising start with training crawlers disallowed and agent traffic blocked on pages with ads. Look at what your own settings are doing.

**Check Search Console.** Google's guide notes that a site must be included in generative AI features in Search Console to be eligible, so confirm the setting is what you intend.

**Use structured data, with modest expectations.** Google says no special schema is needed for AI features. Article, FAQ, Organisation and Product markup still help with rich results and make it clearer who you are. Keep it identical to the visible text.

## Is llms.txt worth adding?

[llms.txt](https://llmstxt.org/) is a proposal, first published by Jeremy Howard in September 2024, for a Markdown file at the root of a site that lists its most useful pages for language models, ideally with clean Markdown versions of those pages. The specification is still open for community input and was revised in August 2026.

Support is uneven. Google states that Search ignores these files. We could not find documentation from OpenAI, Anthropic or Perplexity saying that their search products use the file to choose or rank sources. Where it does earn its place is developer documentation: coding assistants and agents fetch documentation while they work, and several documentation sites, including Anthropic's for Claude Code, publish an llms.txt index for that purpose.

Our advice: generate it from your sitemap at build time if that costs an hour, and expect nothing from it. It is not a substitute for any item in the two sections above.

## Being cited elsewhere

Assistants assemble answers from many sites, so it is reasonable to expect that a brand which appears in independent sources is more likely to be named. Reviews, industry publications, documentation that others link to, and communities where practitioners answer questions all count. This is ordinary reputation work, and it is slower than editing your own pages.

Two cautions. Google's guide says that seeking inauthentic mentions does not help and that its spam systems are built to catch manipulation. And the effect of mentions on any given assistant is speculation. It is widely believed, and the providers have not documented it. Publish original data, contribute where your buyers already read, and keep the basic facts about your company consistent wherever they appear.

## How do you measure it?

Measurement is better than it was a year ago and still partial.

- **Google Search Console.** The generative AI performance report, available to all sites since the end of August 2026, shows impressions in AI features by page, country, device and date. Clicks are counted in the main Performance report.
- **Bing Webmaster Tools.** The AI Performance report shows total citations, cited pages and a sample of the grounding queries behind them.
- **Referral traffic.** Group visits from assistant domains into one channel in your analytics, and look at what those visitors do. Volumes are often small, so judge them by what the visitors go on to do.
- **Server logs.** Hits from search crawlers show what is being indexed. Hits from user-triggered fetchers show that a page was pulled into someone's conversation.
- **A question panel.** Write down 30 to 50 questions your customers ask. Each month, ask them across the assistants your buyers use and record if your brand is mentioned, if a page of yours is cited and which page it was. Answers vary from run to run, so watch the trend and ignore single results.

No tool outside the providers can see inside their ranking. Dashboards that claim a share of AI voice are built on sampled prompts, which is useful for trends and not precise.

## Where to start

A first week:

1. Check robots.txt, your CDN's bot settings and the Search Console setting, so that the search crawlers you care about can reach the site.
2. Fetch your five most important pages without JavaScript and confirm the main content is there.
3. Rewrite the top of those five pages so each opens with a direct answer, and turn vague headings into the questions they address.
4. Add sources and dates to the claims on those pages.
5. Write the question panel and record a baseline.

The first decision is which questions you want to be the answer to. A short list of questions you can answer better than anyone else beats broad coverage. If retrieval is new to you, our guide to [how RAG works](https://westack.ai/blog/retrieval-augmented-generation-guide) explains the mechanics that assistants share. Building fast, server-rendered sites with this structure is part of our [product engineering](https://westack.ai/services/product-engineering) work.

## Frequently asked questions

### What is answer engine optimisation?

Answer engine optimisation, or AEO, is the practice of making a website easy for AI assistants and AI search features to find, read and cite when they answer a question. It overlaps heavily with search engine optimisation. The main differences are a stronger emphasis on direct answers, sourced facts and access for the crawlers that AI products use.

### Do I need an llms.txt file to appear in AI answers?

No. llms.txt is a community proposal, not a standard that answer engines have committed to. Google says its search features ignore it, and we found no documentation from the other large assistants saying their search products rely on it. It is cheap to add and can help coding tools read developer documentation, but it will not get a page cited.

### Should I block AI crawlers in robots.txt?

Separate the two kinds. Crawlers that build search indexes, such as OAI-SearchBot, Claude-SearchBot and PerplexityBot, decide whether you can be cited, so allow them if you want visibility. Crawlers that collect training data, such as GPTBot and ClaudeBot, are a business decision. Blocking them does not remove you from search answers according to the providers' documentation.

### How can I tell whether AI assistants are citing my site?

Use three sources. Google Search Console has a generative AI performance report showing impressions by page, and Bing Webmaster Tools reports citations in Copilot. Your own analytics show referral visits from assistant domains. A fixed list of customer questions, asked across the assistants each month, shows whether your brand appears in the answers at all.
