Guides
How answer engines pick the sources they quote, which AI crawlers cost you something when you block them, and what the published research actually supports — with the limits of each claim stated.
How answer engines choose which sources to quote
Answer engines assemble responses from passages they can lift and trust. Here is what that means in practice, what the research found, and what nobody can honestly promise you.
Read guide 8 min readTraining vs. retrieval AI crawlers: which ones you can safely block
GPTBot governs OpenAI training crawls; OAI-SearchBot governs ChatGPT search discovery. They are different crawlers, the names are similar, and a rule written for one routinely catches the other.
Read guide 7 min readWhat is llms.txt, and is it worth publishing?
A markdown file at your site root that tells a language model what your site covers. The format is simple and settled. Whether any engine reads it is not — and most write-ups skip that part.
Read guide