AI crawler controls
Start by checking which crawlers your site asks to stay away. Training and retrieval serve different purposes, and a robots.txt rule is not the same as denying a request.
Check the site-wide indexing setting
WordPress has a checkbox under Settings → Reading called Discourage search engines from indexing this site. When it is on, your site asks every crawler to stay away — search engines and answer engines alike.
Crawlune highlights this setting so you can check whether it is intentional, especially after moving a staging site live. It asks crawlers not to index the site; it does not make the content private or prevent someone from fetching a public page.
Which crawlers you can control
The crawlers used by the major answer engines are listed individually, each with what it is for. Allow or block each one; Crawlune writes the corresponding rules and shows you the result.
What blocking one costs
Blocking a crawler is a real choice with a real trade-off, and Crawlune states it rather than nudging you one way:
- Training rules express your preference about model training. They do not automatically control a vendor's separate retrieval crawler.
- Retrieval rules affect access for fetching answers. Denying a retrieval crawler can limit its access to your pages. It does not prove that an engine will never link to your site, just as allowing access does not promise a citation.
Crawlune offers six named controls and shows their purpose. For
example, it separates GPTBot from
OAI-SearchBot, and ClaudeBot from
Claude-SearchBot. This lets you choose training and
retrieval rules independently for those vendors.
Robots rules and optional enforcement
robots.txt directives rely on the crawler cooperating.
Crawlune also offers optional request blocking, off by default: it
returns HTTP 403 for requests reaching WordPress whose user-agent
identifies a crawler you have denied. The robots.txt endpoint stays
readable so crawlers can learn the rules.
This checks the user-agent string, not the caller's identity. It cannot stop a client using a different name, and a cached response served before WordPress runs bypasses the check. WordPress has already loaded when it refuses the request; use server or CDN rules when you need to block traffic before it reaches WordPress.