AI crawler controls
If a crawler cannot read your site, nothing else on it matters. This is the first thing to check.
The setting that outranks everything
WordPress has a checkbox under Settings → Reading called Discourage search engines from indexing this site. When it is on, your site asks every crawler to stay away — search engines and answer engines alike.
Crawlune reports this above every other finding, because a site with it switched on cannot be read by anything, and improving your extractability score while it is on achieves nothing. Staging sites get left with it on and go live that way more often than anyone would like.
Which crawlers you can control
The crawlers used by the major answer engines are listed individually, each with what it is for. Allow or block each one; Crawlune writes the corresponding rules and shows you the result.
What blocking one costs
Blocking a crawler is a real choice with a real trade-off, and Crawlune states it rather than nudging you one way:
- Block, and your content cannot be used to train or answer from — which some publishers want.
- Block, and you cannot be cited either. An engine that cannot read a page cannot link to it. If referral traffic from answer engines is what you are after, blocking the crawler that would send it is the opposite of that.
Crawlune offers a control for six agents, and each one does a single
documented job, because its vendor publishes a separate agent for
each — OpenAI splits GPTBot from
OAI-SearchBot, Anthropic splits ClaudeBot
from Claude-SearchBot. That split is what makes the two
decisions separable at all.
Where a vendor runs one agent for both jobs, Crawlune does not offer a control for it. A single toggle there would be one switch over two unrelated decisions — exactly the "block AI bots" control this plugin exists not to ship.
These are requests, not enforcement
robots.txt directives are honoured voluntarily. Reputable
crawlers respect them; nothing forces anyone to. If you need
enforcement, that is a server or firewall matter and no WordPress
plugin can do it for you.