What is robots.txt?
robots.txt is a plain-text file at a site's root that tells crawlers which paths they may or may not request. It controls crawling, not indexing, and it is a public request rather than a security measure.
Also called: robots file, crawler rules.
The distinction between crawling and indexing is where most robots.txt damage happens. Blocking a URL stops a crawler fetching it — but if other pages link to it, it can still appear in results as a bare URL with no description. To keep something out of the index you need a noindex tag on a page crawlers are allowed to read, which is the opposite instinct.
The second trap is blocking assets. Disallow your CSS or JavaScript folders and Google renders your site as a broken skeleton, then judges its mobile usability on that. Modern advice is to let crawlers fetch everything needed to render the page.
And it is public. Anyone can read yours, so listing /admin-secret/ as disallowed is an advertisement, not a lock.
What to actually do about it
- Keep it minimal — block only genuinely useless paths like internal search.
- Never block CSS, JS or image directories.
- Add a Sitemap: line pointing at your sitemap.
- Use noindex, not robots.txt, to remove pages from search results.
- Decide deliberately whether to allow AI crawlers such as GPTBot.
Koo serves a correct robots.txt per site with the sitemap reference included, keeps rendering assets crawlable, and lets you choose whether AI crawlers are welcome.
Related terms
XML sitemap
An XML sitemap is a file listing the URLs on a site that you want search engines to crawl, along with when each was last changed, submitted through Search Console so new and updated pages are found quickly.
llms.txt
llms.txt is a plain-text file at the root of a website that gives AI assistants a curated, markdown-formatted map of its most important pages, so a model can find the substance without crawling and parsing the whole site.
Canonical URL
A canonical URL is the single address you declare as the official version of a page, using a rel="canonical" tag, so search engines consolidate ranking signals on one URL instead of splitting them across near-identical copies.