The robots.txt file is a simple text file published in the root directory of a website that contains instructions for automated indexing programs, known as search engine crawlers or spiders. Its main function is to tell these programs which areas of the site can be explored and which should be avoided. It is not a security mechanism, but a convention widely respected by major search engines such as Google, Bing, and others.
The name comes from the computing tradition of placing files in a domain’s root that define behavior rules for automated access. The robots.txt is read before the crawler starts downloading pages, and the directives it contains influence how the site is scanned. Understanding how it works is essential for anyone doing search engine optimization, because a wrong configuration can prevent important content from being indexed or, conversely, leave pages exposed that you don’t want to show in search results.
What happens when a crawler arrives on the site
When a crawler visits a site, the first request is often for the robots.txt file. If the file exists, the program reads it and interprets its rules. If the file doesn’t exist, the crawler assumes there are no restrictions and proceeds to explore the site freely. Rules are expressed in a simple text format based on pairs of directives and values.
Main directives
The most common directives are User-agent, which identifies the crawler the rule applies to, and Disallow, which indicates paths that should not be explored. There is also Allow, which lets you specify exceptions within forbidden areas. You can also use Sitemap to indicate the address of the sitemap, making it easier to discover pages.
A robots.txt file can contain multiple blocks, each referring to a different User-agent. Rules are applied based on the match between the agent declared by the crawler and the one specified in the file. If there are no matches, the rules of the group marked with an asterisk apply, which serves as the default value.
How to write a robots.txt: syntax and rules
The robots.txt syntax is linear and tolerant, but requires attention to some details. Each line contains a directive followed by a colon and a value. Empty lines and those starting with a hash are ignored and serve as comments. Paths are case-sensitive and must start with a slash.
Example structure
A typical file might start with a block for all crawlers, followed by specific rules for individual agents. For example, you can forbid access to an admin folder while allowing access to a necessary subfolder. Precision in writing paths avoids ambiguity and ensures directives are interpreted correctly.
It’s important to remember that robots.txt does not prevent a malicious crawler from accessing content: it’s a request, not an obligation. To truly protect restricted areas, you need other tools such as authentication or web server rules.
Robots.txt, noindex, and canonical: they are not the same thing
Robots.txt is often confused with other mechanisms that regulate indexing. It’s useful to clarify the differences to avoid mistakes that can compromise a site’s visibility. The following table summarizes the main distinctions.
| Tool | Function | Scope | Effect on indexing |
|---|---|---|---|
| robots.txt | Tells crawlers which paths to explore | Entire site or individual directories | Prevents scanning, not necessarily indexing |
| Meta robots | Provides instructions in the page | Single page | Controls indexing and display |
| Noindex directive | Asks not to index a page | Single page | Removes the page from search results |
| XML Sitemap | Lists pages to index | Entire site | Facilitates discovery, does not impose restrictions |
| HTTP header | Transmits instructions from the server | Single resource | Can block or allow indexing |
Mistakes to avoid and best practices
An incorrect robots.txt configuration can have unintended consequences. Among the most frequent mistakes is accidentally blocking entire sections of the site, which prevents search engines from scanning relevant content. Another mistake is forbidding access to resources needed for rendering, such as stylesheets or image files, compromising page evaluation.
Periodic checks
It’s good practice to regularly check the file’s content and use the tools provided by search engines to test the rules. This way you avoid conflicts between directives and maintain control over site scanning.
Robots.txt and AI crawlers
Robots.txt is also the tool for indicating which AI crawlers can use your content. Some operators publish distinct user-agent names for different purposes: for example GPTBot (OpenAI, training), OAI-SearchBot (OpenAI, search), ClaudeBot (Anthropic), Google-Extended (use of content for Gemini), and PerplexityBot. Blocking Google-Extended does not affect presence in Google Search. Blocking AI search crawlers, however, can reduce the chance of being cited in their answers: it’s a choice to make consciously, also from a GEO perspective. Robots.txt remains a request: bots that don’t respect it must be blocked at the server level.
How to create a robots.txt file step by step
Creating a robots.txt file requires a few steps, but they must be done carefully. Below is a useful operational sequence for those approaching this tool for the first time.
- Check if a robots.txt file already exists in the site’s root directory.
- Decide which areas of the site should be excluded from scanning.
- Open a plain text editor and write the desired directives.
- Specify the User-agent to which the rules apply.
- Enter the paths to block with the Disallow directive.
- Add any exceptions with the Allow directive.
- Indicate the sitemap address, if available.
- Save the file as robots.txt and upload it to the root directory.
Frequently asked questions about robots.txt
How do you create and configure a robots.txt file?
To create a robots.txt file, all you need is a plain text editor: write the directives, save the document as robots.txt, and upload it to the site’s root, i.e., the domain’s main directory. Configuration is based on lines like User-agent, Disallow, Allow, and Sitemap, each followed by a colon and the value. A well-made text file starts with a group for all crawlers and then adds specific blocks. The file must remain reachable at example.com/robots.txt, otherwise search engines consider it absent. Before publishing it, test it with the verification tools provided by search engines to avoid accidentally blocking useful sections.
What is robots.txt for in SEO and crawl budget?
Robots.txt is used to direct crawler scanning, excluding pages and resources you don’t want explored and focusing crawl budget on important URLs. In practice, it lets you block areas like shopping carts, filters, internal search results, or duplicates, preventing them from consuming requests useful elsewhere. Remember that blocking scanning does not always equal blocking indexing: an excluded page can still appear in results if it’s linked from other sources. So when the goal is to keep a page out of the index, robots.txt alone is not enough and must be combined with other tools. Careful configuration improves scanning efficiency and indexing quality.
What are the main commands and directives in robots.txt?
The main robots.txt directives are User-agent, which indicates which crawler the block applies to, Disallow, which lists paths not to explore, and Allow, which opens exceptions within forbidden areas. Sitemap points to the sitemap address and helps page discovery. Crawl-delay is not supported by Google, which manages crawl rate independently, but it is read by other engines like Bing. Rules are written one per line, with the directive followed by a colon and the value, and paths are case-sensitive. When multiple rules apply to the same URL, Google uses the most specific one (the longest path); if specificity is equal, the least restrictive prevails.
What is the robots exclusion protocol and how does robots.txt work?
The robots exclusion protocol is the informal standard that defines how a site communicates access restrictions to crawlers via the robots.txt file. It is not a law or a security system: it’s a convention that major search engines and many other bots voluntarily respect. A crawler that reads the file interprets the directives and, under normal conditions, avoids paths marked with Disallow. Since it’s a request and not an obligation, malicious software can ignore it entirely. To truly protect confidential content, you need authentication, passwords, or server-side rules. The protocol remains the standard tool for managing scanning restrictions in a declared and transparent way.
Why is the robots.txt file important for indexing?
The robots.txt file is important because it’s the first point of contact between a site and search engine bots, and its rules guide the entire indexing process. If the file is missing, crawlers assume there are no restrictions and explore freely; if it’s poorly written, it can exclude pages you want indexed or leave useless areas accessible. Rules influence which URLs are scanned and how often, so they affect index coverage. A correct file doesn’t guarantee ranking, but it prevents relevant content from being left out by mistake. That’s why it should be considered part of every site’s basic technical configuration.
What is the history and standard of robots.txt?
Robots.txt was created in 1994 as a proposal by Martijn Koster, who defined it to coordinate crawler access to web servers, and it has remained a de facto standard for almost thirty years. In 2022 the protocol was formalized by the IETF as RFC 9309, but it remains a voluntary convention: nothing obliges a crawler to respect it. Over time, directives like Allow and Sitemap were added; Google has never supported Crawl-delay. Knowing the history helps understand its current limitations.
How do you create and modify robots.txt and manage its use in WordPress?
In WordPress, robots.txt is generated automatically and normally doesn’t exist as a physical file on the server: you can see it by opening the site followed by /robots.txt. To create and modify rules, you can use a dedicated plugin or, if the site is on hosting with file access, create a robots.txt file in the root, which generally takes precedence over the virtual one. A robots.txt generator helps compose directives without syntax errors. Using it in WordPress then requires testing and troubleshooting with verification tools, because an accidental block can exclude important pages. On managed installations, some changes may be limited by the environment.
How do you block bot access and limit scanning with robots.txt?
To block bot access with robots.txt, use the User-agent directive to identify the crawler and Disallow to indicate folders or paths not to explore. You can limit scanning of entire directories, such as admin areas or internal archives, or individual files. To manage specific bots, declare the exact agent, for example Googlebot or Googlebot-Image, while the asterisk serves as the default rule for all others. Blocking folders is useful to reduce unnecessary requests, but do it with caution: excluding resources needed for rendering, such as stylesheets or images, can worsen page evaluation. After each change, it’s good practice to verify that the rules produce the desired effect.
From glossary to projectWant to know if your site has technical issues holding back visibility?The technical SEO audit analyzes the site to identify errors and ranking opportunities on Google and delivers a prioritized list of actions.Discover the technical SEO audit
Related entries:
Disallow robots.txt Site indexing on Google Advanced technical SEO indexing