Robots.txt
Also written: robots exclusion protocol
A file that tells web robots which pages on a website they can or cannot visit.
Robots.txt is a simple text file that website owners can place on their servers. It acts like a set of instructions for automated programs, often called bots or crawlers, that visit websites to gather information.
These bots are used for many things, like search engines indexing web pages or AI models learning from online content. The robots.txt file tells these bots which parts of the website they are allowed to access and which parts they should ignore. For example, a website owner might use it to prevent bots from accessing private user data or areas under construction.
Robots.txt is a standard protocol defined by the Robots Exclusion Protocol. Webmasters create a file named 'robots.txt' in the root directory of their web server. This file contains directives, such as 'Allow' and 'Disallow', specifying which user agents (bots) can access which URL paths on the site. It is a cooperative mechanism; compliant bots will respect these directives, but malicious bots may ignore them.
OpenAI's recent announcement about controlling AI training data highlights the importance of robots.txt. As AI models are trained on vast amounts of web data, controlling what data is scraped is crucial for privacy and copyright. Robots.txt is a primary tool for website owners to signal their preferences to AI developers about data collection.
- Our approach to data and AIOpenAI · OpenAI News ·
- OpenAI ChatGPT
- OpenAI's approach to data and AI mentions controlling AI training data, which can be managed in part by respecting robots.txt directives.
- Sitemaps
- Sitemaps list pages a site wants bots to crawl, while robots.txt lists
- Robots Meta Tag
- This tag is placed within HTML pages to control crawling and in
The Robots Exclusion Protocol was developed in 1994 by Martijn Koster and others in the web community to help manage the increasing number of web crawlers. Source ↗
You can view a website's robots.txt file by typing '/robots.txt' after the website's domain name in your browser's address bar. For example, try visiting 'https://www.google.com/robots.txt' to see Google's directives.
The Playbook: reusable how-tos, prompts, configs and tips pulled out of the news.
- PromptRole-play a student struggling with math for AI tutor simulationSimulate a student's math struggles and uncertainty for AI tutor evaluation. · 5 min
- TemplateReusable workflow templates for healthcare teamsTeams can create and share templates for common workflows like documentation or prior authorizations. · 15 min
- TemplateTemplate for building an AI tool tailored to your workA structured approach to defining and building a custom AI tool for specific business needs. · 15 min
- TemplateEvals CSV prompt list for agent skillsA CSV file template for listing prompts used in evaluating agent skills. · 20 min
- HeuristicDouble coding agent task completion with docs linksProviding coding agents with a documentation link significantly increases their task completion rate. · 2 min
Questions people ask
- Do all bots follow robots.txt?
- No, well-behaved bots like those from major search engines generally follow robots.txt. However, malicious bots or scrapers may ignore these instructions.
- Can robots.txt block AI training?
- Robots.txt can signal to AI developers that certain data should not be used for training. Compliance depends on the AI developer respecting these directives.
First seen on Subvolts May 7, 2024. Updated Sep 15, 2026. Written by an AI model from the items above and reviewed against them weekly; report an error.