You've built a website for humans, but have you built one for the AI agents that are increasingly browsing on their behalf? This guide shows you how to implement a robots.md file—a simple, markdown-based discovery standard that tells AI exactly what your business does and how to interact with it.
What is robots.md?
Think of robots.md as the modern, AI-first successor to robots.txt. While robots.txt tells crawlers where they can't go, robots.md tells AI agents what they can do. It's a plain text markdown file located at your website's root (yourdomain.com/robots.md).
Step 1: Create Your robots.md File
Open any text editor and create a new file named robots.md. Copy the template below and update it with your business information:
# [Business Name] - AI Agent Discovery ## Organization - **Name:** Your Business Name - **Industry:** e.g., Real Estate, Plumbing, Retail - **Location:** City, Country - **Contact:** [email protected] - **Website:** https://yourdomain.com ## Services 1. Service Name 1 ($Price) 2. Service Name 2 ($Price) ## AI Interaction - **Primary Agent Goal:** e.g., "Help users book a discovery call" - **Contact Endpoint:** https://yourdomain.com/contact - **Sitemap:** https://yourdomain.com/sitemap.xml
Step 2: Upload to Your Website Root
Upload the file to the root directory of your web server so it's accessible at https://yourdomain.com/robots.md.
Step 3: Make It Discoverable
To ensure AI agents find your file, add a link to it in your website's <head> section:
<link rel="ai-discovery" href="/robots.md" type="text/markdown" />