I Built robots.md Because AI Agents Can't Find Your Business

A few weeks ago, I asked Claude to analyse my website, agenti.nz. I wanted a competitive breakdown of my new AI consultancy here in New Zealand. Claude came back with almost nothing. The irony wasn't lost on me. I run an AI consultancy, and AI couldn't find me.

It could see the page title — "agenti nz - Build AI Operating System for Business" — and that was it. The entire content of my site, my services, my case studies, my pricing, my story — invisible. Not to Google. Not to Claude. Not to any AI agent that might want to recommend my business to someone looking for help with automation.

The problem no one's talking about

Here's what happened: my site was built as a JavaScript single-page application. It renders beautifully in a browser. But when an AI agent or crawler requests the page, they get an empty shell. No content. No context. Nothing to work with.

This isn't a niche problem. It's the default state of a huge chunk of the modern web. React, Vue, Next.js (client-side), Angular — millions of business websites render content dynamically in the browser. They look great to humans. They're blank pages to machines.

And here's the thing that really got me thinking: even if my site was crawlable, what would an AI agent actually learn? It would get marketing copy. Hero sections. Buzzwords. Maybe some nav links. What it wouldn't get is the structured information it actually needs: what does this business do? What services are available? What are the prices? Is there an API? How should I interact with this business on behalf of my user?

robots.txt was built for a different era

In 1994, Martijn Koster proposed robots.txt — a simple text file at the root of every website that tells crawlers what they can and can't access. It became a universal standard. Every search engine checks for it. It's been one of the most successful informal standards in the history of the internet.

But robots.txt was designed for a world where the interaction was simple: a crawler indexes pages, a search engine displays links, a human clicks through. The crawler doesn't need to understand the business. It just needs to know which URLs to follow.

We're not in that world anymore.

AI agents are now booking appointments, comparing services, making purchases, filing support tickets, and integrating systems — all on behalf of humans. They don't need a list of URLs. They need to understand what a business is, what it offers, and how to interact with it programmatically.

So I built robots.md

The idea was simple: what if every business had a single, human-readable markdown file at their website root that told AI agents everything they needed to know?

Not a complex schema. Not a YAML config. Not an XML manifest. Just a markdown file — the same format developers use for READMEs, the same format AI models are trained on, the same format that's readable by both humans and machines.

I called it robots.md.

Here's what mine looks like at agenti.nz/robots.md:

# Agenti NZ - AI Agent Discovery & API Documentation

## Organization
- **Name:** Agenti NZ
- **Industry:** AI Automation & Business Process Automation
- **Location:** New Zealand
- **Contact:** [email protected]
- **Website:** https://agenti.nz

## Services
1. AI Discovery Audit ($499)
2. Custom Agent Build (From $2,500)
3. Support & Optimization ($750/month)

## API Endpoints
| Endpoint | Method | Description |
|----------|--------|-------------|
| /api/trpc/content.markdown | GET | Full business content |
| /api/trpc/contact.submit | POST | Contact form |

## Data & Privacy
Privacy Act 2020 compliant. NZ-based servers.
No data shared with third parties.

It's that simple. An AI agent hits /robots.md, gets a complete picture of the business, and knows how to interact with it. The content API endpoint means any agent can pull detailed, structured information about services, pricing, case studies, and FAQs — without scraping a single HTML page.

The moment it actually worked

After I told Claude about the robots.md file, it fetched the content API referenced inside. Suddenly it could see everything: my three service tiers, my case studies (EVX saved $3,200/month, Kiwi Outdoor Gear reduced stockouts by 85%), my implementation process, my FAQ, my compliance posture.

The analysis went from "I can see your page title" to a comprehensive competitive breakdown with pricing recommendations and strategic advice.

I'm not the only one thinking about this

When I started researching, I discovered I was part of a much larger conversation:

The pattern is unmistakable: the industry is converging on the idea that AI agents need a way to discover and understand businesses. The question is what form the standard takes.

Where robots.md fits

robots.md isn't trying to replace any of these. It's the business owner's version — the file that a café owner, a plumber, a real estate agent, or a small consultancy can create and maintain without a developer.

The chicken-and-egg problem (and how to solve it)

I'll be honest about the obvious challenge: no AI agent currently checks for robots.md by default. Claude didn't know to look for mine until I pointed it out. It's not yet a convention.

But this is how every standard starts. robots.txt didn't work until crawlers were programmed to look for it. llms.txt has hundreds of thousands of adopters but no major LLM provider has officially confirmed they read it. AGENTS.md needed OpenAI, Google, and Cursor to agree on the format.

Standards succeed when they solve a real problem with minimal friction. I think robots.md qualifies:

• ✓The problem is real — AI agents can't find or understand most businesses
• ✓The format is universally understood — Markdown
• ✓The cost is near zero — one file, plain text, no tooling required
• ✓The value compounds — every AI agent that reads it gets a better answer

In the meantime, here are practical steps to make robots.md discoverable today:

<link rel="ai-discovery" href="/robots.md" type="text/markdown" />

# AI Agent Discovery
AI-discovery: /robots.md

// GET /.well-known/ai-discovery
{
  "robots_md": "/robots.md",
  "content_api": "/api/content",
  "version": "0.1"
}

# Business Identity
- [AI Agent Discovery](/robots.md): Structured business info for AI agents

What I'm doing next

I'm publishing the robots.md specification as an open standard. I'm submitting a public comment to NIST's AI Agent Standards Initiative. I'm building a free validator tool. And I'm implementing robots.md as a standard deliverable for every AI agent build at Agenti NZ.

If you run a business, you can create a robots.md today in about 15 minutes. The AI agent era is accelerating — Mastercard just tested Agent Pay in New Zealand, OpenAI's Operator browses the web on your behalf, Google's agent ecosystem is scaling fast. The businesses that are machine-readable will get found. The ones that aren't, won't.