Introducing llms.txt: A Simple Standard for LLM Crawling

At Sisif.ai, we believe AI should be built on transparency, respect, and consent. That’s why today, we’re introducing a small but meaningful standard for the AI era: llms.txt.You can check ours out here: https://sisif.ai/llms.txt, and we’re thrilled to be featured on llmstxt.site — a public directory tracking sites that have already adopted the standard.

What is llms.txt?

Think of llms.txt as the LLM-age version of robots.txt. It’s a simple, machine-readable file that tells Large Language Model crawlers how they’re allowed to interact with your content — whether they can crawl, train on it, or use it in inference. And this isn’t hypothetical — major models are already crawling for it.

Why It Matters

The web is being mined, often indiscriminately, by AI models. But creators and website owners deserve a voice in how their content is used. llms.txt is a step toward making AI opt-in by default — or at least giving you the power to say no.This empowers:

• Publishers who don’t want their premium or sensitive content used without consent
• Open source projects to declare training-friendly zones
• Individuals and companies to retain some control over how their data is consumed by LLMs

What Does llms.txt Look Like?

It lives at the root of your site:

We're Just Getting Started

llms.txt is designed to be simple, extensible, and community-driven. We’re already seeing adoption grow — and llmstxt.site is tracking it in real time.

Soon, we’ll be releasing tools to:

• Generate a llms.txt file for your site
  
• Monitor which LLMs are accessing your content
  
• Track changes and violations
  

Join the Movement

If you run a website, add a llms.txt file. If you’re building LLMs, respect it. This isn’t just a file — it’s a signal: that we want an internet where people have agency.

You can follow us at @sisif_ai.

The web has a voice. Let’s make sure AI listens. 🧠🤖