I get the odd one try on my personal site, but usually its just robots.txt... /shrug
literallyroy · 2026-10-25 19:20:30 UTC
I once had oh-my-pi read a website’s llms.txt while doing research many months ago.
naw103 · 2026-10-25 14:05:39 UTC
One trap when you name AI bots specifically is that is that robots.txt groups don’t merge. Crawlers will only follow the most specific group so if GPT bot has its own group named nothing under User-agent: * applies to it anymore so you need to repeat every disallowed line under each group.
Muhammad523 · 2026-10-25 15:50:20 UTC
why? You're helping AI companies to take your writing for free
mmcallister · 2026-10-25 19:00:29 UTC
Whilst I agree with most criticism of AI/LLMs but most of the time at the other end of someone using Claude/ChatGPT there is still a human there, a human that is trying to solve a problem or learn something. Maybe I'm naive and it's just all bots scraping sites to build their corpus of data, but at least the way I use AI is to solve problems and build things I couldn't do before (either because of time, or it was out of my reach)
Most of what the author is doing won't actually help.
> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.
Comments
https://www.rebootonline.com/geo/llms-txt-experiment/
> LLMs ignore Markdown files — log analysis showed no evidence that GPTBot, ClaudeBot, or similar crawlers request or prioritise .md content, even when listed in llms.txt.
https://www.longato.ch/llm-md-files/
> Across all 50 sites, across 200+ days, across every major AI user-agent logged - the `/llms.txt` endpoint was requested zero times.
https://mishamanko.com/research/what-ai-bots-actually-read