You Don’t Need Anubis
you don’t need anubisIn the past years, scrapers operated by LLM training companies have become more relentless. They no longer respect robots.txt, spoof their User Agents and IP addresses and even DDoS small sites with aggessive requests.This seems to mostly apply to Anthropic’s ClaudeBot. OpenAI’s GPTBot is pretty well documented and easy to block.This has led to more and more websites using Anubis, a proof-of-work based bot protection solution that requires all visitors to solve a small cryptographic problem on their device before proceeding.But here’s the thing: Anubis doesn’t work. Well, alright, it does – it can be a good DDoS protection solution, especially for people who don’t want to use Cloudflare. But it seems like most users of Anubis don’t need DDoS protection -…
