Scrapers

Last week AI scrapers got so bad on my tiny server that I couldn’t access any of my git repos over https. I had to disable some features to calm down the CPU (syntax highlighting was worst offender) and it improved the situation, but the root cause is still there.

Last 34 days I got over 14.5 million of valid requests. This is nearly 5 requests per second, but it excludes failed ones (HTTP 404) which I consider cheap. My dashboard reports over 76 million total requests, but frankly I don’t know where this number comes from, so I consider it exaggerated. Most traffic comes from France and Spain now, so I guess it’s Mistral training?

I don’t want to put Anubis in front of git server, because I’m afraid of breaking the ordinary http clones and this server hosts Free Software, which I have obligation to share. (sidenote: Besides obligations, it’s just nice and ethical.) I don’t have obligation to render its source code though, and the whole git history, so I’m thinking about replacing cgit with a list of download links. Less nuclear (and more enjoyable) option would be writing a minimal generator for git repos which would statically render README.md, list of branches, log, file tree (without contents) etc.

It used to be fun to host personal website and git server. Now I have to deal with scale problems even though the only consumers are bots.