The internet sucks111!!!

biyearly post to show i’m still alive.

While going to begin work on an old project, I found that my git server was completely unresponsive. After tail’ing my http server’s logs, I found that I was being endlessly crawled on my git server, into non existent pages.

Problems with GitEA

I believe that GitEA doesn’t serve a proper 404 when a page is not found, and because of this, web crawlers were just endlessly looping through their list of pages to check.

Normally, this is not really an issue as most crawlers will check for the typical exploit paths. Such as /admin, /dashboard, or some PHP related known exploits. However, with the advent of AI summarizers and whatnot, I was getting pings such as GET /mental-health-tools7713/mental-health- assessment-tips7819/issues?assignee=23139&project=-1&sort=nearduedate&state=open&type=all.

These projects and repositories don’t exist anywhere on my site, but I was getting thousands of requests similar in vein to these. My server isn’t quite high powered, it’s a relatively cheap machine that I purchased secondhand. Because of this it was sending the server into a complete slodge and making me unable to access any of repositories

The fix?

I setup a simple robots.txt file to get the ethical crawlers off of my back. Things seemed to clear up after a restart and the new file, so I figured all was well.

…or not?

Just kidding! AI crawlers don’t give a single fuck about a robots.txt. They’ll do whatever the fuck they want. They just want my data. I continued to essentially be DDoSed by these crawlers. Seemingly worse than before.

So I took it a step further, and had to implement an HTTP basic auth system just to be able to use my own private git server.

Is this really what the world has come to? Are we trying to make the internet absolutely unusable. What does this do for me? What does this provide for the average person?

The infinite economic growth model claimed another victim, and it was my energy bill.

I just want to exist in peace in my own private world upon this tiny website. I just want to share my guides and my stupid opionions in peace with the 2 other readers of my content. Why do AI companies and other fortune 500s want to commoditize every little piece of the world until nothing is left to enjoy. Leave me alone!!!