Been stewing at it for longer than you have, buddy.

You’re not correcting anything, dude.

The entire point here is that the bots are being disrespectful.

deleted by creator

They’re not trying to do that in particular, but on a site like lemmy the same content might be linked a dozen different ways:

https://lemmy.world/comment/25863752 https://lemmy.world/post/51995780/25862545 https://lemmy.world/post/51995780/25860801 https://lemmy.world/post/51995780/25858204 https://lemmy.world/post/51995780/25857153 https://lemmy.world/post/51995780/25857033

It’s not necessarily the big companies themselves doing it? But, there’s a push right now to scrape as much data as possible to feed the training of the AI models, and if your business model is entirely based on selling that data you don’t really care who you’re going to piss off. So, in turn these new scraper bots basically behave in the same way that attack bots do where they’ll be behind VPNs and switch IPs and stuff if you block them.

Ordinary search crawlers are throttled and they do stuff like respect your robots.txt and sitemap.xml (directives that tell the crawler what links to scan or not and how frequently). These naive crawlers hit everything at max speed, and on a content-heavy site like lemmy that can easily take out a server.

Well, had one go in an almost brand new to me car over a set of tracks in the dead of winter as a naive kid and didn’t figure it out for quite a while. 😛

Even went autocrossing with it like that and couldn’t figure out why the hell I was sliding into every corner. Was probably losing about 3 seconds over each run it was so bad.

They’re simply not censoring it. The BBC already got thrashed for this bullshit.