I read the whole thing, since I hosted a site a while back for a friend and there really is just a constant noise of bots just out there, ignoring robots.txt, being awful and stupid.
I think the most depressing part is who this is… it’s just fucking there, cloneable, but AI chuds just can’t imagine something being that accessable.
Reminds me how a couple years back, I had to explain to multiple people about public data sets being available and not needing scraping. I laughed then when they wanted to scrape the Dept of Labor for data tables when it’s all literally just pulled from tab delimited files that’re just… there to download. Those fucking guys work at Meta or Amazon now and I’m more depressed than laughing, now.
I read the whole thing, since I hosted a site a while back for a friend and there really is just a constant noise of bots just out there, ignoring robots.txt, being awful and stupid.
I think the most depressing part is who this is… it’s just fucking there, cloneable, but AI chuds just can’t imagine something being that accessable.
Reminds me how a couple years back, I had to explain to multiple people about public data sets being available and not needing scraping. I laughed then when they wanted to scrape the Dept of Labor for data tables when it’s all literally just pulled from tab delimited files that’re just… there to download. Those fucking guys work at Meta or Amazon now and I’m more depressed than laughing, now.