Only tangentially related, but I've recently realized that unless you're some sort of business with a valid requirement of that, you can just block incoming traffic from all of AWS, DigitalOcean, Azure, etc.
I think the spark came from some other HN comment I saw? So thank you for that, whoever wrote that.
That alone is not a sufficient strategy of course, but you can get rid of a lot of bots by just refusing to talk to where they tend to live.
Of course, if everyone were to do that, they'd eventually pivot to residential proxies. But I don't see that happening.
tommica · 2026-10-27 10:52:54 UTC
true! but how does one find all those ips?
hypfer · 2026-10-27 10:53:27 UTC
Just ask RIPE
Xylakant · 2026-10-27 10:54:05 UTC
That’s pretty easy - all major providers have public lists of IP ranges, usually as json files.
gunalx · 2026-10-27 11:03:42 UTC
Allmost all the bottraffic I see is from random residential proxies though.
timpera · 2026-10-27 11:10:24 UTC
Same, I just checked and this week the vast majority of the bot traffic I get is random residential IPs that all pretend to be the exact same Chrome 155 on Linux.
sgerenser · 2026-10-27 11:17:54 UTC
Probably all people that fell for the “superbox” TV streamer scam.
That’s probably a decent starting point, but a little older. I am not sure if the situation has changed any in 2026.
nekusar · 2026-10-27 12:50:45 UTC
Yeah it's definitely sketchy, but a "media access box" that purports to replace $100 a month in streaming bills with pirate media is kinda awesome.
Of course, the countries the Jellyfin and Plex servers along with IPTV dealers aren't illegal there.
And sure, your device becomes a residential proxy. They don't make that fact clear... But if you told people, I think for that $100-$200 a month streaming bills go poof is probably worth it.
BizarroLand · 2026-10-27 16:07:51 UTC
Meh. It's awesome until the links stop working and the app goes dead because of regulators catching on.
I remember there being a plague of kodi plugins that worked for that kind of stuff for a long time, and they would die and be replaced every couple of weeks, sometimes they would have malware in them, it was a whole thing.
I think that if you want streaming and want to save money, the best practice is to only have one streaming service at a time. When you get bored with it, shut it down or pause it and move to the next one. It's a pain, but at least it won't infect every device on your home network and contribute to a bot farm.
sgerenser · 2026-10-27 15:31:25 UTC
I saw these things being sold for $300 at my local State Fair just a week ago. First heard about them a few months back on the Darknet Diaries podcast: https://darknetdiaries.com/episode/172/
severin_letters · 2026-10-27 15:41:52 UTC
Same shape here, and it is easy to miss because they impersonate a browser well enough that the request itself looks human. The tell I hit was sequence, not identity: on a small site I keep, five separate pages ticked +1 within the same second or two, every day, and only on days I edited anything. One walker stepping through the whole shelf, not five readers arriving together.
Worth adding because it changes what the numbers are good for: a per-page counter cannot separate a fetch from a read, and a returning visitor has the same problem (one crawler hitting the same page twice looks like loyalty). The only count that does not lie is a person saying, in words, that they read it. Everything else is a probability estimate shaped like a fact. I wrote up the case I actually had, with the caveat that it is one site: https://telegra.ph/A-count-proves-a-fetch-not-a-read-10-27
cube00 · 2026-10-27 11:04:40 UTC
I initially thought this was a good idea too until another comment pointed out you'll lose most business or personal customers in offices because they connect through VPN gateways running on those cloud services (especially Azure)
hypfer · 2026-10-27 11:12:26 UTC
True but the question is if that matters.
I would argue that more often than not it might not. Really depends on the specifics though.
A lot of stuff on the web actually isn't really relevant for all audiences everywhere I'd say. It just happened to be accessible by all, because academics back in the day dreamt of a better open world that didn't really manifest :(
semiquaver · 2026-10-27 11:22:16 UTC
This is a website about starting businesses.
Let me tell you, at the business I work at, some of our customers work at a company that uses a commercial VDI service for all workstations couldn’t access our service because the VDI egress was in GCP and we had an AWS WAF rule banning datacenter IPs.
Obviously, we removed the rule because it was doing nothing useful but plenty of harm. Real bots come from residential IPs.
hypfer · 2026-10-27 11:23:34 UTC
Hm I mean HN _also_ is that?
But not exclusively.
And even if we run with that assumption, one might still be fine. It's a trade-off, but, depending on the target audience, a few lost sales might be worth savings in other aspects.
anonreplier · 2026-10-27 11:34:15 UTC
Can use cloudflare turnstile or similar instead
bigfishrunning · 2026-10-27 11:36:46 UTC
> This is a website about starting businesses.
It's really not, unless the only qualifier for that is "hosted by a VC incubator". Most of the people here are here to argue about AI or Crypto or whatever, or read announcements from Nvidia and Google. I would bet less then 1% are starting businesses.
wtch98 · 2026-10-27 11:37:41 UTC
If you control the connecting device why not use MTLS?
semiquaver · 2026-10-27 12:53:09 UTC
Who said we control the connecting device? This is a website. Websites are usually pretty arms-length things, and that’s the case with us. They’re our customers but we can’t install certificates on their computers and enforce mTLS. Can you do that with any of your customers?
gverrilla · 2026-10-27 11:42:04 UTC
> day dreamt of a better open world that didn't really manifest :(
it is open and better indeed. open and better for capital - the true, living, flesh-eating god.
wtch98 · 2026-10-27 11:36:43 UTC
All my business traffic comes from either our own AS, or from ZScaler's AS
Do many companies really use azure etc rather than dedicated services like zscaler?
jack_pp · 2026-10-27 12:40:27 UTC
i can't access reddit through nordvpn
trollbridge · 2026-10-27 12:50:44 UTC
Someone accessing a VPN that's in an Azure IP range is going to have a difficult time using half of the Internet.
bearjaws · 2026-10-27 13:43:24 UTC
I would sure hope they split tunnel their VPN so all traffic isn't going through the VPN...
cube00 · 2026-10-27 14:32:55 UTC
Some companies don't split because they want all internet traffic to go through their internal MITM proxies to enforce acceptable internet use and data loss policies.
80fj0348i · 2026-10-27 17:19:48 UTC
Those had to switch over to endpoint agents installed on work devices directly due to TLS 1.3 preventing MITM for domains which you do not control.
jeremyjh · 2026-10-27 11:10:56 UTC
There have been a number of articles on the frontpage of HN about the residential proxy traffic. It is happening. This was one of the most popular:
I know, but it's an extra step, more expensive, and less legally solid.
None of these measures alone are a full solution, but if you stack a bunch of cheese slices, eventually, no holes line up anymore.
trollbridge · 2026-10-27 12:45:49 UTC
... less legally solid? What?
hypfer · 2026-10-27 12:51:10 UTC
Residential proxies are something that is not supposed to exist.
In the sense that you do not have people that actively sell those services via their home internet connection.
The whole thing just exists because there's this whole predatory ecosystem that uses malware and malicious devices to sell that without the owners even knowing.
That makes it legally questionable, even though I guess technically it is legal enough?
Still radioactive, so as a serious business, you do not want to be seen doing that.
nekusar · 2026-10-27 13:13:20 UTC
They exist because of GeoIP databases, and spying on "quality of IP address via ASN".
If you try to use a normal VPN, you get rate limited or blocked everywhere. But if you switch to a residential proxy, all those problems go away.
To be fair, if all you're doing is torrenting or similar copyright infringing actions, then a simple VPN is sufficient. But if you want to do web stuff, you gotta use a residential vpn. Not much way around this.
hypfer · 2026-10-27 13:16:22 UTC
That was not the question.
There are many factors driving crime, yes.
The point however is that this is crime and crime is incorrect and fundamentally shouldn't exist.
But, more importantly, as a serious business, you do not want to be seen doing crime or benefitting from crime or supporting crime or being near crime. At least in a working society that is.
nekusar · 2026-10-27 14:30:54 UTC
First, copyright violation as non-profit is a tort.
Secondly, it's only for-profit copyright violations that are a crime. But as OFCOM learned the hard way, stuff illegal in the US can be legal elsewhere.
Thirdly, for some fucking reason, the US AI companies are immune to criminal statutes of for-profit copyright violations. Job creators, or some bullshit.
And I'm not sure exactly why you're so dead-set on "crime crime crime". It's all politics and manufactured criminality. And well, I don't particularly care if backing up my movies to a Jellyfin and sharing access is "ill eagle".
The AI companies are bulk pirating AND then selling that access.... But I'm the criminal? Fuck that noise.
hypfer · 2026-10-27 14:41:01 UTC
This whole discussion was not about you or what you do, but about the idea of blocking Datacenter ASNs against scraping and other bot activity.
Which then was countered with a broad range of classic HN dysfunction, hostility and pedantry.
When the truth of the matter is that it's actually just a decent idea for some use-cases and a less-decent idea for others.
But instead of acknowledging that nuance, an explosion of nonsense happened here.
I genuinely do not understand why you've suddenly started talking about yourself and your own residential proxy use? Why does that belong here?
itintheory · 2026-10-27 13:27:45 UTC
Sure, that's why they sanitize the User-Ggent header when using these proxies. Good luck attributing the scraping to anyone at all.
hypfer · 2026-10-27 13:29:51 UTC
It just needs one case where someone manages to attribute it to the corp using these illicit services.
Either by technical means or by a whistleblower.
Which might be a gamble worth doing, but also might not. And regardless, the fact that it is a gamble alone slows down velocity.
geraldwhen · 2026-10-27 13:42:30 UTC
If the content is legally obtainable, couldn’t any employee use their own home network to pull data? I don’t think anyone would bother with a resi. Employees have their own network connections available.
hypfer · 2026-10-27 13:46:33 UTC
Can you people please for the love of God stop missing the bigger picture?
Yes, yes there is always something. Yes there can always be workarounds.
But that is not the point. The point is that things are made harder. That is the point. To make things harder
Not impossible. Harder
Jesus Christ. It's all just a game of probabilities and friction. One needs to understand that to understand reality.
ecshafer · 2026-10-27 13:52:31 UTC
The scrapers and AI companies resorting to residential proxy traffic like intelligence and hackers is pretty crazy.
stopsayingsorry · 2026-10-27 11:12:38 UTC
>But I don't see that happening.
Then you are not looking. Already have to switch to resi proxies from time to time when encountering yet another smartass who decided to play security theater and block datacenter ips on his shitpile he calls website, because "OUR USERS MUST BE GOOD BOYS WHO LOVE TO LIVE UNDER SURVILANCE AND ALLOW ISP TO TRACK THEM 24/7" (and before you say something like "others can track you too!", your ISP has the ability to knock on your door, they should be given as little information as possible, let them do metadata analysis on per-person basis if they really want it, but no 1-click mass surveillance shit).
If you are valuable, you will be be scraped. Turning user hostile in a hopeless endeavor to "protect yourself" is not the solution.
hypfer · 2026-10-27 11:16:49 UTC
I can see that you're having strong emotions about this, but that rant is just hard to read and serves primarily as venting and not really to convince anyone of anything.
Unless one zooms out and thinks "okay, these guys seem to react negatively if we do exactly that. We should do more of it", which is probably not what you want.
sampullman · 2026-10-27 11:17:11 UTC
It's hostile to you personally, but I wouldn't call it "user hostile".
Blocking "scrapping [sic]" is also not security theater, you misunderstand the intent of GP.
stopsayingsorry · 2026-10-27 11:31:29 UTC
Anyone who thinks blocking datacenter ips helps with blocking scraping is a clown.
anonreplier · 2026-10-27 11:35:37 UTC
exactly, there are lots of other more effective ways to block the likes of you
stopsayingsorry · 2026-10-27 11:51:00 UTC
Let's hope you enjoy killing your real userbase with collateral damage, because that's the only thing you are going to accomplish.
sampullman · 2026-10-27 11:59:20 UTC
It does, measurably. A massively popular site is going to get hammered with more advanced types of scraping, but they're going to have a team that can more easily mitigate it.
A site that doesn't care about corporate users basically loses nothing.
wtch98 · 2026-10-27 11:42:09 UTC
I trust my ISP far more than I trust bigtech. The downside of connecting from my ISP is that the site I connect to can fingerprint me trivially from the IP range
bruce343434 · 2026-10-27 11:56:36 UTC
Won't this harm SEO? What about agentic software suites where authorized users should be able to drop an app passkey into grokbot or such?
cj · 2026-10-27 13:09:59 UTC
"Yes, but who cares" is the sentiment.
trollbridge · 2026-10-27 12:44:37 UTC
I currently tarpit traffic from AWS, Azure, GCP, etc. since it's not like any of my real customers have their PC inside those networks. (I then use that for experiments with providing bogus data to AI scrapers.)
Doesn't do much to help with the "VPN that uses random consumer PCs and phones" traffic.
xmcp123 · 2026-10-27 14:02:05 UTC
How do you tarpit GCP if Google’s search engine scrapers scrape from their own cloud?
xmcp123 · 2026-10-27 14:00:11 UTC
If you block datacenter IP you’re also going to block all of the legitimate search engine crawlers.
DaSHacka · 2026-10-27 14:55:45 UTC
Depending on the service, you may not want it to be indexed anyway (IE login pages) or, if you do, most commercial crawlers have methods to allow site owners to verify that the traffic is a legitimate crawler and not an impersonator, see:
maybe stupid question but how is it possible to not be protected against supercrawlers like this, when you own large amounts of data, specially if all requests are coming from the same server ? Aren't cyberprotection services like CloudFlare work against this kind of "attacks" ?
Steve16384 · 2026-10-27 11:09:17 UTC
Yeah, I can imagine it being a problem if it was from different IP addresses, but surely putting a limit on requests from a single IP would be a straightforward fix? Although maybe until it happens, you don't think about it.
hmokiguess · 2026-10-27 12:18:57 UTC
"supercrawlers like this"
then in the post "It came from a single Amazon Web Services (AWS) address. It had read the main page of 18,926 projects, which is practically our entire catalogue, in 26,429 requests over a span of 34 hours."
sounds like some R&D dude running solo to be honest, nothing about that is a supercrawler lol
itintheory · 2026-10-27 13:49:13 UTC
> cyberprotection services
These services cost money. Scrapers also cost money in terms of resource utilization and downtime. There's always a tradeoff.
The ROI of services like Cloudflare bot protection, Datadome, HUMAN, AWS bot mitigation etc. is highly dependent on your income from the data. In the case of publicly hosted freely accessible data it can be hard to justify.
caaqil · 2026-10-27 10:55:54 UTC
Who actually cares? Isn't the whole premise of the AI revolution just gorging ourselves with other people's data so we can have AGI? I was under the impression that we have accepted this sacrifice with OAI and friends leading the way.
tempfile · 2026-10-27 11:00:38 UTC
"we" in this sentence is really a handful of encloistered silicon valley weirdos. Most people do not like it and have no real say in the matter.
Steve16384 · 2026-10-27 11:09:50 UTC
Who's "we"?
bulbar · 2026-10-27 11:22:51 UTC
People really still buy into that framing? I think it's pretty clear that it's generally just about money as it has always been.
MSFT_Edging · 2026-10-27 11:58:08 UTC
Tech is full of rubes that eat up the lies.
Crypto, big data, AI. It's all just another middle man hoping to scrape some joy out of everyone's lives to they can put a price tag on it and sell it to rubes and the government(and rubes in the government).
Artoooooor · 2026-10-27 11:58:00 UTC
If the authors of the data are paid...ok. and not just the biggest players - every single one. But they are not.
phoghed · 2026-10-27 12:20:15 UTC
Sam, pay this guy for his shitposts please
KolmogorovComp · 2026-10-27 11:01:40 UTC
I'm sympatic to open3dlab but I think the focus on adult-content is just trying to spark outcry for cheap.
It has been shown times and times against that removing adult content from the training source (at least for diffusion models), results in generally lower-quality output, even outside of adult-content, particularly in terms of representing human bodies.
darthoctopus · 2026-10-27 11:21:08 UTC
would the implied intent (from the inclusion of adult content) of training models on open3dlab content, without asking them, render Roblox's actions _less_ antisocial???
Betelbuddy · 2026-10-27 11:27:01 UTC
The current AI Ethos is morality is just another obstacle on the way to their IPO. ...Lets just hope after they get rich, they stay away, and dont bother us with books, videos and conference speeches on entrepreneurship mentoring...
darthoctopus · 2026-10-27 12:25:40 UTC
hi jared
criddell · 2026-10-27 12:08:47 UTC
They should change their name to openish3dlab.
Viliam1234 · 2026-10-27 18:06:02 UTC
Or specify that it's "open" as in "OpenAI" (i.e. not at all), not as in "open source".
ChrisRR · 2026-10-27 12:34:10 UTC
Not really. Given that roblox is almost entirely children, and given the amount of hot water they're in you'd think they'd stay the hell away from adult content
trollbridge · 2026-10-27 12:58:44 UTC
You'd think! But they don't. Roblox is also notorious for tolerating a ridiculous amount of grooming.
kotaKat · 2026-10-27 13:02:09 UTC
Discord knows I'm an adult and keeps trying to send me "promoted Quests" to go use Roblox. That seems sketchy to keep baiting adults into childrens' spaces, too.
SoftTalker · 2026-10-27 14:30:17 UTC
The number of young adults (18-30) who still play children's video games has always been astonishing to me. We live in a Peter Pan world these days.
flexagoon · 2026-10-27 15:33:32 UTC
An adult who plays Roblox can spend much more money on it than a kid getting some Robux for Christmas
swiftcoder · 2026-10-27 11:02:31 UTC
I wonder at what point the Roblox kids all reach 18 years old, and Roblox can pivot them all to a separate adult version thereof...
phoghed · 2026-10-27 12:09:29 UTC
Never, it gets new players constantly. This is like saying I wonder when YouTube is going to turn into Pornhub
swiftcoder · 2026-10-27 13:49:51 UTC
> This is like saying I wonder when YouTube is going to turn into Pornhub
To a certain extent, it already has. YouTube for Kids and regular YouTube are worlds apart.
lifeisloving · 2026-10-27 12:36:52 UTC
Robloxs original player base is their 20s now and still play.
paxy · 2026-10-27 12:57:25 UTC
You know there are new kids being made right..
embedding-shape · 2026-10-27 13:08:43 UTC
New kids often don't want to be in the same space as where the "new adults" are though, that's uncool. See: literally any social media site since forever.
peridotintellig · 2026-10-27 11:06:12 UTC
I'm not even impressed..
bijowo1676 · 2026-10-27 11:08:07 UTC
if you put stuff available to download, dont be surprised that someone will download it
What is the actual news here, can anyone explain ????
well_ackshually · 2026-10-27 11:32:10 UTC
Man if only there was such a thing as good manners, good behaviour, accepted practices from a community to not massively hammer servers. If only these websites had been up for a decade with very few problems, before a notoriously manipulative, lying, deceiving, abusive company that makes money off of child abusers came in and just went hurr durr free data in the WORST possible way instead of just politely asking for a dump.
bijowo1676 · 2026-10-27 17:16:08 UTC
Internet is untrusted network, there are a lot of malicious and gray actors. There is no community here.
it is in fact engineering incompetence and loose engineering, if you cant handle a burst of traffic
numpad0 · 2026-10-27 11:37:00 UTC
"Leopard ate MY face" ?
offbeatport · 2026-10-27 11:41:13 UTC
They didn't do it in the intended manner.
> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.
Sopel · 2026-10-27 12:39:20 UTC
> did not respect our robots.txt
I don't see a problem
> ignored our Terms of Service
ban them from the service then and case closed?
> and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only
because it's been proven they have been used commercially?
Comments
I think the spark came from some other HN comment I saw? So thank you for that, whoever wrote that.
That alone is not a sufficient strategy of course, but you can get rid of a lot of bots by just refusing to talk to where they tend to live.
Of course, if everyone were to do that, they'd eventually pivot to residential proxies. But I don't see that happening.
That’s probably a decent starting point, but a little older. I am not sure if the situation has changed any in 2026.
Of course, the countries the Jellyfin and Plex servers along with IPTV dealers aren't illegal there.
And sure, your device becomes a residential proxy. They don't make that fact clear... But if you told people, I think for that $100-$200 a month streaming bills go poof is probably worth it.
I remember there being a plague of kodi plugins that worked for that kind of stuff for a long time, and they would die and be replaced every couple of weeks, sometimes they would have malware in them, it was a whole thing.
I think that if you want streaming and want to save money, the best practice is to only have one streaming service at a time. When you get bored with it, shut it down or pause it and move to the next one. It's a pain, but at least it won't infect every device on your home network and contribute to a bot farm.
Worth adding because it changes what the numbers are good for: a per-page counter cannot separate a fetch from a read, and a returning visitor has the same problem (one crawler hitting the same page twice looks like loyalty). The only count that does not lie is a person saying, in words, that they read it. Everything else is a probability estimate shaped like a fact. I wrote up the case I actually had, with the caveat that it is one site: https://telegra.ph/A-count-proves-a-fetch-not-a-read-10-27
I would argue that more often than not it might not. Really depends on the specifics though.
A lot of stuff on the web actually isn't really relevant for all audiences everywhere I'd say. It just happened to be accessible by all, because academics back in the day dreamt of a better open world that didn't really manifest :(
Let me tell you, at the business I work at, some of our customers work at a company that uses a commercial VDI service for all workstations couldn’t access our service because the VDI egress was in GCP and we had an AWS WAF rule banning datacenter IPs.
Obviously, we removed the rule because it was doing nothing useful but plenty of harm. Real bots come from residential IPs.
And even if we run with that assumption, one might still be fine. It's a trade-off, but, depending on the target audience, a few lost sales might be worth savings in other aspects.
It's really not, unless the only qualifier for that is "hosted by a VC incubator". Most of the people here are here to argue about AI or Crypto or whatever, or read announcements from Nvidia and Google. I would bet less then 1% are starting businesses.
it is open and better indeed. open and better for capital - the true, living, flesh-eating god.
Do many companies really use azure etc rather than dedicated services like zscaler?
https://news.ycombinator.com/item?id=49491791
None of these measures alone are a full solution, but if you stack a bunch of cheese slices, eventually, no holes line up anymore.
In the sense that you do not have people that actively sell those services via their home internet connection.
The whole thing just exists because there's this whole predatory ecosystem that uses malware and malicious devices to sell that without the owners even knowing.
That makes it legally questionable, even though I guess technically it is legal enough? Still radioactive, so as a serious business, you do not want to be seen doing that.
If you try to use a normal VPN, you get rate limited or blocked everywhere. But if you switch to a residential proxy, all those problems go away.
To be fair, if all you're doing is torrenting or similar copyright infringing actions, then a simple VPN is sufficient. But if you want to do web stuff, you gotta use a residential vpn. Not much way around this.
The point however is that this is crime and crime is incorrect and fundamentally shouldn't exist.
But, more importantly, as a serious business, you do not want to be seen doing crime or benefitting from crime or supporting crime or being near crime. At least in a working society that is.
Secondly, it's only for-profit copyright violations that are a crime. But as OFCOM learned the hard way, stuff illegal in the US can be legal elsewhere.
Thirdly, for some fucking reason, the US AI companies are immune to criminal statutes of for-profit copyright violations. Job creators, or some bullshit.
And I'm not sure exactly why you're so dead-set on "crime crime crime". It's all politics and manufactured criminality. And well, I don't particularly care if backing up my movies to a Jellyfin and sharing access is "ill eagle".
The AI companies are bulk pirating AND then selling that access.... But I'm the criminal? Fuck that noise.
Which then was countered with a broad range of classic HN dysfunction, hostility and pedantry.
When the truth of the matter is that it's actually just a decent idea for some use-cases and a less-decent idea for others.
But instead of acknowledging that nuance, an explosion of nonsense happened here.
I genuinely do not understand why you've suddenly started talking about yourself and your own residential proxy use? Why does that belong here?
Either by technical means or by a whistleblower.
Which might be a gamble worth doing, but also might not. And regardless, the fact that it is a gamble alone slows down velocity.
Yes, yes there is always something. Yes there can always be workarounds.
But that is not the point. The point is that things are made harder. That is the point. To make things harder
Not impossible. Harder
Jesus Christ. It's all just a game of probabilities and friction. One needs to understand that to understand reality.
Then you are not looking. Already have to switch to resi proxies from time to time when encountering yet another smartass who decided to play security theater and block datacenter ips on his shitpile he calls website, because "OUR USERS MUST BE GOOD BOYS WHO LOVE TO LIVE UNDER SURVILANCE AND ALLOW ISP TO TRACK THEM 24/7" (and before you say something like "others can track you too!", your ISP has the ability to knock on your door, they should be given as little information as possible, let them do metadata analysis on per-person basis if they really want it, but no 1-click mass surveillance shit).
If you are valuable, you will be be scraped. Turning user hostile in a hopeless endeavor to "protect yourself" is not the solution.
Unless one zooms out and thinks "okay, these guys seem to react negatively if we do exactly that. We should do more of it", which is probably not what you want.
Blocking "scrapping [sic]" is also not security theater, you misunderstand the intent of GP.
A site that doesn't care about corporate users basically loses nothing.
Doesn't do much to help with the "VPN that uses random consumer PCs and phones" traffic.
https://developers.google.com/crawling/docs/crawlers-fetcher...
(Though restricting the Google UA to only legitimate Google IPs seemingly would break Brave's crawler, though that's their own fault: https://safe.search.brave.com/help/brave-search-crawler/)
https://duckduckgo.com/duckduckgo-help-pages/results/duckduc...
https://www.bing.com/webmasters/help/how-to-verify-bingbot-3...
https://kagi.com/bot
then in the post "It came from a single Amazon Web Services (AWS) address. It had read the main page of 18,926 projects, which is practically our entire catalogue, in 26,429 requests over a span of 34 hours."
sounds like some R&D dude running solo to be honest, nothing about that is a supercrawler lol
These services cost money. Scrapers also cost money in terms of resource utilization and downtime. There's always a tradeoff.
The ROI of services like Cloudflare bot protection, Datadome, HUMAN, AWS bot mitigation etc. is highly dependent on your income from the data. In the case of publicly hosted freely accessible data it can be hard to justify.
Crypto, big data, AI. It's all just another middle man hoping to scrape some joy out of everyone's lives to they can put a price tag on it and sell it to rubes and the government(and rubes in the government).
It has been shown times and times against that removing adult content from the training source (at least for diffusion models), results in generally lower-quality output, even outside of adult-content, particularly in terms of representing human bodies.
To a certain extent, it already has. YouTube for Kids and regular YouTube are worlds apart.
What is the actual news here, can anyone explain ????
it is in fact engineering incompetence and loose engineering, if you cant handle a burst of traffic
> The crawler did not respect our robots.txt, (which it did retrieve), ignored our Terms of Service, (which forbid scraping), and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only.
I don't see a problem
> ignored our Terms of Service
ban them from the service then and case closed?
> and defied the creators' licenses: 423 of the 712 projects are licensed for non-commercial use only
because it's been proven they have been used commercially?