Really? They think Internet Archive is the problem?
They think AI companies are using it as a “backdoor” to scrape their content. Which is patently ridiculous, but that won’t stop them.
reddit already allows AI to scrape the site, specifically google.
reddit already allows AI to scrape the site
*for a fee
If the wayback machine was an actually workable backdoor to Reddit’s content, none of these companies would ever have any reason to pay them, and that’s what they’re scared about.
They think they want their revenue streams
deleted by creator
its thier excuse to silence dissent, when the time comes to censor overreaching “political trends”
Wait until they find out that AI is scraping their web sites.
Well, Reddit’s got a contract for AI companies to scrape their content, so pig boy Spez is getting paid, he don’t give a fuck
I’m sure they don’t care, or are all about it.
deleted by creator
reddit trying to achieve what FB is doing, meta already has complete control of FB what they push as propaganda.
Ideally, we would have our own personally run AI instances that can give us a probability that what we are reading is LLM generated. It’s still pretty good at recognising itself. That will be an arms race, though.
Content scraping is harming the information business in ways that could not have been foreseen.
What an absolute ridiculous thing to say.
deleted by creator
Gee, I wonder what else could be scraping content from all these websites?
To be fair, the archive indeed got heavily abused into simply reading without paywalls. I know this is a controversial opinion, but seeing comments on other threads like “Remember to support news media”, then “use archive to bypass paywalls” then anger towards said companies for caring about getting paid or growing, makes one question where exactly does Lemmy draw the line between pirating and paid content. Or are we simply altogether against sites like 404Media just because of paywalls?
That’s not the point. The point is content scraping (and crawling) is the cornerstone of the contemporary information environment. It’s how we got to this technological paradigm in the first place.
This whole “people are bypassing paywalls” is a badly evidenced non-issue, and all too convenient. What these companies are really saying is “Content scraping is bad when others do it. Only I and other big fish get to do it and profit billions out of it. Fuck ordinary citizens. Fuck everyone and everything but me and my dreams of endless wealth and power.”
To be fair.
You say bypassing paywalls is a non-issue, but it is basically the only thing I have heard people say to use it for on social media. You can have your problems about data harvesting, but don’t pretend like getting around paywalls was not what the average individual user was using it for.
I’m certain they’ve wanted to do this for a long time, and AI is a convenient way to justify it, rather than admitting they don’t want humans using it to circumvent the paywall. It does solidify for me personally that the LA Times is the paper of record for the United States going forward, rather than the New York Times.
The LA Times also blocks the Internet Archive unfortunately. I’d recommend PBS NPR ProPublica or some other nonprofit organization for your US paper of record.
Ugh. Thanks for the heads’ up — I’ve definitely posted archive links without noticing they’re blocked before. PBS and NPR have really gone downhill with the budget cuts. ProPublica is great, but their coverage is pretty narrow, so there’s a lot of stories they don’t cover at all. It’s getting harder and harder to find a quality source.
Unfortunately, I think most quality sources with broad coverage aren’t free. Even the paid sources almost always have a corporate bias. Of those the financial times probably does the least to editorialize. Beyond that I think you just have to find independent journalists or outlets with a narrower investigative focus that you can trust.
I just got a gift subscription to the NYTimes, for the first time since I quit in 2018, and it’s really gone downhill. I am learning about more big scoops from the guardian from lemmy posts than I see in their paper. I think Israel’s final solution for gaza here broke their brain, they had an identity crisis and sided with Israel and fascism over all the fourth estate democracy mumbo jumbo.
They haven’t broken a single big story that I recall in the past year. Not a single one, even the wall street journal published epstein’s birthday letter from the president. The NYTimes gave up, they are no longer the paper of record, whatever problems before they covered events more thoroughly and had courage to break big stories, and now they don’t.
That’s actually pretty sad. Also a serious problem for the USA. NYT, for all its faults, really was the best one.
The South African billionaire paper that wouldn’t endorse Harris? Well, our options all suck, I guess.
Reminder to donate to the Internet Archive so they can keep fighting the good fight.
Nobody tell NYT about being able to add another “.” Subsequent to”.com” to bypass their paywall.
Awesome, this is the best paywall hack I have ever seen!
I’m probably screwing it up here, but neither of these are working for me
https://www.nytimes.com.2026/02/04/us/politics/supreme-court-california-congressional-map.html
https://www.nytimes…com/2026/02/04/us/politics/supreme-court-california-congressional-map.html
Put the extra “.” after the “.com” so “.com.”
Ah, https://www.nytimes.com/2026/02/04/us/politics/supreme-court-california-congressional-map.html won’t work on my usual browser (which just ends up loading NYTs homepage) but it does work in a Chrome incognito window
Thank you!
I think auto complete or something might have messed with what you intended to post, that link still hits the paywall for me, but using your guidance I was eventually able to figure out that
nytimes.com./2026 etc.
works in a Chrome incognito window. The “.” after “com” and the “/” after that “.” are apparently the critical bits
https://www.nytimes.com/2026/02/04/us/politics/supreme-court-california-congressional-map.html
You are missing the .com. part…
oof I brainfarted - markdown and code auto formats and in doing so autocorrects.
It happens, just wanted to point it out in case people trying it thought it did not actually work
Still getting this bullshit:
beep boop
Why does this work? Is it a deliberate bypass on the NYT’s part?
No idea, but I love it.
Hmmm. Interesting.
Fuck Reddit. That website has been selling our data and using it to train AI… I say fuck 'em
deleted by creator
great catch! you can actually see the AI slop when it pops up. REddit is dead, and you should delete your data from that cesspool
deleted by creator
FYI, any data on Lemmy can be used for the same for free. The federation infra can even be used to give AI models more direct access than even reddit is likely giving them. Just in case anyone is assuming that because this is community-run that it means the data isn’t being sold. It’s not, but it is being accessed by the same entities, if they want it.
The users are still the users though, not the product like Reddit.
Oh yeah, not saying they are generally equivalent, just in that one particular aspect: access to comment data for any purpose.
Yep.
TBH I think it’s kind of silly for the Fediverse to try and block scraping, as long as that scraping isn’t effectively a DDoS. It’s public.
Gotta control the press before you can rewrite history.
I noticed a few days ago when looking into americans leaving loaded firearms in ovens that we are losing archived news. I would find an article or story that is just missing now, all it is is a headline link to no where. And I have seen this trend on all things, we are losing the knowledge and for no other reason then the possibility of an extra dollar at some point. Take this and mix in the overwhelming amount of LLM generated bullshit pretending to be information tailored to peoples perceived interests (if you live in a religious area for example you see more religious bullshit) and we have almost inescapable silos.
I don’t think I need to explain how dangerous this is.
Remember to support your local library. Only physically written words are going to be safe in the coming age.
As someone that has been on my local library board… I got bad news for you on that one. Libraries are culling books like never before, facing licensing issues like never before, and funding issues like never before.
That’s very 1984 of them
Start self-hosting archive box They cant block everyone
If you have a concrete suggestion as to the stack don’t hold back 😃
I run this but not sure if it’s what you want https://wiki.archiveteam.org/index.php/ArchiveTeam_Warrior
Is the Guardian actually blocking the Internet Archive? Seems to work for me
Meanwhile,
e; huh, Mediapost article did in fact start loading on the Archive a few minutes after I posted this
I wander what happened to Archive in 2024 when it was “hacked” and some pages “disappeared”…
Fuck you spez
Buuuut they all say that we need to donate to save free speech! It can’t be a lie right?
Mainly pointing at the guardian here as they are sliding down the same slope that the other two slops did.
By “donate” you mean “buy a subscription”?
