It's frustrating that there's no way for people to (selectively) mirror the Internet Archive. $25-30M per year is a lot for a non-profit, but it's nothing for government agencies, or private corporations building Gen AI models.<p>I suspect having a few different teams competing (for funding) to provide mirrors would rapidly reduce the hardware cost too.<p>The density + power dissipation numbers quoted are extremely poor compared to enterprise storage. Hardware costs for the enterprise systems are also well below AWS (even assuming a short 5 year depreciation cycle on the enterprise boxes). Neither this article nor the vendors publish enough pricing information to do a thorough total cost of ownership analysis, but I can imagine someone the size of IA would not be paying normal margins to their vendors.
It's insane to me that in 2008 a bunch of pervs decentralized storage and made hentai@home to host hentai comics. Yet here we are almost 20 years later and we haven't generalized this solution. Yes I'm aware of the privacy issues h@h has (as a hoster you're exposing your real IP and people reading comics are exposing their IP to you) but those can be solved with tunnels, the real value is the redundant storage.
The illegal side of hosting, sharing, and mirroring technology, as it were, is much more free to chase technical excellence at all costs.<p>There are lessons to be learned in that. For example, for that population, bandwidth efficiency and information leakage control invite solutions that are suboptimal for an organization that would build market share on licensing deals and growth maximization.<p>Without an overriding commercial growth directive you also align development incentives differently.
I was hopeful a few years ago when I heard of chia coin, that it would allow distributed internet storage for a price.<p>Users upload their encrypted data to miners, along with a negotiated fee for a duration of storage, say 90d. They take specific hashes of the complete data, and some randomized sub hashes, of internal chunks. Periodically an agent requests these chunks, hashes and rewards a fraction of the payment of the hash is correct.<p>That's a basic sketch, more details would have to be settled. But "miners" would be free to delete data if payment was no longer available on a chain. Or additionally, they could be paid by downloaders instead of uploaders for hoarding more obscure chunks that aren't widely available.
> a bunch of pervs<p>Not everyone who watches hentai is a perv
Yeah sure, they are just into 9000 year old dragons...
Just don't look up what the word "hentai" means ;)
The fact AI companies are stripping mining IA for content and not helping to be part of the solution is egregious.
How is it "egregious" that people are obtaining content to use for their own purposes from a resource intentionally established as a repository of content for people to obtain and use for their own purposes?
Because nobody who opens a public library does so intending, nor consenting, for random companies to jam the entrance trying to cart off thousands of books solely to use for their own enrichment.<p><a href="https://xkcd.com/1499/" rel="nofollow">https://xkcd.com/1499/</a>
[dead]
Might be easier for them to just pay for the mirrors and do an on-site copy and move the data in a container?<p>That way they would provide some more value back to the community as a mirror?
Has any evidence been provided for this fact?
I would like to be able to pull content out of the Wayback Machine with a proper API [1]. I'd even be willing to pay a combination of per-request and per-gigabyte fees to do it. But then I think about the Archive's special status as a non-profit library, and I'm not sure that offering paid API access (even just to cover costs) is compatible with the organization as it exists.<p>[1] It looks like this might exist at some level, e.g. <a href="https://github.com/hartator/wayback-machine-downloader" rel="nofollow">https://github.com/hartator/wayback-machine-downloader</a>, but I've been trying to use this for a couple of weeks and every day I try I get a HTTP 5xx error or "connection refused."
I wish there were some kind of file search for the Wayback Machine. Like "list all .S3M files on members.aol.com before 1998". It would've made looking for obscure nostalgia much easier.
<a href="https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server" rel="nofollow">https://github.com/internetarchive/wayback/tree/master/wayba...</a><p><a href="https://akamhy.github.io/waybackpy/" rel="nofollow">https://akamhy.github.io/waybackpy/</a><p><a href="https://wiki.archiveteam.org/index.php/Restoring" rel="nofollow">https://wiki.archiveteam.org/index.php/Restoring</a>
Yes, there are documents and third party projects indicating that it has a free public API, but I haven't been able to get it to work. I presume that a paid API would have better availability and the possibility of support.<p>I just tried waybackpy and I'm getting errors with it too when I try to reproduce their basic demo operation:<p><pre><code> >>> from waybackpy import WaybackMachineSaveAPI
>>> url = "https://nuclearweaponarchive.org"
>>> user_agent = "Mozilla/5.0 (Windows NT 5.1; rv:40.0) Gecko/20100101 Firefox/40.0"
>>> save_api = WaybackMachineSaveAPI(url, user_agent)
>>> save_api.save()
Traceback (most recent call last):
File "<python-input-4>", line 1, in <module>
save_api.save()
~~~~~~~~~~~~~^^
File "/Users/xxx/nuclearweapons-archive/venv/lib/python3.13/site-packages/waybackpy/save_api.py", line 210, in save
self.get_save_request_headers()
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
File "/Users/xxx/nuclearweapons-archive/venv/lib/python3.13/site-packages/waybackpy/save_api.py", line 99, in get_save_request_headers
raise TooManyRequestsError(
...<4 lines>...
)
waybackpy.exceptions.TooManyRequestsError: Can not save 'https://nuclearweaponarchive.org'. Save request refused by the server. Save Page Now limits saving 15 URLs per minutes. Try waiting for 5 minutes and then try again.</code></pre>
Pick the items you want to mirror and seed them via their torrent file.<p><a href="https://help.archive.org/help/archive-bittorrents/" rel="nofollow">https://help.archive.org/help/archive-bittorrents/</a><p><a href="https://github.com/jjjake/internetarchive" rel="nofollow">https://github.com/jjjake/internetarchive</a><p><a href="https://archive.org/services/docs/api/internetarchive/cli.html" rel="nofollow">https://archive.org/services/docs/api/internetarchive/cli.ht...</a><p>u/stavros wrote a design doc for a system (codename "Elephant") that would scale this up: <a href="https://news.ycombinator.com/item?id=45559219">https://news.ycombinator.com/item?id=45559219</a><p>(no affiliation, I am just a rando; if you are a library, museum, or similar institution, ask IA to drop some racks at your colo for replication, and as always, don't forget to donate to IA when able to and be kind to their infrastructure)
There are real problems with the Torrent files for collections. They are automatically created when a collection is first created and uploaded, and so they only include the files of the initial upload. For very large collections (100+ GB) it is common for a creator to add/upload files into a collection in batches, but the torrent file is never regenerated, so download with the torrent results in just a small subset of the entire collection.<p><a href="https://www.reddit.com/r/torrents/comments/vc0v08/question_about_archiveorg_torrents_being/" rel="nofollow">https://www.reddit.com/r/torrents/comments/vc0v08/question_a...</a><p>The solution is to use one of the several IA downloader script on GitHub, which download content via the collection's file list. I don't like directly downloading since I know that is most cost to IA, but torrents really are an option for some collections.<p>Turns out, there are a lot of 500BG-2TB collections for ROMs/ISOs for video game consoles through the 7th and 8th generation, available on the IA...
Is running an IPFS node and pinning the internet archive's collections a good way to do this?
> $25-30M per year is a lot for a non-profit<p>$25 million a year is not <i>remotely</i> a lot for a non-profit doing any kind of work at scale. Wikimedia's budget is about seven times that. My <i>local</i> Goodwill chapter has an annual budget greater than that.
You have an extremely skewed view of the average nonprofit
You're being purposefully obtuse. Most non-profits don't function at scale (neither do they do best at scale). They serve their <i>local</i> community
I'd like a Public Broadcasting Service for the Internet but I'm afraid that money would just be pulled from actual PBS at this point to support it.
Facilitating mirroring seems like it would open up another can of liability worms for the IA, as well as, potentially, for those mirroring it. For example, they recently lost an appeal of a major lawsuit brought by book publishers. And then there's the Wayback Machine itself; who knows what they've hoovered up from the public internet over the years? Would you be comfortable mirroring that?
Don’t put any stock into the numbers in the article. They are mostly made up out of thin air.
They have come a very long way since the late 1990s when I was working there as a sysadmin and the data center was a couple of racks plus a tape robot in a back room of the Presidio office with an alarmingly slanted floor. The tape robot vendor had to come out and recalibrate the tape drives more often than I might have wanted.
There is a fundamental resistance to tape technology that exists to this day as a result of all those troubles.
That's sad, but it mirrors my experience with commercial customers. Tape is so fiddly but the cost efficiency for large amounts of data and at-rest stability is so good. Tape is caught in a spiral of decreasing market share so industry has no incentive to optimize it.<p>Edit: Then again, I recently heard a podcast that talked about the relatively good at-rest stability of SATA hard disk drives stored outdoors. >smile<
We had a little server room where the AC was mounted directly over the rack. I don't think we ever put an umbrella in there but it sure made everyone nervous the drain pipe would clog.<p>Much more recently, I worked at a medium-large SaaS company but if you listened to my coworkers you'd think we were Google (there is a point where optimism starts being delusion, and a couple of my coworkers were past it.)<p>Then one day I found the telemetry pages for Wikipedia. I am hoping some of those charts were per hour not per second, otherwise they are dealing with mind numbing amounts of traffic.
Is this some kind of copypasted AI output? There are unformatted footnote numbers at the end of many sentences.
I was thinking the same thing. No proofreading is a sure sign to me. I also feel like I've read parts of this before.
Some of the images are AI generated (see the Gemini watermark in the bottom right), and the final paragraph also reads extremely AI-generated.
Maybe, but I was trying to find the original source of this article and couldn’t, at least not cursorily.
I already stopped when I saw the AI-gen image
I think this was writen wholly by deep research.<p>It just reads like a clunky low quality article
I love to imagine this is all a cover and the Internet Archive is located in a remote cave in northern Sweden and consists of a series of endlessly self replicating flash drives powered by the sun.
This article is way too LLMey for my taste.
Thanks for this, I've always wondered how the Archive operates but always ended up not searching.
IA is hosting a couple more of Rick Prelinger’s shows this month. Looking forward to visiting
Is it still year 2006 and websites haven’t figured out responsive design?
Does any one know how the size of this compares to archive.today?
How long will it take for them to send the PetaBox to space?
Does IA do deduplication?
Hate to be the guy in the comments complaining about the css, but the sides of the text of this article are cut off. It looks like I'm zoomed in, and there's no way I can see the first few columns of the text without going to Reader view. I'm on a modern iPhone using safari, accessibility settings font larger than usual.
Same for me, Safari iOS 18.7.1 no accessibility font size set, no browsers font size set.
FWIW, it's the same for me on FF Android.
this is every data hoarders dream setup haha
Disappointed with the lack of pictures.
Probably because this looks more like a Deep Research agent "delving" into the infrastructure -- with a giant list of sources at the end. The Archive is not just a library; it is a service provider.
What do you want some pictures of?
[flagged]
[flagged]
The IA needs perhaps not just more money, but also more talented people, IMO. I worry that it has stagnated, from a tech pov.
They can offer a perk that literally no other tech job can offer: Someday have a statue of your likeness preserved in ceramic: <a href="https://www.atlasobscura.com/places/internet-archive-headquarters" rel="nofollow">https://www.atlasobscura.com/places/internet-archive-headqua...</a><p>"Inside the church's main room, with its still-intact pews, there are more than 120 ceramic sculptures of the Internet Archive's current and former employees, created by artist Nuala Creed and inspired by the statues of the Xian warriors in China."
We've hired a few dozen people over the past couple of years. We think they're pretty talented.
>And the rising popularity of generative AI adds yet another unpredictable dimension to the future survival of the public domain archive.<p>I'd say the nonprofit has found itself a profitable reason for its existence