In the early days of the web you could do a search on google like<p><pre><code> path:/etc/passwd
</code></pre>
Sometimes there were even shadow passwd files with the hashes exposed on the web. Crazy days.
There's an astounding amount of .DS_Store showing up - I hadn't realised how common it apparently is for people to accidentally upload this.
Its interesting and not interesting at the same time based on some of the search results<p>Almost all of them seem like home projects being deployed with ease in mind than security. The common thread seems to be the fact that most of them are phishing website, not sure if thats a business model to target here?
So is this the crawler that has been constantly hammering all my applications searching for these files from the very second I first issue a TLS cert for them? Thanks to you I've had to put fail2ban on all my public-facing web servers...<p>How about you be a good netizen and make it so people can request to be scanned and don't proactively do it, let alone constantly keep hammering them with requests?
searching for .gov reveals 0 matches... <i>doubt</i>
My guess is that they ran selective search on the domains which get registered with any registrar, thats the trigger to start the search. .gov domains are not managed by your typical registrar which is selling the domain registration information to all these downstream partners/scavengers (for lack of better word)