How to Block Some of the Bots

5 comments

• fxtentacle1 hour ago
  I like the idea of adding a fake cpanel subdomain for 169.254.169.254 so that script kiddies will start port-scanning their own hosting provider, which will likely get them flagged/banned.
  • Bender1 hour ago
    When I first experimented with that I was not expecting anything to happen. Within a few days one person in Amazon EC2 in Germany started trying to do zone transfers for some of my domains <i>likely to figure out which records to avoid</i> and then they just excluded my domains entirely. All of the scanning stopped shortly thereafter. The scanning noise was literally all coming from one person despite the source IP&#x27;s being all over the internet.
• binaryturtle1 hour ago
  410 and a &quot;Sec-Fetch-Mode:&quot; string in the response body. I guess it thinks I&#x27;m a bot? Thanks!<p>Nothing to read, nothing to see, I move along. (Yikes, the modern web sucks!)
  • Bender1 hour ago
    Real browsers send it. [1] Some of the reader apps do not. Most bots aside from those utilizing Chrome Headless do not.<p>People can see a few headers here [2]<p>[1] - <a href="https:&#x2F;&#x2F;caniuse.com&#x2F;?search=sec-fetch-mode" rel="nofollow">https:&#x2F;&#x2F;caniuse.com&#x2F;?search=sec-fetch-mode</a><p>[2] - <a href="https:&#x2F;&#x2F;nochan.net&#x2F;.env" rel="nofollow">https:&#x2F;&#x2F;nochan.net&#x2F;.env</a>
  • gruez49 minutes ago
    I didn&#x27;t even make it that far, I got PR_END_OF_FILE_ERROR which indicates it couldn&#x27;t even get past the tls handshake.
    • ErroneousBosh41 minutes ago
      I mean that just might be the dying gasp of a puddle of scorched fibreglass and boiling copper where the server used to be.
      • Bender39 minutes ago
        Mostly harmless(c)<p><pre><code> load average: 0.00, 0.01, 0.00</code></pre>
• ajsnigrutin30 minutes ago
  There&#x27;s a special place in hell for people who block curl and wget, especially on sites with downloadable files (eg source code tgz&#x27;s, media, etc.), basically anything i might need to wget on a server.
  • Bender22 minutes ago
    A guy gets sent to hell. The demons guide him into the lobby. Satan jumps out and lets out a big evil roar to no effect. Satan then glances down and sees the spot on the mans finger where there was a ring. &quot;Oh... well you&#x27;ve been through worse. The break room is down the hallway to the left, mail room is upstairs to the right...&quot; Satan just walks away.<p>Every step is optional of course. There are ways to make curl work on my site but I choose to add friction as some botters abuse libcurl. I doubt anyone else will do what I do. They could spoof the user-agent but most botters seem to not know how to do that despite the myth that they all do. For what it&#x27;s worth if I had a site specific to sharing code artifacts or archives I would not block curl and I would also enable native rsyncd.
• iririririr49 minutes ago
  most (all?) of those will 100% block valid traffic too
  • Bender40 minutes ago
    I would be interested in some examples of valid traffic that they would block. For the purposes of the document I do not consider anything from a data-center to be valid traffic. People can of course skip any or all steps, experiment with one at a time on a test server <i>as they should.</i>
    • ferngodfather24 minutes ago
      &gt; Obvious proxy is obvious.<p>&gt; if ($http_x_forwarded_for) {....<p>This may block schools and libraries that use content blockers. Often the internal client is left to make abuse tracking easier (or because the overworked admin didn&#x27;t know they could turn it off).
      • Bender18 minutes ago
        <i>This may block schools and libraries</i><p>Oh, well that is ok for me I suppose. I add RTA&#x2F;adult headers that hopefully they also look for and block using parental controls as adult content should not be viewed in a school or library. I could add a note suggesting to skip that step if one wishes schools and libraries that may be using a proxy to view.
        • ferngodfather8 minutes ago
          That&#x27;s fair enough.<p>I think it&#x27;s a great article to be fair. We need more of this cheap and quick bot blocking. The fact the solution to unwanted traffic is often &quot;use Cloudflare&quot; is _not_ great for the internet, and nobody really actually likes deploying or managing ModSecurity. Its a nice middleground.
• Capricorn248121 minutes ago
  Am I the only one that exclusively gets attacks with spoofed user agents and rotating TLS signatures? I feel like every post I see about not needing a CDN has tips that could be overcome in under an hour of scripting.