Nice one.<p>I've been doing the same bit wider scope, for the whole Crux list, pruned to apex domains, and looking for CMS signals - how's your throughput?<p>I'm not doing any headless browser stuff, or many requests, so hyper optimised for speed.<p>I do grab robots.txt - didn't really see much in llms.txt or humans.txt in the wild, does yours?