I’ve seen some things, I’ve done some stuff.

  • 0 Posts
  • 4 Comments
Joined 3 years ago
cake
Cake day: June 22nd, 2023

help-circle


  • Super effective, I tried both of these on a couple of domains I have and the amount of hits they get vs how long crawlers stay in them is insane. I use the AI robot.txt file and if they ignore it will spend hours scraping randomized nonsense text from unlimited internal links. I’m sure large legit ai companies have protection, but I get a lot of traffic from Africa and Asia in particular. Not sure if it’s the source or a VPN, but I just look at the geos and tend not to dig deep.