Techniques

General

Rate Limiting and Bot Behavior

From the StackOverflow link above:

  • General consensus is limit you page requests to 2-5 seconds per request.
  • Identify your requests with a user agent string that identifies your bot.
  • Have a webpage for your bot explaining it's purpose. This URL goes in the agent string.
Go to top