↧
Answer by Paul T. Rawkeen for How to hide an aggressive crawler?
One more solution is to use PROXY server provider (like this one for example) and rotate IP address every X requests. This particular provider has an API to retrieve IPs on the fly. cURL can be used...
View ArticleAnswer by MikeB for How to hide an aggressive crawler?
"Acceptable" is a relative term. Some site owners have enough processing power and bandwidth that they don't think scanning 3000 pages per hour is "aggressive". Some site owners struggle for bandwidth...
View ArticleHow to hide an aggressive crawler?
I'm planning to crawl a specific site. I have 3000 specific pages that I want to crawl once every few months. I've created a crawler, but I don't want to be banned from the site. Is there a way to...
View Article
More Pages to Explore .....