Learn to Crawl Effectively