Communities and Collections
Usage
- 150 views
- 245 downloads
Designing Efficient Topic-Driven Web Crawlers
-
- Author(s) / Creator(s)
-
Technical report TR02-15. Crawlers are essential to web search engines for retrieving high quality web pages automatically and efficiently based on developer defined notions of importance and quality. Due to rapid growth of World-Wide Web and limited resources available to crawlers, developing good crawling strategies and evaluating them are still big challenges. In this paper, we do a comprehensive study of existing and proposed crawling strategies done by other research works. We have developed a topic-driven crawler that uses combinations of two different strategies in evaluating page importance during the crawl. | TRID-ID TR02-15
-
- Date created
- 2002
-
- Subjects / Keywords
-
- Type of Item
- Report
-
- License
- Attribution 3.0 International