Distributed Indexing of the Web using Migrating Crawlers

Odysseas Papapetrou, Stavros Papastavrou, George Samaras
Department of Computer Science, University of Cyprus

Abstract: In this ongoing work, we introduce UCYMicra; a crawling system that utilizes concepts similar to those found in Mobile and distributed Crawling introduced in [3, 4]. UCYMicra extends these concepts and introduces new ones in order to build a more efficient model for distributed Web crawling, capable of keeping up with Web document changes in real time. UCYMicra proposes a complete distributed crawling strategy by utilizing Mobile Agents technology. The goals are (a) to minimize network utilization (b) to keep up with document changes by performing on-site monitoring, (c) to avoid unnecessary overloading of the Web servers by employing time realization, and (d) to be upgradeable at run time.
Keywords: distributed web crawling, mobile agents

author = {Odysseas Papapetrou and Stavros Papastavrou and George Samaras},
title = {Distributed Indexing of the Web using Migrating Crawlers.},
booktitle = {WWW (Alternate Track Papers {\&} Posters)},
year = {2003},
url={http://www2.cs.ucy.ac.cy/~cspapap/publications/www03poster.pdf} }