Distributed location aware web crawling


Odysseas Papapetrou, George Samaras
Department of Computer Science, University of Cyprus
{cspapap,cssamara}@cs.ucy.ac.cy



Abstract: Distributed crawling has shown that it can overcome important limitations of the today.s crawling paradigm. However, the optimal benefits of this approach are usually limited to the sites hosting the crawler. In this work, we propose a location-aware method, called IPMicra, that utilizes an IP address hierarchy, and allows crawling of links in a near optimal location aware manner.
Keywords: distributed crawling, web crawling, location aware crawling

@inproceedings{papapetrou:www04poster,
author = {Odysseas Papapetrou and George Samaras},
title = {Distributed location aware web crawling.},
booktitle = {WWW (Alternate Track Papers {\&} Posters)},
year = {2004},
pages = {468-469},
abstract-url={http://www2.cs.ucy.ac.cy/~cspapap/abstracts/www04.html},
publisher-url={http://portal.acm.org/citation.cfm?id=1013367.1013529} }