Document Type

Article

Publication Date

2006

DOI

10.1016/j.serrev.2006.03.001

Publication Title

D-Lib Magazine

Volume

12

Issue

2

Pages

1-16

Abstract

We describe the observed crawling patterns of various search engines (including Google, Yahoo and MSN) as they traverse a series of web subsites whose contents decay at predetermined rates. We plot the progress of the crawlers through the subsites, and their behaviors regarding the various file types included in the web subsites. We chose decaying subsites because we were originally interested in tracking the implication of using search engine caches for digital preservation. However, some of the crawling behaviors themselves proved to be interesting and have implications on using a search engine as an interface to a digital library.

Original Publication Citation

Smith, J.A., McCown, F., & Nelson, M.L. (2006). Observed Web robot behavior on decaying Web subsites. D-Lib Magazine, 12(2), 1082-9873. doi: 10.1016/j.serrev.2006.03.001

ORCID

0000-0003-3749-8116 (Nelson)

Download

Share

COinS