Document Type

Conference Paper

Publication Date




Publication Title

iPRES 2018 The 15th International Conference on Digital Preservation


10 pp.

Conference Name

iPRES 2018 The 15th International Conference on Digital Preservation, September 24-27, 2018, Boston, Massachusetts


Web archives, a key area of digital preservation, meet the needs of journalists, social scientists, historians, and government organizations. The use cases for these groups often require that they guide the archiving process themselves, selecting their own original resources, or seeds, and creating their own web archive collections. We focus on the collections within Archive-It, a subscription service started by the Internet Archive in 2005 for the purpose of allowing organizations to create their own collections of archived web pages, or mementos. Understanding these collections could be done via their user-supplied metadata or via text analysis, but the metadata is applied inconsistently between collections and some Archive-It collections consist of hundreds of thousands of seeds, making it costly in terms of time to download each memento. Our work proposes using structural metadata as an additional way to understand these collections. We explore structural features currently existing in these collections that can unveil curation and crawling behaviors. We adapt the concept of the collection growth curve for understanding Archive-It collection curation and crawling behavior. Using the growth curves, we can see if most of the mementos in the collection are skewed earlier or later. We also introduce several seed features to describe the diversity and types of seeds present in an Archive-It collection. With these seed features, we come to an understanding of the diversity of resources that make up a collection and the depth of those resources within their seed websites, indicating whether the curator chose to preserve the top-level page or something more specific within a site. Finally, we use the descriptions of each collection to identify four semantic categories of Archive-It collections. Using the identified structural features, we reviewed the results of runs with 20 classifiers and are able to predict the semantic category of a collection using a Random Forest classifier with a weighted average F1 score of 0.720, thus bridging the structural to the descriptive. Our method is useful because it saves the researcher time and bandwidth. They do not need to download every resource in the collection in order to identify its semantic category. Identifying collections by their semantic category allows further downstream processing to be tailored to these categories.


© The Authors.

This is an open-access article published under the Creative Commons Attribution 4.0 International (CC BY 4.0) License.

Original Publication Citation

Jones, S., Nelson, M., Nwala, A., & Weigle, M. (2018). 205.3 The many shapes of Archive-It.


0000-0002-4372-870X (Jones), 0000-0003-3749-8116 (Nelson), 0000-0002-2787-7166 (Weigle)

Weigle-2018TheManyShapesPPT.pdf (111602 kB)
Powerpoint Presentation