How were digital libraries evaluated?

Tefko Saracevic, PhD
School of Communication, Information and Library Studies
Rutgers University
New Brunswick, NJ USA
tefko@scils.rutgers.edu
http://www.scils.rutgers.edu/~tefko/
“Evaluating digital libraries is a bit like judging how
successful is a marriage” (Marchionini, 2000)

Abstract
The purpose is to provide a critical synthesis of works on digital library evaluation that included data. Over 80 evaluation studies are analyzed as to evaluation:
(1) constructs,
(2) context,
(3) criteria and
(4) methods that were used.
Approaches taken in evaluation studies are identified and described; a list of constructs is given as to entities or processes that were evaluated; context or approaches taken in evaluation are enumerated; numerous criteria used as a base of evaluation are classified; and finally the methodologies used are identified. Of the findings only one is generalized: many users have difficulties in using digital libraries; a “versus” hypothesis is proposed describing an adversarial relation between users and digital libraries. Evaluation of digital libraries is not widely practiced. The corpus in this analysis represents the majority of efforts in digital library evaluation that contain data. Conclusions, among others, speculate as to the reasons for a relatively low presence of evaluation in digital library
research and practice.

• Click here for the full version of the article