Tammygoss and hsikaiyang are your discussion facilitators for this unit of A&E. We made the following summaries, followed by proposed discussion topics.
This week’s readings focused on the evaluation and assessment (E&A) of several digital libraries (DL) in an attempt to inform others on the who, what, where, when, and why of evaluation and assessment plans.
The Evaluation of Digital libraries: An Overview by Saracevic contains the building blocks of information that E&A plans should contain: Construct, Context, Criteria, and Methodology. It is up to the information professional and/or planning team to create an E&A plan that best addresses these criteria. The following is a table created from the Saracevic reading, with a little Marchionini (Results column) thrown in at the end.
(Note. Please click the table to enlarge it for better view.)
Using the table, we are able to see the common elements in all of this week’s readings, even those who appear to have radically different approaches to A&E plans. For example, the Perseus Evaluation Project written about in Evaluating Digital Libraries: A longitudinal and Multifaceted View, despite the assertion that it was created as a roadmap rather than specific plan, does contain the elements as shown in the table. The Perseus project contains the entities to be evaluated (operational, augmentations), context (system, human, usability, humanities), criteria (goals – access, collaboration, and freedom), methodology (a wide array were used, depending on the circumstances), and results (disseminated on the web, future planning and follow-up). The Perseus Evaluation Project is an excellent example of a longitudinal A&E plan as an evolving entity for a DL.
Choudhury, et al, in A Framework for Evaluating Digital Library Services make an important statement regarding the Assessment & Evaluation of digital libraries; if brick and mortar libraries can and should be evaluated, so can and should digital libraries. Evaluations were conducted similar to those for non-digital libraries, through the use of surveys (tested with and without incentives), and focus groups segregated into specific groups of granularity.
The Morse article, Evaluation Methodologies for Information Management Systems emphasizes that the A&E should be an explicit part of the entire project and treated as such, including the requirement of deliverables. But by far the most creative use of an evaluation tool has to be the application of Hebb’s Law of Learning to digital library users and their data, showing that hidden in the system logs, there is quantitative data. Jump through some mathematical calculations and taa daa!, you have community context, most of the time. Of course, a different tactic should be employed to capture multi-disciplinary research communities, reinforcing the idea that there is no one way to analyze DLs.
Below are some of the questions we think it is worthy of pondering or practicing. Since each topic may constitute a big issue covered by multiple articles, we think it appropriate to just pick up one or two questions. Please feel free to share other topics that you can provide insights.
- Please use an example to illustrate the relationships among decision making, information, and evaluation. Also describe in details how successful they can work together. If your example is not a success story, please explain why.
- A digital library is of little value if users cannot find needed information in an effective and efficient manner. Precision and recall are traditionally the two primary criteria to evaluate information retrieval. For digital libraries, are there other criteria that have been defined or used? Please list and elaborate all criteria you can find. Also indicate how unique they are in the realm of digital library.
- Three popular methods for usability evaluation are focus group, ethnography, and survey. Please describe your experience in any of these, if any. Otherwise, please describe their pros and cons of each method within the context of digital library evaluation.
- Please apply the methodology of “multi-attribute, stated-preference” to determine the scope of a particular service your library is to offer or evaluate an existing service. For example, helping HathiTrust to address or mitigate the problem of “free riders.”
- Due to resource constraints, no library is able to digitize everything. Please propose a scheme to determine the digitization priorities based on past retrieval patterns specific to your library’s user community. Or describe the inadequacy and deficiency found in common practices.
- Tefko Saracevic discussed the distinction of criteria and measure. Please use more concrete examples to elaborate the ideas. If a specific criterion is not measurable, how would you do evaluation?
- Tefko Saracevic pointed out that “users have many difficulties with digital libraries.” As a librarian or a librarian to be, how would you address Saracevic’s concerns?
- The DARPA IM Program was “intended to explore and evaluate potentially radical alternatives to traditional approaches to information management.” Please summarize what are new in DARPA IM Program. From Emile Morse’s descriptions, do you think whether it has achieved its design goal and how.
- Bibliometrics is a methodology to assess the impact of individuals or communities (for example, journals). Please describe how traditional bibliometrics can be modified to adapt to digital environments, including feasibility and benefit.
- Gary Marchionini distinguished novelty effects from long-term effects. There are ample evidences in his paper illustrating the benefits of taking a longitudinal approach to evaluation. Please pick up any of his examples or your own choice to illustrate the picture.