I found a lot to like about Saracevic, but the line I found most resonant also illustrates the problems with his article: “Every evaluation is also temporal.”
Indeed. Saracevic’s concern that digital libraries are not being evaluated in the way they should be was probably a necessary polemic in 2004, but from my vantage point in 2013, a bit overstated. Think about technology in 2004. WordPress was less than a year old. Flickr came out that same year. Our pal Omeka was four years in the offing. Google Analytics — perhaps the best friend one could hope for in analyzing user traffic and habits on a website — wasn’t available widely until a year after this article was published.
Saracevic’s reasoning included prematurity, funding, and interest among the reasons that evaluation didn’t occur, and those would be the reasons I’d cite as well. But more to my point, tools that eliminated those roadblocks (free tracking software that can be inserted into the source code of any web property, for example) had yet to be developed.
But what else could Saracevic do in 2004? He couldn’t see into the future and neither can we. And yet that’s what makes this article important. “It seems that evaluation theorists and evaluation practitioners do not communicate well, at least not in ways that are visible,” he argues. You can almost hear him sighing as he types that.
I don’t think Saracevic’s article has aged particularly well, at least not in terms of his argument. But the steps he lays out for an evaluation program are as vital as ever, particularly his list of criteria, his description of approaches, and his fundamental belief in the importance of evaluation.