I have to admit to being drawn to Bollen and Luce’s quantitative approach. For one thing, I think that surveying users has a very limited ability to tell you anything, for all kinds of reasons. It has a tendency to give you what users think they do, rather than what they actually do. This is useful for some things, but not everything. Particularly in terms of how much effort it takes to make good surveys, you might as well just use the data you are generating anyway. I realized this when I put together a survey that had lots of “do you check out adult fiction, or what?” then realized that I already know that.
So in other words, we create this data anyway, with server logs that “contain a wealth of information on user preferences and the structure of user retrieval patterns.” It is “just-in-time,” again, unlike user surveys, so that you can get some idea of your evaluation process whenever you are working on the project, without having to stop to bring in outside groups.
They have developed a number of ways of looking at the data which can lessen the impact of some of the shortcomings of server log information. Using their techniques, DL projects should be able to strike a balance, just as we have seen with image quality vs size, that ought to help bring the project closer to its goals.