Seminar 2 Notes

The section on evaluation in the literature treats the reasons to evaluate, its object and its kinds. The primary reason is the difference between the perspectives of the developing team on the one hand and the end users on the other. Because these two groups live and work in different contexts they view the product in different ways so that without evaluation a development team can miss important flaws in the product design. Both user experience and product usability should be evaluated, and evaluation should be performed both during and at the end of development. It seems to me that evaluation should be performed as often as resources permits and after each new stage of product development, often represented by finished implementation of new requirements.

There are three types of evaluation that are separated by the amount of control the evaluators have of the setting. The highest level of control are tests not involving users at all but conducted by experts modeling possible user behaviors. The next level of evaluation takes place in a lab with extensive monitoring capabilities and perhaps modeled after a natural environment where the product will be used when finished. The last level, of least control to the evaluators, takes place in a natural setting where the product is being used in precisely the setting for which it is intended. The advantage of this level is the ability to observe a use of the product in unexpected ways which can easily be missed in a highly controlled environment such as a lab.

The later part of the literature is about different kinds of modeling evaluations, where typical users are simulated by experts or trained evaluators. Heuristic evaluation means using a checklist of usability principles created by researchers to test if a product aligns with them. Cognitive walkthroughs are minute steps that experts take with a small part of a product design to see how users typically would solve problems. There are also similar quantitative methods like analytics (often used to track web activity), GOMS (reminds of a cognitive walkthrough but focuses on goals rather than problems and therefore can be used to compare designs) and KLM (models the amount of time necessary to complete a task by timing key strokes, mouse movements, etc).

I've understood that there is not one evaluation method that fits every need, but if one still would remember only one method from this course (maybe a seed method from which all other methods can be derived?) which one should it be?