Seminar #2 notes

The reading for this seminar presented concepts and tools for evaluating designs. A pillarstone of design evaluation is the fact that the earlier you discover issues, the less of a problem they are likely to pose, and the more likely it is that the design requirements will mature properly, meaning you make the right trade-offs. The reason for this isn't that designs are made to be faulty, but that the process can narrow one's focus too much, and this tunnel vision can often cause obvious design flaws. One example that comes to mind is the famous Cisco router that had the reset button aligned perfectly with a normal cable protector position:





This means that users who were extra cautious and used protected tabs on their networking cables got their whole system reset. Probably not the intention at all, but one may ask how even almighty Cisco could miss this.

How can such fails be avoided? The chapters present some methods, mostly involving external people.

Asking users
Through structured and semi-structured interviews, people from outside the development team can offer their two cents. These could be random people, intended users or field experts. It is also possible to conduct these interviews in groups. It's very important to avoid bringing in bias or direction, the point is to discover design flaws, and this requires a setting that's as neutral as possible.

Questionnaires
Questionnaires are based on the same principles as the interviews - they offer the possibility to ask a larger group, but remove the opportunity to ask follow-up questions immediately. Care must be taken in how the questionnaires are designed, so as not to impose any ideas. Also, the sample group must be representative of the design's intended users.

Inspections
Heuristic evaluation is a technique where an expert can assess the design through informal usability inspection. Parameters for this evaluation can be how the design complies with set design standards, how helpful errors can be etc. Another technique is the cognitive walkthrough, where a user's action process is simulated in order to find any issues or room for improvement.

In the end, the goal is to get good data to analyze. Also, design evaluation is constant work throughout the process, and iteration cycles that take evaluation data into account may very well have much better prototypes to work with in the next evaluation rounds, giving even more relevant data. Also, failing is an important part of the design - the reset button placement probably has a dedicated task group at Cisco by now.

Seminarium 2 - Marcus Dypbukt Källman

The literature for the second seminar examines the role of evaluation when designing a product. Evaluating your design is very important since it affirms that your design is attractive to your target group and to find problems that lowers the usability or hurts the users experience.  What to evaluate depends on the product and it’s important that the evaluation is implemented so benefits the final product. There are generally 3 broad categories of evaluation,

Controlled settings involving users – users’ activities are controlled in staged environments to test and measure how the product is used and to see if the users behave as imagined when interacting with the product. This allows for outside disturbance to be reduced but can lead to troubles capturing the context of use.

Natural Settings involving users – users’ activities are observed in the “real world” to find how the product is being used. Displays how the users would behave outside the test but are often more expensive and harder to conduct.

Any setting not involving users – the product is examined by experts or predicting and modeling how the interface will be used to identify its most obvious faults. This approach is often cheap and easy but can be unreliable since it can miss unpredictable problems.

All of these have different advantages and shed light on different problem areas and are therefore used in different cases. Often a combination of the three should be used to achieve a higher understanding of what to change with the design.

Evaluations not involving users have 4 different methods, heuristic evaluation, walkthroughs, analytics and predictive models. In heuristic evaluation and walkthroughs experts are used to provide feedback on the product. heuristic evaluation lets the expert evaluate the product by a set of pre-determined usability principles to check if the user-interface accommodates these principles. In walkthroughs a task is preformed and its problem areas assessed. Analytics tracks the user’s behavior to see what they do or want from the design. Predictive models are used to measure the user performance of different aspects of the product.

Since we’ve decided our main product is going to be a smartphone app, which of these type can we use to evaluate our design? and which could we realistically execute with our resources?

Notes for Seminar 2 - War

Chapter 13:

We have now in our project come to the part of evaluating our product in the design process where we in this chapter got introduced too. It introduces a process that is very important when designing a product. I personally think that the design process is the most important part in the process of creating a product. When designing a product there is a hard job of making sure that the design of the product is appealing to our target group, the usability and the user experience are great and making sure of that the requirements have been fulfilled.  A great design is often characterize by it have required many iterations until they have accomplish that design and we will evaluate our product by testing, analyze the result and making sure that every requirements have been fulfill. This approach is called iterative design and this process continuous until our product has fulfilled all our requirements and is appealing to our target group.

There are many different methods and techniques that can be used during the evaluation of our product but a good framework is the DECIDE framework and this chapter introduce us to the DECIDE framework which can be used when designing the product. The DECIDE framework can be seen as a checklist that contains a list of which we should go through each point in that checklist iterative. The Decide framework will help us with the evaluation of our product and the DECIDE framework stands for:

1. Determine the goals.
2. Explore the questions.
3. Choose the evaluation methods.
4. Identify the practical methods.
5. Decide how to deal with the ethical issues.
6. Evaluate, analyze, interpret, and present the data. 

This chapter also mentioned that we need to set up goals during the evaluation of our product, because earlier in this text I mentioned that the evaluation is an important process due to the fact that we need to get information about how well our product works, if we have fulfilled our requirements etc. We must therefore formulate questions that aim to seek answers to our questions in a way that we know if we reached our goals.

Chapter 15:

This chapter is also about evaluation but focuses on the different evaluation methods that don’t require the users presence but instead an expert’s input can be used if wanted. The book mentioned these different evaluation methods, heuristic evaluation, walkthroughs, analytics and predictive models.

The heuristic evaluation and walkthroughs are inspection methods that rely on feedback from experts rather then users. This method can be a bit hard to carry out due to the fact that it can be hard to find experts that are willing to evaluate our product. In this specific method the experts evaluate a user-interface and they will also check that the product affiliate with the usability principle know as heuristic.

The walkthrough is the alternative method to heuristic evaluation, which slowly walks though a task using the product and notes any potential problems with the product. We must also distinguish between cognitive and pluralistic walkthroughs.

The predictive models doesn’t rely on users or experts input rather you can use formulas to measure performance and you can have in mind the Fitt’s Law which can be described in interaction design as; predicts the time it takes to point at a target based in the size of the object and the distance to the object. Fitt’s law can be used to measure the usability of our product.

Illustrates the different stages in this Iterative process during the Evaluation, too illustrate the reader that evaluation is an ongoing process that will take many iterations before moving on to the next stage in developing the product. Because a product that have been through many iteration often results in that the product is of high quality.

Question: After reading this chapter I have thought on which evaluation methods we should use, some of these methods can be hard to carry out. So I wonder which evaluation method will benefit us most?

What goals should we set up when we are evaluating our product?

Notes for seminar 2 - Miro

The content for the latest seminar revolves around the evaluation of a design. Evaluations can be done in many different ways, but the common denominator of all evaluations is to find out if a prototype appeals to the target group. Identifying problematic aspects is also a key goal of the evaluation step.

Evaluating a product can be done during the design process, formative evaluation, or when the product is finished, summative evaluation. There are three categories used to define different types of evaluation methods. Evaluating with users in a controlled setting or a natural setting describes the first two. The third is an evaluation made in any setting without involving any users. Depending on the product and the aspect being evaluated one should choose a fitting category. In the case of our idea, an evaluation made in a controlled environment with a user would provide us with interesting information regarding the usability of the application and its interface. However, this might remove the aspect of the ecological validity, since the user isn't affected by the potentially stressful environment the app would be used in with a natural setting.

The content of chapter 14 describes the process of doing usability testing. If we were to do an evaluation, this chapter should be a big part of our focus. Many of our thoughts during the design process have revolved around making the interface as easy as possible to use, while also providing a fairly wide variety of different functions. Measuring the amount of time it takes a user to complete a task and observing the user's navigation of the interface could provide us with data showing potential changes needed in the design. We could also benefit from hypotheses testing. During the design process, we chose to go with a certain interface based on our own justifications of what we thought would be easier for the personas. A test of these hypotheses might be in order, after all we could be wrong.

Chapter 15 revolves around methods of evaluating a product by understanding users, not by involving them in the evaluation itself. A couple of methods are presented, the one I see most fit do discuss is heuristic evaluation. This method of evaluation uses the help of experts to evaluate certain criterias a product should fulfill. The criterias are called heuristics  and are generally different depending on the type of product. I believe this method is of certain use to us even though we might lack the resources needed to hire an expert. When searching for smartphone heuristics, several documents and presentations show up and they all describe valid and important things to consider when designing a smartphone application, can we use these to evaluate our design from a different perspective than our own? This is my question for this seminar. 

Seminar 2 Notes

The section on evaluation in the literature treats the reasons to evaluate, its object and its kinds. The primary reason is the difference between the perspectives of the developing team on the one hand and the end users on the other. Because these two groups live and work in different contexts they view the product in different ways so that without evaluation a development team can miss important flaws in the product design. Both user experience and product usability should be evaluated, and evaluation should be performed both during and at the end of development. It seems to me that evaluation should be performed as often as resources permits and after each new stage of product development, often represented by finished implementation of new requirements.

There are three types of evaluation that are separated by the amount of control the evaluators have of the setting. The highest level of control are tests not involving users at all but conducted by experts modeling possible user behaviors. The next level of evaluation takes place in a lab with extensive monitoring capabilities and perhaps modeled after a natural environment where the product will be used when finished. The last level, of least control to the evaluators, takes place in a natural setting where the product is being used in precisely the setting for which it is intended. The advantage of this level is the ability to observe a use of the product in unexpected ways which can easily be missed in a highly controlled environment such as a lab.

The later part of the literature is about different kinds of modeling evaluations, where typical users are simulated by experts or trained evaluators. Heuristic evaluation means using a checklist of usability principles created by researchers to test if a product aligns with them. Cognitive walkthroughs are minute steps that experts take with a small part of a product design to see how users typically would solve problems. There are also similar quantitative methods like analytics (often used to track web activity), GOMS (reminds of a cognitive walkthrough but focuses on goals rather than problems and therefore can be used to compare designs) and KLM (models the amount of time necessary to complete a task by timing key strokes, mouse movements, etc).

I've understood that there is not one evaluation method that fits every need, but if one still would remember only one method from this course (maybe a seed method from which all other methods can be derived?) which one should it be?

Pain points

1: Very important - - - - 5: Not needed

Issues / Opportunities	Primary persona	Secondary persona
Trouble getting a ticket	1	1
Trouble finding the dock at Slussen	2 (whiny family, no patience for being lost)	3 (less pressure to get things right)
Difficulties finding parking	5 (no car)	5 (no car)
Trouble finding desired location from Allmänna gränd	3 (annoying, but you have lots of people to ask)	3
Confused about where to step off the boat	2 (being on watch takes some fun out of the journey)	3 (less pressure to get things right)
Smartphone app with suggestions of nearby activities	2 (for when she goes away on her own)	5 (doesn’t know the benefits of such a function)
Smartphone app with family tracking	3 (fun function for when the family has been doing different things and need to find each other)	5
Guide on the boat	4 (americans and asians care about this)	5 (enjoy the silence)
Language issues with existing signage/apps	2 (Questionable English skills)	5 (International Vacuum sales have perfected his English)
Having all of the above seemlessly integrated in one single app.	2	3 (Important to be easy to get started with)

Persona and scenario 2, Luís.

Persona 2 (Luís)   

       

Luís is 52 years old, was born in a small rural town outside Rio de Janeiro but moved to the big city for his university studies. He had begun a 3 year program in economics but dropped out when he got a job at a company selling bags for vacuum cleaners. At the time it was a small firm but it has grown since then and he now has a managerial position, overseeing employees. He met his wife Ana in one of his regular visits to a church which she also used to visit. They have been married for 24 years and have four kids, Gonya, Piero, Maribel and Lucia.

Luís is a devoted Catholic and reads a page from the bible every evening. He loves cooking and throws famous barbecue parties for his family and friends. He is a jovial person who is easily recognized in social gatherings and has a reputation for being a lot of fun to be around. He does not care much about technology but has a smartphone and uses apps when someone (for instance one of his kids) has helped him with installation and given him a tutorial.

As the company at which Luís works has grown, it has gained customers abroad and is now trying to strike a deal with a retailer in Sweden, for a brand of vacuum cleaning bags sized for a common vacuum cleaner. The company decided to send Luís to negotiate the final terms of the deal in Stockholm. He is to land on Tuesday evening, meet his clients on Wednesday morning and go back home in the evening.

Apart from his business goal of getting a good deal, he wants to explore Stockholm on the evening of Wednesday. He has not yet made up his mind, but he is choosing between dining at a good restaurant, visiting a museum or a grand swedish church.

Scenario 1

Luís is currently residing in Stockholm visiting a work related fair. After a long day of meetings and seminars he and a new acquaintance he met during the day decide to treat themselves to a nice dinner. Neither of them are Swedish speakers and none has previously been in Stockholm so finding and getting to a restaurant which suits them quickly proves somewhat difficult and time consuming.  

Scenario 2

Luís visit in Stockholm is nearing its end, he feels satisfied with the weekend from a work related perspective, but after checking out of his room he finds himself with quite a lot of time to burn before his flight with departs later that evening. He isn’t very appealed at the thought of spending all that time waiting at the airport. Having opened his eyes for Swedish culture during the weekend Luís feels an itch for exploring. He would love to visit some iconic or exiting place in his close proximity but has no idea of what, or how to get there.