8. Conducting Heuristic Evaluation
Conducting Heuristic Evaluation (expert reviews)
Heuristic evaluation (also known as expert review, usability audit, usability inspection) is a usability evaluation of a product, which involves a small set of expert evaluators who examine the level of its compliance with recognized usability principles, also called heuristics. Expert evaluators are needed, although it could be carried out by less experienced people, provided that they receive some training and a checklist, however, fewer problems get discovered that way. Each evaluator inspects the interface individually, the findings are then aggregated. It differs from user testing, because the responsibility of analyzing the interface is on evaluators, not a researcher or moderator.
There are multiple benefits of letting experts evaluate usability:
- They provide quick feedback to designers, relatively easy to do compared to other methods;
- Cheaper than many other methods, such as user testing.
- Could be carried out before user testing, so that biggest usability problems are discovered and addressed before they prevent participants from discovering harder to spot, their workflow specific issues. Doing this would save a lot of money, since you would need fewer rounds of user testing.
- Reviews are also excellent in competitive benchmarking, since they allow comparing the usability of your product to your competitor’s product.
Heuristic Evaluation vs User Testing
Heuristic evaluation is never a substitute for user testing, since it does not provide any insights on how actual users use the system, it is hard even for experts to predict it.
Nielsen’s Usability Heuristics
Nielsen’s Heuristics are commonly used usability principles in expert reviewing.
Visibility of system status
The system should always keep users informed about what is going on, through appropriate feedback within reasonable time.
Match between system and the real world
The system should speak the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order.
User control and freedom
Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
Consistency and standards
Users should not have to wonder whether different words, situations, or actions mean the same thing. Follow platform conventions.
Even better than good error messages is a careful design which prevents a problem from occurring in the first place. Either eliminate error-prone conditions or check for them and present users with a confirmation option before they commit to the action.
Recognition rather than recall
Minimize the user’s memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate.
Flexibility and efficiency of use
Accelerators — unseen by the novice user — may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.
Aesthetic and minimalist design
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility.
Help users recognize, diagnose, and recover from errors
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution.
Help and documentation
Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the user’s task, list concrete steps to be carried out, and not be too large.
Other usability standards
- Iso’s 7 dialogue principles (http://www.userfocus.co.uk/resources/iso9241/part110.html): suitability for the task, suitability for learning, suitability for individualization, conformity with user expectations, self descriptiveness, controllability, error tolerance.
- Schneiderman’s 8 golden rules of interface design (https://www.cs.umd.edu/users/ben/goldenrules.html): strive for consistency, cater to universal usability, offer informative feedback, design dialogs to yield closure, prevent errors, permit easy reversal of actions, support internal locus of control, reduce short-term memory load.
- Arnie Lund’s “Expert Ratings of Usability Maxims” (http://www.theuxbookmark.com/2009/11/ux-articles-and-research-papers/usability-engineering/expert-ratings-of-usability-maxims/).
- Bruce Tognazzini’s “Principles of Interaction Design” (http://asktog.com/atc/principles-of-interaction-design/)
Most UX practitioners have their favorite sets of usability principles; most of these principles overlap.
Typical process of heuristic evaluation
- Identify the business goals of the product being evaluated, create short profiles of users, typical tasks. When carrying out an evaluation for a client, it is a good practice to send them a quick questionnaire that asks about the users and typical tasks or just call them to discuss it.
- Decide which heuristics you are going to use.
- Create a checklist for that particular application domain.
- Review the product: firstly use the product to get a general overview of its structure and functionality, then go through each task and note usability issues.
- You could also explore the elements not related to the tasks, e.g. additional web pages. It is a good practice to go through the interface several times.
- Invite 2-4 additional evaluators, brief them on the product, the tasks, the checklist etc.
- Ask each evaluator to evaluate the product.
- Put the problems found by each evaluator together, remove duplicates.
- Write up a report that describes the problems, assigns severity ratings to them and suggests solutions. The report should be easy to understand to the audience who are not usability professionals, terms such as “lack user control and freedom” are not straightforward, you should clearly describe each problem. Often experts do not map each problem directly to a specific heuristic, since the audience is less interested in the heuristics than in how they can address the problems.
- Present the findings to the design team.
Evaluation checklists (templates)
It is a good practice to use detailed evaluation checklists when carrying out an evaluation since using only high level principles risks missing some usability issues, also high level principles (such as Nielsen’s heuristics) might appear too vague when applied to new technologies, thus domain specific checklists are needed.
A checklist for evaluating usability of websites – UserFocus web usability checklist. It is very detailed, thus covers all aspects of web usability.
You can download the Excel checklist here http://www.userfocus.co.uk/resources/guidelines.html. The document has several worksheets that correspond to the main elements and aspects of websites: Homepage, Task Orientation, Navigation, Forms, Trust, Content, Layout and others.
Also, it calculates usability scores, which is excellent for competitor benchmarking, you could compare usability scores of your website to competitors’.
How to use it: For each checklist item, enter a rating between -1 and 1; -1 means that it does not comply with the guideline, 0 – kind of complies, +1 means complies. If a guideline is not applicable, leave it blank. Each guideline has a field for entering comments when needed. After you do that for all checklist items, you can see the scores – just go to the “Results” worksheet and you will get numerical ratings of compliance there.
N.B. It is not organized according to heuristics, but that is not a problem, often designers prefer that, they are less interested in the level of “User control and freedom”, more interested in what they can improve on the homepage.
Another checklist for evaluating usability of websites – UX for the Masses usability review template.
You can download it here: http://www.uxforthemasses.com/blog/wp-content/uploads/2011/02/Usability-review-template.xls . It is less detailed than the template above, but also very straightforward to use.
How to use it: enter a score (very poor, poor, moderate, good, very good) and comments for each of the best practice usability principles. The principles cover features and functionality, navigation, search, feedback and other aspects of an interface. At the bottom you will get the overall usability score (out of 100).
A checklist for desktop software – Heuristic evaluation system checklist.
You can download it here ftp://cs.uregina.ca/pub/class/305/lab2/example-he.html . It is based on Nielsen’s heuristics, each heuristic has a fairly long review checklist. It is very detailed, thus makes sure that even small issues are likely to be detected.
How to use it: For each checklist item, highlight Yes, No or N/A option. No in-built score calculation.
Psychological usability heuristics – looks at usability from the psychological view and focuses on aspects such as human memory, mistakes etc.
It is not as detailed as most of the other checklists, however, worth having a look at it, especially if you are developing your own checklist.
If you are evaluating a domain that is not a website of software application, you would need to create your own checklist. It takes time, however, you will have a re-usable checklist tailored to your field (visibility of system status might means different things in desktop software and in virtual reality games). In order to do that, you need to look at each heuristic and decide what exactly it means in your particular domain.
When designing your custom checklist, you will need some way to rate usability problems. You could create your own scale or use an existing one.
Nielsen’s rating scale is commonly used:
0 = I don’t agree that this is a usability problem at all
1 = Cosmetic problem only: need not be fixed unless extra time is available on project
2 = Minor usability problem: fixing this should be given low priority
3 = Major usability problem: important to fix, so should be given high priority
4 = Usability catastrophe: imperative to fix this before product can be released
The traffic-light color scheme could also be used:
- example of a best practice (Green)
- minor problem (Yellow)
- serious problem (Orange)
- critical problem (Red)
Number of evaluators
One evaluator will not be able to find all usability problems, regardless of the level of expertise, different evaluators discover different usability problems; having more evaluators means more insights and less chance of getting fixated on a minor issue (though it is better to have one evaluator than not doing any evaluation). It is recommended to use three to five evaluators, since larger numbers do not gain much additional information per extra participant.
More evaluators should be used in situations where usability is critical (e.g., mission-critical software) and all problems need to be discovered.
The figure below shows the percentage of problems found by each number of evaluators:
N.B. Evaluators tend to find more problems when using detailed checklists than just using heuristics, since checklists make sure they cover the things they might miss otherwise; even though no empirical study has been carried out to estimate the percentage of issues found when evaluators use checklists, it is likely that you could find the same amount of problems with fewer evaluators.
Nielsen’s heuristics: https://www.nngroup.com/articles/ten-usability-heuristics/
Schneiderman’s 8 golden rules of interface design: https://www.cs.umd.edu/users/ben/goldenrules.html
Arnie Lund’s “Expert Ratings of Usability Maxims”: http://www.theuxbookmark.com/2009/11/ux-articles-and-research-papers/usability-engineering/expert-ratings-of-usability-maxims/.
Bruce Tognazzini’s “Principles of Interaction Design”: http://asktog.com/atc/principles-of-interaction-design/
UserFocus web usability checklist can be downloaded here: http://www.userfocus.co.uk/resources/guidelines.html.
Heuristic evaluation system checklist ftp://cs.uregina.ca/pub/class/305/lab2/example-he.html
UX for the masses usability review template http://www.uxforthemasses.com/blog/wp-content/uploads/2011/02/Usability-review-template.xls