In the last decade we’ve seen a huge shift from lab-based testing to remote moderated and remote unmoderated testing. As the technology gets better and cheaper, I think we will continue to see testing done outside the lab and in combination with other easily collected metrics like analytics and log-files.

• How to measure the usability of an application.
• Thinking in terms of specific and measureable outcomes for software and products.
• Knowing what statistical tests to use and how to interpret the results.
• How to compute, report and interpret confidence intervals around key usability metrics.
• Understanding both the limits and power of small sample
• sizes.
• Interpreting and reporting usability metrics and appropriate statistical test outcomes.

While this depends on the context, the major metrics are completion rates, task-times, some measure of satisfaction and usability problem descriptions. These correspond to the ISO 9241 pt 11 definitions of effectiveness, efficiency and satisfaction.

I use a combination of task-performance and attitudinal measures like standardized question- naires, like the System Usability Scale, where possible. It’s nice to know both what is wrong with an interface, then how the users’ attitudes of its usability compare to other interfaces. It’s important to understand the complementary nature of perception and performance. Measures of task-performance like completion rates and task-times usually correlate highly to task-level measures of ease-of-use. That is, if a user isn’t able to accomplish she wants to do or takes a long time, they’re often aware of this poor experience. However, asking a user about their overall impression of the usability of an application often has a loose correlation to task-performance because the user is associating other experiences and expectations about the interface that aren’t tested in a study. Having both types of measures provides a better picture of the user experience.

All measures should use the appropriate average and confidence intervals (margins of error). I often report them graphically as «error-bars».

Jeff er opprinnelig fra Montreal, Canada, men har slått seg ned i Denver, Colorado. Han har aldri vært i Skandinavia før, og gleder seg til å oppleve Norge. Du møter ham på Yggdrasil i Tønsberg 17. – 18. oktober. Vi sees!