Usability Testing & UX Metrics

Qualitative usability testing (the workhorse)

5 users per round finds most severe issues for a given segment/ design (Nielsen & Landauer 1993 cost-benefit model; Virzi 1992) — iterate in small rounds rather than one big study. Caveats: 5 is for problem-DISCOVERY, not statistics; distinct user groups need their own ~5; comparative/quant studies need far more (NN/g suggests ~20+ per design for quantitative benchmarking).
Think-aloud protocol (Ericsson & Simon's verbal-report methodology, adapted): users narrate while doing realistic tasks; moderator stays neutral ("what are you thinking?" not "did you like it?"), never rescues early, never asks leading questions.
Task design: realistic goal + scenario, no interface words in the prompt ("You want to see last month's spending" not "open the Reports tab"), defined success criteria before the session.
Severity rating (frequency × impact × persistence, Nielsen's scale 0–4) turns findings into a prioritized fix list.
Unmoderated remote (UserTesting-style platforms) scales breadth; moderated depth for complex flows and probing.

Qualitative rigor

Evaluator effect (Hertzum & Jacobsen 2001): different evaluators watching the same sessions report substantially different problem sets — use 2+ analysts for consequential studies and merge findings.
Interrater agreement: when coding qualitative data that will drive decisions, have a second coder code a sample and report agreement (Cohen's kappa or Krippendorff's alpha are the standard statistics); disagreements refine the codebook, not just the numbers.
Keep the observation/interpretation line explicit in reports; tie every finding to session evidence (clip or timestamp).

Quantitative instruments

SUS (Brooke 1996): 10-item questionnaire, 0–100 score; average across large samples ≈68 (Sauro's normative database); grades/adjectives mapping in Bangor, Kortum & Miller (2008/2009). Cheap, reliable, comparable — the default post-study instrument.
SUPR-Q (Sauro 2015, JUS 10(2)): 8 items measuring website usability, trust, appearance, loyalty; scored as percentile ranks against a commercial normative database (licensed) — useful when "how do we compare to other sites" is the question, not just "is it usable."
SEQ (Single Ease Question, 1–7 after each task); NASA-TLX (Hart & Staveland 1988) for workload-heavy domains; UMUX-Lite (Lewis, Utesch & Maher 2013) as a 2-item SUS alternative that correlates well with SUS.
Task metrics: success rate, time-on-task, error rate — ISO 9241-11's effectiveness/efficiency/satisfaction triad operationalized.
NPS (Reichheld 2003): loyalty question; widely used, widely criticized psychometrically — fine as a trend, weak as a diagnosis; always pair with the "why" verbatims.

Tree-test metrics (Optimal Workshop-style convention): task success (right node), directness (reached it without backtracking), and time — low directness with eventual success still means weak scent.
First-click accuracy on real pages: a correct first click predicts higher task success (Bailey & Wolfson's 2006–2009 live-site analyses reported 87% vs 46%; unpublished gray literature — MeasuringU's replication found a much smaller advantage). Use first-click as a comparative diagnostic between designs, not the 87/46 as a constant.

Benchmarking practice

Benchmark BEFORE redesigning (baseline), then re-run the identical tasks + instrument each release cycle — deltas on your own product beat absolute scores; external norms (SUS 68, SUPR-Q percentiles) contextualize but don't replace your baseline.
Quant benchmarks need real samples (~20+ per design/segment) and confidence intervals; for small-sample completion rates use the adjusted-Wald interval (Sauro & Lewis) rather than raw percentages.
Keep tasks, wording, and recruiting criteria frozen across waves; changing any of them resets the trend line.

Product analytics frameworks

HEART (Google: Rodden, Hutchinson & Fu, CHI 2010): Happiness, Engagement, Adoption, Retention, Task success — each with Goals→Signals→Metrics. The standard way to make "UX quality" measurable at product scale.
Funnels (activation, checkout), cohort retention curves, rage-click/ dead-click detection, search-with-no-results rate, support-contact rate per feature — behavioral smoke alarms.

Telemetry & experimentation ethics

Session replay and analytics collect personal data: disclose in the privacy notice, honor consent requirements (GDPR/CCPA), and mask input fields by default — replay tools can otherwise capture passwords, card numbers, and health data keystroke by keystroke.
Minimize and aggregate: report cohorts, not identifiable individuals; set retention limits on recordings.
Don't A/B test manipulative patterns "because the metric went up" — a conversion win from confusion is a liability (see Persuasion, Ethics & Dark Patterns); guardrail metrics should include trust signals (support contacts, refunds, opt-outs).

A/B testing hygiene

Hypothesis stated in advance (metric + direction + rationale); adequate sample & duration (whole business cycles; no peeking-and-stopping — sequential-testing corrections exist if you must); guardrail metrics so a "win" can't secretly damage retention/support load; and remember A/B tells you WHICH, not WHY — pair with qualitative for the why (Kohavi, Tang & Xu 2020 is the rigor bible).

Try it — score a product with the real SUS. The ten System Usability Scale items (Brooke 1996), scored live with the actual algorithm. Rate a product you use daily and watch where it lands against the norms.

1 = strongly disagree · 5 = strongly agree

Sources

Nielsen, J. & Landauer, T. (1993). "A mathematical model of the finding of usability problems." CHI/INTERCHI '93; Virzi, R. (1992). Human Factors, 34(4).
Hertzum, M. & Jacobsen, N. E. (2001). "The evaluator effect." IJHCI, 13(4).
Brooke, J. (1996). "SUS: A 'quick and dirty' usability scale." In Usability Evaluation in Industry; Bangor, Kortum & Miller (2008, IJHCI; 2009, JUS); Sauro, J. & Lewis, J. (2016). Quantifying the User Experience (2nd ed.). Morgan Kaufmann.
Sauro, J. (2015). "SUPR-Q: A comprehensive measure of the quality of the website user experience." Journal of Usability Studies, 10(2).
Hart, S. & Staveland, L. (1988). "Development of NASA-TLX." In Human Mental Workload. North-Holland.
Lewis, J., Utesch, B. & Maher, D. (2013). "UMUX-Lite." Proc. CHI '13.
Rodden, K., Hutchinson, H. & Fu, X. (2010). "Measuring the user experience on a large scale (HEART)." Proc. CHI '10.
ISO 9241-11:2018 — usability definitions.
Kohavi, R., Tang, D. & Xu, Y. (2020). Trustworthy Online Controlled Experiments. Cambridge UP.
Reichheld, F. (2003). "The One Number You Need to Grow." HBR.
Bailey, B. & Wolfson, C. — first-click analyses (webusability.com, gray literature); replication discussion at measuringu.com.