Formative and summative usability testing: two different questions

Usability testing is often described as a single activity that happens somewhere between design and launch. In practice it answers two very different questions, and confusing them wastes effort in both directions. Formative testing asks what is wrong with a design while there is still time to change it. Summative testing asks whether the finished design is safe enough to release, and while the two look similar on paper they differ in timing, participants, analysis and the consequences of a finding.

Two questions behind every usability study

Formative work is diagnostic. Its purpose is to surface use errors, confusion and workarounds early, when a screen layout or a label can still be redesigned cheaply. Findings are treated as leads to investigate, and a single serious problem discovered by one participant is enough to justify a change.

Summative work is evidential. Its purpose is to demonstrate that the design, as finalized, supports safe use in the intended environment. Findings are treated as outcomes, and the study has to be planned so that its results can be defended to a regulator, a notified body or an internal safety committee.

Formative work: finding problems while change is cheap

Formative studies belong early, often with prototypes, partial interfaces or simulated workflows. Participants are asked to think aloud while they attempt realistic tasks, and the moderator probes hesitation, misreading and recovery. The output is a prioritized list of design problems rather than a pass or fail verdict.

Because the goal is discovery, the protocol stays flexible. Sessions can be short, tasks can be adjusted between participants and the team can iterate on the design mid-study. A finding that changes the next prototype has done its job even if the study never produces a clean number.

Formative testing is also the natural place to explore the edges of use: a distracted night shift, a patient cable routed across a control panel, a clinician working in gloves. Those conditions rarely appear in a tidy laboratory session, and they are exactly where human factors in device design tends to break down.

Summative work: evidence that the design is safe enough

Summative evaluation happens on a design that is intended to be final, with production-representative instructions, labelling and interface. It tests whether users can complete critical tasks without use errors that could cause harm, and it is normally tied to the hazard analysis rather than to general impressions of ease of use.

The protocol is fixed in advance. Tasks, success criteria, participant profile and analysis method are defined before the first session, because changing them afterward undermines the evidence. Moderators avoid coaching, and any intervention is recorded, since assistance given during a critical task changes what the result means.

Where a critical task fails, the outcome is not simply a design tweak. The failure feeds back into risk management for medical devices, which may require a design change, an additional risk control or a justified conclusion that the residual risk is acceptable.

How the two differ in practice

The practical differences are easier to see side by side. Most arguments about usability testing come from applying formative expectations to a summative protocol, or the reverse, rather than from disagreement about method.

Participants, tasks and use scenarios

Both types need participants who resemble the intended users, including the ones who are not the design team's mental model of a user. Training level, shift pattern, eyesight, hearing, manual dexterity and language all shape how an interface is read. A study staffed with experienced specialists will miss what a new graduate or an agency nurse encounters.

Tasks should come from real use rather than from the feature list. Setup, cleaning, alarm silencing, battery change, troubleshooting and handover are usually more revealing than the flagship measurement function. For summative work, tasks must map to the hazardous situations identified earlier, so that the evidence covers the risks that matter.

Common mistakes

Most weak usability programs repeat the same handful of errors, and they are procedural rather than technical. Recognizing them early saves a great deal of rework.

• Running a summative study on a prototype that differs from the released device.
• Coaching participants through a critical task and then reporting it as a success.
• Testing only with expert users, or only with colleagues from the same building.
• Treating formative findings as a checklist to close rather than a design input.
• Leaving instructions for use out of the session, when they are part of the use situation.
• Recording observations but not the reasoning behind a design decision.

From test findings to field evidence

Usability work does not end at release. Use errors that survive into clinical practice appear in incident reports, in help desk calls and in the informal workarounds staff invent to get through a shift. Reading those signals against the original test findings shows whether a problem was missed, misjudged or introduced later by a software change.

That loop is what makes reporting device-related incidents useful to design teams rather than only to safety officers. A pattern of the same error in the field is stronger evidence than any single laboratory session, and it should reopen the design question instead of being filed as user error.

The discipline is to be clear about which question is being asked. Formative testing buys better designs; summative testing buys defensible evidence. A program that runs both, at the right moments and with honest reporting, ends up with devices that are easier to use and a file that survives scrutiny.