William Hyman: Do We Know How to Evaluate and Compare Risks and Benefits?

It is well known that few things in life are risk free and that we therefore take on risk whenever we seek the benefits of healthcare. This leads to the straightforward notion that we should evaluate risks and benefits, and then compare them in order to make a rational decision about undertaking the activity. While it is often said that this is being done, there is no consistent methodology for assessing either risk or benefit. And even if we could determine a measure for risk and one for benefits, we wouldn’t know how to rationally compare them because they are different concepts that would be measured in different ways. In addition, how much benefit offsets how much risk? Do benefits have to just edge out risk or be significantly greater by some amount? Similarly, do benefits offset risks even when those risks could be reasonably mitigated?

These kinds of evaluations as applied to medical devices are addressed in a recent FDA draft guidance document called Factors to Consider Regarding Benefit-Risk in Medical Device Product Availability, Compliance, and Enforcement Decisions. This document presents FDA’s thinking on the elements of risks and benefits that should be considered in reaching a risk-benefit determination with regard to certain postmarket regulatory issues, including recalls. The core issue here is that a device that is a candidate for a recall may have current benefits that should be considered relative to those risks. In a stop-use recall, these benefits could be lost, perhaps temporarily, subject to the availability of alternative devices or treatments.

The components of risk considered in the FDA document include the common attributes of severity and likelihood, along with the duration of exposure, uncertainty, detectability, and patient tolerance of risk. Of these, severity and likelihood have often been subject to quasi-numerical analysis by dividing each into a finite number of steps and assigning a numerical value to each step. A risk score then might be obtained by multiplying the individual rankings. While common, this type of calculation has little fundamental basis and a number of problems.

Uncertainty can be considered by recognizing that the assignment of a severity or likelihood level is not absolute. Similarly, detectability might be used to modify likelihood since the idea is that the potential for harm will be noticed before it actually causes injury. This is common in device manufacturing where inspections are meant to catch defects before they reach patients and cause injury. Similarly, a clinical alarm is meant to alert staff to a patient issue (detection) before that issue becomes a source of harm.

Duration of exposure might be used to modify severity, but this is but one of several multipliers or weighting factors that might be used. In assigning varying weights, we face the fundamental question of whether and to what degree some factors are more important than others. Patient tolerance is quite another issue. What level of knowledge is assumed in assessing a patient’s willingness to undertake risk? Is this all patients (measured somehow) or is it individualized? Is the patient’s willingness an overriding factor or just part of the assessment? If just a part, how important?

Combining multiple risks is also problematical. If numerical values are used it is tempting to add the scores for each risk but this has no underlying basis and it creates additional issues. For example, are multiple small risks “equal” to fewer larger risks? Even if a risk result is achieved, there is still no inherent basis for deciding whether or not the risk is acceptable, along with the caveat of acceptable to whom.

In the FDA document, the notion of benefits is also multifactorial, including type of benefit, magnitude of benefit, likelihood of the patient seeing the benefit, duration of the benefit, and patient preference. Each of these factors also could be subject to scales, descriptive words, and scores. For example, the magnitude of the benefit might be labeled very high, high, moderate, low, and very low. Duration might be designated as extended, medium, or short, noting that the various factors need not have the same number of levels. Patient preference has the same issues as patient tolerance of risk. Is this all patients, subgroups of patients, or individual patients? What is their preference based on—or does this matter? How good are people at deciding what is best for them, and is this an unfettered decision or one that can be second guessed by family, regulators, courts, providers, etc.? Again a hierarchy of importance is needed in order to weigh the various factors. For example, is duration more or less important than magnitude of benefit?

And if we did have a rational assessment of risk, perhaps reduced to a score, and an assessment of benefit, perhaps also scored, then what? What do the assessments or numbers mean for things that are fundamentally different and not inherently quantifiable? How do we compare them? Do we need to just tip the scale toward benefit or do we need a more demanding standard? As an engineer, I am used to having specifications that are measurable. Which is greater, stronger, or stiffer is a question that can then be answered by objective evidence using a consistent set of units and measurements. I can describe the evaluation and others can see exactly what I did. But I don’t know how to compare strength or weight to, say, color.  Nor do I know how to compare risks to benefits, even if I knew how to actually measure each, which I don’t.

The FDA does not have any suggestions in its draft document for how to combine factors and make comparisons, but perhaps if this draft ever emerges as an actual guidance document it will be more forthcoming. If not, we will have a list of things to consider (which might be of value) but not how to rate and combine them. If in an analysis each component was carefully described, we might be able to follow the analyst’s thinking if decisions. But decisions based on this process will not necessarily be consistent and transparent which is part of the FDA’s goals for the guidance document. These challenges are not limited to this document or the FDA. Whenever people speak of comparing benefits and risks, they are on shaky ground with respect to how they are making their determinations of each, and how they are making the comparison.

William Hyman, ScD, is professor emeritus of biomedical engineering at Texas A&M University. He now lives in New York where he is a consultant and adjunct professor of biomedical engineering at The Cooper Union.

2 thoughts on “William Hyman: Do We Know How to Evaluate and Compare Risks and Benefits?

  1. The cost side of cost/benefit is easier than risk. We know the units of cost and we know that if there are multiple costs that you can in general get the total cost by adding. However, cost can also be more theoretical than real. Is personnel cost real? It is if they do the assessment as extra paid time. But what if they did it instead of checking Facebook or during a break?

    The benefits side is still challenging. What exactly are the benefits? How do you measure them? How do you value them? And if you can value them, how do you decide if they are worth the cost? Also the benefits of the assessment have to be in the form of action items, and those actions must actually be taken.

    There is also some perversity here. If you detect a risk and you don’t do anything about, are you worse off if that risk manifests itself? On the other hand not doing the analysis can make you look careless. If something bad happens it is probably better to say we did this, this, and this and followed through than to say “huh?” when asked what you did to prevent it. And you never want to argue that someones suffering wasn’t worse the cost of preventing it. Yet I mock the “if you could save just one life” argument since it ignores the reality of resource allocation. My absurdist example is that I live in a 29-story apartment building with many old people (a group I have joined). A life might be saved if an ambulance was parked in front of my building at all times. But this is true for every other building and is thus not practical in terms of allocation of limited resources.

    All is well does not mean you wasted the effort. Thinking this might be like having term life insurance and being mad that you didn’t die. But it might mean that you can increase the interval between assessments. This is an issue in medical device “preventive” maintenance. Can you adjust your procedures and/or intervals based on what you find?

    Can’t afford to be safe is similar to my suggested motto of too busy to be safe. And it also can be made to be absurd. For example: We don’t sterilize our instruments because of our low margins.

  2. Dr. Hyman,

    You got me thinking. I am a nurse studying for my DNP. My project concerns awareness and use of the SAFER guides an ONC/HHS risk assessment tool for hospitals & other care settings. The guides evaluate health IT risk and present”elements for consideration.” No true score to benchmark against, and no definitive answer in regards to actions to take, other than recommended best practices.To complete the assessment, there is a cost to the organization in time personnel etc.
    Not surprisingly, because these assessments are not mandated, many administrators, nursing, operations, and IT are not doing them or in some cases even aware they “should” be done. What happens if a major flaw is found? Is the benefit of knowing your systems are at risk of failure worth the cost of deploying employees to investigate?
    What if you determine all is well? Has the exercise then been a waste of valuable healthcare dollars? I am interested in your thoughts about the risk-benefit analysis of doing a risk-benefit analysis in an industry where margins are slim and the cost of error is high in terms of dollars and human suffering. I can be reached at makarelv@gmail.com. Thank you for a very interesting piece . I hope to begin the data collection portion of my DNP project soon.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s