New York Times Opinion: Why Health Care Tech Is Still So Bad

This was an opinion piece published 21 March 2015 in the New York Times written by Robert M. Wachter, Professor of Medicine, University of California, San Francisco and author of "The Digital Doctor: Hope, Hype, and Harm at the Dawn of Medicine’s Computer Age” also published in the New York Times.

Here's the link to the article: http://www.nytimes.com/2015/03/22/opinion/sunday/why-health-care-tech-is-still-so-bad.html?smid=nytcore-ipad-share&smprod=nytcore-ipad

I have commented on several quotes from the article.

1. "Even in preventing medical mistakes — a central rationale for computerization — technology has let us down. (My emphasis.) A recent study of more than one million medication errors reported to a national database between 2003 and 2010 found that 6 percent were related to the computerized prescribing system.

At my own hospital, in 2013 we gave a teenager a 39-fold overdose of a common antibiotic. The initial glitch was innocent enough: A doctor failed to recognize that a screen was set on “milligrams per kilogram” rather than just “milligrams.” But the jaw-dropping part of the error involved alerts that were ignored by both physician and pharmacist. The error caused a grand mal seizure that sent the boy to the I.C.U. and nearly killed him.

How could they do such a thing? It’s because providers receive tens of thousands of such alerts each month, a vast majority of them false alarms. (My emphasis.) In one month, the electronic monitors in our five intensive care units, which track things like heart rate and oxygen level, produced more than 2.5 million alerts. It’s little wonder that health care providers have grown numb to them."

Comments: Before I read the third paragraph, I was thinking How can you blame the computer when it provided you with an alert regarding the prescribing error that you made? 

It is well known that systems that produce a high percentage of false alarms, that those alarms over time will be ignored or discounted. I consider this is a devastating indictment. We must do better.

I have been a human factors engineer and researcher for decades. One of the mantras of human factors is preventing errors. That's central to what we're about. But if the systems we help engineer generate false alarms at a rate that has our users ignoring the correct ones, then we have failed and failed miserably.

I think the problem of false alarms requires further research and commentary.


2. "... despite the problems, the evidence shows that care is better and safer with computers than without them."

Commentary: This is nice to read, but we as medical technologists need to do better. We really need to follow up on the repercussions of our technology we create when it's deployed and used in the field.


3. "Moreover, the digitization of health care promises, eventually, to be transformative. Patients who today sit in hospital beds will one day receive telemedicine-enabled care in their homes and workplaces."

Commentary: I agree. Of course that's a central theme of this blog.


4. "Big-data techniques will guide the treatment of individual patients, as well as the best ways to organize our systems of care. ... Some improvements will come with refinement of the software. Today’s health care technology has that Version 1.0 feel, and it is sure to get better.

... training students and physicians to focus on the patient despite the demands of the computers.

We also need far better collaboration between academic researchers and software developers to weed out bugs and reimagine how our work can be accomplished in a digital environment."

Commentary: Agreed again. But, I believe that technologist just can't dump these systems into the healthcare environments without significant follow-up research to insure that these systems provide or suggest the correct treatment programs and effectively monitor patients. Investment in systems like these will be cost effective and improve lives, but only if the necessary level of care and follow-up is performed.


5. "... Boeing’s top cockpit designers, who wouldn’t dream of green-lighting a new plane until they had spent thousands of hours watching pilots in simulators and on test flights. This principle of user-centered design is part of aviation’s DNA, yet has been woefully lacking in health care software design."

Commentary: All this is true. And as noted above that it would be a good idea to do more extensive research on medical systems before we deploy them to the field as well. That this is not done may be a regulatory issue that the FDA has not required the kind of rigorous research as performed in aircraft cockpit design. They should require more research in real or simulated environments. Right now, all that appears to be required is a single verification and single validation test before allowing commercialization. I think it would be valuable for regulators to require more research in real or simulated settings before allowing companies to commercialize their products.

Or, requiring more extensive follow-up research. Grant companies the right to sell their medical products on a probationary basis for (say) 1 year after receiving initial commercialization certification. During that year, the company must perform follow-up research on how their medical product performs in real environments. If there are no significant problems ... such as overly abundant number of false alarms ... then the product no longer on probation and would be considered fully certified for commercialization.
However, if significant problems emerge, the FDA could:

a) continue to keep the product in a probationary status pending correction of those problems and another year of follow-up research or

b) it could require the withdrawal of the product from sale. A product that had been withdrawn would have to go through the entire commercialization certification process just as if it were a new product before commercialization and sale would be allowed.


A final thought ... I think there's a reality in commercial aviation that is not true in medicine. If commercial aircraft killed and injured as many people as are killed and injured by medical practitioners, then the commercial aviation would come to a halt. People would refuse to fly because they perceive it to be too dangerous. But, if you're sick, then you have little choice but the clinic, ER or hospital.

Are Electronic Prescription Systems Failing to Trap Errors?

A Brief Introduction

Before I jump into the topic of electronic prescription systems, I want to make known how I came across the article I am about to post. I am creating a website that includes a substantial portion of the human factors related work I have produced over the years. That website also includes posting articles on the home page related specifically to human factors - and that includes article related to medical errors: a topic of interest to me.

The new human factors website is not yet ready for viewing. I have just created a usable home page. The bulk of the work is to come. I'll post the address when it's reached a usable state.


What's Going on with Electronic Prescription Systems?


Bloomberg news recently reported the results of a study that indicated that prescription errors are as frequent whether handwritten or written through an electronic prescription system. Here is the address of the Bloomberg article:
http://www.bloomberg.com/news/2011-06-29/errors-occur-in-12-of-electronic-drug-prescriptions-matching-handwritten.html

I have not yet had the opportunity to read the study. However, I shall and I'll continue to update this blog on this topic based on what I find. 


With respect to the Bloomberg article, this quote caught my eye:


"The most common error was the omission of key information, such as the dose of medicine and how long or how many times a day it should be taken, the researchers said. Other issues included improper abbreviations, conflicting information about how or when to take the drug and clinical errors in the choice or use of the treatment, the researchers said."


I have been a human factors professional for a long time and as I read the quote above my jaw dropped. The errors described in the quote are some of the most fundamental and easily trappable and correctable errors. It seems beyond belief that an electronic prescription system would allow a user to make such errors. In the environments where I have worked, I have designed and installed subsystems to insure that users do not make the kinds of errors as described in the Bloomberg article. When I have a chance to read the report, I'll cover specific errors, their detection and correction. And means to insure that patients are not harmed.


Here's a link to another publication that reported on the same study:


http://www.eurekalert.org/pub_releases/2011-06/bmj-oep062811.php

HE-75, Usability and When to Prototype and Usability Test: Take 1

Prototyping and Testing will be a topical area where I shall have much to contribute.  Expect numerous articles to appear on this topic.

I had a discussion a few days ago with one of my colleagues who has worked as a user interface designer, but has little knowledge of human factors.  He was completely unaware of the concepts of "top-down" and "bottom-up" processes to user interface design.  I provide for you the essence of that discussion.

Top-Down Approach

The top-down approach begins with a design.  Most often the initial design is a best or educated guess based on some set of principles.  Could be aesthetics or "accepted" standards of good design, or something else.  The design is usability and/or acceptance tested in some manner.  (Anywhere from laboratory testing to field-collected data.)  In response to the data, the design reworked.  The process is continual.  Recent experience has suggested that the top-down approach has become predominant design methodology, particularly for the development of websites.

Top-down is a valid process, particularly for the deployment of new or unique products where the consequences of a failed design do not lead to serious consequences.  It can get a design into user hands more quickly.  The problem with a top-down approach (when practiced correctly) is that it relies on successive approximations to an ill-defined or unknown target.  To some degree it's similar to throwing darts blindfolded with some minimal correction information provided after each throw.  The thrower will eventually hit the bull's eyes, but it may take lots and lots of throws.

The top-down approach may have a side benefit in that it can lead to developing novel and innovative designs.  Although, it can have the opposite effect when designs are nothing more than "knock-offs" of the designs from others.  I have seen both coming out of the top-down approach.

Bottom-Up Approach

HE-75 teaches the use of a bottom-up approach where first one defines and researches the targeted user population.  Contextual Inquiry is also a bottom-up approach.  Since I have already discussed researching the targeted user population in depth, I'll not cover it here.  

With the bottom-up approach, the target is clear and understood.  And tailoring a design to the user population(s) should be a relatively straight forward process.  Furthermore, the bottom-up approach directly addresses the usefulness issue with hard data and as such, more likely to lead to the development of a system that is not only usable, but useful.

Useful vs. Usable

I'll address this topic more deeply in another article.  It suffices to say that usability and usefulness are distinctly different system qualities.  A system may be usable, that is, the user interface may require little training and be easy to use, but the system or its capabilities are not useful.  Or, and this is what often happens particularly with top-down approaches, much of what the system provides is not useful or extraneous.

Personal Preference

I am a believer in the bottom-up approach.  It leads to the development of systems that are both usable and useful sooner than the top-down approach.  It is the only approach that I would trust when designing systems where user error is of particular concern.  The top-down approach has its place and I have used it myself, and will continue to use it.  But, in the end, I believe the bottom-up approach is superior, particularly in the medical field. 

HE-75 Topic: Meta Analysis

The definition of a "meta-analysis" is an analysis of analyzes.  Meta analyzes are often confused with a literature search, although a literature search is often the first step in a meta-analysis.

A meta-analysis is a consolidation of similar studies on a single, well defined topic.  The each study may have covered a variety of topics, but with the meta-analysis, each study will have addressed the common topic in depth and collected data regarding it.

The meta-analysis is a well-respected means of developing broad-based conclusions from a variety of studies.  (I have included a book on the topic at the end of this article.)  If you search the literature, you will note that meta-analyzes are often found in the medical literature, particularly in relationship to the effectiveness or problems with medications.

In some quarters, the meta-analysis is not always welcome or respected.  Human factors (Human engineering) is rooted in experimental psychology, and meta-analyzes are not always respected or well-received in this community.  It is work outside of the laboratory.  It is not collecting your own data, but using the data collected by others, thus the tendency has been to consider the meta-analysis as lesser.

However, the meta-analysis has a particular strength in that it provides a richer and wider view than a single study with a single population sample.  It is true that the studies of others often do not directly address all the issues that researchers could study if those researchers performed that research themselves.  In other words, the level and the types of research related controls were employed by the researchers themselves.  But, again, the meta-analysis can provide a richness and the numeric depth that a single study cannot provide.

Thus the question is, to use or not to use a meta-analysis when collecting data about a specific population?  Should a meta-analysis be used in lieu of collecting empirical data?  

Answer.  There are no easy answers.  Yes, a meta-analysis could be used in lieu of an empirical analysis, but only if there are enough applicable studies recently performed.  However, I would suggest that when moving forward with a study of a specific, target population that the first response should be to initiate a literature search and perform some level of a meta-analysis.  If the data is not available or is incomplete, then the meta-analysis will not suffice.  But, a meta-analysis is always a good first step, and a relatively inexpensive first step, even if the decision is made to go forward with an empirical study.  The meta-analysis will aid in the study's design and data analysis.  And will act as a guide when drawing conclusions.

Additional Resources

Meta Analysis: A Guide to Calibrating and Combining Statistical Evidence (Wiley Series in Probability and Statistics)

Remote Monitoring and Preventing Unnecessary ICD Shocks

In 2009 there was an interesting editorial written by Joseph E. Marine from Johns Hopkins University School of Medicine, published in the journal, Europace (European Society of Cardiology).  The title of the editorial was "Remote monitoring for prevention of inappropriate implantable cardioverter
defibrillator shocks: is there no place like home?"  The entire article can be found at the following location: http://www.europace.oxfordjournals.org/content/11/4/409.full.pdf
  
For those of you unfamiliar with ICD's (implantable cardioverter
defibrillator), the ICD delivers a relatively high-voltage shock to the heart when conditions indicate that the heart may be about to go ventricle fibrillation (a rapid irregular heartbeat that will likely lead to death) or that the heart ceases beating.  The latter condition is easily detected, however, determining the former condition is more difficult.  Because the conditions are not always clear, ICD (and a companion system, the CRT-D) too frequently deliver shocks unnecessarily. (I have discussed issues related to detection in other articles in the blog.  Here are the links to those discussions:  http://medicalremoteprogramming.blogspot.com/2009/11/remote-monitoring-sensitivity-and.html, http://medicalremoteprogramming.blogspot.com/2009/11/remote-monitoring-update-to-sensitivity.html
http://medicalremoteprogramming.blogspot.com/2009/11/remote-monitoring-predictability.html)  Another reason that an ICD might deliver unnecessary shocks would be because of sensor lead failure or near failure. 


Joseph Marine examined the value of remote monitoring to the prevention of unnecessary shocks.  He concluded that remote monitoring was particularly suited to providing early detection of failing sensor leads.  However, ...

[f]inally, most inappropriate ICD shocks are not caused by
lead failure, but rather by supraventricular arrhythmias, and this
study does not provide any evidence that home monitoring
reduces risk of inappropriate shocks from this cause.

In other words, remote monitoring could not aid with improving the false positive rate - the delivery of unnecessary shocks.

To those who have not been involved with ICDs, it may seem that the delivery of an unnecessary may not be so bad given the alternative, that a failure to deliver a shock will likely lead to the patient's death.  And there are many cardiologists who will argue the case for a "hair-trigger" system - acceptance of false positives, but no acceptance of false negative: that is a failure to deliver a shock when conditions warrant.

However, unnecessary shocks will do damage over time.  Furthermore, those patients who have received a shock describe it as feeling like "... a mule kicked" them in the chest.  I know of situations where patients who a received shocks eventually have the ICD removed. 

So, I want to make the case to the medical device industry that remote monitoring may be the key to solving the false positive problem.  In that the data that remote monitoring systems collect and transmit may lead to better detection and discrimination.  In addition with reference to my article on prediction, remote monitoring may enable physicians to tune ICDs based on specific predecessor events that could enable remotely adjusting the parameters on the ICD to allow better targeting.


I'm not an expert in this area.  However, I know enough about indicator conditions in other areas that can be used to adjust systems and improve their accuracy.

Remote Monitoring/Programming and Diabetes Management

Diabetes management is a personal area of concern for me.  No, I'm not diabetic.  However, my late mother-in-law was.  She had Type II diabetes; however, she was not overweight.  She died of a sudden cardiac arrest that was a direct result of her diabetes.  Although she did a great deal to manage her diabetes, her insulin would swing widely.  Those wide swings damaged her heart muscles leading to a cardiac arrest.  I can't help but believe if remote monitoring had been available to her, that she should would be alive today.

In the past my primary topical area has been cardiac rhythm management.  I plan to broaden my focus. Diabetes management using remote monitoring and even remote programming will be a topical area of increasing focus in this blog.  In later weeks I plan to branch out into COPD.

For those of you who have domain expertise in diabetes management and COPD, I would appreciate your comments.  You can make your comments in the comment area of this blog or email them to me.  Whatever way you feel the most comfortable.

To get things started, I have three links that I would like share.  The first link is a blog article titled, "Finding patterns in diabetes treatment may be key for telemedicine."  The article is a brief discussion about a presentation by Dr. David Klonoff of Mills-Peninsula Health Center and UC San Francisco.  His focus was on Type I diabetics, however, I believe what he discussed has significant implications for Type II diabetics as well.  Dr. Klonoff's interest is technology "...for automatic measurement of blood glucose, automatic dose calculation, and automatic insulin delivery."  From the article ...

For this ideal scenario to develop, five technologies need to be solved, and Klonoff sees printed electronics being used in every one:

• Self-monitoring of blood glucose
• Continuous (and ultimately non-invasive) monitoring of blood glucose
• Alternate routes for delivering insulin rather than needles, such as micro-needles. (Klonoff referred to work being done at UC Berkeley; I saw some demonstrated at the University College Cork/Ireland (PDF poster here) although using traditional semiconductors, not printed electronics.)
• Artificial pancreas
• Telemedicine

 In the quotation above, there are several links.  The one of greatest interest to me and to this forum, is the "non-invasive" link.  This will link you to an article titled, "The Search for Noninvasive Glucose Technology That Works: Where It Stands Now".


The article is a discussion of a need for a means for non-invasive monitoring of glucose levels.  The capability of having a non-invasive means of monitoring glucose levels would go a long ways towards supporting automatic, remote monitoring of glucose levels.  This could be an extension of the body area networks work (BANs).  So if anyone has any ideas in this area, apparently this is a wide open area for invention.

Finally, I want to provide a link to a brief report by the Whittier Institute of Diabetes.  The report is undated, but a brief review of the document's properties indicated that it was created in 2004.  It's not as recent as I would like, however, I believe that it's findings are relevant.  In summary, it showed that even relatively crude means for monitoring diabetes could lead to some positive outcomes at relatively low cost. 

 

Remote Monitoring: Sensitivity and Accuracy ... using wine tasting as a model

This article focuses on measurement accuracy, sensitivity and informativeness.  Sometime later I shall follow will an article that will focus on predictability.  

I discuss measurement accuracy, sensitivity and informativeness in this article in the abstract and use an example, wine tasting. However, in later articles when I drill-down into specific measurements provided by remote monitoring systems.  I shall make reference to concept foundation articles such as this one when I discuss specific measurements and measurement systems.



For remote monitoring to be a valuable tool, the measurements must be informative.  That is, they must provide something of value to the monitoring process - whether that monitoring process is an informed and well trained person such as a physician or software process.  However, there are conditions that must first be met before any measurement can be considered informative.

For any measurement to be informative, it must be accurate.  It must correctly measure whatever it was intended to measure.  For example, if the measurement system is designed to determine the existence of a particular event, then it should register that the event occurred and the number of times that it did occur.  Furthermore, it should reject or not respond when conditions dictate that the event did not occur - that is, it should not report a false positive.  This is something that I covered in detail on my article on Signal Detection.  Measurement extend beyond mere detection and to the measurement tied to a particular scale, e. g., such as the constituents in a milliliter of blood.


A constituent of accuracy is granularity.  That is, how fine is the measurement and is it fine enough to provide meaningful information.  Measurement granularity can often be a significant topic of discussion, particularly when defining similarities and differences.  For example, the world class times in swimming are to the hundredth of second.  There have been instances when the computer that sensed that two swimmers touched the end simultaneously and that the times were identical.  (I can think of a particular race in the last Olympics that involved Michael Phelps and the butterfly.)  At the resolution of the computer touch-timing system (and I believe it's down to a thousandth of a second), the system indicated that both touched simultaneously and that they had identical times.  However, is that really true?  If we take the resolution down to a nanosecond, one-billionth of a second, did they touch simultaneously?  

However, at the other end, if measurements are too granular, do they lose their meaningfulness?  This is particularly true when defining what is similar.  It can be argued that with enough granularity, every measurement will differ from all other measurements on that dimension. How do we assess similarities because assessing similarities (and differences) is vital to diagnosis and treatment.


We often make compromises when in comes to issues of granularity and similarity by categorizing.  And often times, categorization and assessments of similarities can be context-specific.  This is something that we do without thinking.  We often assess and reassess relative distances.  For example,  Los Angeles and San Diego are 121 miles from each other.  (I used Google to find this distance.)  To people living in either city, 121 miles is a long distance.  However, to someone is London, England, these two cities would seem to be nearly in the same metropolitan area.  They appear within the same geographic area from a far distance. 



Sensitivity is a topic often unto itself.  Since I discussed it at some length when I discussed Signal Detection, I shall make this discussion relatively short.  In the previous discussion, I discussed the issue related to a single detector and its ability to sense and reject.  I want to add the dimension of multiple detectors and the capability to sense based on multiple inputs.  In this case I am not discussing multiple trials to test a single detector, but multiple measures on a single trial.  Multiple measurements on different dimensions can provide greater sensitivity when combined even if the accuracy and sensitivity of each individual measurement system is less accurate and sensitive than the single measurement system.  I'll discuss this more in depth in a later article.


Informativeness ... this has to do with whether the output of the measurement process - its accuracy (granularity) and sensitivity - provides one with anything of value.  And determining the value depends on what you need that measurement to do for you.  I think my example provides a reasonable and accessible explanation.


Wine Tasting - Evaluating Wine


Over the years, people interested in wine have settled on a 1-100 scale - although, I do not know of an instance where I have seen anything less than an 80 rating.  (I am not a wine expert by any stretch of the imagination.  I know enough to discuss it, that's all.  If you're interested, here's an explanation, how ever they will want to sell you bottles of wine and some companies may block access, nevertheless, here's the link: http://www.wine.com/v6/aboutwine/wineratings.aspx?ArticleTypeId=2.)   Independent or "other" wine raters use a similar rating system.  Wine stores all over the US often have their own wine rater who "uses" one of these scales.  In theory, you'll note that they're reasonably similar.  In practice, they can be quite different.  Two 90 ratings from different wine raters don't always mean the same thing.


So, what is a buyer to do?  Lets look at wine rating in a mechanistic way.  Each wine rater is a measuring machine who is sensitive to the various constituents of a wine and how those constituents provide an experience.  Each rating machine provides us with a single number and often a brief description of the tasting experience.  But, for most people buying wine, it's the number that's the most important - and can often lead to the greatest disappointment.  When we're disappointed, the measurement has failed us.  It lacks informativeness.

How to remedy disappointment of expectation and often times, over payment?  I think of four ways:

1. Taste the wine yourself before you buy it.  The wine should satisfy you.  You can determine if it's worth the price.  However, I've met many who are not always satisfied with this option for a variety of reasons, ranging from they do not trust their own tastes or lack of "wine knowledge" to the knowing that they are not in a position to taste the wide variety of wines available to professional wine tasters, and thus are concerned about "missing out."  Remote monitoring provides a similar situation.  A patient being remote monitored is not in the presence of the person doing the monitoring, thus the entire experience of seeing the patient along with the measurement values is missing.  However, remote monitoring provides the capability to provide great deal of information about many patients without the need to see each individual.  The problem is, the person doing the monitoring needs to trust the measurements from remote monitoring.
   
2. Find a wine rater who has tastes similar to yours.  This might take some time or you might get lucky and find someone who likes wine the way you like it.  Again, this all boils down to trust.
   
3. Ask an expert at the wine store.  The hope is that the person at the store will provide you with more information, ask you about your own tastes and what you're looking for.  Although this is not experiential information, you are provided with more information on more dimensions with the ability to re-sample on the same or different dimensions (i. e., ask a question and receive an answer).  In this sense, you have an interactive measurement system.  (At this juncture, I have added by implication remote programming to mix.  Remote programming involve adjusting, tuning or testing additional remotely monitored dimensions.  In this sense, the process of remote monitoring can be dynamic, inquiry-driven.  This is a topic for later discussion.)
4. Consolidate the ratings of multiple wine raters.  Often several wine raters have rated the same wine.  This can get fairly complicated.  In most cases not all wine raters have rated the same wine and you'll probably get a different mix of raters for each wine.  This too may involve some level of tuning based on the "hits" and "misses." 
   

This ends this discussion of measurement.  Measurement is the foundation of remote monitoring.  For remote monitoring what its measuring and the accuracy and sensitivity of that measurement and whether that measurement is informative is key to its value.  We've also seen a place for remote monitoring as a means for getting at interesting measurements; changing measurement from a passive to an active, didactic process.


Next time I discuss a recent development with respect to physiological measuring systems.  Here's a link to an article that I believe many will find interesting.  http://mobihealthnews.com/5142/tedmed-wireless-health-has-killed-the-stethoscope/