Article: Establishing Trust in Wearable Medical Devices

Many of the topics covered in this article I have covered in this blog. Most recently, my discussion of signal detection and the Apple Watch (https://medicalremoteprogramming.blogspot.com/2019/12/signal-detection-and-apple-watch.html) that I suggest that you read after reading this article from Machine Design.

Here's the link to the article: https://www.machinedesign.com/medical-design/article/21122718/establishing-trust-in-healthcare-wearables?utm_source=MN+MD+Medical+Design&utm_medium=email&utm_campaign=CPS200206038&o_eid=7596G0176945B2B&rdx.ident%5Bpull%5D=omeda%7C7596G0176945B2B&oly_enc_id=7596G0176945B2B

Here are a few quotes from the article.

To say that we can get personal health insight from continuous monitoring presumes a “chain of trust.” In other words: 

• The interpretation of any data must not only be accurate but reliable. The challenge lies in handling “borderline” data. Any interpreting strategy or algorithm faces data sets that it finds ambiguous. For an algorithm to be reliable, users must be able to quantitatively understand its detection limits and error characteristics.

• The data and/or its interpretation must reliably reach the decision-maker for it to become actionable.

• The data must be correctly associated with historical records of the patient for it to have context.

• The data must be proven to be authentic to trigger any meaningful action.

However, using clinical equipment to capture vital signs that are representative of the wearable use cases is often difficult and sometimes inaccurate. To avoid a rash of false positives or false negatives, one must carefully select the population of test subjects and carefully develop the representative use cases. It’s also important to compare data from the patient’s own history or baseline, keeping in mind that this baseline isn’t static as the patient ages and undergoes other changes.

Apple Watch 5: Heart Monitoring Capabilities -- Afib

The Apple Watch 5 has a heart rhythm monitoring capability that is tuned to detecting the presence of atrial fibrillation, AKA, Afib. Apple categorically states that the watch is unable to detect a heart attack. (And by implication, the likelihood of a heart attack occurring within minutes or hours.)

You have to manually enable your heart monitoring system (Watch and iPhone) to detect Afib. This not part of the default configuration. Here's the link for setting it up: https://support.apple.com/en-us/HT208931#afib

Here's what Apple says about the capabilities of their system and note that it requires both the Apple Watch 5 and an iPhone: 

INDICATIONS FOR USE (NON-EU REGIONS)

The Irregular Rhythm Notification Feature is a software-only mobile medical application that is intended to be used with the Apple Watch. The feature analyzes pulse rate data to identify episodes of irregular heart rhythms suggestive of atrial fibrillation (AF) and provides a notification to the user. The feature is intended for over-the-counter (OTC) use. It is not intended to provide a notification on every episode of irregular rhythm suggestive of AF and the absence of a notification is not intended to indicate no disease process is present; rather the feature is intended to opportunistically surface a notification of possible AF when sufficient data are available for analysis. These data are only captured when the user is still. Along with the user’s risk factors, the feature can be used to supplement the decision for AF screening. The feature is not intended to replace traditional methods of diagnosis or treatment.

The feature has not been tested for and is not intended for use in people under 22 years of age. It is also not intended for use in individuals previously diagnosed with AF.

INTENDED PURPOSE (EU REGION)

Intended Use

The Irregular Rhythm Notification Feature (IRNF) is intended to pre-screen and notify the user of the presence of irregular rhythms suggestive of atrial fibrillation (AF). The feature can be used to supplement a clinician’s decision to screen for possible AF. The feature is intended for over-the-counter (OTC) use.

The feature has not been tested for and is not intended for use in people under 22 years of age. It is also not intended for use in individuals previously diagnosed with AF.

Indications

The feature is indicated to pre-screen for irregular rhythms suggestive of AF for anyone aged 22 years and over.

USING THE IRREGULAR RHYTHM NOTIFICATION FEATURE Set-Up/On-boarding

• Open the Health app on your iPhone.
• Navigate to “Heart”, then select “Irregular Rhythm Notifications”.
• Follow the onscreen instructions.

Receiving a Notification

Once the feature is turned on, you will receive a notification if the feature identified a heart rhythm suggestive of AF and confirmed it on multiple readings.
If you have not been diagnosed with AF by a GP, you should discuss the notification with your doctor.

All data collected and analysed by the Irregular Rhythm Notification Feature is saved to the Health app on your iPhone. If you choose to, you can share that information by exporting your health data in the Health app.

SAFETY AND PERFORMANCE

In a study of 226 participants aged 22 years or older who had received an AF notification while wearing Apple Watch and subsequently wore an electrocardiogram (ECG) patch for approximately one week, 41.6% (94/226) had AF detected by ECG patch. During concurrent wear of Apple Watch and an ECG patch, 57/226 participants received an AF notification. Of those, 78.9% (45/57) showed concordant AF on the ECG patch and 98.2% (56/57) showed AF and other clinically relevant arrhythmias. A total of 370 irregular rhythm notifications with readable ECG patch data was received by the 57 participants. Of those 370 notifications, 322 (87.0%) were assessed to be AF, 47 (12.7%) were arrhythmias other than AF and 1 (0.3%) was sinus rhythm. These results demonstrate that, while in the majority of cases the notification will accurately represent the presence of AF, in some instances, a notification may indicate the presence of an arrhythmia other than AF. No serious device adverse effects were observed.

CAUTIONS

The Irregular Rhythm Notification Feature cannot detect heart attacks. If you ever experience chest pain, pressure, tightness or what you think is a heart attack, call emergency services.

The Irregular Rhythm Notification Feature is not constantly looking for AF and should not be relied on as a continuous monitor. This means the feature cannot detect all instances of AF and people with AF may not get a notification.

• Not intended for use by individuals previously diagnosed with AF.
• Notifications made by this feature are potential findings, not a complete diagnosis of cardiac conditions. All notifications should be reviewed by a medical professional for clinical decision making.
• Apple does not guarantee that you are not experiencing an arrhythmia or other health conditions even in the absence of an irregular rhythm notification. You should notify your GP if you experience any changes to your health.
• For best results, make sure your Apple Watch fits snugly on top of your wrist. The heart rate sensor should stay close to your skin.

From the information provided I am unable to determine how the Afib monitoring system detects Afib. It does seem use an additional capability beyond heart rate system, but from what little I can understand, it uses software running on either the watch and/or the iPhone and uses as input the data from the heart rate system.

I have no idea what algorithms the Apple heart monitoring system is using to detect atrial fibrillation (AF), but if you read the study above, you'll note that apparently, the Apple system has significant false positive rate. Walking through the study, to qualify as a subject for the study, you had to have had a positive indication of AF by the Apple system. That's the one clear message from the study. Another clear message is that both the Apple system and the AF patch can detect heart arrhythmia  other than AF, but what those were is unclear. Unfortunately the way the data is reported does not provide full clarity into the procedure and results. So there's not much more that I can comfortably conclude.

I feel comfortable stating that if you're wearing the Apple Watch and using the AF detection system and you get an AF indication, it's worth your time to get it checked out even knowing full well that the indication is more than likely to be a false positive.

However, high AF false positive rate of nearly 60% is concerning from the standpoint of those who have the Apple AF detection system activated and receive false positive indications. Information like this gets around and users may have tendency to ignore the AF indications when in fact they should be paying attention to them. To curb the possibility that someone ignores an accurately reported AF indication from the Apple system, it would behove Apple to include with the AF notification a check list displayed on the iPhone the walk the user through to determine if in fact this is an AF event.

Welch Allyn Published Patent Application: Continuous Patient Monitoring

I decided to review this patent application in light of the New York Times Opinion piece I commented on. Here's the to my commentary: http://medicalremoteprogramming.blogspot.com/2015/03/new-york-times-opinion-why-health-care.html

Also, I've gone back to the origins of this blog ... reviewing patents. The first patent I reviewed was one from Medtronic. Here's the link: http://medicalremoteprogramming.blogspot.com/2009/09/medtronics-remote-programming-patent.html

The issue raised of particular interest was the high "false alarm" rate generated reported by the author that would lead medical professionals to disregard warnings generated by their computer systems. I wrote that I wanted to follow-up on the issue of false alarms.

The patent application (the application has been published, but a patent has not yet been granted) describes an invention intended to 1) perform continuous automated monitoring and 2) lower the rate of false alarms.

Here are the details of the patent application so that you can find it yourself if you wish:

The continuous monitoring process from a technical standpoint is not all that interesting or new. What is interesting is the process they propose to lower the false alarm rate and determine whether this process in turn will not lower the false negative rate.

Proposed Process of Lowering False Alarms

As mentioned in my earlier article, it appears that false alarms have been a significant issue for medical devices and technology. Systems that issue too many false alarms issue warnings that are often dismissed or ignored. Or waste the time and attention of caregivers who spend time and energy on responding to a false alarm. This patent application is intended to reduce the number of false alarms. However, as I mentioned earlier, can it do that by not increasing the number of false negatives, that is, failure to detect when there is a real event where an alarm should be going off.

Getting through all the details of the patent application and trying to make sense of what they're trying to convey, the following is what I believe is the essence of the invention:

• Measurement a sensor indicates an adverse patient conditions and an alarm should be initiated.
• Before the alarm is initiated, the system cross-checks against other measurements that are: 

              1) from another sensor measuring essentially the same physiological condition as the
                  sensor that detected the adverse condition, the measurement from the second sensor
                  would confirm the alarm condition or indicate that an alarm condition should not exist; or
              2) from another sensor or sensors that take physiological measurements that would confirm
                  the alarm condition from the first sensor or indicate that an alarm condition should not
                  exist.

In this model at least two sensors must provide measurements that point to an alarm state.

Acceptable Model for Suppressing False Alarms and Not Increasing False Negatives?

Whatever you do in this domain of detecting adverse patient conditions, you don't want to lower your accuracy of detecting the adverse condition. That is, increase your false negative rate.

So is this one way of at least maintaining your currently level of detecting adverse events and lowering your false alarm rate? On the face of it, I don't know. But it does appear that it might be possible.

One of the conditions the inventors suggest that initiates false alarms are those times when patients move or turn over in their beds. This could disconnect a sensor or cause it to malfunction. A second sensor taking the identical measurement may not functioning normally and have a measurement from the patient indicating that nothing was wrong. The alarm would be suppressed ... although, if a sensor was disconnected, one would expect that there would be a disconnected sensor indicator would be turned on.

Under the conditions the inventors suggest, it would appear that cross checking measurements might reduce false positives without increasing false negatives. I would suggest that care should be given to insure that a rise in false negative rates do not increase. With array of new sensors and sensor technology becoming available, we're going to need to do a lot of research. Much of it would be computer simulations to identify those conditions were an adverse patient condition goes undetected or suppressed by cross-checking measurements.

Post Script

For those who do not know, I am on numerous patents and patent applications (pending patents). Not only that I have written the description section of a few patent applications. So I have a reasonable sense of what is what is not patentable ... this is in spite of the fact that I'm an experimental, cognitive psychologist and we're not general known for our patents.

So, what is my take on the likelihood that this applications will be issued a patent? My sense is not likely. As far as I can tell there's nothing really new described in this application. The core of the invention, the method for reducing false alarms, is not new. Cross-checking, cross-verifying measurements to determine if the system should be in an alarm state is not new. As someone who has analyzed datasets for decades, one of first things that one does with a new dataset is to check for outliers and anomalies - these are similar alarm conditions. One of the ways to determine whether an outlier is real, is to cross check against other measures to determine if they're consistent with and predictive of the outlier. I do not see anything that is particularly new or passes what known in patent review process as the "obviousness test." For me cross checking measures does not reach the grade of patentability.

Remote Monitoring: Sensitivity and Accuracy ... using wine tasting as a model

This article focuses on measurement accuracy, sensitivity and informativeness.  Sometime later I shall follow will an article that will focus on predictability.  

I discuss measurement accuracy, sensitivity and informativeness in this article in the abstract and use an example, wine tasting. However, in later articles when I drill-down into specific measurements provided by remote monitoring systems.  I shall make reference to concept foundation articles such as this one when I discuss specific measurements and measurement systems.



For remote monitoring to be a valuable tool, the measurements must be informative.  That is, they must provide something of value to the monitoring process - whether that monitoring process is an informed and well trained person such as a physician or software process.  However, there are conditions that must first be met before any measurement can be considered informative.

For any measurement to be informative, it must be accurate.  It must correctly measure whatever it was intended to measure.  For example, if the measurement system is designed to determine the existence of a particular event, then it should register that the event occurred and the number of times that it did occur.  Furthermore, it should reject or not respond when conditions dictate that the event did not occur - that is, it should not report a false positive.  This is something that I covered in detail on my article on Signal Detection.  Measurement extend beyond mere detection and to the measurement tied to a particular scale, e. g., such as the constituents in a milliliter of blood.


A constituent of accuracy is granularity.  That is, how fine is the measurement and is it fine enough to provide meaningful information.  Measurement granularity can often be a significant topic of discussion, particularly when defining similarities and differences.  For example, the world class times in swimming are to the hundredth of second.  There have been instances when the computer that sensed that two swimmers touched the end simultaneously and that the times were identical.  (I can think of a particular race in the last Olympics that involved Michael Phelps and the butterfly.)  At the resolution of the computer touch-timing system (and I believe it's down to a thousandth of a second), the system indicated that both touched simultaneously and that they had identical times.  However, is that really true?  If we take the resolution down to a nanosecond, one-billionth of a second, did they touch simultaneously?  

However, at the other end, if measurements are too granular, do they lose their meaningfulness?  This is particularly true when defining what is similar.  It can be argued that with enough granularity, every measurement will differ from all other measurements on that dimension. How do we assess similarities because assessing similarities (and differences) is vital to diagnosis and treatment.


We often make compromises when in comes to issues of granularity and similarity by categorizing.  And often times, categorization and assessments of similarities can be context-specific.  This is something that we do without thinking.  We often assess and reassess relative distances.  For example,  Los Angeles and San Diego are 121 miles from each other.  (I used Google to find this distance.)  To people living in either city, 121 miles is a long distance.  However, to someone is London, England, these two cities would seem to be nearly in the same metropolitan area.  They appear within the same geographic area from a far distance. 



Sensitivity is a topic often unto itself.  Since I discussed it at some length when I discussed Signal Detection, I shall make this discussion relatively short.  In the previous discussion, I discussed the issue related to a single detector and its ability to sense and reject.  I want to add the dimension of multiple detectors and the capability to sense based on multiple inputs.  In this case I am not discussing multiple trials to test a single detector, but multiple measures on a single trial.  Multiple measurements on different dimensions can provide greater sensitivity when combined even if the accuracy and sensitivity of each individual measurement system is less accurate and sensitive than the single measurement system.  I'll discuss this more in depth in a later article.


Informativeness ... this has to do with whether the output of the measurement process - its accuracy (granularity) and sensitivity - provides one with anything of value.  And determining the value depends on what you need that measurement to do for you.  I think my example provides a reasonable and accessible explanation.


Wine Tasting - Evaluating Wine


Over the years, people interested in wine have settled on a 1-100 scale - although, I do not know of an instance where I have seen anything less than an 80 rating.  (I am not a wine expert by any stretch of the imagination.  I know enough to discuss it, that's all.  If you're interested, here's an explanation, how ever they will want to sell you bottles of wine and some companies may block access, nevertheless, here's the link: http://www.wine.com/v6/aboutwine/wineratings.aspx?ArticleTypeId=2.)   Independent or "other" wine raters use a similar rating system.  Wine stores all over the US often have their own wine rater who "uses" one of these scales.  In theory, you'll note that they're reasonably similar.  In practice, they can be quite different.  Two 90 ratings from different wine raters don't always mean the same thing.


So, what is a buyer to do?  Lets look at wine rating in a mechanistic way.  Each wine rater is a measuring machine who is sensitive to the various constituents of a wine and how those constituents provide an experience.  Each rating machine provides us with a single number and often a brief description of the tasting experience.  But, for most people buying wine, it's the number that's the most important - and can often lead to the greatest disappointment.  When we're disappointed, the measurement has failed us.  It lacks informativeness.

How to remedy disappointment of expectation and often times, over payment?  I think of four ways:

1. Taste the wine yourself before you buy it.  The wine should satisfy you.  You can determine if it's worth the price.  However, I've met many who are not always satisfied with this option for a variety of reasons, ranging from they do not trust their own tastes or lack of "wine knowledge" to the knowing that they are not in a position to taste the wide variety of wines available to professional wine tasters, and thus are concerned about "missing out."  Remote monitoring provides a similar situation.  A patient being remote monitored is not in the presence of the person doing the monitoring, thus the entire experience of seeing the patient along with the measurement values is missing.  However, remote monitoring provides the capability to provide great deal of information about many patients without the need to see each individual.  The problem is, the person doing the monitoring needs to trust the measurements from remote monitoring.
   
2. Find a wine rater who has tastes similar to yours.  This might take some time or you might get lucky and find someone who likes wine the way you like it.  Again, this all boils down to trust.
   
3. Ask an expert at the wine store.  The hope is that the person at the store will provide you with more information, ask you about your own tastes and what you're looking for.  Although this is not experiential information, you are provided with more information on more dimensions with the ability to re-sample on the same or different dimensions (i. e., ask a question and receive an answer).  In this sense, you have an interactive measurement system.  (At this juncture, I have added by implication remote programming to mix.  Remote programming involve adjusting, tuning or testing additional remotely monitored dimensions.  In this sense, the process of remote monitoring can be dynamic, inquiry-driven.  This is a topic for later discussion.)
4. Consolidate the ratings of multiple wine raters.  Often several wine raters have rated the same wine.  This can get fairly complicated.  In most cases not all wine raters have rated the same wine and you'll probably get a different mix of raters for each wine.  This too may involve some level of tuning based on the "hits" and "misses." 
   

This ends this discussion of measurement.  Measurement is the foundation of remote monitoring.  For remote monitoring what its measuring and the accuracy and sensitivity of that measurement and whether that measurement is informative is key to its value.  We've also seen a place for remote monitoring as a means for getting at interesting measurements; changing measurement from a passive to an active, didactic process.


Next time I discuss a recent development with respect to physiological measuring systems.  Here's a link to an article that I believe many will find interesting.  http://mobihealthnews.com/5142/tedmed-wireless-health-has-killed-the-stethoscope/