Submission of the Human Engineering File to the FDA and Other Regulatory Bodies: Part III

In this article we cover section 4 of how to file the human engineering file to the FDA:

4	Summary of known use problems  Known use problems with previous models of the subject device Known use problems with similar devices, predicate devices or devices with similar user interface elements Design modifications implemented in response to  post-market use error problems

Section 4: Summary of known use problems

First, if this device is an update and replacement for an earlier version of the same device, summarize by category (if appropriate) the use related problems that have been detected. These will likely be problems reported from users in the field. This section of the narrative should reference field reports or summaries that should be included with this submission to your approving regulatory body. All problems need not be included in the summary, but be sure that your summary discloses the most concerning and potentially harmful problems that the new release of your device corrects -- and point this out. The narrative need not include specifics on how the use problems of the previous version have been corrected, but be sure to include references to the documentation included with the submission where this is described.

Second, the device being submitted for approval may well be one of many functionally identical or nearly identical devices. In the narrative identify any significant problems found in any of these devices. Be sure to mention how your device avoids or overcomes any of the significant known problems of other devices. This is particularly important if your device is your first version of this type of device. If your device is the first of its kind, then consider referencing devices with similar characteristics including ones your company has created. And address any field detected use related problems and how they have been addressed in the design of the device being submitted for approval.

Collecting information about competitors devices can be a long process. However, most companies I know have people who do competitive market analysis. I have found that much of the information related to other companies products including many issues and concerns has been collected and cataloged by competitive market analysts. Furthermore, information from competitive market analysts can often be an input to the user interface design process. Finally, if the device being submitted for approval has solved many if not most of the user interface issues found in other's devices, that's a competitive edge and something that marketing should want to know about and share with customers and potential customers.

The one issue of concern will be how much overlap should there be between the foundation documentation and the summary included in the narrative. Since the narrative is a summary and a guide for the reviewer, it may be acceptable to have this summary be found only in the narrative. Or the narrative could include a brief summary and a pointer to the foundation document or documents. There is no set rule in this matter. Whatever you chose, you will need to make clear to your reviewer the path you have chosen to take to make this information available. 

Submission of the Human Engineering File to the FDA and Other Regulatory Bodies: Part II

These are the sections we'll cover in this article.

Sec	Contents
1	Conclusion The device has been found to be safe and effective for the intended users, uses and use environments. Brief summary of HFE/UE processes and results that support this conclusion Discussion of residual use-related risk
2	Descriptions of intended device users, uses, use environments, and training Intended user population(s) and meaningful differences in capabilities between multiple user populations that could affect user interactions with the device Intended use and operational contexts of use Use environments and conditions that could affect user interactions with the device     Training intended for users
3	Description of device user interface  Graphical representation of device and its user interface Description of device user interface Device labeling Overview of operational sequence of device and expected user interactions with user interface

With the exception of Section 1 each one of these sections will have a document or document attached to that section. Each section of the narrative document will be a summary of the foundational document or documents attached to that specific section. Make sure that the foundational documents are clearly referenced in each section.

Section Content Analysis

Section 1: Conclusion

This section is an abstract that includes the conclusion of the research program: that the device is "safe and effective" for those intended to use it. I suggest that this section include a brief description of the device at the beginning, its purpose and what medical issues it addresses. For example, if the medical device was a pacemaker, describe what it does, for whom it is intended and what medical issues it addresses. This description should be about one paragraph in length. And the paragraph should end with the "safe and effective" statement and that as to why this conclusion had been reached is briefly listed below in this section. You can add that a full and complete summary follows this section. 

A brief description of the device may well be helpful to the reviewer(s). Although submitted as a complete package that includes a full and detailed description of the device, the package is often split among several reviewers with specific specialties such as human factors. Therefore, the reviewer of the human engineering file may not read or even see the entire submission. Thus a brief description of the device may well provide some welcome context to the reviewer.

The brief summary of human engineering processes can be as simple as a list of steps that were performed. That could include observational studies/contextual analysis, structured interviews of members of the user population and/or subject-matter experts, prototype testing, use-error identification, etc.

Finally, this would be the place to disclose any lingering use-error problems that remain and how they're being addressed. Also, note that these lingering use-error problems have been determined to be low risk: before or after risk mitigations.

Section 2: Descriptions of intended device users, uses, use environments, and training

This section consists of a discussion who would be using the device -- their characteristics, training, education, certification, etc. -- where and how it would be used and the circumstances were it would be used. And finally, a description of the instructions or training that may be required in order to operate the device.

This section will likely be lengthy and comprehensive. Not as lengthy as the foundation document or documents, but it should include a significant portion of what that document(s) contains. You should include enough information to provide the reviewer with a full understanding of who is expected to use the device, how and when it should be used, where it's expected to be used and what positive outcomes would be expected when it is used. And finally what if any are the circumstances such as  the use environment that could how it might be used or effect whether or not it is used correctly.

Your foundational documents will likely include several specific device use scenarios. Include some of the most illustrative of those scenarios in your narrative. 

Devices are often targeted for use by specific classes of users in specific environments. For example, a device may be targeted for use by ICU nurses in the ICU. However, the device could conceivably be used in the general hospital setting by general practice nurses, especially inexperienced or recently graduated nurses. This section should discuss this possibility and what might be needed to be done to adapt the device to other environments and more groups of users with different skills and skill levels.

Finally, summarize the research evidence collected as well as the analysis performed that pertain to this section. Be sure to reference all the foundational documents as well as any internal or outside, published research not specifically performed for this device but applicable to this section. 

Section 3: Description of device user interface 

It may be worth considering lifting the entire description with the images from the foundation document and placing it in the narrative. And then consider what you can remove and still provide the reviewer with adequate information to evaluate the operation of the device. The entire user manual will likely not be necessary to be included in the narrative. You could consider having someone who is not familiar with the device read the description to determine if enough information has been provided. If the person can accurately describe how users would interact with the user interface in order to accomplish specific task (with the images of the user interface that you have provided), then your description is adequate.

Although not stated, this would be an acceptable section to summarize the design process, the evidence that lead to the finalized design and how that evidence was used to both create the initial design and iterate the design until it reached its final form. 

Submission of the Human Engineering File to the FDA and Other Regulatory Bodies

Background

About a year ago I was asked what I thought was the most difficult phase of the medical device human engineering process. Frankly, I'd never before considered such a question. I could not identify any phase of the process that I considered more difficult than any other. I considered whether early phase formative research and testing or risk identification, use errors and risk management would be the most difficult. No, actually for me these phases have always proven to be the most interesting phases in the process. Challenging, yes; difficult, no.

The questioner had an answer in mind: she believed that the validation testing was the most difficult. I disagreed. I believe that the validation testing was most often the least difficult of all the phases of the process. Why? Because validation comes at the end of the entire process and is based on all of the work that you've done previously to get one to the point of validation testing. Thus the procedure of a validation test should flow naturally and easily from the earlier work.

In the end, we agreed to disagree. However, the question never left me. 

I have finally come up with the answer. The most difficult phase of the process is the creation of the narrative for the regulatory reviewers. What I refer to is more than putting together the folder of documents of all of the human engineering related activities. It is the construction of a cohesive and understandable narrative the provides to a reviewer an overall view of the reasoning and logic of the steps taken and that the procedures performed that will demonstrate that the human engineering process was sound and resulting from it is a system that will be safe to use. This is the most difficult phase of the human engineering process, especially if writing the narrative comes at the end.

Human Engineering Pre-Market Submission

Here is the outline of what the FDA expects in a Human Engineering Pre-Market Submission as provided by the FDA on their website:

Sec.	Contents
1	Conclusion The device has been found to be safe and effective for the intended users, uses and use environments. Brief summary of HFE/UE processes and results that support this conclusion Discussion of residual use-related risk
2	Descriptions of intended device users, uses, use environments, and training Intended user population(s) and meaningful differences in capabilities between multiple user populations that could affect user interactions with the device Intended use and operational contexts of use Use environments and conditions that could affect user interactions with the device     Training intended for users
3	Description of device user interface  Graphical representation of device and its user interface Description of device user interface Device labeling Overview of operational sequence of device and expected user interactions with user interface
4	Summary of known use problems  Known use problems with previous models of the subject device Known use problems with similar devices, predicate devices or devices with similar user interface elements Design modifications implemented in response to  post-market use error problems
5	Analysis of hazards and risks associated with use of the device Potential use errors Potential harm and severity of harm that could result from each use error Risk management measures implemented to eliminate or reduce the risk Evidence of effectiveness of each risk management measure
6	Summary of preliminary analyses and evaluations Evaluation methods used Key results and design modifications implemented in response Key findings that informed the human factors validation test protocol
7	Description and categorization of critical tasks  Process used to identify critical tasks List and descriptions of critical tasks Categorization of critical tasks by severity of potential harm Descriptions of use scenarios that include critical tasks
8	Details of human factors validation testing Rationale for test type selected (i.e., simulated use, actual use or clinical study) Test environment and conditions of use Number and type of test participants Training provided to test participants and how it corresponded to real-world training levels Critical tasks and use scenarios included in testing Definition of successful performance of each test task  Description of data to be collected and methods for documenting observations and interview responses Test results: Observations of task performance and occurrences of use errors, close calls, and use problems  Test results: Feedback from interviews with test participants regarding device use, critical tasks, use errors, and problems (as applicable)   Description and analysis of all use errors and difficulties that could cause harm, root causes of the problems, and implications for additional risk elimination or reduction

Each one of the items listed will include one or more documents that address in some manner each one of the sub-points listed in each one of the items. You could place these documents into a folder, identify where they belong and submit this to the FDA or other regulatory body. However, I advocate something more: write an over arching narrative that takes the reader through the path of not only what was performed but why as well. The narrative can be and probably should be based on the outline above. (I'm sure there are exceptions, but beginning with the outline above should be your starting point.)

The narrative should provide a comprehensive flow that interconnects each of the phases and explains reasoning for what was done including the rationale behind the design and operation of the system as well as why it's appropriate for use by the identified user population as well as other likely populations who might encounter it.

Why Write a Narrative?

Is a narrative required as part of a submission? From all that I can tell: no, it's not a requirement as part of a submission to regulatory bodies. 

However, consider the fact that if you can't explain the logic of what you did to yourselves, how hard will it be for a reviewer to comprehend? And if the reviewer can't comprehend what you did -- including the reasoning and logic behind it -- could your submission be at risk for rejection? The answer is "yes," you may be putting your submission at risk for rejection.

The narrative is analogous to a completed jigsaw puzzle. A human engineering file without a narrative is analogous to just the jigsaw puzzle pieces. Yes everything is there, but what is it suppose to be? 

Submitting a human engineering file that includes a comprehensive narrative can insure that your submission is understandable: to you as well as your reviewers. It can insure that there are no gaps or issues that should have been included in your submission are left out. Again, going back to the jigsaw puzzle analogy, you don't know that you've got a missing piece or pieces until you've assembled the puzzle.

The narrative provides reviewers with framework to understand what you've done. Interestingly enough this will likely minimize any questions reviewer might have about your submission. And will likely minimize the likelihood that you'll get questions that you cannot answer.

One more thing to note: if your narrative is clear and comprehensive, it's likely that the reviewer or reviewers will often read no further or will simply scan the foundational documents to insure that the foundational documents do in fact support what is stated in the narrative. This could speed the regulatory review and acceptance. 

More Articles on This Topic

I'll be writing a series of articles on the topic of human engineering file submission narratives over the next week or two. I'll focus on specific areas of the narrative and discuss some of what I have done with regards to putting together narratives for submission to regulators. 

Commentary: HE-75 and IEC 62366 and Cleaning Up the Messes

I received a reminder recently when I was made aware of the International
Consortium of Investigative Journalists' database of medical device recalls of what human factors professionals working in the area of human engineering for medical devices are often called on to do: clean up the mess created by a failed design process that somehow failed to incorporate research. (Note that medical device development isn't the only domain where this kind of failure occurs, however, the impact of medical device failures can often result in fatalities.) The persons responsible for designing an awful, unusable and in some case, useless user interface expect the usability expert to come in, take one look and create a beautiful user interface. This is absurd!

Writing from my own perspective, there is nothing that a usability professional likes to do less than to correct a failed design that resulted from a failed design process. This week I was asked to save a group of programmers and user interfaced designers from the monstrosities that they had created. What was particularly strange was that the leader of the project thought that I could just redesign something by looking at what they had created. It was bizarre. Unfortunately, I had to deliver several harsh messages regarding the design process and the design, that were not well received. (Nevertheless, that is my job.)

Here is the point I want to make clear to anyone who reads this: Process and the resulting design should be considered as two sides of the same coin. The outcome of a good design process generally results in a good design. A nonexistent or poor design process often times leads to a poor design and a design that gets worse with each design iteration when attempts are made to fix problems or incorporate enhancements.

The processes and design direction provided by HE-75 and IEC 62366 can serve as a foundations for research and designing systems with user impacts within nearly any industry, particularly in those industries where the potential for harm is likely.

UK Perspective Regarding FDA Regulatory Requirements

A Linked-In colleague posted a link to this article. I read it and found it interesting enough to post the link and comment on it. It's by a UK publication and discusses the FDA regulatory process as it relates to Human Engineering requirements for device approval for commercialization.

Here's the link:
http://www.emdt.co.uk/daily-buzz/what-are-fda-usability-testing-requirements-device-approval?cid=nl.qmed01.20150325

In addition, I provide my own perspective on the article in the "Commentary" section below. I do not critique the article. I only attempt to expand on a few points from it.

But first, a brief summary of the article.

Article Summary

Medical errors have become an increasing concern of the FDA. I became interested in medical errors when I was a consultant at St. Jude Medical Cardiac Rhythm Division in Sylmar, CA. During my time at St. Jude (2009-2010), deaths by medical error were being reported as being 100,000 to 120,000 per year. Last year, I posted links to two articles that stated that deaths by medical errors could be closer to 400,000 per year. (http://medicalremoteprogramming.blogspot.com/2014/07/rip-death-by-medical-error-400000-year.html)

It has been noted by the FDA a large proportion of medical errors can be attributed to poorly designed medical device user interfaces. Since a fundamental mission of the FDA is increasing patient safety and reducing injuries and fatalities in the practice of medicine, the FDA has begun placing greater emphasis on improving the usability of medical device user interfaces.

This article provides measures that show the FDA's increasing emphasis on usability and human factors issues by showing the increasing frequency that companies seeking medical device clearance for the US market mention the terms "usability" and "human factors." Figure 1 from the article clearly shows the increasing usage of these terms in company filings.

The focus should be on the trends, not the absolute numbers because not all filing documents have been included in the count. But the trend clearly shows an increased emphasis by companies to increasingly use the terms "usability" and "human factors" in their filings with the FDA. The two figures that follow suggest the degree that companies have incorporated the FDA prescribed human factors engineering process and design guidance documentation.

The documents listed below are specifically targeted to defining and supporting the human factors engineering process and the development of the Human Engineer File that's included as part of a company's filing to the FDA.

• ISO 62366, Medical Devices - Application of Usability Engineering to Medical Devices
• AAMI / ANSI HE75:2009, Human Factors Engineering - Design of Medical Devices (General)

I'll discuss the documents above in greater detail and describe how they're intended to fit within the human factors engineering process when developing medical devices.

• IEC 60601-1-6 Medical electrical equipment - Part 1-6 General requirements for Safety - Collateral standard: Usability
• IEC 60601-1-8 Ed. 1, Medical Electrical Equipment - Part 1-8: General Requirements for Safety - Collateral Standard: Alarm Systems - Requirements, Tests and Guidance - General Requirements and Guidelines for Alarm Systems in Medical Equipment (General)

The two documents above are engineering standards. They're engineering specifications that medical devices must meet. They are technical and specific.

I show Figure 3 from the article before showing Figure 2. 

The increasing reference of 60601-1-8 is not surprising given the increased emphasis on safety. My real interest is in the significant increase in reference to ISO 62366. As mentioned, this is process standard the lays out how human factors engineering should be engaged to reduce "use errors." The emphasis in this standard is on the reduction of risk. Risk management is extremely well embedded in the medical device design and engineering process. It would seem that from a cultural perspective, ISO 62366 fits with the medical device engineering process. 

I want to contrast the dramatic, increasing references to ISO 62366 with the references to AAMI/ANSI HE75 shown in Figure 2 below.

References to AAMI/ANSI HE75 rise and fall from 2010 to 2013 instead of a steady upward trend that you see with ISO 62366 in Figure 3. I would like to emphasize that ISO 62366 and AAMI/ANSI HE75 should be considered as companion documents. (I'll expand on this in the Commentary section below.)

Commentary

The article does support the contention that the FDA and the companies it regulates are paying increasing attention to usability and human factors. That they're paying enough attention is another matter entirely. As new medical devices are introduced we should see two things. First, the use error rate for the newly introduced medical devices (once users have adapted to them) should decline in relationship to other similar devices currently in use. Second, we should see over time the number of per year of deaths and injuries from medical errors begin to decline. This will take time to detect.

Without a doubt, the push by the FDA to define a human engineering process in the design and testing of medical devices, and to press for testing under actual or simulated conditions is needed. In many ways the FDA is mirroring many of the processes that have already been adopted by the US Department of Defense (DoD) in the area of human engineering. Admittedly, the DoD doesn't always get it right, there is an understanding within the DoD that it is important ... life saving, battle-winning important ... to insure that those at the controls can do their jobs quickly, effectively and with as few errors as possible.  So from that standpoint, the FDA has adopted processes from programs that have proven effective. But the FDA has just passed the starting line. And much more will be required going forward.

ISO 62366 vs AAMI/ANSI HE75

As I mentioned earlier ISO 62366 and AAMI/ANSI HE75 should be consider complementary or companion documents. HE75 is a much larger document than 62366 and includes a significant amount of device design guidance and guidelines. 62366 is almost entirely a process document that's devoted to directing how to go about managing the research and development process of a medical device. In addition, the focus of 62366 is managing risks, risks in the realm of reducing use errors.

I found it interesting that references to HE75 were not increasing at the rate as references to 62366. I would have expected Figures 2 and 3 to have a similar appearance with respect to 62366 and HE75 in large part because the documents significantly overlap. In fact I might have reasonably expected references to HE75 to outpace 62366 because HE75 includes design specific guidelines in addition.

One possible reason for references to HE75 not being referenced in the same accelerated way as HE75 may have to do with the fact that the European Union has not adopted HE75, so it's required for medical devices that will be marketed in the EU (CE).  (I am currently unaware of the regulatory requirements of other countries on this matter.) Medical device companies are international companies and the documents that they file in one country are generally the same in each country. Thus since the EU hasn't adopted HE75, references to HE75 and HE75's use as a foundational process and design document may be less.

DESIGN RATIONALE

I'm not sure that this is true at this point in time, but I am certain that the following will be true going forward at some time in the future. I believe that the FDA will hold companies to account for their user interface designs. I believe that the FDA will demand that companies clearly define how they came up with their user interface designs and that those designs are well-grounded in empirical evidence.

This is what I mean ... the FDA will demand that the design choices ... these include: controls, placement of controls, number of controls, actions performed by controls, the way the control responds, methods for interacting with the device (e. g., touch screen, buttons, mouse), size of the display, etc. ... for medical device user interfaces must be grounded in empirical data.

Commercial websites are often designed by graphic artists. Often times the design of webpages reflect the artist's aesthetic sensibilities. Layout appear they way that they do because they look good.

I believe that the FDA will require that user interface designs for medical devices have an empirically grounded design rationale. Companies will be required to point to specific research finding to justify the design and the design choices that they made. Furthermore, as the design of the user interface evolves with each iteration of testing, the FDA will require that changes to the design be based on research findings.

Finally, I believe that soon if it is not occurring already, that the FDA will require:

1. That companies submit documentation to show in detail the full evolutionary design process beginning from product inception, including ...
2. Detailed pre-design research ... population(s), method(s), research questions and rationale, etc ... as well as the findings and what they suggest for the design of the user interface
3. A design that includes with a full discussion of the design rationale ... why was it designed the way it was ... 
4. A detailed description of the evolution of the design that include full and clear justification(s) for each change in the design ... and require that changes be grounded empirical data 
5. A full description of pre-commercialization testing process and method ... with a clear justification for why this testing meets FDA testing requirements
6. And a complete and clear analysis of the testing data.

What I'm suggesting above is that the process of designing and testing a medical device user interface should be more than going through the prescribed steps, collecting the data, doing the tests, etc. There should be a clear thread that ties all the steps together. When in a subsequent step, one should be able to point back to the previous steps for the rationale to explain why the user interface was designed to appear and operate the way it does ... to this point.

As near as I can tell, what I described above is rigorous than is currently required by the FDA. However, I believe that it would be in any company's best interest to follow what I've suggested because there may come a time when the FDA's enforcement becomes more rigorous. 

Another reason may be lawsuits. If a company can show that it went beyond the FDA's regulatory requirements at the time, those suing would likely have less of a chance of collecting damages. And if damages were awarded, they may likely be lower. Also, if the company went beyond the FDA requirements, it would be likely that there would be fewer people injured and that should lower damages.

FINALLY

This article has been a springboard for me to discuss a number of topics related to human engineering for medical devices user interfaces. This topic will remain a central part of this blog. I'll return this within a week or two, and discuss in depth other topics related to the human engineering process for medical device user interfaces. 

HE-75 Topic: Cleaning Up the Mess

I received a reminder this week of what usability professionals are often called on to do – cleaning up the mess created by a failed process. Somehow, the persons responsible for designing an awful, unusable and in some case, useless user interface expect the usability expert to come in, take one look and create a beautiful user interface. This is absurd!  It was the "nightmare" come true - something related to one of my other postings: HE-75 topic: Design first and ask questions later

Writing from my own perspective, there is nothing that a usability professional likes to do less than to correct a failed design that resulted from a failed design process. This week I was asked to save a group of programmers and user interfaced designers from the monstrosities that they had created. What was particularly strange was that the leader of the project thought that I could just redesign something by looking at what they had created. It was bizarre. Unfortunately, I had to deliver several harsh messages regarding the design process and the design, that were not well received. (Nevertheless, that is my job.)

Here is the point I want to make to anyone who reads this. Process and the resulting design should be considered as two sides of the same coin. Good design process nearly always results in a good design. A nonexistent or poor design process leads to a poor design. HE-75's design process can serve as a foundation design process for designing user interface in nearly any industry, particularly in those industries where the harm is particularly severe. Where I am currently working, I plan to use HE-75 as one of the foundation documents to set user interface design standards. And as I mentioned, I am not currently working in the medical or medical device industry. However, I have come to believe that in this industry, the level of harm can be significant. Thus, I shall incorporate HE-75.

 

Next time, I'll review so of the literature that might be of some use to the community.

Useful Verses Usable

This is a discussion that you are not likely to read in usability texts – the topic of useful verse usability, and the value of each. Just recently I had a discussion with someone on just this topic. Furthermore, I have had numerous discussions with others and each time it surprises me that people often do not know the difference between the two and the value of each.

Useful and Usable

If you go to the dictionary, you will discover that “useful” means “to be of serviceable value, beneficial” and in addition “of practical use.” Pretty straight forward.

On the other hand, the definition of “usable” is “capable of being used” and “convenient and viable for use.” Also a straightforward definition.

However, if you probe more deeply into the definitions, you will note that “useful” is the first, necessary quality of a tool or system. It must be useful or why use it? Usable is a quality of a tool or system. However, it is not primary in relationship with the quality of the being “useful.” It is secondary. Necessary, yes, nevertheless, it is still secondary.

The usefulness as a quality of a tool or system is not addresses in HE-75, or any other usability standard that I have encountered. (If any one knows of a standard where usefulness is addressed, please add a comment to this discussion.) Usefulness is assumed.

However, I have learned that in the real world, the usefulness of any tool or system should not be assumed. It should be tested. Furthermore, with complex systems, the fundamental capabilities of a system or tool are often useful. However, not all of the capabilities of that system may be.

I direct experience with a remote monitoring system where the primary or fundamental capabilities of the system have clear use. However, with each release of this system, as more capabilities are added, the useless capabilities may be on the verge of out numbering the useful ones.

Bottom Line

• 
  
  Usefulness is always more important than usability. If it is not useful, it is fundamentally worthless or at best, excess baggage, and a drag on the actual and perceived quality of the tool or system.
• 
  
  Usefulness should never be assumed. It should be demonstrated. I know of too many projects where usefulness was not demonstrated. This lead to capabilities being developed that waste time and money, and can damage reputations.

HE-75, Usability and When to Prototype and Usability Test: Take 1

Prototyping and Testing will be a topical area where I shall have much to contribute.  Expect numerous articles to appear on this topic.

I had a discussion a few days ago with one of my colleagues who has worked as a user interface designer, but has little knowledge of human factors.  He was completely unaware of the concepts of "top-down" and "bottom-up" processes to user interface design.  I provide for you the essence of that discussion.

Top-Down Approach

The top-down approach begins with a design.  Most often the initial design is a best or educated guess based on some set of principles.  Could be aesthetics or "accepted" standards of good design, or something else.  The design is usability and/or acceptance tested in some manner.  (Anywhere from laboratory testing to field-collected data.)  In response to the data, the design reworked.  The process is continual.  Recent experience has suggested that the top-down approach has become predominant design methodology, particularly for the development of websites.

Top-down is a valid process, particularly for the deployment of new or unique products where the consequences of a failed design do not lead to serious consequences.  It can get a design into user hands more quickly.  The problem with a top-down approach (when practiced correctly) is that it relies on successive approximations to an ill-defined or unknown target.  To some degree it's similar to throwing darts blindfolded with some minimal correction information provided after each throw.  The thrower will eventually hit the bull's eyes, but it may take lots and lots of throws.

The top-down approach may have a side benefit in that it can lead to developing novel and innovative designs.  Although, it can have the opposite effect when designs are nothing more than "knock-offs" of the designs from others.  I have seen both coming out of the top-down approach.

Bottom-Up Approach

HE-75 teaches the use of a bottom-up approach where first one defines and researches the targeted user population.  Contextual Inquiry is also a bottom-up approach.  Since I have already discussed researching the targeted user population in depth, I'll not cover it here.  

With the bottom-up approach, the target is clear and understood.  And tailoring a design to the user population(s) should be a relatively straight forward process.  Furthermore, the bottom-up approach directly addresses the usefulness issue with hard data and as such, more likely to lead to the development of a system that is not only usable, but useful.

Useful vs. Usable

I'll address this topic more deeply in another article.  It suffices to say that usability and usefulness are distinctly different system qualities.  A system may be usable, that is, the user interface may require little training and be easy to use, but the system or its capabilities are not useful.  Or, and this is what often happens particularly with top-down approaches, much of what the system provides is not useful or extraneous.

Personal Preference

I am a believer in the bottom-up approach.  It leads to the development of systems that are both usable and useful sooner than the top-down approach.  It is the only approach that I would trust when designing systems where user error is of particular concern.  The top-down approach has its place and I have used it myself, and will continue to use it.  But, in the end, I believe the bottom-up approach is superior, particularly in the medical field. 

HE-75 Touch Screen Recommendations

I have found HE-75 to be one of the best human factors standards ever produced.  However, I have found their analysis and recommendations regarding touch screens lacking, and out of date.  To place a perspective on the HE-75 touch screen recommendations ... in the late 1980's and early 1990's, I ran a user interface design and implementation project inside of a larger project at Bell Laboratories.  To make a long story short, one of the user interfaces we needed to design and produce was a touch screen interface.  The touch screen used a CRT as a display device and it was as flat as we could make it.  In addition, the distance between the touch screen surface and the display was about 35 mm.  When I read of the issues related to touch screens and the recommendations in HE-75, I experience deja vu and I feel as if I've been transported back to that time.

Some of the most significant advances in user interfaces have been in the areas of display technology and touch screens with respect to hardware and in particular software.  Apple Computer has been a leader in combining the advances in both display technology, touch screen design and touch screen interface software.  I would have expected the HE-75 committee to have incorporated these advances and innovations in touchscreen software into the standard.  However, what I have found appears to me as ossified thinking or ignoring what has transpired.  

People in the medical field are using smart phones with their advanced touch screen interfaces in their medical practice.  Smart phone touch screens and now the Apple iPad have become the de facto standard in touch screen technology.  My previous article related to consistency ... here's a consistency issue.  Is it wise to suggest that medical device touch screen interfaces look and operate in a way different from the accepted standard in the field?  I know this is not a simple question, but I think it is one that will need to be addressed in future editions of HE-75.

HE-75 Topic: Risk Management

One more HE-75 topic before proceeding into design and design related activities.  The topic, risk management.

Reading HE-75, you will note that this document continually discusses risk management and reducing risk.  In fact, the entire document is fundamentally about reducing risk, the risks associated with a poor or inappropriate design.

If you drive a car, especially if you have been driving cars for more than a decade or two, you will note that a driving a car with well-designed controls and well-laid out and designed displays seems inherently easier than one that is poorly designed.  Furthermore, it has been demonstrated time and again that driving safety increases when a driver has been provided well-designed controls and displays, driving become less risky for everyone concerned.

Car makers now see safety as selling point.  (Look at a car that was built in the 40s, 50s or 60s and you'll note how few safety features the car included.)  Manufacturers are beginning to include in their luxury models driver-error detection systems.  For example, one manufacturer has a system that signals the driver of the existence of an other vehicle the space the driver wants to move to.  One of the qualities of a well-designed user interface is the ability to anticipate the user and identify and trap errors or potential user errors, and provide a means or path for preventing or correcting the error without serious consequences.  Car manufacturers have been moving in this direction.  I suggest that the adoption of HE-75 will be the FDA's way of pushing medical manufacturers in the same direction.


Risk Management: Creating a Good Design and Verifying It


My many blog postings on HE-75 will address the specifics of how to create a good design and verify it, and the process of incorporating these design and verification processes in the a company's risk management processes.  In this posting I want to address a two issues at a high level.


First, I want to address "what is a good design and how to do to create it." Creating a good design requires a process such as one outlined by HE-75.  I am often amused at hiring managers and HR people who want to see a designer's portfolio having no conception regarding how the design were created.  A good design for a user interface is not artistry, it is a result of an effective process.  It should not only look good, but it should enable users to perform their tasks effectively and with a minimum of errors.  Furthermore, it should anticipate users and trap errors and prevent serious errors occurring.  And finally, it should provide users with paths or instructions on how to correct the error.  This is what HE-75 teaches in that it instructs researchers and designs   And to that end, the design process should reduce risk.  Think this is not possible?  Then I suggest you spend some time in the cockpit of a commercial airline.  It is possible.


Second, HE-75 teaches that design verification should be empirical and practiced often throughout the design process.  This is an adjunct to classic risk management that tends to be speculative or theoretical in that it relies on brainstorming and rational analysis.  HE-75 teaches that medical device and system manufacturers should not rely just on opinions - although opinions provided by subject-matter experts can provide valuable guidance.  HE-75 instructs subjects drawn from the targeted population(s) should be used to guide and test the design at each stage of the process.  This becomes the essence of risk management and risk reduction in the design of user interfaces.



Additional Resources

I have this book in my library.  It provides some good information, but it's not comprehensive.  Unfortunately, it's the only book I know of in this field.  
















These books I do not owe, but provide you with the links for information purposes.  I am surprised at how few books in the field of medical risk management there are.  It may go a long ways to explain the large number medical errors, especially the ones that injure or kill patients.

Risk Management Handbook for Health Care Organizations, Student Edition (J-B Public Health/Health Services Text) 

Medical Malpractice Risk Management 

HE-75: Collecting Data and Modeling Tasks and Environment

This article expounds on my earlier article related to AAMI HE-75: Know what thy user does and where they do it. 


Collect and Represent the Data


Ideally the first steps in the design process should occur before a design is ever considered.  Unfortunately, in virtually every case I have encountered, a design for the user interface has already been in the works before the steps for collecting user and task related data have been performed.


Nevertheless, if you are one of the people performing the research, do as much as you can to push the design out of your mind and focus on objectively collecting and evaluating the data.  And, in your data analysis, following the data and not your or the preconceived notions of someone else.


There are a variety of means for collecting data and representing it.  The means for collecting the data will generally involve:

• Observation - collecting the step-by-step activities as a person under observation performs their tasks.
• Inquiry - collecting data about the a person's cognitive processes.

Once the data has been connected, it requires analysis and representation in a manner that is useful for later steps in the design process.  Data representations can include:

• Task models - summary process models (with variants and edge cases) of how users perform each task.  This is different from workflow models in that in task models no references to specific tools or systems should be included in the task model.  A task model should be abstracted and represented at a level without reference to actions taking place on a particular device or system.
• Workflows - summary process models (with variants and edge cases) similar to the task flows with reference to a particular device or system.  For example, if the user interface consists of a particular web page, there should be a reference to that webpage and the action(s) that took place.
• Cognitive models - a representation of the cognitive activities and processes that take place as the person performs a task.
• Breadth analysis - I have noted that this is often overlooked.  Breadth analysis organizes the tasks by frequency of use and if appropriate, order of execution.  This is also the place to represent the tasks that users perform in their work environment but were not directly part of the data collection process.

Detailed Instructions


I cannot hope to provide detailed instructions in this blog.  However, I can provide a few pointers. There published works on how to collect, analyze and model the data by leaders in the field.

Here are three books that can recommend and several can be found in my library:


User and Task Analysis for Interface Design by  J. Hackos & J. Redish


I highly recommend this book.  I use it frequently.  For those of us experienced in the profession and with task and user analysis, what they discuss will seem familiar - as well it should.  However, what they do are provide clear paths and methods for collecting data from users.  The book is well-structured and extremely useful for practitioners.  I had been using task and user analysis for a decade before this book came out.  I found that by owning this book, I could throw all my notes away related to task and user analysis, and use this book as my reference.


Motion and Time Study: Improving Work Methods and Management 
by F. Meyer
Motion and Time Study for Lean Manufacturing (3rd Edition) by F. Meyer & J. R. Stewart


Time and motion study is a core part of industrial engineering as the means to improve the manufacturing process.  Historically, time and motion studies go back to Fredrick Taylor (http://en.wikipedia.org/wiki/Frederick_Winslow_Taylor) who pioneered this work in the later part of the 19th and in early part of the 20th Century.  I have used time and motion studies as a means for uncovering problematic designs.  Time and motion studies can be particularly useful when users are engaged in repetitive activities and as a means for improving efficiency and even as a means for reducing repeated stress injuries.  The first book I have in my library however it is a bit old (but very inexpensive) so I include the second book by Meyers (and Stewart) that more recent.  I can say that the methods of time and motion can be considered timeless, thus adding a book published in 1992 can still be valuable.

Time and motion studies can produce significant detail regarding the activities that those under observation perform.  However, these studies are time-consuming and as such, expensive.  Nevertheless, they can provide extremely valuable data that can uncover problems and improve efficiency.


Contextual Design: Defining Customer-Centered Systems (Interactive Technologies) by H. Beyer & K. Holtzblatt &

Rapid Contextual Design: A How-to Guide to Key Techniques for User-Centered Design (Interactive Technologies) by K. Holtzblatt, J. B. Wendell & S. Wood


The first book I have in my library, but not the second.  I have used many of the methods described in Contextual Design before the book was published.  The contextual design process is one of the currently "hot" methods collecting user and task data, and as such, every practitioner should own a copy of this book - at least as a reference.


I believe what's particularly useful about this contextual inquiry is that it collects data about activities not directly observered.  It's able but that affect the users and the tasks that they perform.  For example, clinicians engaged in the remote monitoring of patients often have other duties, many of them patient related.  Collecting data exclusively targeting remote monitoring activities (or the activities specific to a targeted device or company) can miss significant activities that impact remote monitoring and vice versa. 


Additional Resources


As a graduate student, I had the privilege of having my education supported by Xerox's Palo Alto Research Center.  I was able to work with luminaries of the profession, Tom Moran and Allen Newell on a couple of projects.  In addition I was able to learn the GOMS model.  I have found this model useful in that it nicely blends objectively observed activities with cognitive processes.  However, the modeling process can be arduous, and as such, expensive.  

Allen Newell and Herbert Simon are particularly well known for their research on chess masters and problem solving.  They were well-known for their research method, protocol analysis. Protocol analysis is a method that has the person under observation verbally express their thoughts while engaged a particular activity.  This enables the observer to collect data about the subject's thoughts, strategies and goals.  This methodology has been adopted by the authors of contextual inquiry and one that I have often used in my research.


The problem with protocol analysis is that it cannot capture cognitive processes that occur beyond the level of consciousness, such as the perception.  For example, subjects are unable to express how they perceive and identify words, or express how they are able to read sentences.  These processes are largely automatic and thus not available to conscious processes.  (I shall discuss methods that will enable one to collect data that involves automatic processes when I discuss usability testing in a later article.)  However, protocol analysis can provide valuable data regarding a subject's thoughts particularly when that person reaches a point where confusion sets-in or where the person attempts to correct an error condition.

Here's a link from Wikipedia: http://en.wikipedia.org/wiki/GOMS.


Another book that I have in my library by a former Bell Labs human factors researcher, Thomas K. (TK) Landauer, is The Trouble with Computers: Usefulness, Usability, and Productivity.


This is fun book.  I think it's much more instructive to the professional than Don Norman's book, The Psychology Of Everyday Things.  (Nevertheless, I place the link to Amazon just the same.  This is a good book for professional in the field to give to family members who ask "what do you do for a living?")  

Tom rails against the many of the pressures and processes that push products, systems and services into the commercial space before they're ready from a human engineering standpoint.  Although the book is relatively old, many of the points he makes are more relevant today than when the book was first published.  The impluse to design user interfaces without reference or regard for users has been clearly noted by the FDA, hence the need for HE-75.

More on Knowing Thy Target User Population

Before moving forward into product development, I want to elaborate on the issues in my first two articles. This article elaborates on the importance of knowing the target population and ways to gather that information.  

The next article will discuss  I have had some recent experiences that reinforced that importance of defining and clearing understanding the targeted user population. And the importance of fully understanding and documenting what those members of the user population do and the environment(s) wherein they live and work.

Before proceeding any further, please review my previous article on understanding your target population. The link to the article is below:

http://medicalremoteprogramming.blogspot.com/2010/03/know-thy-target-population.html

HE75 clearly emphasizes the importance of understanding your target population.   The standard instructs that companies who develop medical devices should:

1. Know their targeted user population
2. Involve users early and often
3. Accommodate user characteristics and capabilities. And in order to do this, one must first know what they are.

The information gathered about a target population should enable one to clearly define the qualities and characteristics of that population.  This can be particularly important when designing medical devices, particularly when those devices are targeted to patients. 

I have seen organizations a company, organizations that include program management, marketing and engineering assume that they know the characteristics of the targeted population.  Once the product is deployed, the company comes to a rude awakening and learns that their assumptions were often times false.  Neither the company nor the targeted user population(s) benefit from such a failure.

Methods for Gathering Target Population Data

The target population data is the most elemental data in the product development process.  All the descriptions about the targeted user population, their characteristics, culture and capabilities originate from this step in the research and development process.

So, how is this crucial data gathered? First, a confession ... the amount of work I have performed at this stage of the process has been limited.  My training is in cognitive psychology and computer science.  Most often I have been the recipient of such information about the targeted user population.  I have used the results of this first step as a means for recruiting subjects in my usability experiments and evaluations.  The training that is most suited to gathering this kind of data is anthropology and sociology.  The process of collecting target user population data draws on ethnographic and participant observation research methodologies.  The research can be observational.  It can be based on questionnaires administered orally or in writing.  It can be structured interview.  It can participant observation where the observer becomes participates in the activities of the target population.  It can be a combination of a variety of methods and include methods not listed above.  

The objective is the development well-grounded description that captures the important, defining characteristics of the target population.  The description can be provided in variety of ways, verbal or graphic.  The description should use the clearest and most appropriate methods available to covey that information to the members of the product development organizations.

Interestingly enough, I have used the data gathering methods I listed above.  However, I used those methods to collect data for the second step, Knowing what the user does and where they do it.  In other words, to gather task and environmental data.

Potential Costs for Failure to Correctly Define the Target User Population

Consider the following scenario ... that I collect task and environmental data about the wrong population, about a population that is not the target population.  What is the value of the results of my research?  And what could be the cost to the company for this failure?  What could be the cost to the target user population, to have a device with a user interface unsuited to their needs?

In reality, the cost could be high, but the product may not be a dismal failure.  Given the fact that we are all human, we share a wide variety of characteristics.  However, in the more stringent regulatory environment that is anticipated, it could mean delay, additional research, engineering and product development costs.  If the product is intended to provide a new capability to providers and/or patients, a delay could mean that a competitor could be first to the market the product.  Thus company could miss the competitive advantage to being first.

I have recent experience with two products targeted to patients. In one case the target population was well understood and well defined, and members of that population were used in usability testing.  In another case, there was a limited understanding of the target population by the research and development organization. And no member of the target population involved at any stage of the research and development process or in the development of the user interface.   In the first case where the target population was well understood and well defined, the user interface research and development process was clear and logical.  On the other hand, the research and development process that did not have a clear understand of the target population is struggling, it is learning as it goes.  Each time it learns something new about its target population, the user interface has to be updated.  It has been a costly process with constant reworks of the user interface.  So many reworks that the integrity of the original design has been lost.  It appears deconstructed.  At some point the entire user interface will have to be redesigned and that will likely come at the behest of the FDA enforcing HE75.

A Final Thought

HE75 instructs that medical product user interfaces should accommodate a diverse groups of users and should be maximally accessible. I see this as design objective of any user interface in that vernacular should be limited as much as possible and that limiting qualities should not be designed in or should be removed when detected. However, all products may not be accessible to all users but should be clearly accessible to the target population.  And I believe that the FDA will insist on this.

Know What Thy User Does and Where They Do It

Review ...



Last time, I discussed the importance of knowing your target population and their use environment. That first step identifies and specifies the population for inclusion. It is the means for including who should be included and excluded, and the environment where they work. For example, a targeted population for a particularly medical product could be surgical nurses who work in hospitals. The target population does not include all nurses or even all surgical nurses. In addition, the use environment in which the targeted population performs their work needs to be a part of the definitional equation.
Thus, the first step in the design process is defining the properties of the target population, determining who is and who is not part of that population. And include a complete description of their working environment, the environment where the product or service will be used. Field research will be necessary to establish the target population and its characteristics and the work environment. When this step is finished, the next step is perform additional research to establish the details of the work of interest and the environment in which it performed. (The means for collecting this information and form of the analytic product will be discussed in a later article.)

Know What Thy User Does and Where They Do It

Knowing what the user does consists of documenting the tasks that the target user population would perform with the product or service that a company plans to provide.

Once the product or service has been conceptually defined, the following information from the target population must collected:

1. 
   
   The preconditions that lead to performing each task,
2. 
   
   The steps required to perform a task, and
3. 
   
   How frequently each is performed (in absolute terms and in relationship to other tasks.

The information collected focus on the actions performed by a user localized to the product or service in development. The data would have little or no reference to the use environment – the full set of activities and environmental conditions wherein this product or service will be used. Thus once having collected the task data specific to the product or service in development, the next step would be placing this product or service within the environment wherein it will used.

A full and complete description or representation of the use environment may not always be possible. Moreover, often times there are multiple use environments. And a description or descriptions may be only of a representative sample. Nevertheless, it can be extremely useful to understand how the product or service in development will be used in context.

The final research products resulting from this step in the product development process are:

1. 
   
   Task analyzes: pertaining only to the product or service in development.

That include:

• 
  
  Breadth analysis, that consists of:
• 
  
  The number of tasks users would perform using the product or service in development
• 
  
  Their frequency of performance
• 
  
  Depth analysis
• 
  
  Define how each task is performed
• 
  
  The level is detail required will vary with the complexity of the task
• 
  
  Each task analysis should include likely errors and the steps required to correct them.
• 
  
  There are a variety of means to represent task analyze. The representation method should be agreed on by the affected parties.

2. 
   
   Task execution within the wider use environment

• 
  
  The tasks that users will perform with the product or service in development will be performed within a larger context.
• 
  
  The tasks within the wider context and how those other tasks relate to each other requires documentation.
• 
  
  The other tasks require a breath analysis. Rarely is a depth analysis required unless tasks are intermingled.

I discuss the benefits of knowing what your user does and where they do it in my next article. I shall discuss with reference to HE75 and what the FDA will likely require from the medical product companies.

Where in a company's organizational structure is this work performed?

Knowing your 1) target population and their use environment, and 2) what your users do (task analysis) are the first two steps in the product formation stage of development. This precedes requirements gathering stage of product development. Thus, this would require the engagement of human factors engineers working with marketing and other product and field-focused organizations to engage those working at this early stage. St. Jude Medical with whom I consulted for 15 months has placed their all their human factors engineers in systems engineering. Thus, the placement of human factors engineering only in systems engineering means that important data impacting the beginning of product development is either not produced, or produced at a later stage in the development process where its impact is minimal or non existent.

As I shall discuss later, it is important that human factors engineering be involved from concept to deployment, thus human factors engineers should be distributed throughout an organization.

Stirrings in the Regulatory Environment

Medical product producers are regulated by the Federal Government. Anyone who reads this blog is highly likely to know that. Drug companies are acutely aware of governmental regulation in that their products must be “safe and effective.” Drug companies must prove through research safety and effectiveness. Implanted device manufacturers must demonstrate that the implanted devices themselves are safe and effective. Devices and drugs that deliver therapy have to prove to the FDA safety and effectiveness.

But what about the user interfaces for devices that enable users to make changes to the operation of implanted devices, deliver therapies, provide information about the patient, etc., where's the proof in the form of empirical data to prove that they're safe and effective?

In the US, more people are killed by medical errors each year than are killed in automobile accidents and in the military service combined. Yet, the FDA has placed few relatively requirements on the process for designing user interfaces on medical products and services. This is a disgrace and the FDA knows it. FDA mandates for insuring the usability of medical devices, products and services has been merely to determine only that a usability process is in place. FDA mandates for usability have not reached the level of the Departments of Defense or Transportation. Companies that design and build medical systems have not been required to prove to the FDA that their products are usable in their use environment. Yet, it is clear that usability is just as important in medical practice as with combat systems and cars – particularly when one considers the number of injuries and deaths resulting from medical errors. And with more powerful and complicated systems are being designed and planned, the need for the FDA to act and act effectively in the area of usability grow substantially.

In my personal experience, I saw a device under development that if used improperly, could injure or in one case, lead the to death of a patient. In fact, I uncovered a condition in which the device when used properly could lead to death. And, it would have been surprisingly easy to do. Of course, I raised my concerns regarding this device and its potential for injuring patients. Nevertheless, in the current regulatory environment, I believe that it would be possible that the device could be approved for use by the FDA because of the lack of a clear standard from the FDA that the company prove empirical data regarding that the device is usable and safe.

The Department of Defense has been particularly forceful with contractors regarding insuring that members of the target population will be able to use systems within the environment of their intended use. It does not take a great deal of contemplation to understand the value of insuring that a system can be operated effectively by a soldier in a combat environment. A system that could save the lives of fellow soldiers or civilians would be useless if it could not be used effectively by a member of the target population (i. e., a soldier) in combat. If the user interface is too cumbersome or complicated when used in the stress and difficulties of the combat environment, then all the time and effort taken to create that system has been wasted. NASA and the FAA have taken a stance similar to the DoD. The Department of Transportation in conjunction with Congress have taken a strong stance with respect to the design of user interface of vehicles.

I believe that the FDA will begin to strong stance similar to other regulatory bodies of the Federal Government regarding insuring that the user interfaces of medical devices meet specific usability standards and the meeting of these standards must be demonstrated experimentally with empirical data. I think that one of the first step in the process of ever increasing regulation of the user interfaces of medical will be the adoption of HE75 by the FDA. This would start the process towards mandating that companies prove their products meet user performance standards.