Submission of the Human Engineering File to the FDA and Other Regulatory Bodies, Section 8: Part VI

This is the easiest for me to cover largely because the requirements for the validation section are clearly spelled out in detail.

8

Details of human factors validation testing Rationale for test type selected (i.e., simulated use, actual use or clinical study) Test environment and conditions of use Number and type of test participants Training provided to test participants and how it corresponded to real-world training levels Critical tasks and use scenarios included in testing Definition of successful performance of each test task  Description of data to be collected and methods for documenting observations and interview responses Test results: Observations of task performance and occurrences of use errors, close calls, and use problems  Test results: Feedback from interviews with test participants regarding device use, critical tasks, use errors, and problems (as applicable)   Description and analysis of all use errors and difficulties that could cause harm, root causes of the problems, and implications for additional risk elimination or reduction

These requirements are largely self explanatory. However, I would like to make a few comments and additions.

• Validation testing -- including verification testing -- are often performed by outside consulting firms. Thus it is extremely important that you spell out how your testing should be performed and the measurements to be collected and reported. I've noted that often times consulting company is asked to write both the protocol and the testing script. This is a mistake. The organization that performed the work up to the validation testing stage should be responsible for creating the protocol and the script, because it is this organization that will be responsible for the submission of the HE file to the FDA and/or other regulatory bodies. It's important that the research and development as well as the submission be responsible and in full control of what takes place during the validation step.
• Verification and Validation testing. Verification testing takes place under laboratory conditions using as testing participants members of the targeted user population. This is an additional check on the usability of the system or device. Validation testing takes place in actual or simulated actual conditions -- with all the distractions and problems that users will likely encounter.
• Rationale for type of testing performed and the conditions chosen for validation testing can be extremely important especially if you have chosen a testing procedure less rigorous than performance testing under real or simulated real conditions. Consult IEC 62366 and AAMI HE75 for guidance.
• Testing procedure should insure that full testing of critical tasks are performed and likely to be repeatedly performed by testing participants.
• Suggested additional measurement: If your system or device has error trapping and redirecting capabilities, be sure to report how often these capabilities were triggered and if they enabled the testing participant to successfully complete the task. This could be labeled as: task successfully completed, close call. However, a system or device with the capability to protect against use errors is a capability worth pointing out. 

What to include in your narrative?

Include the abstract or abstracts of your validation testing in your narrative. 

If you haven't included any significant issues or root cause analysis in your abstract, be sure to include this in your narrative. Be sure you surface all issues or concerns in your narrative, if you don't it could appear to a reviewer that you're trying to hide any problems that you encountered. Even the appearance of hiding problems could cause problems with receiving approval for your system or device. 

Submission of the Human Engineering File to the FDA and Other Regulatory Bodies, Sections 6 and 7: Part V

I cover Sections 6 and 7 in this article as shown below:

6	Summary of preliminary analyses and evaluations Evaluation methods used Key results and design modifications implemented in response Key findings that informed the human factors validation test protocol
7	Description and categorization of critical tasks  Process used to identify critical tasks List and descriptions of critical tasks Categorization of critical tasks by severity of potential harm Descriptions of use scenarios that include critical tasks

I consider Sections 6 and 7 together because the information for these two sections should have come from the formative stage of the research and design process. These two sections could be combined into a single section. However, it is apparent that the FDA (and probably other regulatory bodies as well) considers Section 7, Description and categorization of critical tasks, as important enough to have its own, separate section. 

Importance of Getting It Right

The contents of these sections, the descriptions and explanations provided, can be the difference between: 

• An easy, unquestioned acceptance of what you've done or
• A difficult, question-riddled review of the work that you performed resulting in:
• Approval delays, 
• A reworking of the submitted materials 
• Requests for additional research to be performed, or 
• Rejection of the human engineering file 

To fully address what should be included in Sections 6 and 7, you need to examine your entire HE process in the context of the research and development program of your medical device or system and determine whether your HE process can adequately address reporting requirements of these two sections. These sections form the core of the report of your research and design process up to the point immediately before you begin your final phase of testing, namely verification and validation (summative) testing.

Section 6: Summary of the Preliminary Analysis and Evaluations

What Should be in Section 6

I briefly cover the points of what should be included in Section 6. Assuming that you are a human engineering professional, you should already have a reasonable understanding of the meaning of each of three requirements listed below. 

1. Evaluation methods used

This comprises the entire body of research performed including all of the data collected before the implementing a foundational or initial design, and all of the testing performed on the design.

2. Key results and design modifications implemented in response

What findings from your research lead to you to creating your initial design and what where the factors that lead you to modifying your design?

3. Key findings that informed the human factors validation test protocol

How did your arrive at creating your research protocol for summative/validation testing? How do you know that your validation protocol is appropriate and will verify that your system or device is safe for use?

That's the brief overview of what should be in Section 6. However, what should be included in Section 6 are the logical threads of justifications for doing what you did: for creating your research and development plan, the initial/foundational design and how you went about modifying that design. 
   
Don't be deceived by the seeming simplicity of Section 6. It is far more complicated and demands much more investigative and design process rigor than one might imagine. 

Human Engineering (HE): Research and Development

Section 6 is the section where you lay out all research and development performed in relationship to human engineering. Thus, Section 6 becomes the place where you make your case for the research that you performed and the design choices that you made. After reading Section 6, the reviewer should have a clear understanding and be in agreement with the research and design process that was undertaken. This includes the rationale for the research plan as proposed and undertaken including the rationale for any changes made to the plan on the basis of research findings. It will include the rationale for the design process, including the initial or foundational design and the reasons for changes made through the design iteration process.

Human factors is the study of how human interact with or operate systems and devices. Its fundamentally research. Human engineering incorporates the human factors, but encompasses and  incorporates design and the design process that should be at its foundation, driven by research. The research that directs and informs design and the design process includes field, laboratory, library, risk or research-based standards. And in the absence of the ability to collect empirical data: scenarios and interaction walk-throughs and analysis. 


You will need to defend your rationale for the specific research projects undertaken and the design choices made. Because the narrative is an overview, it's often a good place to explain the much of the logic for the research undertaken and the design choices made.

Defending HE Research and Design Planning and Choices

Adequate and effective justification of your research and development plan and design choices will often be the key to insuring unquestioned acceptance of your submission. Here are some suggestions:

1. Justifying the Research and Development Plan -- the means for creating a usable and low-likelihood use-error and low risk system or device. Reasoning and justifications for the creating a research and development plan for this system or device include:
• Compliance with IEC 62366 (part 1).
• Conformance to FDA HE program guidance (on the FDA website).
• Guidance from AAMI/ANSI HE-75
• Guidance from previous, similar and accepted plans 
• This system or device is a next generation release of a currently, commercially available product. Thus the research and development performed along with field collected data provide guidance for research and design plan for this next generation product.
2. Justification for performing specific research include:
• Planned research
• Research fits within the guidelines set within the research plan.
• Research is designed to answer specific research questions. Often during a research program, questions arise that may be human performance, design specific, etc. that may not have been specified in the research plan. Often times these types of studies are applicable to the research and development of a variety of device and systems. In this case the research is "question-driven." Those research questions need to be clearly defined out within the research protocol and become the clear justification for the research and the applicability and potential value of the findings.
• Findings from planned research suggest the need for new research not originally planned.
3. Justification for the foundational design: is initial design that is prototyped, usability tested and then iterated. The foundational design establishes the basic design philosophy (appearance and operation) that will likely be commercialized. While the foundational design will likely be updated and improved throughout the research and development process; fundamentally, it will likely maintain the same design philosophy. Thus, establishment of the foundational design maybe the most consequential step in the research and development process. Justifications for the foundational design include:
• Updated version of an earlier, accepted design: using the same design philosophy. Updates and improvement driven-by field research, customer feedback, research on the use of the system under actual conditions.
• Findings from formative research as defined by the research and development plan undertaken before initiating a design.
• Compliance with accepted design standards, e.g., AAMI HE75. (There are a wide array of design standards issued and accepted by the US agencies as well as other agencies of a variety of countries. When localization of a design is required, the design standards issued by the targeted country should be considered and referenced.)
4. Justification for changes made to the foundational and modified designs.
• Findings from prototype testing.
• Findings from expert reviewers: resulting from design walkthroughs/reviews and/or interactions with the device or system.
• Limited field tests of prototypes.
5. Justification that the design has reached the stage for verification and validation (summative) testing. And that a research protocol can be written that can effectively and realistically test the system or device to demonstrate that the system or device will be safe for use by members of the targeted population in the intended use environment(s).
• This is the hand-off point to the summative testing phase.
• Justification that that the system or device is read to hand off: The formative testing up to this point should have subjected the system or device to the all of the testing that it would be subjected-to multiple times. And the system or device should have passed those tests multiple times. Thus, if the research and development plan was properly executed nothing of any concern should come from verification and validation testing. If there are findings that are the least bit concerning, then it is time to reexamine your research and development planning and protocols. 
• Finally, if your formative testing, meaning all of the testing performed up to this point, has been comprehensive,  rigorous and complete, then that testing should dictate the verification and validation research protocols.

What Should be Included in the Section 6 Narrative

I suggest that your narrative should be written in the form of a story. It should be a narration that describes in a linear fashion (from the beginning to immediately before the validation step) what you did, why you did it: 

• if it's research, summarize what you did and what you found, 
• if it's your foundational design, provide an high level description of how you arrived at this design (include enough figures to be sure that a reviewer will understand your description) and
• if it's a design update, explain what change or changes were made and why.

Be sure to include references to your submitted materials in your HE file.

Section 7: Description and categorization of critical tasks

Identifying the critical tasks that will be performed on your system or device should be part of formative research. Often the ability to identify the set of critical tasks is beyond the expertise of the human engineering professional and identifying as well as categorizing the critical tasks requires the support of subject-matter experts (who should be included from the beginning of the formative research stage). My experience has been to integrate subject-matter experts into the research and design process from product inception.  

The list of requirements for Section 7 include:

1. Process used to identify critical tasks

With your subject-matter experts, describe the process used to identify your critical tasks. 

2. List and descriptions of critical tasks

Include with this your justifications and reasoning for this list. 

3. Categorization of critical tasks by severity of potential harm

In addition, if any of your critical tasks have the possibility of inflicting moderate to critical harm, I suggest that mitigations developed to minimize the likelihood that harm would ever occur. 


4. Descriptions of use scenarios that include critical tasks

These use scenarios should form a fundamental part of both your testing as well as justification and rationale for your design (and updates to your design.)

Section 7 Narrative

I suggest that in your narrative that you include a table with the information from items 2 and 3 above. I would add a brief summary of the process that was used to identify your critical tasks. Finally, include a reference to the use scenarios that include the critical tasks. You don't need to include them in your narrative, a reference should be sufficient. 

______________________
Note: I plan on periodically updating this article as I learn more and reconsider what I have written. With each update, I'll include at the top of this article, when it was updated and list some of the changes that I have made. 

Submission of the Human Engineering File to the FDA and Other Regulatory Bodies

Background

About a year ago I was asked what I thought was the most difficult phase of the medical device human engineering process. Frankly, I'd never before considered such a question. I could not identify any phase of the process that I considered more difficult than any other. I considered whether early phase formative research and testing or risk identification, use errors and risk management would be the most difficult. No, actually for me these phases have always proven to be the most interesting phases in the process. Challenging, yes; difficult, no.

The questioner had an answer in mind: she believed that the validation testing was the most difficult. I disagreed. I believe that the validation testing was most often the least difficult of all the phases of the process. Why? Because validation comes at the end of the entire process and is based on all of the work that you've done previously to get one to the point of validation testing. Thus the procedure of a validation test should flow naturally and easily from the earlier work.

In the end, we agreed to disagree. However, the question never left me. 

I have finally come up with the answer. The most difficult phase of the process is the creation of the narrative for the regulatory reviewers. What I refer to is more than putting together the folder of documents of all of the human engineering related activities. It is the construction of a cohesive and understandable narrative the provides to a reviewer an overall view of the reasoning and logic of the steps taken and that the procedures performed that will demonstrate that the human engineering process was sound and resulting from it is a system that will be safe to use. This is the most difficult phase of the human engineering process, especially if writing the narrative comes at the end.

Human Engineering Pre-Market Submission

Here is the outline of what the FDA expects in a Human Engineering Pre-Market Submission as provided by the FDA on their website:

Sec.	Contents
1	Conclusion The device has been found to be safe and effective for the intended users, uses and use environments. Brief summary of HFE/UE processes and results that support this conclusion Discussion of residual use-related risk
2	Descriptions of intended device users, uses, use environments, and training Intended user population(s) and meaningful differences in capabilities between multiple user populations that could affect user interactions with the device Intended use and operational contexts of use Use environments and conditions that could affect user interactions with the device     Training intended for users
3	Description of device user interface  Graphical representation of device and its user interface Description of device user interface Device labeling Overview of operational sequence of device and expected user interactions with user interface
4	Summary of known use problems  Known use problems with previous models of the subject device Known use problems with similar devices, predicate devices or devices with similar user interface elements Design modifications implemented in response to  post-market use error problems
5	Analysis of hazards and risks associated with use of the device Potential use errors Potential harm and severity of harm that could result from each use error Risk management measures implemented to eliminate or reduce the risk Evidence of effectiveness of each risk management measure
6	Summary of preliminary analyses and evaluations Evaluation methods used Key results and design modifications implemented in response Key findings that informed the human factors validation test protocol
7	Description and categorization of critical tasks  Process used to identify critical tasks List and descriptions of critical tasks Categorization of critical tasks by severity of potential harm Descriptions of use scenarios that include critical tasks
8	Details of human factors validation testing Rationale for test type selected (i.e., simulated use, actual use or clinical study) Test environment and conditions of use Number and type of test participants Training provided to test participants and how it corresponded to real-world training levels Critical tasks and use scenarios included in testing Definition of successful performance of each test task  Description of data to be collected and methods for documenting observations and interview responses Test results: Observations of task performance and occurrences of use errors, close calls, and use problems  Test results: Feedback from interviews with test participants regarding device use, critical tasks, use errors, and problems (as applicable)   Description and analysis of all use errors and difficulties that could cause harm, root causes of the problems, and implications for additional risk elimination or reduction

Each one of the items listed will include one or more documents that address in some manner each one of the sub-points listed in each one of the items. You could place these documents into a folder, identify where they belong and submit this to the FDA or other regulatory body. However, I advocate something more: write an over arching narrative that takes the reader through the path of not only what was performed but why as well. The narrative can be and probably should be based on the outline above. (I'm sure there are exceptions, but beginning with the outline above should be your starting point.)

The narrative should provide a comprehensive flow that interconnects each of the phases and explains reasoning for what was done including the rationale behind the design and operation of the system as well as why it's appropriate for use by the identified user population as well as other likely populations who might encounter it.

Why Write a Narrative?

Is a narrative required as part of a submission? From all that I can tell: no, it's not a requirement as part of a submission to regulatory bodies. 

However, consider the fact that if you can't explain the logic of what you did to yourselves, how hard will it be for a reviewer to comprehend? And if the reviewer can't comprehend what you did -- including the reasoning and logic behind it -- could your submission be at risk for rejection? The answer is "yes," you may be putting your submission at risk for rejection.

The narrative is analogous to a completed jigsaw puzzle. A human engineering file without a narrative is analogous to just the jigsaw puzzle pieces. Yes everything is there, but what is it suppose to be? 

Submitting a human engineering file that includes a comprehensive narrative can insure that your submission is understandable: to you as well as your reviewers. It can insure that there are no gaps or issues that should have been included in your submission are left out. Again, going back to the jigsaw puzzle analogy, you don't know that you've got a missing piece or pieces until you've assembled the puzzle.

The narrative provides reviewers with framework to understand what you've done. Interestingly enough this will likely minimize any questions reviewer might have about your submission. And will likely minimize the likelihood that you'll get questions that you cannot answer.

One more thing to note: if your narrative is clear and comprehensive, it's likely that the reviewer or reviewers will often read no further or will simply scan the foundational documents to insure that the foundational documents do in fact support what is stated in the narrative. This could speed the regulatory review and acceptance. 

More Articles on This Topic

I'll be writing a series of articles on the topic of human engineering file submission narratives over the next week or two. I'll focus on specific areas of the narrative and discuss some of what I have done with regards to putting together narratives for submission to regulators.