Click here for Update: The Final Rule for the 2015 edition certification has been released.

On page 191 of the 431 page document from ONC about the proposed changes to the Meaningful Use program for Stage 3, ONC begins to identify several significant changes to the Safety-enhanced Design (a.k.a.usability) testing requirements.

ONC is actively seeking public comment on these proposed changes. The PDF of the new 2015 Edition IT certification criteria and public comment form can be found here: https://www.federalregister.gov/articles/2015/03/30/2015-06612/2015-edit...

We have created an informational page about the required testing at http://www.theusabilitypeople.com/ehr-usability-testing-onc-2015-certifi...

Below is a cut and paste of the section on Safety-Enhanced Design (from pages 191 to 196 of the ONC document referenced above)

Safety Enhanced Design 2015 Edition Health IT Certification Criterion § 170.315(g)(3) (Safety-enhanced design)

We propose to adopt a 2015 Edition “safety-enhanced design” (SED) certification criterion that is revised in comparison to the 2014 Edition “safety-enhanced design” criterion.

We propose to add certification criteria to this criterion that we believe include capabilities that pose a risk for patient harm and, therefore, an opportunity for error prevention. We propose to provide further compliance clarity for the data elements described in NISTIR 7742 174 that are required to be submitted as part of the summative usability test results and to specifically include these data elements as part of the certification criterion.

Certification Criteria Identified in the SED Criterion for UCD Processes

We propose to include seventeen (17) certification criteria (seven are new) in the 2015 Edition SED certification criterion, as listed below (emphasis added for new criteria). For each of the referenced certification criteria and their corresponding capabilities presented for certification, user-centered design (UCD) processes must have been applied in order satisfy this certification criterion.

§ 170.315(a)(1) Computerized provider order entry – medications
§ 170.315(a)(2) Computerized provider order entry – laboratory
§ 170.315(a)(3) Computerized provider order entry – diagnostic imaging
§ 170.315(a)(4) Drug-drug, drug-allergy interaction checks
§ 170.315(a)(5) Demographics
§ 170.315(a)(6) Vital signs, BMI, and growth charts
§ 170.315(a)(7) Problem list
§ 170.315(a)(8) Medication list
§ 170.315(a)(9) Medication allergy list
§ 170.315(a)(10) Clinical decision support
§ 170.315(a)(18) Electronic medication administration record
§ 170.315(a)(20) Implantable device list
§ 170.315(a)(22) Decision support – knowledge artifact
§ 170.315(a)(23) Decision support – service
§ 170.315(b)(2) Clinical information reconciliation and incorporation
§ 170.315(b)(3) Electronic prescribing
§ 170.315(b)(4) Incorporate laboratory tests/results

The continued submission of summative usability test results promotes transparency and can foster health IT developer competition, spur innovation, and enhance patient safety. With this in mind, we also seek comment on whether there are other certification criteria that we omitted from this proposed SED criterion that commenters believe should be included.

NISTIR 7742 Submission Requirements

In the 2014 Edition final rule, we specified that the information listed below from the NISTIR 7742 “Customized Common Industry Format Template for Electronic Health Record Usability Testing” (NIST 7742)175 was required to be submitted for each and every one of the criteria specified in the 2014 Edition SED criterion (77 FR 54188). For the 2015 Edition SED criterion, we propose to include the information below in the regulation text of the 2015 Edition SED criterion to provide more clarity and specificity for the information requested to be provided to demonstrate compliance with this certification criterion. The findings that would be required to be submitted for each and every one of the criteria specified in the 2015 Edition SED criterion (and become part of the test results publicly available on the Certified Health IT Product List (CHPL)) are:

• Name and version of the product
• Date and location of the test
• Test environment
• Description of the intended users
• Total number of participants
• Description of participants as follows:
  • Sex
  • Age
  • Education
  • Occupation/role
  • Professional experience
  • Computer experience
  • Product experience
• Description of the user tasks that were tested and association of each task to corresponding certification criteria
• List of the specific metrics captured during the testing
  • Task Success (%)
  • Task Failures (%)
  • Task Standard Deviations (%)
  • Task Performance Time
  • User Satisfaction Rating (Scale with 1 as very difficult and 5 as very easy)
• Test results for each task using metrics listed above
• Results and data analysis narrative:
  • Major test finding
  • Effectiveness
  • Efficiency
  • Satisfaction
  • Areas for improvement

There are illustrative tables on pages 11 and 20 in NISTIR 7742 that provide examples of the presentation of test participants and test results data. We specify that all of the data elements and sections specified above must be completed, including “major findings” and “areas for improvement.” Pages 18 and 19 of the NISTIR 7742 contain a table with suggested instructions for data scoring specifically noting that for task success, a task is counted as successful if the participant was able to achieve the correct outcome without assistance and within the time allotted on a per task basis. Likewise, for task satisfaction a 5 point Likert scale is recommended with scores ranging from “1 - very difficult” to “5 – very easy.” The NISTIR 7742 includes several sections: Executive Summary, Introduction, Method, and Results. In each of these sections, there are required data elements – and some of these elements call for the reporting of the number of study participants, their level of experience with EHR technology and other pertinent details.

We recommend following NISTIR 7804176 “Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records” for human factors validation testing of the final product to be certified. In accordance with this guidance, we recommend a minimum of 15 representative test participants for each category of anticipated clinical end users who conduct critical tasks where the user interface design could impact patient safety (e.g., physicians, nurse practitioners, physician assistants, nurses, etc.). The cohort of users who are selected as participants will vary with the product and its intended users; however, the cohort should not include employees of the developer company. We specify the submission of demographic characteristics of the test participants comparable to the table on page 11 of NISTIR 7742 because it is important that the test participant characteristics reflect the audience of current and future users. In accordance with NISTIR 7804 (page 8), we recommend that the test scenarios be based upon an analysis of critical use risks for patient safety which can be mitigated or eliminated by improvements to the user interface design.

In lieu of simply providing guidance on the number of, and user cohort for, test participants, we request comment on whether we should establish a minimum number(s) and user cohort(s) for test participants for the purposes of testing and certification to the 2015 Edition under the ONC Health IT Certification Program.

New Requirements and Compliance Guidance

As we noted in the 2014 Edition final rule (77 FR 54188), examples of method(s) that could be employed for UCD include ISO 9241-11, ISO 13407, ISO 16982, ISO/IEC 62366, ISO 9241-210 and NISTIR 7741. The UCD process selected by a health IT developer need not be listed in the examples provided in order to be acceptable. We do, however, strongly advise health IT developers to select an industry standard process because compliance with this certification criterion requires submission of the name, description, and citation (URL and/or publication citation) of the process that was selected. In the event that a health IT developer selects a UCD process that is not an industry standard (that is, not developed by a voluntary consensus standards organization), but is based on one or more industry standard processes, the developer may name the process(es) and provide an outline of the process in addition to a short description as well as an explanation of the reason(s) why use of any of the existing UCD standards was impractical.

Health IT developers can perform many iterations of the usability testing, but the submission that is ultimately provided for summative usability testing and certification must be an expression of a final iteration. In addition, we expect the test scenarios used to be submitted as part of the test results. Last, we note that we do not expect developers to include trade secrets or proprietary information in the test results.

Request for Comment on Summative Testing

We understand that some health IT developers are concerned that the summative testing report may not adequately reflect the design research that has been performed throughout a product’s lifecycle. We request public comment regarding options that we might consider in addition to – or as alternatives to – summative testing. For example, if formative testing reflects a thorough process that has tested and improved the usability of a product, could a standardized report of the formative testing be submitted for one or more of the 17 certification criteria for which summative testing is now required? What would be the requirements for this formative testing report, and how would purchasers evaluate these reports?

Retesting and Certification

We believe that ONC-ACB determinations related to the ongoing applicability of the SED certification criterion to certified health IT for the purposes of inherited certified status (§170.550(h)), adaptations and other updates would be based on the extent of changes to user interface aspects of one or more capabilities to which UCD had previously been applied. We believe that ONC-ACBs should be notified when applicable changes to user-interface aspects occur. Therefore, we include these types of changes in our proposal to address adaptations and updates under the ONC-ACB Principles of Proper Conduct (§ 170.523). Please see section IV.D.6 of this preamble for further discussion of this proposal.

We are glad that ONC has expanded the number of areas that need to be tested, provided some guidance on the number of participants required, and has shed some light on when an EHR needs to be retested due to significant changes to the user-interface. We are looking forward to providing public comment on these, and some of the accessibility issues (see pages 164-166 of the ONC Document) as part of the ONC HITPC Implementation, Usability, & Safety Workgroup.