User-Centered Design: Helping users become Effective, Efficient, and Satisfied

Starting at Stage 2 of meaningful use, The Office of the National Coordinator for Health Information Technology (ONC) released a set of certification and meaningful use requirements for electronic health records (EHRs). These requirements have been expanded for the ONC 2015 Edition certification (MACRA,MIPS). These require that EHR vendors include evidence of user-centered design and user test results in their certification submission. To be able to obtain the 2015 Edition certification, EHR vendors must follow a formal User Centered Design (UCD) process and perform summative usability testing on specific areas of the product.

In a public comment on the certification process, Drummond (one of the ONC ATCB's) stated that

"...the usability test effort is not a small undertaking for EHR vendors. In fact, it {is} essentially one of the most difficult criteria to complete."

Click here to sign up for Summative testing for your EHR:

We recommend that EHR vendors follow the ISO 9241-11 standard. ISO-9241 Part 11: (1998) pertains to the extent to which a product can be used by specified users to achieve specified goals with Effectiveness (Task completion by users), Efficiency (Task on time) and Satisfaction (responded by user in term of experience) in a specified context of use (users, tasks, equipment & environments).

Summative testing

Summative usability testing is an industry standard usability methodology (see http://bit.ly/13WYqTU ) During the testing a trained user experience professional works with the EHR vendor to create a series of representative tasks and then measures the amount of time it takes to complete the task (time on task), how many and what type of errors occur (error rates), and satisfaction with the task interaction (user satisfaction). The results —as measured by things like task times, assisted and un-assisted completion rates and standardized satisfaction scores --can be benchmarked against current industry standards.

EHR developers can and should perform many iterations of user testing. These early iterations are often called Formative testing. We also recommend that an Expert (Heuristic) review be performed before any of the more formal usability testing. ( see also: Please don’t air our dirty laundry! ) This is a cost-saving measure that can significantly reduce the number of iterations required and ultimately reduce the number of “items needed improvement” presented in the ONC submission.

The submission that is ultimately provided for testing and certification should be the expression of a final iteration in which few areas for improvement would be identified.

How should the Results Presented?

Results from the summative study should be created and presented to the ONC using the Customized Common Industry Format (CCIF) Template for EHR Usability Testing (NISTIR 7742 see http://1.usa.gov/gAZFDt )These reports typically include an Executive Summary, an introduction, a method section, discussion of the results and any appendices
.

The following are required for the 2015 Edition submission to the ONC:

Name and version of the product
Date and location of the test
Test environment
Description of the intended users
Total number of participants
Description of participants as follows:
Sex
Age
Education
Occupation/role
Professional experience
Computer experience
Product experience
Description of the user tasks that were tested and association of each task to corresponding certification criteria
List of the specific metrics captured during the testing
Task Success (%)
Task Failures (%)
Task Standard Deviations (%)
Task Performance Time
User Satisfaction Rating (Scale with 1 as very difficult and 5 as very easy)
Test results for each task using metrics listed above
Results and data analysis narrative:
Major test finding
Effectiveness
Efficiency
Satisfaction
Areas for improvement

Summative testing for the 2015 certification needs to test the following 12 areas:

Section 170.315(a)(1) Computerized provider order entry – medications
Section 170.315(a)(2) Computerized provider order entry – laboratory
Section 170.315(a)(3) Computerized provider order entry – diagnostic imaging
Section 170.315(a)(4) Drug-drug, drug-allergy interaction checks
Section 170.315(a)(5) Demographics (New)
Section 170.315(a)(6) Problem list (New)
Section 170.315(a)(7) Medication list
Section 170.315(a)(8) Medication allergy list
Section 170.315(a)(9) Clinical decision support
Section 170.315(a)(14) Implantable device list (New)
Section 170.315(b)(2) Clinical information reconciliation and incorporation
Section 170.315(b)(3) Electronic prescribing

For more information on the Safety-enhanced Design requirements for the 2015 Edition Certification, see SafetyEnhancedDesign.com

For expert Usability evaluations of your companies EHR: Contact The Usability People


Usability reports produced by The Usability People adhere to the NISTIR 7742 CCIF format suitable for submission to an ONC-ACB. We can also provide a set of prioritized, detailed, and actionable feedback that you can take directly to your design and development teams for implementation before your summative testing.