Case Study: System Rewrite and UI Testing

The Challenge

Take a 30-year-old system, which we will call the rating engine, that already boasts a 99.7% rate of automation and rewrite it in upgraded technologies without losing any automation. On top of that, reorganize the UI to improve the user’s speed of decision making for the 0.3% that requires manual intervention. No documentation existed, and each user interacted with the system differently.

Part of the challenge was understanding how to improve user efficiency with software when we did not know how they used the software to make their decisions.

To gain insight, we enlisted the help of research professionals: graduate and PhD students. The TD MASH development team partnered with the local university and its lab staff to study how users interacted with the 30-year-old software.

Background of Knowledge Work

Work that requires individuals to interact with computer data to make decisions is called knowledge work. At the time of this project, no formal process for evaluating knowledge work existed. Traditional labor had tried-and-true practices for evaluating work because the work could be easily observed and measured.

You could watch someone pick up a box 50 times in a day. You could measure the weight of the boxes to determine that the maximum weight was 50 pounds. You could observe and time how long someone needed to be on their feet to complete the job. These benefits of observation and measurement allowed management to clearly define job requirements and reorganize operations to improve efficiency.

With knowledge work, observation and measurement are not as simple as watching a task. Thus, the research question became: how can the TD MASH development team improve user efficiency with software when we do not know how users use the software to make their decisions?

The Study

To gain clarity around the knowledge work needed for operating the rating engine, graduate and doctoral students designed a study using advanced technologies. They developed a multi-staged approach that involved a survey, work observation, and interview.

The survey asked users questions regarding their interactions with the rating engine. It asked users to order the information they used to make decisions. It also asked users to measure their satisfaction with the system.

The work observation utilized eye-tracking software, screen-recording devices, and galvanic skin response devices to capture and measure user interactions with the rating engine. In this stage, users were asked to complete their daily work while the devices captured data.

Lastly, the researchers interviewed the users in a manner similar to film review. They took the screen recordings from the previous stage, played them back with the users, and asked the users to talk through what they were looking for, where they found the information they needed, and what they did with that information. The results from this final stage were codified to find similarities across the user base.

Technologies and platforms used included:

Google Forms
Eye-tracking Software
GSR Devices
Screen Recording Software
Balsamiq
Java
Angular
HTML
CSS
Ricola
Spring Boot Microservices
Oracle

Study Findings

The study allowed us to measure how many screens were used, how many clicks were required, and how long it took for users to make decisions. Quantifiable data that previously was not available became visible. It also allowed us to observe similarities in user work patterns and redundancies in the software. All this data proved invaluable in the redesign of the rating engine.

For the students, they were able to design a methodology for evaluating knowledge work. That led to a publication in a journal for IO psychology and a presentation at SIOP’s annual conference.

The Redesign

The TD MASH development team was able to take the findings and redesign the rating engine’s UI with fewer screens.

The UI was also able to reorganize the data on the screen, so the most useful information was at the top left of each screen. The results were a piece of software that was easier to support and one that provided users with the most prevalent information in fewer clicks.

In other words, the redesign improved efficiency on the front end and back end.