This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.
Clinical decision support systems (CDSS) are important tools to improve health care outcomes and reduce preventable medical adverse events. However, the effectiveness and success of CDSS depend on their implementation context and usability in complex health care settings. As a result, usability design and validation, especially in real world clinical settings, are crucial aspects of successful CDSS implementations.
Our objective was to develop a novel CDSS to help frontline nurses better manage critical symptom changes in hospitalized patients, hence reducing preventable failure to rescue cases. A robust user interface and implementation strategy that fit into existing workflows was key for the success of the CDSS.
Guided by a formal usability evaluation framework, UFuRT (user, function, representation, and task analysis), we developed a high-level specification of the product that captures key usability requirements and is flexible to implement. We interviewed users of the proposed CDSS to identify requirements, listed functions, and operations the system must perform. We then designed visual and workflow representations of the product to perform the operations. The user interface and workflow design were evaluated via heuristic and end user performance evaluation. The heuristic evaluation was done after the first prototype, and its results were incorporated into the product before the end user evaluation was conducted. First, we recruited 4 evaluators with strong domain expertise to study the initial prototype. Heuristic violations were coded and rated for severity. Second, after development of the system, we assembled a panel of nurses, consisting of 3 licensed vocational nurses and 7 registered nurses, to evaluate the user interface and workflow via simulated use cases. We recorded whether each session was successfully completed and its completion time. Each nurse was asked to use the National Aeronautics and Space Administration (NASA) Task Load Index to self-evaluate the amount of cognitive and physical burden associated with using the device.
A total of 83 heuristic violations were identified in the studies. The distribution of the heuristic violations and their average severity are reported. The nurse evaluators successfully completed all 30 sessions of the performance evaluations. All nurses were able to use the device after a single training session. On average, the nurses took 111 seconds (SD 30 seconds) to complete the simulated task. The NASA Task Load Index results indicated that the work overhead on the nurses was low. In fact, most of the burden measures were consistent with zero. The only potentially significant burden was temporal demand, which was consistent with the primary use case of the tool.
The evaluation has shown that our design was functional and met the requirements demanded by the nurses’ tight schedules and heavy workloads. The user interface embedded in the tool provided compelling utility to the nurse with minimal distraction.
Clinical decision support systems (CDSS) are important tools to improve health care outcomes and reduce preventable medical adverse events [
However, the effectiveness and success of CDSS depend on their implementation context and usability in complex health care settings (eg, [
In particular, CDSS implementations often suffer from poor usability, which directly impacts their adoption and effectiveness. For instance, user interface (UI) workarounds have been shown to greatly diminish the effectiveness of widely used CDSSs [
In this study, we developed a novel CDSS for the CHRISTUS St. Michael health system (a 350 bed acute care hospital) to help frontline nurses better manage critical symptom changes in hospitalized patients. The CDSS is currently undergoing clinical pilots inside the hospital. The goal of the CDSS was to reduce preventable failure to rescue (FTR) cases in the hospital. Since the nursing work environment is subject to constant interruptions and is error prone [
In this paper, we will discuss the design, evaluation, implementation, and validation of the CDSS UI. We will present several innovations in nursing CDSS UI design, especially on large touch screen devices. The internal algorithmic design and the validation of decision rules, however, are beyond the scope of this paper. In the next section, we will start with a brief clinical background of the nursing CDSS tool.
The FTR is a leading patient safety indicator with the highest incident rates among all indicators according to a recent large-scale study [
FTRs are often considered preventable because the symptoms of a deteriorating patient could present hours before the rescue starts. Examples of such critical symptom change include patient complaint of a new pain, mental status change, and difficulty breathing etc. Studies have indicated that many FTRs could have been averted if the critical symptoms in patients were captured, evaluated, and communicated early.
It was suggested that the nurses’ early recognition, evaluation, and decision making of symptom signs could play an important role in FTR [
Simply detecting and evaluating the critical symptom changes is not enough. The potential complication must be communicated to the rest of the clinical team, and be escalated to the right team members in order to organize effective interventions. It was argued that FTRs are often caused by the failure to communicate [
Frontline nurses are often the first to notice critical symptom changes. Their decisions at the point-of-care are crucial factors determining whether FTR events can be reduced. However, at the same time, nurses are ill equipped to manage critical symptom changes in hospitals.
The frontline nursing staff in most hospitals have very high workloads, need to manage extensive multitasking, and are fatigued [
The average skill and training levels of nurses do not adequately prepare them to evaluate potentially complex symptom changes. A study found that a 10% increase in the proportion of nurses holding a bachelor’s degree was associated with a 5% decrease in the odds of FTR [
While the RRT is a proven effective intervention for FTR, RRT resources can be under-utilized [
The hieratical structure in hospitals is known to impede nurse decision-making process [
A specially designed CDSS could potentially help the nurse address the above issues related to critical symptom changes and FTRs. Such CDSS requires special design considerations for two reasons.
First, the system must be tailored to the nurses’ training and cognitive levels, and generate action items that are appropriate for the nurse. Most floor nurses have gone through less than 4 years of medical training after high school, and they do not have independent authority to treat the patient without the physician’s prescription.
Second, the system must be adapted to the fast paced workflow during a rescue operation. The tool must be ubiquitous, instant on, and provides useful feedback in merely minutes. The application should enhance real-time communication across team members, as opposed to bringing in another computer that impedes face-to-face communication.
Both challenges highlight the need for a novel design, and formal evaluation of the system UI and workflow.
Human-computer interaction and workflow designs are crucial for the success of clinical informatics projects. A large body of research has been devoted to study methods and techniques to evaluate usability of systems.
Early efforts focused on creating human models and breaking down tasks into small pieces that could be directly measured and optimized for user performance. For instance, the goals, operators, methods, and selection rules family of frameworks [
For cognitive systems, analysis of the UI itself is a key aspect of usability design, because UI design often has a deterministic effect on user performance. Research in cognitive theory has indicated that different visual representation of the same underlying work problem could produce dramatically different user performance in terms of ability to complete tasks correctly and productivity [
Furthermore, complex work often requires collaboration of multiple users. It was demonstrated that cognition can be distributed across multiple users working on the same system [
A popular design approach that works with the above cognitive design principles is the work-centered design (WCD) [
A particularly interesting application of distributed cognition and WCD in the medical informatics field is the UFuRT (user, function, representation, and task analysis) [
User analysis is used to identify users and stakeholders of the work product, and document their needs and objectives. The user requirements are translated into system design requirements in this process.
Function analysis aims to generate an essential description of the work. The UFuRT process calls for a 4-step analysis to detail the dimensions, constraints, relations, and finally operations.
Representational analysis is the design process to identify and determine the implementation representations of relations among the dimensions identified in the functional analysis. The representation includes UIs and workflows for different types of users of the system. Representational analysis is a crucial step of the design process since it has been convincingly demonstrated that different representations of the same task can have very different impacts on the user’s efficiency and productivity [
Task analysis is to identify steps by a specific user on a specific representation in order to carry out an operation.
In the context of our project, we used UFuRT framework to analyze software requirements and inform the specification. Hence, we focused on user analysis and UI design aspects of representation analysis. We performed a high-level functional analysis and did not perform task analysis in the design stage. The reason was that complete functional and task analysis require full knowledge of every detail of the product, which would not provide enough flexibility for our iterative software development process.
The overall objective of the system was to help prevent patient safety events during critical changes. Through interviews with hospital-based clinicians, we have specifically identified symptom evaluation and escalation as the 2 main functional goals of the CDSS.
While nurses do not make diagnoses, they are the first to recognize and evaluate the patient symptom changes. Based on their evaluation, they would decide how to (or whether to) coordinate further care, and their evaluation results are often accepted by the team as the basis of a formal diagnosis.
Existing diagnostic CDSS tools provide a proven framework to help reduce errors in diagnostic evaluation, and improve documentation of the clinical findings that lead to diagnoses. Specially, the CDSS needs to provide 2 core functionalities.
For many critical symptom changes, there are multiple possible diagnoses. An example is that a hospitalized patient suddenly feels chest pain. The chest pain could be an indicator of heart attack, which needs to be attended to by a cardiologist or surgery team immediately; or the chest pain could indicate reflux or indigestion, which is a rather common condition that is simple to treat.
The frontline nurses typically do not have enough medical training and experience to thoroughly evaluate those potential diagnostic outcomes. The CDSS should provide specific instructions for the nurse to follow, and then make recommendations on what to do next. For instance, it should provide specific instructions on whom to call and what to say during the call for each potential diagnosis. The system does not replace human decision-making or training, but it provides support to help nurses deal with complicated emergent situations to the best of their capabilities.
Common cognitive errors that lead to diagnostic errors include premature closure, anchoring, confirmatory bias, and framing [
To reduce framing and premature closure, the CDSS should encourage and prompt the clinicians to check all possible diagnostic outcomes, especially severe outcomes that lead to FTRs. The CDSS should also prompt the clinicians to verify all important symptoms and findings related to major diagnostic outcomes to minimize missed diagnoses.
To reduce anchoring or confirmatory bias, the CDSS should present an objective estimate of likely diagnoses and suggested clinical actions based on the current findings. The objective probability estimate could reduce the user’s reliance on reconceived decision biases.
Teamwork is one of the few proven approaches to improve patient safety and care quality in hospitals [
As we discussed in the clinical background, RRT is an effective approach to help reduce FTR when it is deployed correctly. Our CDSS aimed to improve the effectiveness of the RRT by activating RRT early and making RRT mandatory when the nurse detects certain warning signs.
The CDSS needs to provide an easy and non-intrusive way to automatically alert the RRT at appropriate times. The RRT consists of more experienced clinicians, and they can decide whether or when to respond to those alerts. At the same time, it is important for the CDSS to clearly notify the nurse when it sends alerts to the RRT and the status of the alerts. The user must feel that he/she is in full control in order to effectively utilize the system.
If the floor nurse determines that the patient needs assistance from a physician, he/she would call the physician and explain the situation. The conversation could be a frustrating experience for both the nurse and the physician due to different expectations. That could result in the physician losing confidence in the nursing staff, and nurses delaying calls to physicians. The system should provide tools to help nurses communicate better with physicians in emergency situations.
We used the UFuRT framework as a conceptual guide to develop the software specification for the CDSS tool. Specifically, we identified users of the system, and documented use case stories for each user (ie, user analysis). We identified high-level functions the system must perform to meet the user requirement (ie, functional analysis). And finally, we created visual representations of the UI that can best accomplish those functions (ie, representational analysis). The UFuRT task analysis was not conducted at the design stage. Instead, the tasks were evaluated as part of the user evaluation process described later in this article.
Users of the proposed CDSS were members of the clinician team responsible for rescuing patients in the hospital. They included floor nurses, RRT nurses, and physicians. The user roles described in this section were based on interviews with hospital clinicians.
The primary users of the CDSS were the floor nurses. The system presented information and actions that were appropriate to the floor nurses. Specifically, the system could not present medical content that required MD-level training to understand, or ask nurses to make diagnostic decisions on their own. The CDSS also could not instruct the nurse to perform clinical actions that he/she was not authorized or qualified to do, such as performing advanced examinations, ordering labs, or writing prescriptions. Furthermore, a key characteristic in the floor nurse’s work environment is that they are very busy and have established workflows. The system added minimal overhead to the existing workflows.
If the floor nurse detected a potential problem, the RRT nurse was the next escalation step. RRT nurses are typically paged by the hospital internal communication system, and hence the CDSS must support paging the RRT. The system should give RRT nurses more options as they have the authority to perform standing orders on patients. Finally, when the RRT nurse arrived at the bedside, in order to minimize errors at the hand-off of care, it was important for the CDSS to have clear documentation on the findings and actions that have been performed by the floor nurse so far.
The physician in charge of the patient should be notified when there is a probable problem with the patient. The system should provide accurate and concise summaries of the patient condition for the nurse to read to the physician when talking on the phone.
Once the user requirements were determined, we developed a list of high-level functions the system must perform. Please note that we did not create a detailed catalog of functions at this stage of development. Instead, we focused on high-level operations in order to provide implementation flexibility. Key operations of the system include the following:
Identify the symptom change that triggers the use of the system
Identify a list of potential diagnoses
Identify a list of potential clinical findings that will reject or affirm those diagnoses
Enter clinical findings
Re-evaluate the probabilities for each diagnosis after each finding
Repeat for all finds until a diagnosis becomes highly likely
Identify the action items for this diagnosis
Identify the escalation path for this diagnosis
Perform operations required in the action items list
In addition, we have also identified non-essential operations that were related to the specific design of the system. Such operations included user login to the system with badge number, synchronization of the device content with online repositories, user entry of the patients’ room number, and user configuration of the device for display options.
The UI of the product was designed to address operations listed in the previous section. It aimed to present a familiar and non-intrusive interface to the user at the point-of-care. In this section, we describe key features of the UI.
We decided to implement the UI on a touch screen consumer tablet device. The reason behind choosing a tablet device was that it can be accessed anytime, anywhere, and could be carried around by the clinician or be made available at the bedside. The tablet device was connected to the hospital secure WiFi system to access medical records, alert RRT and other teams, and update clinical content as needed.
The choice of a consumer tablet, as opposed to a dedicated medical device, was due to two reasons. First, the consumer device was much cheaper to deploy. A consumer iPad costs less than one third of a special purpose tablet PC on the market. Second, the consumer device featured an UI that the nurses were already familiar with due to his or her use of similar devices at home.
The most widely used and user-friendly consumer tablet device on the market is the Apple iPad, which we chose as the implementation platform for the CDSS device.
Most existing diagnostic decision support tools use decision trees [
Instead, we decided to use another UI metaphor that is commonly used in hospital environments—the medical checklist. The main UI of the system was a dynamic checklist for the nurse to go over and examine clinical findings related to the patient. Checklists have been shown to reduce medical errors [
The list on the left shows potential causes for the patient's critical change (ie, the diagnostic outcomes). The causes were listed in order of their probabilities based on the current findings from the checklist items on the right panel.
All the user needed to do was to follow the checklist and enter a simple yes/no answer to the findings. With each yes/no answer, the system automatically recalculated and redisplayed the diagnostic outcome probabilities and the priorities of the remaining checklist items.
The nurses could go through the findings checklist in any order. The nurses could also undo any choices to go back to any previous state. That allowed the nurses to pick and choose tasks that happen to fit the existing workflow at any point of the process. There was no need to interrupt the flow just to provide a finding required by the software.
This is different than the typical decision tree or flow chart decision models, where the workflow is dictated by the software system.
The main split screen user interface of the decision support system.
The CDSS was connected to the hospital communication system, and it automatically sent out pages to the RRT as the nurse works on the patient. The RRT members could then decide whether to intervene depending on how severe the patient condition was as reported by the nurse through the device.
If the RRT decided to intervene, they could simply take over the CDSS device, which has documentation of the findings the nurse had already completed.
The CDSS provided a standard list of items for the nurses to go through with the physicians when a likely diagnosis emerged (
The action items were reviewed and approved by the physicians in the hospital, and they were designed to enable physicians to make quick decisions over the phone.
The action items for the nurse after a likely outcome is reached.
The CDSS system was implemented as a client-server computer application. The main component of the system was an iPad application developed in Objective C using the Apple iOS software development kit. The iPad application provided all the UI elements described in the design, and it was the only UI device the nurses needed to interact with during the patient evaluation process. The iPad application contained a SQLite-based relational database to store decision rules, medical content, user credentials, and usage logs. The application required access to the hospital’s secure WiFi network in order to send paging messages to the RRT members. Except for the RRT page, the iPad device could function entirely without network connectivity, and only needed to occasionally synchronize with the backend database for content updates.
The second component of the system was an online content management system (CMS) to manage the decision rules, medical contents, and authorized users and devices. The system was designed as a Web application built on Java Enterprise Edition running on Tomcat and MySQL database servers. The interface with the iPad device was programmed as RESTful XML Web services. The CMS had a human UI that visualized the content and allowed CRUD (create, retrieval, update, and delete) operations of the content items from any Web browser. Proper user authorization was enforced in the CMS so that only users with certain roles (eg, physicians and managers) could update the content.
The CMS also provided an interface for the physician reviewers to review cases based on the usage log of the iPad device. That supplemented the brief information recorded in formal medical records and provided insights into how to improve the system in the future.
In the next two sections, we will discuss evaluations and validations we performed on the CDSS, especially the iPad UI.
The clinical rule editor in the Web-based CMS.
The UI and workflow design of the product was evaluated using heuristic evaluation and performance-based end user evaluation. The heuristic evaluation was done after the first prototype, and its results were incorporated into the product before the performance-based evaluation was conducted.
Heuristic evaluation is a formal UI evaluation method designed to uncover potential problems in a product [
In this project, we incorporated heuristic evaluation into the iterative product design and development process. Based on the functional requirements outlined earlier in this paper, we built a first prototype, conducted heuristic evaluation, and then improved the prototype by addressing the heuristic violations identified by the evaluators.
It was demonstrated that the evaluators who are experts in both UI design and the specific application domain tend to be most effective in identifying heuristic violations [
The evaluators went through all UI elements in the application, and used the 10 heuristics in the computer software for evaluation [
The heuristic violations were entered into an issue tracking system for the engineering team. The product reached its first release after all heuristic violations rated 3 and above were fixed.
Once the first release of system was developed, we assembled a panel of nurses to evaluate the UI and workflow via simulated use cases. The panel consisted of 10 nurses from our target user group in the hospital. The panelists had varied education background and experience levels. There were 3 licensed vocational nurses and 7 registered nurses on the panel. All of them were non-rapid response nurses working full time on the floor. Their work experience ranged from 1 to 39 years, with a median of 23 years. The simulation study was conducted as follows.
The nurse enters a patient room to meet the study monitor. The monitor gives a trigger symptom verbally to the nurse.
The nurse goes back to the station and fetches the tablet device. On the way, he/she will enter badge number, room number, and select the trigger symptom from a list.
When the nurse enters the room again, he/she can go through the checklist in any order. The nurse will verbally ask the monitor questions on the checklist, and the monitor will provide a yes/no answer.
When the nurse has received enough information, he/she decides on a likely diagnostic outcome for the patient.
The nurse will read out aloud each of the action item associated with the diagnostic outcome.
The process was repeated 3 times for each nurse. The tablet device automatically logged usage during the sessions.
We recorded whether each nurse successfully completed each session. The first session for each nurse was considered a training session to get the nurse familiar with the device, and was not included in the evaluation results. The success criterion was to have the nurse walkthrough the entire process and reach the action items without external help.
For each session, we recorded the entire duration from the time the nurse walked into the room to the point where the nurse finished reading the action items. The completion time was an estimate of how much overhead time the use of the device added to the whole workflow. Since the product was designed to help nurses make quick decisions in urgent situations, it was crucial that the tool does not introduce too much overhead on its own. The evaluation criterion for the tool was that it should add less than 5 minutes of overhead to the existing clinical workflows.
After each session, the nurse was asked to use the National Aeronautics and Space Administration (NASA) Task Load index [
In
A total of 83 heuristic violations were identified in the studies.
The released version of the product had all heuristic violations rated 3 and above fixed. In this study, heuristic evaluation conducted by experts improved the usability of the product.
The 10 nurses on the panel successfully completed all 30 sessions of the performance evaluations. All nurses were able to use the device after a single training session with the instructor.
For each nurse, we took the median completion time from the 3 sessions, and then calculated the mean and standard deviation across the 10 nurses. On average, the nurses took 111 seconds (SD 30 seconds) to complete the simulated task. That is well within the 5 minutes overhead goal that we had set.
The NASA Task Load Index results indicated that the work overhead on the nurses was low. In fact, most of the burden measures were consistent with zero, as seen in
Example heuristic violations.
Heuristics violated | Place of occurrence | Severity | Usability problem description |
Visibility of system status | Start | 3.8 | When syncing the application, there was no way to know if it will take 15 seconds or 10 minutes. It would be nice to know that it will take approximately 1 minute or show a percent completion. |
Match between system and the real world | Outcome | 3.4 | List the outcomes as percentages instead of just a number without percentages. |
User control and freedom | Checklist | 4 | The user should have the ability to change an answer once it has gone down to the list of answered questions. I can see frustration with the process if you have to completely start over to change an answer. |
Consistency and standards | Outcome | 1 | Color code should be far apart along the visible spectrum so that the outcome can be clearly distinguished. |
Error prevention | Checklist | 4 | Have the user confirmation when backing out of a screen that would cause the user to have to reenter all data. |
Recognition rather than recall | Checklist | 2 | Abbreviations are used in the checklist. It should follow a simple primary rule. |
Flexibility and efficiency of use | Checklist | 3 | If we add future triggers, there needs to be a way to ensure that when the keyboard displays that it does not cover the last triggers. Currently it is not a problem but should build this into system now. |
Aesthetic and minimalist design | Outcome | 3 | There were too many "start over" displays currently. It would be simpler to have 1 button with a drop down screen listing the options: trigger, patient, or user. The questions also need to be reviewed by Dr. Finley and the RRT as currently there are a few questions that ask the same thing, but are just worded differently, and duplicating the questions is unnecessary. |
Help user Recognize, diagnose, and recover from errors | Start | 4 | When a user accidentally hit the home button on iPad, the system will close without any warning and all data will be lost. Restarting within 1 minute allows you to get back to where you were. Otherwise the program will close. |
Documentation and help | Outcome | 3 | The outcomes are in different colors. I am not sure that the staff will know what the color-coding means. Define the color scheme. |
Number of the heuristic violations across the heuristics.
Heuristics violated | Count of usability problem description |
Aesthetic and minimalist design | 4 |
Consistency and standards | 10 |
Documentation and Help | 13 |
Error prevention | 6 |
Flexibility and efficiency of use | 4 |
Help user recognize, diagnose, and recover from errors | 12 |
Match between system and the real world | 10 |
Recognition rather than recall | 4 |
User control and freedom | 8 |
Visibility of system status | 12 |
Grand total | 83 |
Severity of the heuristic violations.
Heuristics violated | Average of severity |
Aesthetic and minimalist design | 2.25 |
Consistency and standards | 1.49 |
Documentation and Help | 3.01 |
Error prevention | 3.88 |
Flexibility and efficiency of use | 2.88 |
Help user recognize, diagnose, and recover from errors | 2.48 |
Match between system and the real world | 2.50 |
Recognition rather than recall | 2.20 |
User control and freedom | 3.13 |
Visibility of system status | 2.93 |
Places of the heuristic violations occurrence.
Places of occurrence | Count of heuristics violated |
Action | 13 |
Checklist | 33 |
Outcome | 13 |
Start | 24 |
Grand total | 83 |
The task burdens measured by the NASA Task Load Index.
Task burden | Average out of 100 (SD) |
Mental demand | 10.0 (7.4) |
Physical demand | 1.8 (2.1) |
Temporal demand | 20.4 (24.8) |
Performance | 10.7 (11.3) |
Effort | 4.5 (4.9) |
Frustration | 1.6 (2.5) |
We have demonstrated that the usability of the CDSS is suitable for nurses in hospital environments. However, the ultimate success of the CDSS tool depends on many factors beyond usability, such as training and culture. In the next phase of the project, we have received generous funding from the Center for Medicare and Medicaid Innovations and CHRISTUS Health System to deploy the CDSS in 17 acute and long care facilities in a 3-year clinical deployment. The direct measurement of FTR cases and preventable complications at the deployment sites will provide the ultimate validation of the efficacy of the tool in improving patient safety and hospital care.
In this paper, we discussed the UI design and evaluation of a new decision support tool for nurses. The system was designed to help nurses recognize and escalate early warning signs of patient deterioration in acute care settings. The system will be used by floor nurses to evaluate patients on a daily basis. It will automatically alert the RRT when probable diagnoses are reached.
Using established cognitive design framework UFuRT as a guide, we were able to identify key requirements for the product, create a high-level functional specification, and then translate those functions into UI designs. During the implementation of the product, we performed heuristic evaluation to iteratively identify 83 usability issues, and fixed all issues rated as severe. These design and implementation approaches can be widely used in many different types of software development projects.
After the product was developed, we validated the design by performing end user usability tests, including performance tests and NASA Task Load Index evaluation. The evaluation has shown that our design was functional and met the requirements demanded by the nurses’ tight schedules and heavy workloads.
UI design and implementation were critical factors contributing to successful deployment of the CDSS tools, but they were not the only factors. In follow-up research, we will deploy the solution in a working hospital environment, and evaluate the clinical outcome measures to determine the barriers and efficacy of the overall solution.
clinical decision support systems
content management system
electronic medical record
failure to rescue
information technology
National Aeronautics and Space Administration
rapid response team
user, function, representation, and task analysis
user interface
work-centered design
None declared.