Data from all implemented projects were collected according to routine by the WestChronic system. Response rates were calculated as the minimal response rate (RR1) [23].

We divided the function of the PRO system into a number of elements, the first of which is ubiquitous in any PRO application, while the other three are optional (Table 1 and Figure 1).

PRO data may be collected in the outpatient clinic or at a distance (eg, from home). We will refer to the latter as TelePRO. The mode to record the PRO data may be based on paper forms or electronic devices. Although paper modes may involve sophisticated electronic procedures like on-demand printing and optical character recognition, we will restrict the term ePRO to Web-based interfaces, tablet computers, other hand-held devices, and interactive voice response [17].

The base element is mandatory for any application of PRO, and may range from a photocopied paper form handed out to patients on arrival, filled in and used as is in the subsequent visit, to advanced computerized systems handling all processes. A PRO system may implement and manage some or all relevant elements of a protocol and may also support the logistics for the collection of PRO data. Crucial issues include the definition of content and development of the actual PRO questionnaire: the validity, reliability, acceptability to patients and clinicians, and the relevance with respect to the purpose of the collection of PRO data [24,25]. Other issues include the process of implementation, pilot-tests, and methods to collect and integrate user feedback and experiences. Issues of importance to response rate and usability, include the offered modalities (Web-based, paper-based), the offered options, and level of automation of the protocol (administration of implementations, items, handling of subjects, reminders, data import and export).

A PRO system may enable the clinician to access and overview systematically collected PRO data on symptoms, functional status, and health-related quality of life that can support symptom monitoring, consultation support, and clinical decision-making [11,26]. The results of PRO assessments may be used longitudinally to monitor the course of symptoms and to flag symptoms that need further attention during an outpatient visit. The procedure may range from using the paper form as is as a checklist during the interview, to using graphical display systems fully integrated within an electronic health record (EHR) system, in which an overview is presented to the clinician, who can use it for clinical decisions together with other available clinical and laboratory data. It is crucial that the items used for decision support are relevant for the situation seen from the point of view of the patient as well as the clinician.

A PRO system may be designed to make automatic decisions. As a screening tool, PRO assessments may be used to identify patients that need attention as well as patients that do not need attention at the moment. The design may range from a simple score calculated by hand by the clinician and compared with published cut-off values, to automated computer algorithms that include actual absolute scores and intra-individual changes with respect to previous scores. Crucial issues include the risk of false-positive and false-negative results of the algorithm. In statistical terms this is expressed by sensitivity, specificity, and predictive values. Furthermore, the algorithm should be acceptable and meaningful for both patient and clinician.

Normally, researchers or clinicians define the content of the PRO questionnaire, prompt the patient to answer, and collect the data, but some functions in a PRO system may go beyond the one-way, one-to-one flow of information. It is noteworthy that the definition of PRO does not impose strict demands on the origin of a PRO assessment or prerequisites regarding who initiates the communication.

Figure 1 displays the eight possible combinations of the base element and the three optional elements.

PRO data were collected in mixed-mode with paper- and Web-based questionnaires in all projects except three, where only the Web-based method was applied. Three reminders were applied in project type G and two in all other, except in project type A/stroke, where no reminders were applied.

The participation rates for the implemented projects are displayed in Table 2. In all projects, the patient was clearly informed that participation was voluntary. The range in response rates to the first PRO questionnaires was between 75% and 93% (median value 85%). The highest rates (93%) were found in project type G/epilepsy (clinical decision support, n=2882 patients) and project type B/prostatic cancer patients (hospital assessment, n=7423), while the lowest (77%) were found among stroke patients (epidemiological research, n=3575). During follow-up, the rates were between 82% and 99% (median value 96%). Due to different protocols (eg, number of reminders varied between zero and three), the rates are not directly comparable.

In project types A, B, and C (Table 2) PRO data from 8 different diagnostic groups were collected for use at a group level, and therefore none of the three optional elements were in use. Experiences related to the content of the PRO questionnaires for these projects will be described here.

In project type A, the three projects, include patients with breast cancer, ischemic heart disease, and stroke. Patients are monitored with PRO by multiple measurements over a span of 2 to 6 years. The aim is to describe prognosis using PRO data regarding symptoms and functioning, and to analyze PRO variables as risk factors for medical and social outcomes [20,27-30].

In project type B, PRO data are collected nationwide for patients with four malignant diseases: prostatic cancer, renal cancer, esophageal cancer, and lung cancer. The aim is to include PRO measures in existing national clinical registers used for research and hospital performance assessment.

In project type C, in an ongoing reorganization of the treatment of stroke in the Central Region of Denmark, it was decided to collect PRO data consecutively in all patients to monitor possible effects.

PRO data collected for use at the group level (clinical epidemiological research and hospital performance assessments) must comply with the usual demands regarding validity and reliability [1], and these issues will not be described further here. Due to the generic property of WestChronic, valid scales can be applied across projects, which make it easy to implement new projects using scales already in the bank simply by selecting them from drop-down menus. In such cases, new projects may be implemented very quickly.

Experiences related to the content of PRO for projects with applications at a patient level will be described below in connection with the corresponding optional element.

In projects aiming at clinical decision support, the core element is a graphical overview over the course of PRO. The clinician is presented with a graphical view of the course of selected PRO variables displayed within an EHR in the same context as the clinical data. A screen shot capturing the AmbuFlex II implementation is shown in Figure 2. Each column represents a PRO assessment. Color codes signal the severity of the symptom. The actual wording of the question as well as the answer is displayed as a “pop-up tip” when the user puts the mouse icon over the displayed bar. Vertically, the overview presents the actual situation and horizontally the PRO course over time with regard to symptoms, functional level, etc.

The PRO overview is used in two different situations: (1) in telePRO to evaluate and decide whether the patient needs a visit, and (2) as consultation support to identify and flag important symptoms that need focus and attention at an outpatient visit or in a telephone consultation.

Patients with chronic heart failure often need treatment with multiple pharmacological substances. During the period in which patients are seen in the outpatient clinic, medical therapy is up-titrated, and patients are scheduled for frequent visits to monitor treatment results, identify side effects, and ensure compliance. PRO questionnaires were filled out by the patient before the visit to facilitate more efficient visits by flagging important symptoms. Furthermore, the overview is used for telephone consultations, enabling these to be shorter and more comprehensive [31]. The application as a screening tool before a telephone consultation was not a specific aim, but turned out to be the most significant issue with respect to both quality and time saving in PRO projects using a graphical PRO overview. The same method is used in patients with epilepsy, rheumatoid arthritis, renal failure, sleep disorders, neuromuscular diseases, lung and prostatic cancer, and endometriosis (Project type D+G).

Patients with epilepsy are normally followed-up as outpatients at a neurological clinic, usually with 1 to 4 appointments yearly. PRO questionnaires are used to evaluate whether the patient needs a visit or not. If not, the patient automatically receives a new PRO questionnaire after a preset interval (eg, 3 months). The procedure consists of two steps: an automated decision in patients with obvious clinical problems and patients with no obvious problems at the moment (optional element 2) and a PRO-based clinical decision support in the remaining patients. Overall, for 48.75% of the PRO questionnaires no additional contact to the patient was needed, while the remaining 51.25% had a subsequent follow-up visit or a telephone consultation. The same method is used in patients with sleep disorders, neuromuscular diseases, and prostatic cancer.

The PRO-based overviews use a PRO assessment to reflect clinical aspects as they are met in the daily clinical practice for that particular group of patients. The clinician, who makes the decision based on the PRO overview, still has the professional responsibility in case of an erroneous decision. Our experience is that it is vital that clinicians have full confidence in the system, even at the item level (face validity). The content of the PRO is negotiated based on iterative inputs from clinicians, review of the literature, and anthropological interviews with patients [25,32]. For new items constructed in this way, identical 5-point Likert scales are used to assess severity and frequency of symptoms. Pilot tests and semistructured interviews before implementation are used to identify problems such as relevance of items, clarity of wording, ambiguity of items, and lack of important issues. After a pilot test, the PRO application is put in operation, and a parallel iterative process is launched in which experiences are continuously evaluated, and items and mode of display revised as a running process until saturation is reached after 2 to 4 months. We have experienced that such projects are easily transferred to outpatient clinics for the same patient group without any modification, even though the clinicians are invited to suggest such. Thus, an implementation seems to be specific for a patient group, not for a location. The implementations have been evaluated from a clinical as well as a patient perspective with positive conclusions [33].

An automated PRO algorithm was applied in project types E, F, and G. As a part of the implementation process, an algorithm for each specific group of patients was developed and programmed into the server software.

According to Danish clinical guidelines, all patients discharged from hospital after an ischemic heart attack should be screened for depression and anxiety 6 weeks after admission. Due to logistic challenges, this is rarely accomplished. The patients are recruited consecutively from hospital discharge registers, and 6 weeks after admission they are mailed a generic questionnaire on depression and anxiety [34]. An automated algorithm based on published cut-off values divide patients into nine groups according to no, moderate, or severe symptoms on the two scales. Based on these values, WestChronic automatically generates a personalized letter with the results of the screening, and if moderate or severe symptoms are present, the patient is advised to consult a general practitioner and bring along the letter. The same method is now extended to heart transplanted patients.

According to guidelines, patients with hip and knee replacements are invited to a follow-up visit 3 months after surgery. Several studies have documented that few of these visits have any clinical consequences and could be cancelled if satisfactory information on, for example, pain and difficulty in walking were available [35]. Patients were included at the preoperative examination in selected hospitals in 4 of 5 Danish regions. The automated algorithm was based on published values from well-established disease-specific questionnaires on symptoms and functioning [36,37]. At the beginning of the questionnaire we included an additional item: “You may wish to have an outpatient clinic visit regardless of these clinical factors and you can indicate your preference here”. This option was ticked off by 291 patients (27.3%). If scores were below thresholds and the patient did not indicated an absolute wish for a follow-up visit, the department was electronically informed that the scheduled follow-up visit should be cancelled. The same method is used in patients with endometriosis, where the algorithm is based solely on the patient’s wish of a visit.

Clinical decision support among outpatients with epilepsy is described above. First, however, patients obviously needing attention and those that do not are handled automatically. Based on the incoming PRO data, the server algorithms simultaneously categorize the patients’ present condition into red, yellow, or green status (red status: the patient should be seen or contacted; green status: no action is needed). In the latter case, clinicians are not notified or involved at all, and at the scheduled time (eg, 3 months) a new questionnaire is automatically printed out and mailed or emailed to the patient. The PRO assessments that could not be processed automatically are assigned yellow status, meaning that a clinician shall inspect the PRO overview (decision support; optional element 1). Examples of inducers of red status are self-reported aggravation of seizures or planning of pregnancy. We allow the patient to overrule the automated decision with the same question as mentioned above. Nonresponders and patients who indicate they want a personal clinical contact are categorized to red status. WestChronic keeps track of patients with red and yellow status, and, if no action is taken by a clinician before a deadline, the server software reacts with reminders to the clinician on duty and, if ignored, alarms the system supervisors. Among 2766 questionnaires (November 2013) 37.8% were handled automatically (10.3% green and 27.5% red). Among patients with epilepsy, 27.3% indicated an absolute wish for a clinical contact.

For simple screening purposes with defined binary outcomes (depression, inadequate function after surgery) existing PRO scales with documented sensitivity and predictive values were used. In more complex clinical decisions in which the whole situation of a patient needs to be evaluated, in several cases no PRO instrument was available or applicable. In these applications the goal is to have a false negative rate of zero, whereas the rate of false positive is of less concern. When PRO is intended for automated decisions, both content (items) and threshold need to be defined and documented with respect to sensitivity and specificity. Even if an obvious candidate for the PRO questionnaire exists, it is only possible to extract a cut-off value from the literature if the aims are identical. This was the case in screening for depression and canceling of postoperative visits. In the other projects, we had no predefined cut-offs to rely on. Initially, we gave priority to sensitivity in order to identify all cases that would be identified in a normal practice. When experience was gathered, cut-offs were adjusted by consensus conferences.

In some projects there was need for information beyond that provided by predefined PRO questionnaires delivered from the patient to one or more clinicians.

In the very first PRO project (breast cancer, project type A), we gave the patient the possibility to log on and review her own course of symptoms over time. However, to comply with the demands from the Danish Data Protection Agency, we had to apply rather complicated procedures. A shared secure log-on procedure has now been provided at the national health website from which the patient can obtain a link to the personal site at WestChronic. In future implementations, the patient will be able to see an overview similar to what is presented to the clinician (optional element 1).

The treatment of patients with severe ADHD may involve several therapists and social workers (project H). The project attempts to promote an overview of the situation among a group of complex patients with often quick shifts in condition and surroundings, and where continually shared updated knowledge is crucial for all partners, including the patient and relatives. The system includes PRO as well as a Web-based communication area in which structured as well as unstructured information can be shared between all parties, including the patient. The psychiatric department creates a record for new patients and decides which partners are relevant in each particular case. Partners are labeled according to their role (eg, patient, community psychiatric nurse, municipal social worker, outpatient clinic nurse, relative). Each patient is assigned 1 main contact person at a time. At the beginning of treatment, this would typically be a member of the psychiatric team. Any partner, including the patient, can create a new communication. The communication element is accessed from the PRO-based graphical overview for the particular patient. The patient participates in the following ways: first, s/he fills in the PRO questionnaire, which is graphically displayed at the initial page where all partners enter the system. Second, s/he has access to exactly the same written information as all other partners. Finally, s/he may create a communication to all partners. The psychiatric team accesses the system through the EHR system, while other professional partners obtain accesses after logon to their local area network. The patient obtains access via secure login at the national health website.