This study was a component of a larger systematic review and realist synthesis of clinical governance in primary care [16]. The CMR had the allure of being an unrealized tool to support clinical governance, measuring quality, conducting clinical audit, and ensuring safety (Textbox 3). We therefore undertook an analysis of CMR systems used in Australia, exploring the extent to which the CMR supported clinical governance, including to what extent this reflects contextual factors that may be unique to the Australian context. In keeping with the main study framework, we performed a structured analysis in conjunction with key themes emerging in the main study from a literature review and informant interviews. We analyzed seven CMR products used in Australia, and also their capacity to deliver clinical governance. We concluded by developing a maturity framework for CMRs in relation to clinical governance, and classified the maturity of the various CMRs.

We carried out a review from a realist perspective, mirroring the approach of the main study [17], modifying the approach previously used to explore the success and failures of the UK National program for IT [18]. A realist perspective is useful in assessing complex interventions as it aims to develop explanatory analyses of why and how these interventions may work in particular settings and contexts. The realists mantra is: “Context (C)” plus causal link with an appropriate “Mechanism (M)” results in an “Outcome (O)”; in other words, “C+M=O”. Part of the realist perspective is that effects are reported according to the three Ws: “What Works, for Whom, and in What circumstances.”

In realist evaluations, there can sometimes be difficulty in distinguishing context and mechanism. In this analysis, the context (C) is the Australian primary health care context, and the mechanism (M) is the CMR system used at the point of care. Our outcome measure (O) was the ability to produce clinical governance outputs through the ability to monitor quality of care against given criteria and standards. This combination describes how in the Australian context (C), the CMR, might contribute as a mechanism (M) to deliver the outcome measure (O), clinical governance (C+M=O) (Figure 1).

We mapped the primary care context using Lusignan’s 4 component classification [8]:

The contextual features discovered through this review were then analyzed in concert with the mechanisms of the CMR described below to develop a nuanced understanding of how the CMR operated in this particular environment to produce governance outcomes.

To identify the ways in which the CMR operated in relation to clinical governance, we used the Donabedian [20] classification of structure and process elements to describe the three types of mechanism by which CMRs may enable the delivery of improved clinical governance: structures, processes of care and review, and processes that impact on outcome. In this study, the software settings were considered to be process elements. For example, a key enabler of clinical governance-such as the presence of a unique patient identifier within the system, essential for data aggregation-would be listed as a key component of the mechanism provided by the CMR.

Each of these categories was subcategorized to produce the detailed tool across the categories (Table 1). The CMR structure was divided using the Open EHR model of the four separate components of a CMR system: interface, clinical archetypes, coding system, and database.

We explored factors related to clinical governance outcomes occurring at the level of the patient, the health care provider, and the setting (ie, impact at the population- or health system-level) [21].

We created a new assessment tool (Multimedia Appendix 1), a bi-axial tool, where the previously-described taxonomies of mechanism and context occupied each of the axes. The cells of the grid are populated with outcomes related to clinical governance for patients/clients, the provider, and the broader population level.

The top six CMR systems measured in terms of user base [22] were evaluated using this tool (Textbox 4). We also examined a CMR system with a small installation base (and therefore less organizational resources within the company) as a comparator. For each system, we used either software in demonstration mode or installed software in training mode. The testing was done with simulated patient data, and independently of the software providers, to explore how the system might retain clinical data and enable clinical governance activities. The tool was applied by one researcher and checked for accuracy by experienced users of each system. We elected not to disclose or publish comparison between brands, instead keeping our focus on whether the current generation of CMRs provide a viable mechanism for implementing clinical governance.

We developed a CMR maturity model, again using the Donabedian classification into structures (including IT architecture issues), process and outcomes, using existing consensus about CMR maturity [23-25]. At the structural level, we looked at the number of vendors and their market share, use of standards and interoperability, and the use of unique patient identifiers and clinical coding (eg, single national coding system). The processes were graded from passive reporting through to active decision support-again looking at individual patient, practice or locality and population levels. Outcomes data were expressed in terms of feedback about quality (Figure 2).

The process and potential of the CMR to influence clinical governance outcomes were graded into a four-level model (Table 2). This grading is multi-dimensional: (1) the agency of the CMR: namely, does the CMR play a passive or dependent role compared to an active or autonomous role in delivering clinical governance, (2) the level of complexity of the transaction and whether or not it is adaptive [26], (3) the degree of integration with other information systems, and (4) the physical integration and linkage processes underpinning it.

Only four of the CMR systems were able to participate in regional data quality activities. These activities revolve around the Australian Primary Care Collaborative program, The Practice Health Atlas and the PEN Clinical Audit tool [31]. All these activities require the use of an external tool to interrogate the program’s database and generate pooled data. One other package had its own tool to perform similar functions. All programs were able to generate pay-for-performance lists, according to the particular funding initiative.

No system had inbuilt data quality checks (prescribing insulin without a diagnosis of diabetes for example). One system had its own ‘in-house’ sentinel/research network ability; no other program had such a designated function.

At the structural level, Australian CMRs are well developed but there is scope for further progress against our maturity framework. Lacking are open standards, as yet no implementation of a standard coding system, and probably too many vendors in a relatively small market.

Australian primary care is therefore largely at level 2, with some systems only supporting level 1 and with some systems offering level 3 models. There was no evidence of level 4 systems. Some CMR systems had features that from the international perspective must be a developmental blind alley. The local coding systems are one of these; it is unlikely to ever become part of an interoperable health system.

In the Australian context, at practice and locality level the CMR works well, and is being used to facilitate clinical governance activities. Nearly all practices have systems with search functionality that enable participation in clinical audit.

However, while practices and localities are widely engaged in clinical governance processes, these are usually being done in an uncritical way. In particular, there is little attention given to data quality, or the obligation to code clinical conditions in standardized extractable fashion.

The record structures are often proprietary and there is a dearth of open architectural models, with many mission critical functions happening within a black box.

Benchmarking standards at a national or international level will be challenging if poor data quality and the disparate nature of record systems and system architecture remain unaddressed. Although not a registration based system, denominators such as those who attended in the last year can be used to make comparisons between practices and systems.

It is not possible to have lossless conversion of data held in one coding system to another, and the use of idiosyncratic coding systems increases the risk of data loss. While statistical techniques, in particular multiple imputation [41], can be used to compensate for missing data, this is never the same as having complete data. Black box data extraction processes and audit systems tend to foster uncertainty about the validity of findings.

Disease registers are much more challenging to set up when there is incomplete coded data, and patients with a condition not on a disease register are not going to benefit from computerized prompts or recall. Their standard of care may also be lower. This data quality and use issue will become a major problem as more information is shared.

The complexity of the clinician-patient-computer interaction, touched on in the introduction, is reflected elsewhere in the literature. Patient-centered care [38,42] and relationship-centered care [43] have taken hold and been shown to affect the outcomes. Computerization is changing the balance of power in the interaction [44].

There is no requirement for CMR systems to provide any specific functionality whatsoever, no set of criteria over information use, and no standards over usability or even formally recommended testing protocols [37]. The ‘Swiss cheese’ model of error [45] highlights how gaps in complex systems can result in errors, which in turn can raise patient safety issues. Drug interaction checking is an example of this, with interaction resources needing to be integrated into the prescribing package and then used by the clinician. While the UK National Program for IT has been much criticized, the one area that appears to have stood the test of time has been the rigid implementation of a drug dictionary and messaging standards [46].

Patient access to their records has become the norm elsewhere [47] and increased openness may help ensure good governance. Australia has aspirations to provide patient access through the National “Personally Controlled Electronic Health Record” program. Online access is no panacea; however, uptake of access to very different models of online summaries of care has been poor uptake in both England and France [48].

A comparison with the UK system of CMR driven pay-for-performance suggests that there may be quality gaps that computer mediated incentives might help close [49,50]. Additionally, the UK Primary Care Information Service (PRIMIS) has promoted data quality through a wide range of initiatives. The PRIMIS approach has been one of facilitation and feedback rather than financial incentives. These have been clinically focussed and included looking at disparities in data quality heart disease and improving patient safety [51,52]. However, more recently, the English NHS has gone through a game changing transformation with extraction of data on a National Scale through a system called the General Practice Extraction Service This gives the potential to extract data to measure quality and clinical governance on a national scale. The GPES system has its own Independent Advisory Group (SdeL is the Royal College of General Practitioners representative) to: “Consider the risks and benefits in order to assess whether the extraction is in their view appropriate and in the public interest.”

The evaluation took place at a point in time in 2009, and each package has gone through several upgrades over that time. As such, this analysis is not meant as a detailed critique of the packages with recommendations. It is quite possible that many of the comments here may no longer apply to a specific package. Moreover, it is the first discussion of the increasing influences on clinical governance by CMRs, with a framework that can be applied serially or in different contexts. What is lacking in planning and development is a consistent approach to thinking about CMRs and clinical governance, and what systemic controls should be there.

We might have explored the extent to which the standardization of record formats might have aided comparison and measurement of quality. The Royal College of Physicians, UK, has been very active in trying to standardize records reporting handovers, including admission and discharge [53]; it is likely that such a process would facilitate the implementation of clinical governance process. Although we make reference to the Open EHR initiative in our method we have not fully described its potential impact on standardization, and therefore toward being able to contribute to governance by facilitating the measurement and compare clinical standards on different systems. The two elements of Open EHR we believe that contributes most are its clinical models program, which enables researchers and practitioners to build sharable archetypes of clinical concepts [54]; and the specification program that defines data, services, and application program interfaces and offers the allure of quality certification of systems [55].

There are also other models we might have used for example: Yousuf et al have proposed an adoption model that includes: user attitude and skills base together with good leadership, IT-friendly environment, and good communication [56]. Lau et al have identified factors that influence adoption, and that it includes people, organization, implementation, and the macro environment [57].

These models share some similarity in that they both identify socio-technical aspects of implementation. Had we used either of these models, our subheadings might have been different but our findings are unlikely to have changed. Our selection driven by the wish to emphasize the interaction between organization (which included governance), the individual clinician, the clinical task (which should be of quality) and the technology; and not predefined success factors or progressive levels that should be reached.

The observations in this study have not been tested in a controlled trial and are retrospective in nature. Although we have approached this study from a neutral position of identifying factors that helped and hindered there may be bias. One author (CMP) is very familiar with many of the brands of Australian CMR and may have been susceptible to familiarity bias [58], and pointed out issues he was previously aware of. However, SdeL does not share this bias, instead having his experience framed in other countries’ CMR systems. Our assertion is that the CMR is as an instrument of and for clinical governance. At the very least, the CMR provides the tools to enable clinical audit and retrospective analysis of data. At its best the CMR can flag, recall, remind to monitor, and provide information support, and taking an ever more active role in the consultation. The current use of CMRs in Australia supports clinical governance at the individual patient, practice and possibly locality level; but provides no insights at the national level. Where the CMR does not facilitate clinical audit, individual practitioners are blocked from raising quality standards. We need to further test this hypothesis in prospective trials.

We have developed a framework for evaluating how CMR systems support clinical governance in a particular context; and whether the CMR has helped to achieve those goals. By applying the tool to several different brands of Australian CMRs, we have highlighted the issues that exist today, but importantly shown a graded way forward using a simple model and maturity framework that we hope can be readily followed by clinician users of these systems.

The limitations of the process relate to the heterogeneity of the data and their sources, the continuing change over time, but above preeminent is the lack of implementation of standards. While CMR implementation in Australia has enabled better clinical governance improving systems technical capability and rigorous standardization is needed to enable more comprehensive assessment of quality and outcomes for patients.