This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.
Medication is the most common intervention in health care, and written medication information can affect consumers’ medication-related behavior. Research has shown that a large proportion of Australians search for medication information on the Internet.
To evaluate the medication information content, based on consumer medication information needs, and usability of 4 Australian health websites: Better Health Channel, myDr, healthdirect, and NPS MedicineWise .
To assess website content, the most common consumer medication information needs were identified using (1) medication queries to the healthdirect helpline (a telephone helpline available across most of Australia) and (2) the most frequently used medications in Australia. The most frequently used medications were extracted from Australian government statistics on use of subsidized medicines in the community and the National Census of Medicines Use. Each website was assessed to determine whether it covered or partially covered information and advice about these medications. To assess website usability, 16 consumers participated in user testing wherein they were required to locate 2 pieces of medication information on each website. Brief semistructured interviews were also conducted with participants to gauge their opinions of the websites.
Information on prescription medication was more comprehensively covered on all websites (3 of 4 websites covered 100% of information) than nonprescription medication (websites covered 0%-67% of information). Most websites relied on consumer medicines information leaflets to convey prescription medication information to consumers. Information about prescription medication classes was less comprehensive, with no website providing all information examined about antibiotics and antidepressants. Participants (n=16) were able to locate medication information on websites in most cases (accuracy ranged from 84% to 91%). However, a number of usability issues relating to website navigation and information display were identified. For example, websites not allowing combinations of search terms to be entered in search boxes and continuous blocks of text without subheadings.
Of the 4 Australian health information websites tested, none provided consumers with comprehensive medication information on both prescription and nonprescription medications in a user-friendly way. Using data on consumer information needs and user testing to guide medication information content and website design is a useful approach to inform consumer website development.
Medication is the most common intervention in health care [
Evaluations of Web-based health information have typically utilized instruments with various criteria [
Although there have been numerous evaluations of Web-based health information [
A mixed-method approach was used in this study comprising (1) an assessment of consumer medication information needs, (2) a website content evaluation using the consumer information needs, and (3) user testing of websites (including qualitative interviews).
Consumer medication information needs were determined by examining the most frequently used medications in Australia and the most frequent consumer medication queries made to the healthdirect helpline. The top 5 most commonly used prescription and nonprescription medications in Australia were extracted from the Pharmaceutical Benefits Scheme (PBS) [
The consumer medication queries made to the healthdirect helpline in November 2014 were extracted and reviewed to determine the medication therapeutic class that was the subject of the call and the query regarding the medication (eg, what to do if a dose was missed). The healthdirect helpline is a free 24-hour health advice telephone line that covers approximately 56% of Australia’s population. The telephone service receives between 60,000 and 70,000 calls per month, with medication queries the most frequent clinical issue discussed [
We aimed to identify websites for the evaluation that consumers would most frequently encounter when using the Google search engine for medication queries. Search terms related to the top medications and queries outlined above were entered into Google. The 4 most frequently generated Australian health websites providing consumer medication advice were selected. Websites specific to a condition, a medication, or a population group were excluded (eg, beyondblue, a website targeting mental health [
First, each website’s content on the prescription and nonprescription medications identified from the PBS and National Census of Medicines Use was assessed with respect to the extent to which it covered the 3 most frequent medication queries to the healthdirect helpline (ie, how to take a medication, interactions, and side effects). Additionally, whether each website had an information page about the therapeutic classes of these medications was examined. For example, for atorvastatin, websites were evaluated on the extent to which they covered information on how to take the medication (how much to take, what do to if a dose is missed, and what to do in an overdose), interactions, and side effects as well as whether there was a general page on hypolipidemic medications.
Second, each website’s content was assessed with respect to whether it covered the most frequent medication classes that were the subject of calls to the healthdirect helpline and each class’s most frequent queries. Whether each website included information on the most frequent medication within each class was also examined. For example, when providing information on antibiotics, whether the website covered missed doses, interactions, and stopping an antibiotic were assessed, along with whether there was any specific information on amoxicillin. For analgesics and antipyretics, information on paracetamol, ibuprofen, and paracetamol and codeine was examined, as these were by far the most frequent medication types queried and these medicines are available over the counter (nonprescription).
A coding system was applied to indicate the extent to which information on each medication query was available on the websites. Two investigators (MZR and LR) initially tested the coding system to ensure it was suitable and reliable (ie, produced the same code when applied independently by 2 reviewers). Subsequently, a single investigator (LR) coded the queries on all websites. Each query was coded as either covered (C), that is, the information provided on the website was comprehensive enough to fully answer the query; partially covered (PC), that is, there was information related to the query, but it did not answer the query specifically; referred (R), that is, the website referred users to another site that answered the query; and not covered (and not referred; NC). By way of example, for the query “what medications does aspirin interact with?” a website was coded as covering the information if it provided a list of medications with which aspirin interacts but partially covered if it only stated that aspirin interacted with some medications and asked the user to seek advice from a health professional.
Website content assessment was conducted in January 2015.
We developed scenarios for testing based on the medication calls made to the healthdirect helpline outlined above. The most frequently queried medications were combined with the most frequently asked questions to create the scenarios. The scenarios consisted of 8 questions, all of which had answers available on the test websites (
1. Can I take Panadeine Forte (paracetamol 500 mg and codeine phosphate 30 mg per dosing unit) while breastfeeding?
2. It is safe to take my antibiotic (Keflex: cephalexin) with Panadol (paracetamol)?
3. I missed a dose of my antibiotic (Amoxil: amoxicillin), what do I do?
4. What is warfarin (Coumadin) used for?
5. I’m feeling better, can I stop my antibiotic (erythromycin: Eryc)?
6. Is nausea a side effect of my antidepressant (Zoloft: sertraline, a selective serotonin reuptake inhibitor)?
7. Does warfarin interact with Nurofen (ibuprofen, a nonsteroidal anti-inflammatory drug)?
8. Is it safe to take Telfast Decongestant tablets (fexofenadine hydrochloride 60 mg and pseudoephedrine hydrochloride 120 mg per dosing unit) while pregnant?
Website sequences used for user testing scenarios.
Sequence | Website 1 | Website 2 | Website 3 | Website 4 |
1 | healthdirect | NPS MedicineWise | myDr | Better Health Channel |
2 | Better Health Channel | healthdirect | NPS MedicineWise | myDr |
3 | myDr | Better Health Channel | healthdirect | NPS MedicineWise |
4 | NPS MedicineWise | myDr | Better Health Channel | healthdirect |
Consumers who were unfamiliar with the target websites took part in user testing. To recruit participants, posters were displayed at Macquarie University, Sydney campus, Australia. Participants received a complimentary lunch for taking part.
Participants were observed by a single investigator (MTB) while performing the 8 scenario tasks using the websites. To complete each task, the user was required to answer each medication-related query by locating relevant information on a website. Tasks were completed in a fixed order but the order of website use varied between subjects to minimize any learning effects, with each participant randomly allocated to 1 of 4 sequences listed in
The variables collected by the observer during each scenario were time taken to locate the desired medication information; number of screens required to locate the piece of medication information; number of new searches a user performed (ie, new entries into a search box); the user’s search method (eg, whether he or she used the search box or browsed subheadings); whether the user was successful in completing the task (ie, answered the question correctly); and any obvious negative affect (eg, frustration).
After completion of 4 scenarios (on 2 websites) participants took a short break and were asked to comment on the 2 websites they had just used. They were asked to indicate which website they preferred and why, to describe good and bad features of the websites, and to comment on the layout of information on the screen and on how understandable the website content was. Participants then completed the remaining 4 scenarios and were interviewed about the 2 additional websites. Finally, participants were asked to indicate which of the 4 websites was their preferred website and why and to describe an ideal website for locating medication information.
Nonparametric Friedman tests were used to detect the differences across the websites on time taken to locate information, number of screens required, and number of new searches. A generalized estimating equation approach, with consideration of the correlation of measurements from the same participant, was used to compare the websites on proportion of tasks successfully completed. Results were considered significant when
Website coverage of information on the most frequently used prescription medications in Australia.
Prescription medication information | healthdirect | NPS MedicineWise | myDr | Better Health Channel | |
How to take ita | Rb | Cc | C | C | |
Interactions | R | C | C | C | |
Side effects | Cd | C | C | C | |
How to take ita | R | C | C | C | |
Interactions | R | C | C | C | |
Side effects | Cd | C | C | C | |
How to take ita | R | C | C | C | |
Interactions | R | C | C | C | |
Side effects | R | C | C | C | |
How to take ita | R | C | C | C | |
Interactions | R | C | C | C | |
Side effects | R | C | C | C | |
How to take ita | R | C | C | C | |
Interactions | R | C | C | C | |
Side effects | R | C | C | C |
aHow to take it includes how much to take, what to do if a dose is missed, and what to do in an overdose. Inclusion of all 3 resulted in a rating of C; if only 1 or 2 items were covered, then a rating of PC (partially covered) was given.
bR: referred to an external site.
cC: covered.
dCovered on a general page about statins, which mentions atorvastatin and rosuvastatin.
Total percentage of website coverage of information on the most frequently used prescription medications in Australia.
Total coverage (%) | healthdirect | NPS MedicineWise | myDr | Better Health Channel | |
Covered | 13 | 100 | 100 | 100 | |
Partially covered | 0 | 0 | 0 | 0 | |
Referred | 87 | 0 | 0 | 0 | |
Not covered | 0 | 0 | 0 | 0 |
Website coverage of information on most frequently used nonprescription medications in Australia.
Nonprescription medication information | healthdirect | NPS MedicineWise | myDr | Better Health Channel | |
How to take ita | Rb | PCc | Cd | C | |
Interactions | R | C | C | C | |
Side effects | PC | C | C | C | |
How to take ita | R | PC | NCe | PC | |
Interactions | R | C | NC | NC | |
Side effects | R | C | PC | NC | |
How to take ita | R | PC | NC | NC | |
Interactions | R | PC | NC | C | |
Side effects | R | C | NC | NC | |
How to take ita | R | PC | C | C | |
Interactions | R | NC | C | C | |
Side effects | R | PC | C | C | |
How to take ita | R | PC | PC | C | |
Interactions | R | C | NC | C | |
Side effects | R | C | C | C |
aHow to take it includes how much to take, what do to if a dose is missed, and what to do in an overdose. Inclusion of all 3 resulted in a rating of C; if only 1 or 2 items were covered then a rating of PC was given.
bR: referred to an external site.
cPC: partially covered.
dC: covered.
eNC: not covered (and not referred).
Total percentage of website coverage of information on the most frequently used nonprescription medications in Australia.
Total coverage (%) | healthdirect | NPS MedicineWise | myDr | Better Health Channel | |
Covered | 0 | 47 | 47 | 67 | |
Partially covered | 7 | 47 | 13 | 7 | |
Referred | 93 | 0 | 0 | 0 | |
Not covered | 0 | 6 | 40 | 27 |
Website coverage of queries to the healthdirect helpline related to simple analgesics and antipyretics.
Queries for analgesics and antipyretics | healthdirect | NPS MedicineWise | myDr | Better Health Channel | |
How much to take (children) | PCa | PC | Cb | PC | |
Interactions | PC | C | C | NCc | |
Overdose | C | C | NC | C | |
How much to take (children) | PC | PC | C | C | |
Interactions | Rd | C | C | NC | |
Overdose | R | NC | C | C | |
Interactions | PC | PC | PC | C | |
Overdose | PC | C | C | C | |
Vomited after taking, is it safe to take another dose | NC | NC | NC | NC |
aPC: partially covered.
bC: covered.
cNC: not covered (and not referred).
dR: referred to an external site.
Total percentage of website coverage of queries to the healthdirect helpline related to simple analgesics and antipyretics.
Total coverage (%) | healthdirect | NPS MedicineWise | myDr | Better Health Channel | |
Covered | 11 | 44 | 67 | 56 | |
Partially covered | 56 | 33 | 11 | 11 | |
Referred | 22 | 0 | 0 | 0 | |
Not covered | 11 | 22 | 22 | 33 |
Website coverage of queries to the healthdirect helpline related to antibiotics, antidepressants, antihistamines, and anticoagulants and antithrombotic agents.
Queries | healthdirect | NPS MedicineWise | myDr | Better Health Channel |
Interactions | Ra | PCb | PC | NCc |
How long to take it for | R | Cd | C | NC |
Missed dose | NC | PC | NC | NC |
Interactions | PC | C | PC | NC |
Starting and stopping | PC | C | C | C |
Missed dose | NC | NC | NC | NC |
How much to take (children) | R | NC | NC | NC |
Interactions | R | NC | NC | NC |
Use in breastfeeding | R | Ce | NC | NC |
Interactions | R | C | C | NC |
Side effects | R | C | C | NC |
aR: referred to an external site.
bPC: partially covered.
cNC: not covered (and not referred).
dC: covered.
eInformation on use of antihistamines while breastfeeding was on a page about medication use in breastfeeding.
Website coverage of general information queries to the healthdirect helpline related to antibiotics, antidepressants, antihistamines, and anticoagulants and antithrombotic agents.
Queries | healthdirect | NPS MedicineWise | myDr | Better Health Channel |
General information? | Yes | Yes | Yes | No |
Specific information on amoxicillin? | No | Yes | Yes | Yes |
General information? | Yes | Yes | Yes | Yes |
Specific information on sertraline? | No | Yes | Yes | Yes |
General information? | No | No | No | No |
Specific information on promethazine? | No | Yes | Yes | Yes |
General information? | Yesa | Yes | Yes | No |
Specific information on warfarin? | Yesa | Yes | Yes | No |
aInformation provided on a page about stroke treatment.
Total percentage of website coverage of queries to the healthdirect helpline related to antibiotics, antidepressants, antihistamines, and anticoagulants and antithrombotic agents.
Total coverage (%) | healthdirect | NPS MedicineWise | myDr | Better Health Channel | |
Covered | 0 | 55 | 36 | 9 | |
Partially covered | 18 | 18 | 18 | 0 | |
Referred | 64 | 0 | 0 | 0 | |
Not covered | 18 | 27 | 45 | 91 |
A total of 16 consumers were recruited for user testing. The median age was 27 years (range 18-66) and 8 out of 16 participants (50%) were male. When asked how frequently they used the Internet, all participants reported that they used the Internet “multiple times a day.” When asked how often they used the Internet to find medication information, typical responses were “never” (n=7), “rarely” (n=4), and “once per month” (n=2).
Of the 16 participants, 11 participants (69%) indicated NPS MedicineWise was their preferred website, 4 participants (25%) said myDr, and 1 participant preferred healthdirect to the other websites.
Results of scenario-based user testing of websites.
Key results | healthdirect | NPS MedicineWise | myDr | Better Health Channel |
Median time taken to complete a scenario (range)a | 3 minb 37 sc |
2 min 56s |
2 min 41s |
4 min 34s |
Median number of screens viewed (range) | 10 (3-34) | 7 (2-17) | 7 (3-15) | 10 (5-17) |
Median number of new searches (range)a | 2 (0-7) | 0 (0-4) | 2 (0-6) | 2 (0-9) |
Percentage correcta | 75% | 75% | 81% | 69% |
a No evidence to indicate differences between websites.
b min: minute.
c s: second.
Overall, it appeared to be more difficult for participants to locate the appropriate page of information for each scenario than to find the relevant piece of information on a page. In 84 of the 128 (65.6%) scenarios observed, participants began the scenario by entering a keyword or keywords into the home page search box. In 32 of the 128 scenarios (25.0%), participants looked for the website’s medicine page before searching for a particular medication name. When trying to locate a piece of information on a webpage, more than half the participants (n=10) used a keyboard shortcut (ie, control-F) to find a keyword (eg, pregnancy) on the page, whereas the remaining participants scrolled through the information.
Participant interviews provided further information on their perceptions of the usability of websites. All participants reported that information displayed on websites was presented at the right level of difficultly, although some suggested that content was understandable because they were students or researchers:
I think there would be a not insignificant proportion of the population that would struggle with it, because they would baulk at the terminology that’s used.
The inability to search using combinations of search terms (eg, Panadeine Forte AND pregnancy) was identified to be a negative aspect of websites by participants:
Yeah, so if it would work more like Google where I would type in the keywords of what I was looking for then that would be much easier. That was something that I noticed all the websites didn’t do.
Users also reported that being directed to PDF versions of consumer medicines information (CMI) leaflets was problematic:
It’s really wordy and the format of it, because it’s set up, to me, it’s set up like a physical pamphlet, so if I had that in my hands, that’s fine but on the screen, the three column thing with the same format and the font of a physical pamphlet doesn’t work...it makes skimming much harder because, I mean, I could do it but it just took me longer.
A large number of menus and drop-down menus on the home page resulted in participants using the search box rather than browsing the website:
Yeah, so it’s very all over the place really. You really don’t know where to start with that one so you’re almost forced to go to search this side.
The listing of medications only by generic names or brand names, not both, was identified as a barrier to finding relevant information, as users were not necessarily familiar with both terms:
It can be confusing with generic versus brand names, because I know one, when that I was initially looking for Zoloft I looked at that list of anti-depressants and I think they only had the generic names so I didn’t see Zoloft on that list, so I think it’s important to have both written.
Websites presenting medication information in a separate location from health information was perceived as a problem because users generally viewed medication information as a subset of health information, not as a separate category of information:
I think it should be more integrated because I think that that’s logically how people think. They see health as being the generic term and medicine a subset within.
The large number of results being generated from search-box queries was also considered a barrier to locating information:
It’s like when you’re searching, it kind of gives you every possible result rather than the one you probably want, the common one. Well, I don’t know, I mean, it’s hard to get that balance between only throwing up a few common ones, and the person could miss out on what they want to see, or throwing up everything and the person just gets, like, what is all this?
Participants viewed features of websites that broke up large amounts of text (eg, subheadings, highlighting, or hyperlinked subheadings) as helpful:
Subheadings are very good. Especially when you know what you are looking for.
I think that in terms of NPS [MedicineWise] there was more bold so I found it easier to read because then I would just skim and if the bold didn’t apply then I would just ignore the regular font.
Bullet points are good. I mean, you don't want massive slabs of information that you need to search through.
You don’t want to sit there reading through it all. Having those little jumping links is helpful if you are looking for a particular bit of information.
Additionally, auto-completion of search terms in search boxes was reported to be a positive feature of websites:
I like that if you search something there are suggestions for what you are searching.
Several limitations were identified in the medication information available on 4 Australian health websites in relation to both content and usability. Although detailed information on specific prescription medication was provided, information on nonprescription medication and medication classes was less comprehensive. Several website features affected how quickly and easily users were able to locate medication information.
Information on the most common prescription medications and most frequent medication queries made to the healthdirect helpline were covered or referred by all 4 websites. The healthdirect website was the only website to refer consumers to other websites. This is because the healthdirect website acts as a portal site that directs consumers to other sources of reliable health information. The majority of the prescription medication information was available through CMI leaflets, either embedded into webpages (NPS MedicineWise, myDr) or as a link to a PDF file (Better Health Channel). CMI content is regulated by the Australian Government and is prepared by and the responsibility of pharmaceutical companies [
The way CMI content is displayed on the websites also appeared to be problematic. The inclusion of a link to a PDF version of the “paper” leaflet (as on Better Health Channel), which includes three columns of text, was not viewed favorably by consumers. This was primarily because the layout required users to continuously scroll up and down to read the text. Consumers preferred CMI content to be embedded into the webpage in a single column, as done on NPS MedicineWise and myDr. Interestingly, an assessment of consumer needs in relation to printed CMI leaflets also noted that consumers preferred a single-column layout [
Information on nonprescription medication was less comprehensively covered on the websites than prescription medication. The nonprescription medication or medication classes examined in this study included paracetamol, ibuprofen, paracetamol and codeine (formulations with codeine≤12 mg/unit), antihistamines, aspirin, fish oil supplements, glucosamine, calcium, and cholecalciferol. Nonprescription medications do not require a CMI leaflet according to Australian regulations. Instructions for use typically appear in or on the packaging. However, as is evident from the large volume of calls made to the healthdirect helpline about these medications, consumers may not always read, keep, or understand packaging instructions, or all the required information may not be provided on packaging instructions.
Of the nonprescription medicines examined in this study, 5 were complementary medicines. Complementary medicines are a subset of nonprescription medicines that can be defined as herbal, natural, or alternative medicines and include vitamins, minerals, herbs, and nutritional supplements. Australian studies estimate that 50% of complementary medicine users also take conventional medicines [
Although the content of websites is important, it is also crucial for information to be easily located. Usability issues related to both website navigation (ie, locating the correct page) and information display (ie, locating information on a page) were identified in this study. A key navigation issue was that websites did not allow users to search using multiple keywords, as is typically the case in search boxes. This caused users to become extremely frustrated and resulted in delays. Information layout was important for locating content on a page, with participants preferring text to be broken up using subheadings, highlighting, or bullet points. These features are in line with those identified in a previous assessment of consumer needs related to printed CMI leaflets [
This study had a number of limitations. The assessment of consumer medication information needs was based on calls to a national health helpline and the most commonly used prescription and nonprescription medications. Consumer queries from other sources, such as health professionals (physicians and pharmacists), were not captured. Additionally, there may be important medication safety issues not recognized by consumers but for which there is limited information, and our methods would not have captured these. The accuracy or the readability level of the medication information on the websites was not evaluated as part of this study. However, a recent study assessing the readability of 251 Australian health webpages found that their readability was above the average Australian levels of reading [
This study applied a unique approach, guided by consumer medication information needs, to assess the content and usability of medication information on 4 Australian websites. Several gaps were identified with respect to website content, and several usability issues were identified with respect to navigation and information presentation. Results showed that the 4 Australian websites tested did not provide consumers with comprehensive medication information on both prescription and nonprescription medications in a user-friendly way. Additional content (eg, on nonprescription medication) and some simple redesign of content (eg, single-column text with bullet points) would improve both the content and usability of widely used Australian websites.
consumer medicines information
Pharmaceutical Benefits Scheme
This research formed part of a project, supported by funding from Healthdirect Australia, evaluating Healthdirect’s telephony and digital medication services. MB and MR are employed at Healthdirect Australia. MB and MR contributed to the study design and revisions of the manuscript but were not involved in data analysis and interpretation.
MTB, MB, MR, JIW, MZR, AT, and LL contributed to the study design. LL analyzed medication query call data. MZR and MTB designed the user testing scenarios and website content assessment. LR assessed websites for medication information content. MTB conducted user testing and analyzed qualitative user testing data. LL analyzed quantitative user testing data. MTB, JIW, MZR, AT, LL, and LR contributed to the analysis and interpretation of the data. All authors reviewed the manuscript for content and approved the final version for publication.
None declared.