This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.
Directing patients suffering from third molars (TMs) problems to high-quality online information is not only medically important, but also could enable better engagement in shared decision making.
This study aimed to develop a scale that measures the scientific information quality (SIQ) for online information concerning wisdom tooth problems and to conduct a quality evaluation for online TMs resources. In addition, the study evaluated whether a specific piece of readability software (Readability Studio Professional 2012) might be reliable in measuring information comprehension, and explored predictors for the SIQ Scale.
A cross-sectional sample of websites was retrieved using certain keywords and phrases such as “impacted wisdom tooth problems” using 3 popular search engines. The retrieved websites (n=150) were filtered. The retained 50 websites were evaluated to assess their characteristics, usability, accessibility, trust, readability, SIQ, and their credibility using DISCERN and Health on the Net Code (HoNCode).
Websites’ mean scale scores varied significantly across website affiliation groups such as governmental, commercial, and treatment provider bodies. The SIQ Scale had a good internal consistency (alpha=.85) and was significantly correlated with DISCERN (
The developed SIQ Scale was found to demonstrate reliability and initial validity. Website affiliation, DISCERN, and HoNCode were significant predictors for the quality of scientific information. The Readability Studio software estimates were associated with scientific information comprehensiveness measures.
Wisdom teeth removal is the most commonly performed oral surgical procedure [
It is not always possible to provide adequate information for patients suffering from TMs problems, because it might be limited by the available consultation time allocated to each patient, given the fact that clinics are often overbooked [
There can be a potential limitation in the current clinical practice in referring patients to high-quality Internet resources due to clinicians’ lack of time and/or lack of knowledge [
Understanding health information is a major domain in health literacy, allowing patients to make appropriate health-related decisions [
The aims of this study were to (1) develop and validate a scale that measures SIQ; (2) evaluate the quality and readability of online health information concerning TMs problems; (3) validate the Readability Studio Professional 2012 software for measuring comprehensibility of online information; and (4) explore factors that could predict the SIQ of online health information.
To identify high-quality online resources, a cross-sectional sample of websites was selected on October 14, 2013, using advanced search options in Google, Yahoo!, and Bing search engines, with output limited to English language, any location, and specific phrases in the page title. The 3 phrases used were “wisdom tooth removal” OR “wisdom tooth extraction” OR “impacted wisdom tooth problems.” The first 50 results of each search engine output were selected after excluding websites identified as advertisements. A total of 150 websites were initially included. Websites were then filtered by removing duplicates and were reviewed for their relevance as a source for patient information. During this stage, nonfunctional, nonrelevant news articles or blogs were excluded. If a website was found to be relevant, it was categorized as having high, medium, or low relevance based on reporting the predetermined information sections of the SIQ Scale. Only websites of high relevance, according to this classification, were selected for content analysis.
Website content analysis flowchart.
A number of website characteristics that might have an association with the quality of provided information were determined. Website affiliation (to which organization the website belongs to) was recorded as governmental, educational institute, treatment provider (hospital/medical or dental practice), nonprofit organization, commercial [
To assess different quality aspects of websites under evaluation, several scales were used [
The Usability Scale consisted of items that were partially based on the Minervation Tool (LIDA [
A single-item Binary Scale that was used as a proxy for compliance with Web Content Accessibility Guidelines [
The Trust Scale was developed for this study and comprised a 4-item Binary Response Scale measuring trust in a website. Items for this scale were display of the Health on the Net (HoN) seal [
Mean readability grades were computed using Readability Studio Professional 2012 that provides readability grades estimates based on 6 different formulas recommended for the health care industry, which are FORCAST, Fry, Gunning Fog, New Fog, Raygor Estimate, and SMOG. Text from websites was extracted to MS Word (Microsoft, Redmond, WA, USA) where they were prepared for evaluation by the software. In addition, videos were transcribed by the author (KH).
The Scientific Information Comprehension (SI Comprehension) Scale was developed specifically for this study. It comprises a 9-item scale that measures the understandability of each section of the scientific information shown in
Evaluation criteria for scientific information quality concerning wisdom tooth problems.
Criteria (assessed on a 5-point Likert-like scale: range from 1 for poor to 5 for excellent) | Description |
Overview | Number of third molars (TMs), age of eruption, and etiology of impaction |
Presentation | Mild pericoronitis to severe infection of facial spaces, swelling, trismus, periodontitis, decay, cyst, or tuners with incidence. No evidence supports the association between TMs and late teen crowding. |
Diagnosis and investigations | Diagnosed by a dentist/oral surgeon, medical and dental history, clinical and radiographic examination, and other radiographs in high-risk TMs |
Treatment options | Retain functional TMs, symptomatic TMs with untreatable conditions or associated with pathology should be removed, no evidence supports the removal of asymptomatic impacted TMs, shared decision making. Anesthetic options (local anesthetic, intravenous sedation, or general anesthetic). Pathway (minor oral surgery, hospital day case, or hospital inpatient). |
Risk and benefits | Incidence of risks associated with retaining TMs, general surgical risks (pain, bleeding, swelling, etc), anatomical-related risks (numbness of lip or tongue, oroantral fistula), rare risks (tuberosity/mandible fracture) |
Surgical procedure | Draping, anesthesia, flap, bone removal, tooth sectioning, tooth removal, socket irrigation, socket inspection, bone filing, suture, and gauze pack |
Postoperative care and recovery | Postoperative instructions, how to control pain, bleeding, swelling, infection, and dry/infected socket. Information about diet and oral hygiene. Expected recovery. |
Costs associated with the treatment | Depend on pathway: direct cost (surgeon, anesthetize, and/or hospital fees), indirect cost (time off work), insurance information |
More information for intravenous sedation and general anesthesia/dental anxiety management | Conscious sedation (oral, inhalation and intravenous sedation), general anesthetic |
The Scientific Information Reporting (SI Reporting) Scale is a 9-item binary scale that was developed for this study based on reporting information topics, which can be found on the assessed website. Items for this scale were scored 1 if the information section was covered and 0 if the information section was not covered in the examined website. Full scale scores range from 0 to 9. The SI Reporting Scale was used to identify websites of high relevance as a source of information.
The Scientific Information Referencing (SI Referencing) Scale is a 9-item Binary Scale that was developed for this study to measure referencing different information sections on the assessed website. Items for this scale were scored 1 if the information section was referenced and 0 if the information section was not referenced, and the full scale scores again range from 0 to 9.
The SIQ Scale is a 9-item Likert-like scale, which was developed to assess various aspects of information that should be provided to patients, based on literature review and authors’ experience in the field (
This was a 14-item scale that was developed by authors [
This is a 16-item scale developed by Charnock [
The website total score was used as a measure of the total website quality. It was calculated as an unweighted sum of website usability, trust, SIQ, scientific information comprehensiveness, scientific information referencing, scientific information reporting, accessibility, DISCERN, HoNCode Scales, and the reverse-coded mean readability grade. The website total scores range from 57 to 222.
To allow the evaluator (KH) to provide qualitative feedback on the assessed websites, the researcher commented on areas of biased/unbalanced information. In addition, the researcher commented on factors that might affect information readability and the recommended treatment options. These comments were then coded into themes and subthemes for analysis.
Data were analyzed using IBM SPSS Statistics for Windows version 22.0 (IBM, NY, USA) [
To explore predictors for SIQ scores, linear regression was performed after creating dummy variables for website affiliation groups. A block of website affiliation dummies (Model 1) was entered in linear regression, where the “other” group was used as a reference category. In Model 2, DISCERN was added, and in Model 3, DISCERN was removed and replaced by the HoNCode score while statistically controlling for website affiliation. Websites were ranked according to their SIQ score and to their total (unweighted) score. The correlation between the 2 ranking orders was examined using Spearman ranking correlation.
The website reviewer’s (KH) comments were analyzed using NVivo 10 [
Of the 50 websites available for content analysis, a majority of the reviewed websites (54%, 27/50) were related to a treatment provider after adding 1 website to this group from the “educational institute” group that has a teaching hospital attached to it. A total of 7 of the 50 (14%) websites were related to commercial websites, and governmental and nonprofit organizations websites were equally represented (8%, 4/50). There were 7 “other” group websites (hub pages, blogs, news, and wiki, 14%). A combination of text and image was the most commonly used information format (40%, 20/50). Question and answer was the most predominant information communication method either alone (34%, 17/50) or in combination with fact sheets (22%, 11/50).
Most websites were open access (74%, 37/50), and the most common form of content editing was posting comments (14%, 7/50). All websites were accessible without either registration or subscription. A majority of websites were judged easy to navigate (62%, 31/50) while slightly above half of the websites (52%, 26/50) had no search facility. Facebook (23% of Web 2.0 applications, 28/121) and Twitter (20% of Web 2.0 applications, 24/121) were the most commonly used Web 2.0 applications.
The developed SIQ Scale had good internal consistency (Cronbach alpha=.85). Furthermore, the SIQ scores were significantly correlated with DISCERN scores (
DISCERN had high internal consistency (Cronbach alpha=.91), whereas that for HoNCode was slightly lower (Cronbach alpha=.80). DISCERN and HoNCode were significantly correlated with each other (
One-Way ANOVA showed a significant association between website affiliation and SIQ (
Quality and readability scores by website affiliation.
|
Website affiliationa |
|
|||||||||
Commercial | Treatment provider | Government | Nonprofit organization | Other |
|
||||||
Mean | SD | Mean | SD | Mean | SD | Mean | SD | Mean | SD |
|
|
Scientific Information Quality (SIQ) | 32.6a | 9.5 | 26.0a | 7.8 | 34.8a | 10.9 | 29.0a | 9.8 | 17.0b | 7.3 | .01 |
DISCERN | 62.3a | 9.3 | 46.0b | 10.7 | 59.3a | 17.6 | 61.8a | 12.1 | 44.1b | 13.5 | <.01 |
Health on the Net Code | 35.4a | 4.3 | 26.8b | 3.7 | 35.3a | 2.5 | 35.8a | 4.9 | 30.7a | 7.8 | <.01 |
Mean readability grade | 10.3a | 0.9 | 10.6a | 1.4 | 10.4a | 3.7 | 11.3a | 2.9 | 9.0a | 2.7 | .34 |
SI Comprehension | 9.9a,c | 1.3 | 12.8b | 2.6 | 9.8a,b,c | 2.2 | 13.5a,b | 7.6 | 8.3c | 4.5 | .01 |
SI Referencing | 10.6a | 2.4 | 9.1b | 0.5 | 9.8a,b | 0.5 | 11.8a | 3.1 | 10.1a,b | 2.3 | .01 |
Trust | 5.0a | 1.4 | 4.1b | 0.4 | 4.8a,b | 1.0 | 4.5a,b | 1.0 | 4.0b | 0.0 | .03 |
Usability | 10.3a | 1.4 | 8.4b | 1.1 | 10.5a | 0.6 | 9.8a | 1.3 | 10.4a | 1.4 | <.01 |
SI Reporting | 16.9a | 1.6 | 16.2a | 1.4 | 17.0a | 2.0 | 16.3a | 2.2 | 13.9b | 2.1 | .01 |
Accessibility | 1.0a | 0.0 | 1.1a | 3.0 | 1.5b | 0.6 | 1.0a | 0.0 | 1.0a | 0.0 | .02 |
Total score | 153.8a | 26.1 | 114.3b | 22.3 | 152.6a | 29.9 | 144.9a | 26.9 | 114.0b | 25.4 | .02 |
aValues in the same row and subtable not sharing the same subscript are significantly different at
Linear regression models (
Scientific Information Quality score prediction models.a
Model | Unstandardized coefficients | 95% CI for B | Standardized coefficients |
|
Significance ( |
Model summary | |||||
|
Standard error | Lower bound | Upper bound | Beta | Adjusted |
|
Significance |
||||
1 | Constant | 17.00 | 3.16 | 10.64 | 23.36 |
|
5.38 | <.01 | .21 | .28b | .01 |
Commercial | 15.57 | 4.47 | 6.58 | 24.57 | .58 | 3.49 | <.01 | ||||
Treatment provider | 8.96 | 3.53 | 1.85 | 16.08 | .48 | 2.54 | .02 | ||||
Governmental | 17.75 | 5.24 | 7.20 | 28.30 | .52 | 3.39 | <.01 | ||||
Nonprofit organization | 12.00 | 5.24 | 1.45 | 22.55 | .35 | 2.29 | .03 | ||||
2 | Constant | -9.50 | 3.16 | -15.87 | -3.14 |
|
-3.01 | <.01 | .76 | .51c | <.01 |
Commercial | 4.68 | 2.71 | -0.77 | 10.13 | .17 | 1.73 | .09 | ||||
Treatment provider | 7.83 | 1.96 | 3.87 | 11.79 | .42 | 3.99 | <.01 | ||||
Governmental | 8.68 | 3.05 | 2.55 | 14.82 | .25 | 2.85 | .01 | ||||
Nonprofit organization | 1.43 | 3.09 | -4.80 | 7.66 | .04 | 0.46 | .65 | ||||
DISCERN | 0.60 | 0.06 | 0.48 | 0.72 | .85 | 10.09 | <.01 | ||||
3 | Constant | -2.45 | 8.49 | -19.56 | 14.67 |
|
-0.29 | .76 | .29 | .09d | .02 |
Commercial | 12.59 | 4.41 | 3.70 | 21.47 | .47 | 2.85 | .01 | ||||
Treatment provider | 11.43 | 3.50 | 4.38 | 18.48 | .61 | 3.27 | <.01 | ||||
Governmental | 14.88 | 5.11 | 4.59 | 25.17 | .43 | 2.91 | .01 | ||||
Nonprofit organization | 8.81 | 5.14 | -1.54 | 19.16 | .26 | 1.72 | .09 | ||||
Health on the Net Code Scale | 0.63 | 0.26 | 0.11 | 1.16 | .39 | 2.45 | .02 |
aThe “other” website affiliation group was used as a reference category.
b
c
d
The mean (SD) readability grade (
Box plot of readability grades and mean readability grade.
Websites were ranked according to their SIQ scores. Results showed that the Bupa-UK website [
The comment section was analyzed using thematic analysis. Biased or unbalanced information provided was coded. TMs and late teen crowding were the most frequently reported biased information (41% of reported biased/unbalanced information, 13/31). Forcing patients to undergo “sleep dentistry” (16% of reported biased/unbalanced information, 5/31) was an example of unbalanced information, where information providers limited the anesthetic options to general anesthesia or sedation without providing local anesthetic as an option. The treatment provider group was associated with the highest frequency of biased/unbalanced information (74% of reported biased/unbalanced information, 23/31), which was also confirmed by the weighted frequencies of biased/unbalanced information across different website affiliation groups.
Comprehensibility of information was affected by the use of terminologies without explanation (28% of reported readability issues, 10/35), and/or the use of illustrations that were incorrectly labeled (5% of reported readability issues, 2/35), or inadequately labeled (5% of reported readability issues, 2/35), or sometimes not relevant at all (11% of reported readability issues, 4/35). In addition, poor information presentation and organization (11% of reported readability issues, 5/35) played an important role in the ability of finding information. Furthermore, repetition was found in some of the reviewed websites (11% of reported readability issues, 5/35).
The most frequently reported treatment option was the removal of symptomatic wisdom teeth and to seriously consider removal of asymptomatic ones (30% of reported treatment options, 7/23), while 4 websites (17% of reported treatment options) recommended the prophylactic removal of all wisdom teeth to “get peace of mind.” A number of websites (28% of reported treatment options, 6/23) recommended the removal of only symptomatic ones. There were instances where patients were advised to get a second opinion (17% of reported treatment options, 4/23) before making a treatment choice related to their wisdom teeth. Coronectomy (removing the crown and retaining the root) as a treatment option for high-risk wisdom teeth was rarely mentioned.
In this study, we aimed to provide a guide to assess the quality and readability of online health information with an application on Internet-related information concerning TMs problems using a scale developed for this purpose. The study also identified a shortlist of high-quality resources that might be recommended by clinicians to patients having TMs problems. Because online resources are dynamic, the researchers explored predictors for SIQ that might be used for a quick and easy identification of high-quality online resources.
To identify high-quality resources, a search was carried out using 3 common search engines (Google, Yahoo!, and Bing), and 3 keywords thought to be used by an average patient. While some authors have claimed that patients do not normally go beyond the first 25 results [
Internet information was delivered using mainly question and answer format either alone or together with fact sheets. Preferences of dental patients in relation to information delivery format need further investigation as there is a knowledge gap in the existing literature in this area. In addition, treatment providers should consider using online forums on their websites supported by health professionals to allow for a better engagement with patients [
Evaluating the quality of scientific information was challenging, especially with the lack of reliable and valid assessment tools. In addition, evaluating the scientific content requires a person who has extensive knowledge in the field. This paper demonstrated that the newly developed SIQ Scale has a high internal consistency and also displayed convergent validity with information credibility tools (DISCERN and HoNCode), which can be used by other researchers. Website affiliation was found to have a significant association with SIQ, usability, accessibility, trust, DISCERN, and HoNCode.
Linear regression models were used to explore the predictors for SIQ. The importance of this step is to make clinicians spend less time and effort to identify high-quality Internet resources, where no content analysis study is available. Website affiliation was able to significantly predict SIQ. Among different groups of website affiliation, governmental websites were found to be associated with the highest predicted SIQ score compared with the reference category. Credibility indicators—either DISCERN or HoNCode—were able to significantly predict SIQ after statistically controlling for website affiliation. A majority of variance in SIQ scores were explained by website affiliation and DISCERN. This finding is important because it might not only improve clinicians’ ability to identify high-quality online resources but also improve patients’ ability to find these resources by reviewing the governmental websites in light of DISCERN criteria.
Among the reviewed websites, the recommended treatment options were a reflection of the clinical uncertainty related to asymptomatic wisdom teeth [
Among the used readability-grade estimates, the New FOG readability grade was the most powerful in predicting scientific information comprehensibility. The significant correlation between the mean readability grade and Scientific Information Comprehension Scale score suggested convergent validity and consequently that the Readability Studio software could be used to assess information comprehensibility. In this study, the estimated mean readability grade was higher than Grade 8 as recommended by some health authorities [
The strong and significant correlation between websites’ ranking according to their SIQ and their ranking according to total scoring suggested that websites associated with the SIQ were also associated with other quality aspects such as readability, usability, trust, and credibility. Such results suggest that future research might focus on the SIQ Scale, readability-grade estimate, and DISCERN to limit the evaluation process.
The main limitation of this study lay in 2 main areas: sampling bias and examiner bias that were known to the researchers when conducting data collection and analysis. However, effort was made to minimize their impact by using predetermined assessment criteria and to statistically validate the measurements used. In addition, websites were evaluated by the main author who has appropriate academic qualifications and clinical experience—an approach that has been used in previous research [
The strengths of our study were (1) the contribution to the field of health informatics such as the development and initial validation of the SIQ Scale and the validation of Readability Studio Professional 2012; (2) contribution to current clinical practice by providing a shortlist of high-quality websites (however, clinicians need to consider the dynamic nature of online resources); (3) the development of criteria for patient information concerning wisdom tooth problems (see
This study provides clinicians with guidance in assessing Internet resources for patients suffering from wisdom tooth problems. However, clinicians may apply similar techniques when recommending websites to patients who suffer from other dental problems. Consumer health information providers should consider evidence-based information, use of multimedia, and information readability during the process of information production. Readability Studio Professional 2012 was found to be valid as a software application for assessing comprehensibility of online health information. Website affiliation and DISCERN were found to play a major role in the prediction of SIQ. Governmental websites were associated with the highest prediction for SIQ. DISCERN and HoNCode as online information credibility tools were significantly able to predict the SIQ. In instances where no guidance is available, patients could review governmental websites in light of DISCERN criteria to identify high-quality information. The developed SIQ Scale had high internal consistency and established convergent validity, suggesting its use in the future to assess the SIQ of online dental information.
Criteria used for evaluating scientific information quality concerning wisdom teeth problems.
analysis of variance
Health on the Net
Health on the Net Code
scientific information quality
Scientific Information Comprehension
Scientific Information Referencing
Scientific Information Reporting
third molars
This paper was part of the principal author’s PhD research project, for which an Adelaide Scholarship International was received. The research received support from the Australian Research Centre for Population Oral Health, The University of Adelaide.
None declared.