JMIR Publications

Interactive Journal of Medical Research

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 12.05.17 in Vol 6, No 1 (2017): Jan-Jun

This paper is in the following e-collection/theme issue:

    Original Paper

    Thoracic Surgery Information on the Internet: A Multilingual Quality Assessment

    1University of Melbourne, Melbourne, Australia

    2University Hospitals Case Medical Centre, Cleveland, OH, United States

    3Austin Hospital, Department of Surgery, University of Melbourne, Melbourne, Australia

    Corresponding Author:

    Nathan Lawrentschuk, MBBS, FRACS (Urology), PhD

    Austin Hospital

    Department of Surgery

    University of Melbourne

    Suite 5, 210 Burgundy Street, Heidelberg

    Melbourne,

    Australia

    Phone: 61 394553363

    Fax:61 394575829

    Email:


    ABSTRACT

    Background: Previous data suggest that quality of Internet information regarding surgical conditions and their treatments is variable. However, no comprehensive analysis of website quality exists for thoracic surgery.

    Objective: The aim of this study was to quantify website quality in a multilingual setting using an international standard for assessment.

    Methods: Health On the Net (HON) principles may be applied to websites using an automated toolbar function. We used the English, French, Spanish, and German Google search engines to identify 12,000 websites using keywords related to thoracic conditions and procedures. The first 150 websites returned by each keyword in each language were examined. We compared website quality to assess for tertile (is the quality better in first, second, or third 50 websites returned) and language differences. A further analysis of the English site types was undertaken performing a comparative analysis of website provider types.

    Results: Overall, there are a considerable number of websites devoted to thoracic surgery: “lung cancer” returned over 150 million websites. About 7.85% (940/11,967) of websites are HON-accredited with differences by search term (P<.001) and tertiles (P<.001) of the first 150 websites, but not between languages. Oncological keywords regarding conditions and procedures were found to return a higher percentage of HON-accreditation. The percentage of HON-accredited sites was similar across all four languages (P=.77). In general, the first tertile contained a higher percentage of HON-accredited sites for every keyword.

    Conclusions: Clinicians should appreciate the lack of validation of the majority of thoracic websites, with discrepancies in quality and number of websites across conditions and procedures. These differences appear similar regardless of language. An opportunity exists for clinicians to participate in the development of informative, ethical, and reliable health websites on the Internet and direct patients to them.

    Interact J Med Res 2017;6(1):e5

    doi:10.2196/ijmr.6732

    KEYWORDS



    Introduction

    Background

    As patients are diagnosed with serious conditions and await complex procedures, it is accepted that they inherently will explore the Internet for answers. Over 80% of patients, health care professionals, and other invested groups utilize the Internet to seek medical information, seeing it as a reliable, trustworthy, and accessible source [1-3]. Industry groups, clinicians, and health care institutions may construct websites with commercial interests in mind [1,4]. In contrast, only a minority of websites are sponsored by government or educational organizations and nonprofit organizations, which may provide objective, unbiased, and hence more accurate information, compared with other sponsors [5,6]. Therefore, the Internet’s accessible source of health information, and frequency of use by the majority, substantiates the need to assess its quality and validity.

    Thoracic surgery is a common mode of treatment for many patients with lung cancer. However, patients face a range of extensive and unregulated information regarding conditions and procedures on the Internet, often affecting their expectations and informed decision-making [7,8]. Moreover, language affects the quality of information [9-14], which impacts on multicultural societies and non-English speaking patients who require reliable information.

    Health on the Net

    Clinicians also require tools both to identify quality information for themselves and also to direct their patients to reliable, high quality Internet resources [11-13]. High quality and reliable health information can be found through the help of several tools [10,11,15]. The Health On the Net (HON) Foundation is one such tool. HON is a not-for-profit multilingual accreditation body that aims to accredit health websites according to its key principles of authority, complementarity, confidentiality, attribution, justifiability, transparency of authorship, sponsorship, and advertising [11]. The HONcode offers directions for users in evaluating and creating a trustworthy and reputable website [16,17]. Of note, website quality has been tested using the HONcode tool across a range of specialties with only a small percentage of websites (7-27%) being routinely accredited [6,18-20].

    A comprehensive literature search regarding website information within the sphere of thoracic surgery was undertaken, yielding no studies that evaluate the quality of thoracic surgery-related information on the Internet. In this study, we aimed to evaluate the quality of current Internet information on thoracic surgery websites based on HON principles, and to compare differences between English, French, German, and Spanish language sites. The effect of language relates to what websites appear on different Google search engines (English, French, German, and Spanish), and whether there are any differences in HON-accredited websites. Our secondary goal was to assess and compare information quality based on types of website sponsors.


    Methods

    Search Engine and Search Terms

    Our methodology has been previously described and used [11-13,21]. On this occasion, however, we used the corresponding Google search engine for each respective language search. We performed an Internet search of 20 terms in December 2014 to March 2015 (Table 1) and assessed 12,000 websites. As formal medical terminology has been used for search terms, the same search term used in English was used for the French, German, and Spanish searches on their respective Google search engines. The terms searched were “pectus excavatum,” “pectus carinatum,” “Nuss procedure,” “Ravitch procedure,” “Lorenz bar repair,” “lung cancer,” “nonsmall cell lung cancer,” “small cell lung cancer,” “VATS,” “video-assisted thoracic surgery,” “lung resection,” “lung wedge resection,” “pneumonectomy,” “thoracotomy,” “mediastinoscopy,” “bronchoscopy,” “EBUS,” “endobronchial ultrasound,” and “lung lobectomy.” An expert thoracic surgeon deemed these terms the most common and pertinent medical conditions and procedures for review in this study. These search terms were selected because they are the most objective terms that patients would hear during a consultation. By searching these terms, more meaningful data from websites can be ascertained. Ethics or Institutional Review Board (IRB) approval was not required for this study, since it does not involve patients but only Web-based review of publicly accessible websites.

    Internet Searching for Accredited Websites

    International and independent, qualified accrediting bodies check HON status at regular times, ensuring that HON certification meets the strict internationally accepted requirements. Moreover, the HON function has been evaluated by many studies, and judged to be a high caliber tool [10-13,22,23].

    Access beyond the first page of results by patients is rare [24]. Thus, the first 150 websites yielded by each search were identified and sequentially screened for quality as defined by the HON Foundation. HON principles through the HONcode toolbar function (downloaded from http://www.hon.ch/ for use on any personal computer. HONcode toolbar is easily installed, providing an accessible and visual cue for users) were then applied. According to the HON Foundation website [10], there are 8 criteria evaluated for HONcode certification of a website. These are (1) authoritative (indicate qualifications of authors), (2) complementarity (information should support, not replace, the doctor-patient relationship), (3) privacy (respect privacy and confidentiality of personal data submitted to the site by visitor), (4) attribution (cite the sources of published information, date medical and health pages), (5) justifiability (site must back up claims relating to benefits and performance), (6) transparency (accessible presentation, accurate email contact), (7) financial disclosure (identify funding sources), and (8) advertising policy (clearly distinguish advertising from editorial content). This toolbar automatically activates if a website is accredited by the HON Foundation (HONcode+), as opposed to the toolbar not lighting up, indicating that the website is not HON-accredited (HONcode−). On the basis of the previous studies, approximately 5% of websites could be deemed HONcode+, but have not been accredited yet [10-13].

    Analysis of Accredited Websites’ Likelihood of Being Viewed

    A secondary analysis of the first 150 websites encountered for each search term was undertaken, as previously described [6,18,25]. First, all returned websites for each search term were divided into tertiles (first 50, middle 50, and last 50). The proportion of accredited sites in each tertile and language was then analyzed and compared by the chi-square test. The purpose of this analysis was to determine whether accredited websites were appearing preferentially—that is, in the pages least likely (last 50) versus most likely (first 50) to be viewed.

    Quality Control

    For quality control, an English-language search of the control term, “lung cancer,” had nonaccredited sites within the first 150 discovered websites manually evaluated using the HON criteria to determine their HON status to ascertain if they fulfilled the criteria despite not being officially accredited.

    Logistic Regression Examining Variables Associated With HON Status

    This test was conducted using the three major variables of our study, namely a search term, language, and tertile, of the first 150 websites returned. The reference groups for each variable were excavatum, the first tertile, and English, respectively.

    Analysis of Website Sponsors

    For all search terms, an analysis was undertaken from English-language websites to determine who the website sponsors were. Only English language websites were examined due to the authors’ lack of proficiency in the other languages. The site sponsors were organized into the following groups: (1) lawyers, (2) nonprofit organizations, (3) government organizations or educational institutions, (4) commercial, (5) thoracic specialists and their professional organizations, (6) Books, articles, and references, (7) other health care professionals, (8) other (social media, forums, personal websites, newspapers, and (9) unrelated.

    Sponsorship was determined independently by information on the retrieved Web page regarding its origin; if sponsorship was not obviously apparent, the website was explored until sponsorship could be determined. The concept of sponsorship is not to be confused with the Google terminology of “sponsored links,” which either highlights pages at the start of retrieved search or lists links on the side of the page under a banner. As in a previous analysis, such pages were not included in this study [11].

    Statistical Analysis

    Comparisons of proportions across types of cancer and language were performed by the chi-square test (or Fisher exact test when counts were <5). All statistical tests were two-sided. Odds ratio and 95% CI were also calculated from the logistic regression analysis. The data analysis for this study was generated by SAS software version 9.1. (SAS Institute Inc).


    Results

    Internet Search Results for Accredited Websites

    The total number of websites for each thoracic surgery-related search term is variable (Table 1). “Lung cancer” had the most websites with approximately 150 million websites followed by “small cell lung cancer” with approximately 112 million websites. “Ravitch procedure” returned the least number, with only 159,890 websites.

    The total percentage of HON-accredited sites was notably low across all search terms (median 8%; see Table 1). “Lorenz bar repair,” “EBUS,” “endobronchial ultrasound,” and “VATS” had less than 5% of HON-accredited sites (Table 1).

    Regarding linguistic differences (see Table 2 and Figure 1), there was a similar number of HON-accredited thoracic websites across all languages evaluated. English (8%) and German (8%), French (7%) and Spanish (7%) had a similar percentage of HON-accredited sites.

    Tertiles were examined to ascertain where HON-accredited websites were more likely to appear. HON accreditation was seen statistically more commonly in the first tertile (0-50 sites) of websites (see Table 3 and Figure 2).

    Table 1. Number and percentage of HON-accredited websites.
    View this table
    Figure 1. Column graph of median percentage of Health On the Net (HON)–accredited sites for all keywords arranged according to language. Each keyword was searched on native Google search engine of respective countries. The graph indicates the median percentage of HON-accredited websites.
    View this figure
    Figure 2. Clustered column graph of percentage of Health On the Net (HON)–accredited websites for keywords arranged by tertiles. The color “blue” indicates percentage HON-accredited websites in first tertile, “red” indicates percentage HON-accredited websites in second tertile, and “green” indicates percentage HON-accredited websites in third tertile.
    View this figure
    Table 2. Percentage of HON-accredited websites by language.
    View this table
    Table 3. Percentage of HON-accredited websites by tertile.
    View this table
    Quality Control

    For the first 150 “lung cancer” (English) results, we found that 20 sites were accredited by the HON toolbar and 130 were not. We found that 6.9% (9/130) of those nonaccredited sites met HON criteria when assessed manually and 13.2% (79/600) of cancer-related thoracic surgery websites are HON-accredited.

    Logistic Regression Examining Variables Associated With HON Status

    Odds ratios calculated by search term, language, tertile, and between groups, highlighted significant differences (Table 4). For language, English compared with French, German, or Spanish was just as likely to return an accredited site. The second tertile of websites (51-100) assessed were more likely than the third tertile (101-150) to have accredited sites.

    Table 4. Odds ratio and 95% CI. Illustration of odds ratio of a search having HON-accreditation in relation to referent. The higher the ratio, the less likely a search term has HON-accreditation. The lower the ratio, the more likely a search term has HON-accreditation.
    View this table
    Table 5. Website sponsor analysis.
    View this table
    Analysis of Website Sponsors

    The sponsor analysis of the 150 websites in English (Table 5) indicated that the most commonly encountered sponsors were “books, articles, and references” (47.1%, 1399/2967) followed by “government or education” (27.7%, 821/2967), “others (social media, forums, personal websites, newspapers” (6.7%, 199/2967), “nonprofit organizations” (5.0%, 149/2967), “commercial” (4.5%, 134/2967), and “thoracic specialists or professional organizations” (3.4%, 100/2967). “Lawyer” (<1%, 9/2967) and “other health care professionals” (<1%, 3/2967) sponsored far less sites. A small percentage (6.3%, 188/2967) of sponsor websites were unrelated to medicine.

    Search terms with a larger percentage of “government or education” or “books, articles, and references” were the terms with a larger percentage of HON-accredited websites: “lung cancer,” “nonsmall cell lung cancer,” “small cell lung cancer” with P value <.001; “lung lobectomy,” “lung resection,” and “lung wedge resection” with P value .001; “pneumonectomy,” “bronchoscopy,” and “thoracoscopy” with P value .001.


    Discussion

    Principal Findings

    The aim of this study was to quantify information quality on thoracic surgery-related websites on the Internet. Clinicians may become aware of the lack of quality information regarding thoracic surgery and help to educate patients about the pitfalls of information on the Internet, and direct them to better quality websites.

    In summary, the total number of websites for keyword searches varies considerably. The total percentage of HON-accredited websites was markedly low across all search terms. There were minimal linguistic differences in HON-accredited websites, with HON-accredited websites most likely to appear in the first tertile. Nearly half of the websites were books, articles, or references, whereas nearly one-third were governmental or educational.

    Comparison With Prior Work

    The Internet has developed into an accessible source of health information for everyone. Health websites are guides for patients wanting to better understand their conditions [26]. Web-based health information was sought by 72% of adult Internet users over the last few years [27], a number predicted to grow. Clinicians directing patients to reliable information has many benefits: improving patient-doctor relationships, reinforcing consultation discussions, assisting informed decision-making, providing education before and after events, and helping patients seek appropriate consultation for sensitive topics (eg, urology, gynecology).

    There is a stark discrepancy between reliable health information and quality resources that disseminate it. The number of websites providing accurate information for thoracic surgery is not ideal. Only 13% of cancer-related thoracic surgery websites overall were HON-accredited. This is less than in our previous studies, uro-oncology websites [6] in 2009 and surgical oncology websites in 2012 [18], which each returned 18% of oncology-related HON-accredited websites. Similarly, there were 15% of HON-accredited gynecological oncology-related websites [20]. Worse still, only 9% of benign prostate hyperplasia websites were HON-accredited [19]. This reflects our hypothesis that reliable, high-quality health information on the Internet is lacking, specifically for thoracic surgery as well as in a broader context. In the latter study [19], only 7% of nononcology-related websites such as “surgical treatments” were HON-accredited. This figure is comparable with our 10% “Lungsurg” HON-accredited websites. These results are concerning because they imply that patients will encounter unreliable information about their condition, regardless of cancer type. Evidently, this makes website assessment difficult for patients and clinicians alike, potentially leading to distrust of Internet thoracic surgery resources.

    It has previously been acknowledged that website quality differs by language [10,11,14]. In our study, whereas English language searches returned more websites overall, both German and English searches returned 8% HON-accredited sites, and French and Spanish searches returned 7% HON-accredited sites. Thoracic surgery information is far more uniform across languages than results from our previous studies [6,18-20], albeit still alarmingly low. It is evident that there is a paucity of high quality, comprehensive information on thoracic surgery available around the world on the Internet, regardless of language. Similarly, HON-accredited websites are more likely to appear in the first tertile overall than in the second or third tertiles. This tertile discrepancy was expected since the Google algorithm generally places the most relevant websites first. Further analysis into the proportion of HON-accredited websites on the first page compared with the first tertile overall may yield interesting results, since it has been known that patients rarely move past the first search page.

    Websites also act as a conduit for advertising. Health information is increasingly being controlled by marketing and commercial interests, taking advantage of a significant proportion of the population searching for health information [28]. Consequently, unbiased views are sacrificed for the type of health information offered. However, the majority of sponsors in this study were composed of (1) academic books, articles, and references and (2) government or education. The absence of commercial bodies or marketing in this area implies that thoracic surgery information might not be biased or skewed for marketing purposes, compared with other medical fields previously analyzed [6,18-20]. Notably, the search terms with these sponsors were those with more website results and more HON-accredited websites. This suggests a conscious effort to provide high quality information for these conditions and procedures. Although our study only revealed 1% of websites sponsored by lawyers, a search performed in the United States may show otherwise. This illustrates the unpredictable nature of the Internet.

    HONcode is a simple means by which a clinician or patient can objectively correlate a website with high quality information. Compared with other instruments for evaluating website quality, it appears to be a straightforward, valuable tool, and fulfills its goal of identifying reliable health websites [29]. However, HONcode is by no means the only way to rate quality. The DISCERN instrument [30] and LIDA tool [31] are freely available online, designed to help users evaluate the quality of health information on the Internet. The ODPHP’s National Quality Health Website Survey instrument provides a sophisticated method to assess website quality, though is quite time-consuming and subjective [32]. Thus, compared with other, more intensive search tools, HONcode can be used to access reliable information easily by patients and clinicians, who have no prior experience or knowledge. Furthermore, it has been previously shown that website affiliation with HONcode is a significant predictor for scientific information quality [23]. Due to the growing number of websites, the HONcode certification seal is now obtained by voluntary application. However, many high quality websites lack the HONcode seal. In our study, 6% of websites in the control term could have met the criteria and this is consistent with prior research [6,18-20]. Currently, no studies evaluate awareness of HON certification in organizations and patients. Hence, shortcomings of HON may include voluntary application and lack of public awareness. Patients may bypass trustworthy websites, whereas organizations may not actively apply for HON certification. In a wider context, there is a notable lack of congruence of criteria between health information quality assessment tools [33]. Future research may be needed to streamline assessment tools, or streamline health website guidelines so that quality information is standardized. However, this is out of the scope of this paper. More immediately, further research is required to anal awareness of HON. Depending on these results, appropriate steps could then be taken to help clinicians, patients, and organizations to be exposed to HONcode, enabling access to reliable sources of information.

    Limitations

    It must be said that HONcode is a predictive indictor for high quality websites, which has its drawbacks. Thus, a proportion of websites with objectively high quality information may not fulfill HONcode criteria, and vice versa. As of 2015, HONcode certification is provided as a paid service. This can distort the validity of website information with HONcode criteria.

    An inherent limitation of this study involves the search terms used. It cannot be guaranteed that patients would use these terms in their own research of their condition. It is in dispute whether informal search terms would yield websites with better quality information. Conversely, it may result in unrelated website results. However, given that the search terms used in this study are the most formal and objective, informal search terms would likely defer to pages with the formal terms by the Google search algorithm. One solution to this limitation is to encourage clinicians to use the formal medical terms for their patients, thereby empowering patients to research their condition better, ultimately resulting in greater patient education.

    As with any Internet study, its dynamic and diverse character produces inherent limitations. In our study, we only performed searches in Melbourne, Australia. It would be interesting to perform multiple searches at various times and locations, analyzing any differences found. “Google” is the most popular search engine (http://searchenginewatch.com), having been used in other studies [10]. However, studies have also shown the impact of social media and health-related videos on YouTube on health care [34]. As these media are not appropriately standardized for health promotion and education, these studies highlight the need for caution among users. Search engines rely on language filters to determine sites returned, but Google enables a multilingual approach. A key advantage of Google may be for clinicians and patients who speak the languages analyzed here, which have a low number of accredited websites. Google translate may provide people with wider access to information online, though quality may vary. The impact of the validity of HON certification once a website has been translated by Google was not investigated in this study.

    Conclusions

    In conclusion, clinicians must appreciate the lack of validated information of most thoracic surgery websites. Discrepancies are apparent in quality and number of websites between search terms, tertiles, and language. Awareness of this lack of quality can facilitate clinicians in educating patients by using the formal medical term to empower patients to research their condition more comprehensively and thus gain a greater level of understanding. Clinicians must be proactive in identifying and directing patients to trustworthy and accurate information on websites. HONcode is an uncomplicated search tool and can serve as the vanguard to detect appropriate and trustworthy websites.

    Authors' Contributions

    MD was involved in data collection, data analysis, and manuscript writing and editing. RA helped in data analysis. SB and NL were involved in manuscript writing and editing.

    Conflicts of Interest

    None declared.

    References

    1. Risk A, Dzenowagis J. Review of internet health information quality initiatives. J Med Internet Res 2001;3(4):E28 [FREE Full text] [CrossRef] [Medline]
    2. Couper MP, Singer E, Levin CA, Fowler Jr F, Fagerlin A, Zikmund-Fisher BJ. Use of the Internet and ratings of information sources for medical decisions: results from the DECISIONS survey. Med Decis Making 2010;30(5 Suppl):106S-114S. [CrossRef] [Medline]
    3. Menon M. Editorial comment on: laparoscopy in German urology: changing acceptance among urologists. Eur Urol 2009 Dec;56(6):1080-1081. [CrossRef] [Medline]
    4. DaJusta DG, Mueller TJ, Barone JG. Accreditation Council for Graduate Medical Education competency-based on-line computer course in pediatric oncology for urology residents. Urology 2008 May;71(5):818-820. [CrossRef] [Medline]
    5. Babamiri K, Nassab RS. The availability and content analysis of melanoma information on YouTube. Plast Reconstr Surg 2010 Jul;126(1):51e-52e. [CrossRef] [Medline]
    6. Lawrentschuk N, Abouassaly R, Hackett N, Groll R, Fleshner N. Health information quality on the internet in urological oncology: a multilingual longitudinal evaluation. Urology 2009;74(5):1058-1063. [Medline]
    7. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. J Am Med Assoc 2001;285(20):2612-2621 [FREE Full text] [Medline]
    8. Menon M, Bhandari M. Unhappy patients: musings of two surgical nihilists. Eur Urol 2008 Oct;54(4):723-725. [CrossRef] [Medline]
    9. Chen X, Siu LL. Impact of the media and the internet on oncology: survey of cancer patients and oncologists in Canada. J Clin Oncol 2001 Dec 01;19(23):4291-4297. [CrossRef] [Medline]
    10. -. HON code of conduct for medical and health Web sites. Am J Health Syst Pharm 2000 Jul 01;57(13):1283. [Medline]
    11. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. Br Med J 2002 Mar 09;324(7337):573-577 [FREE Full text] [Medline]
    12. Gaudinat A, Grabar N, Boyer C. Machine learning approach for automatic quality criteria detection of health web pages. Stud Health Technol Inform 2007;129(Pt 1):705-709. [Medline]
    13. Kaimal AJ, Cheng YW, Bryant AS, Norton ME, Shaffer BL, Caughey AB. Google obstetrics: who is educating our patients? Am J Obstet Gynecol 2008 Jun;198(6):682.e1-682.e5. [CrossRef] [Medline]
    14. Eastham JA. Robotic-assisted prostatectomy: is there truth in advertising? Eur Urol 2008 Oct;54(4):720-722. [CrossRef] [Medline]
    15. Avery KN, Blazeby JM, Lane JA, Neal DE, Hamdy FC, Donovan JL. Decision-making about PSA testing and prostate biopsies: a qualitative study embedded in a primary care randomised trial. Eur Urol 2008 Jun;53(6):1186-1193. [CrossRef] [Medline]
    16. Smith RP, Devine P, Jones H, DeNittis A, Whittington R, Metz JM. Internet use by patients with prostate cancer undergoing radiotherapy. Urology 2003 Aug;62(2):273-277. [Medline]
    17. Wang AJ, Bhayani SB. Robotic partial nephrectomy versus laparoscopic partial nephrectomy for renal cell carcinoma: single-surgeon analysis of >100 consecutive procedures. Urology 2009 Feb;73(2):306-310. [CrossRef] [Medline]
    18. Lawrentschuk N, Sasges D, Tasevski R, Abouassaly R, Scott AM, Davis ID. Oncology health information quality on the Internet: a multilingual evaluation. Ann Surg Oncol 2012 Mar;19(3):706-713. [CrossRef] [Medline]
    19. Chen EC, Manecksha RP, Abouassaly R, Bolton DM, Reich O, Lawrentschuk N. A multilingual evaluation of current health information on the Internet for the treatments of benign prostatic hyperplasia. Prostate Int 2014 Dec;2(4):161-168 [FREE Full text] [CrossRef] [Medline]
    20. Hewitt E, Mulcahy A, Lawrentschuk N, Jobling T, Abouassaly R, Bolton D. Health information quality on the internet in gynecological oncology: a multilingual evaluation. Int J Gynecol Cancer 2016;37(4):478-483. [CrossRef]
    21. Ekman A, Hall P, Litton J. Can we trust cancer information on the Internet? A comparison of interactive cancer risk sites. Cancer Causes Control 2005 Aug;16(6):765-772. [CrossRef] [Medline]
    22. Air M, Roman S, Yeo H, Maser C, Trapasso T, Kinder B, et al. Outdated and incomplete: a review of thyroid cancer on the World Wide Web. Thyroid 2007 Mar;17(3):259-265. [CrossRef] [Medline]
    23. Hanna K, Brennan D, Sambrook P, Armfield J. Third molars on the internet: a guide for assessing information quality and readability. Interact J Med Res 2015 Oct 06;4(4):e19 [FREE Full text] [CrossRef] [Medline]
    24. Killeen S, Hennessey A, El HY, Killeen K, Clarke N, Murray K, et al. Gastric cancer-related information on the Internet: incomplete, poorly accessible, and overly commercial. Am J Surg 2011 Feb;201(2):171-178. [CrossRef] [Medline]
    25. Alkhateeb S, Lawrentschuk N. Consumerism and its impact on robotic-assisted radical prostatectomy. BJU Int 2011 Dec;108(11):1874-1878 [FREE Full text] [CrossRef] [Medline]
    26. Hoppe IC. Readability of patient information regarding breast cancer prevention from the Web site of the National Cancer Institute. J Cancer Educ 2010 Dec;25(4):490-492. [CrossRef] [Medline]
    27. Nagler RH, Gray SW, Romantan A, Kelly BJ, DeMichele A, Armstrong K, et al. Differences in information seeking among breast, prostate, and colorectal cancer patients: results from a population-based survey. Patient Educ Couns 2010 Dec(81 Suppl):S54-S62 [FREE Full text] [CrossRef] [Medline]
    28. Mayer MA, Karkaletsis V, Stamatakis K, Leis A, Villarroel D, Thomeczek C, et al. MedIEQ-Quality labelling of medical web content using multilingual information extraction. Stud Health Technol Inform 2006;121:183-190. [Medline]
    29. Boyer C, Baujard V, Griesser V, Scherrer JR. HONselect: a multilingual and intelligent search tool integrating heterogeneous web resources. Int J Med Inform 2001 Dec;64(2-3):253-258. [Medline]
    30. Charnock D, Shepperd S. Learning to DISCERN online: applying an appraisal tool to health websites in a workshop setting. Health Educ Res 2004 Aug;19(4):440-446. [CrossRef] [Medline]
    31. Grewal P, Alagaratnam S. The quality and readability of colorectal cancer information on the internet. Int J Surg 2013;11(5):410-413 [FREE Full text] [CrossRef] [Medline]
    32. Devine T, Broderick J, Harris LM, Wu H, Hilfiker SW. Making quality health websites a national public health priority: toward quality standards. J Med Internet Res 2016 Aug 02;18(8):e211 [FREE Full text] [CrossRef] [Medline]
    33. Pauer F, Göbel J, Storf H, Litzkendorf S, Babac A, Frank M, et al. Adopting quality criteria for websites providing medical information about rare diseases. Interact J Med Res 2016 Aug 25;5(3):e24 [FREE Full text] [CrossRef] [Medline]
    34. Gabarron E, Fernandez-Luque L, Armayones M, Lau AY. Identifying measures used for assessing quality of YouTube videos with patient health information: a review of current literature. Interact J Med Res 2013;2(1):e6. [CrossRef] [Medline]


    Abbreviations

    HON: Health On the Net
    HONcode: toolbar function for website accreditation recognition by HON principles
    IRB: Institutional Review Board
    PoHONA: percentage of HON-accredited sites
    WHO: World Health Organization


    Edited by G Eysenbach; submitted 01.10.16; peer-reviewed by A Lau, F Pauer, K Hanna; comments to author 08.01.17; revised version received 02.02.17; accepted 21.02.17; published 12.05.17

    ©Myles Davaris, Stephen Barnett, Robert Abouassaly, Nathan Lawrentschuk. Originally published in the Interactive Journal of Medical Research (http://www.i-jmr.org/), 12.05.2017.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.