Readability scores were obtained using online readability analysis tools. The text from the entire website was copied into word processing software using the “text only” option, meaning features such as bold, italics, or hyperlinks were removed. All pictures, tables, and bullet points were also discarded, and lines of text following bullet points were not included if they did not form full sentences. This text was then pasted in its entirety into the automated readability tool. The online utility tool [30] used for English text analysis was extensively used to measure English readability in several Anglophone countries [21,31-33] and provided scores for each of the following reliable and validated readability formulae: the Fog Index, the Simplified Measure of Gobbledygook (SMOG), and the Flesch Kincaid Grade level [34-36]. These formulae calculate the grade level necessary to understand content based on the characteristics of text such as number of syllables per word and number of words per sentence; higher scores reflect lower readability. Average grade level readability scores were computed from the Fog, SMOG, and Flesch Kincaid Grade Level scores. The practice of averaging readability scores from several tools is widely endorsed [21,37,38] as a method of ensuring reliability.

French text was evaluated using the Système d'Analyse de Textes par Ordinateur (SATO) [39]. This tool was developed by researchers and government agents to assess French readability [40], and provides an adapted version of the Fog Index for the French language. This was the only automated readability score calculated for French content, given a lack of freely accessible online tools to calculate French text readability.

Two trained raters completed suitability and quality assessments using the Suitability Assessment of Materials (SAM) and the DISCERN. These tools provide a standardized method for evaluating both print and electronic resources and have established reliability and validity.

The SAM checklist [23] is a 22-item measure that assesses how suitable material is for patient purposes based on 6 factors that affect both readability and comprehension. Content areas are assessed for how easily a reader understands the stated purpose of the material, the scope of information provided in relation to its purpose and the provision of examples of behavior that help problem-solve. This section also evaluates if an overall summary that retells key messages is provided, ideally using visual aids or examples. Section 2, literacy demand, determines the reading grade level determined using the Fry formula [41] and analyzes the complexity of the writing style reflected in sentence length, the “voice” of the narrative and its vocabulary. Ideal terms are common and explicit, offer non-value judgments, and bring forth mental images. The order of information is considered: context should be provided before new information is described and followed by an “advanced organizer” such as a header that informs the reader what to expect next. The third section, graphics, reviews all illustrations, charts, lists, and tables. Cover graphics are rated on their friendliness, attractiveness, and clarity of purpose, while illustrations are evaluated for complexity and if they provide familiar and easy to remember concepts. The relevance of illustrations is based on whether key messages are expressed in visual terms and without distraction. Other non-text information such as charts or tables are scored on whether or not they are coupled to a set of instructions for navigation and comprehension, and explained in a caption. Layout and typography, section 4, contains explicit criteria to determine if layout meets superior standards based in part on whether text and images are arranged in a logical sequence and are of an appropriate color, how contrast and white space is used, and if visual cuing (arrows, shading, boxes) is incorporated. Type font and size, along with whether or not information is divided into manageable subsections, is evaluated. Section 5, learning stimulation and motivation, assesses whether readers are asked to actively engage with the material by asking questions or by problem-solving, whether appropriate behavior or skills are clearly modeled, and how well topics are divided into understandable or doable components that might ultimately lead to reader self-efficacy. The final section of the SAM determines the cultural appropriateness of the material by assessing how well the logic, language, and experience of the instruction and the reader match. In addition, images or examples of this culture should be presented in a positive light, not exaggerated nor caricatured.

In all categories, items are rated from 0 to 2, where higher scores reflect increased suitability. To obtain a total score, individually scored items are summed and divided by the total possible score for those items, which is represented as a percentage. These total scores are interpreted as follows: ≥70% “superior”, 40% to 69% “adequate”, and <40% “not suitable”. This tool has been extensively used in the assessment of Web-based health education materials by researchers and government bodies [42,43].

Researchers modify the SAM for on-screen patient educational material assessment by excluding individual items (as outlined in the original Doak and Doak scoring manual) while others [43] have omitted entire categories that are not applicable. One study replaced typography features such as the contrast of paper to typeface with online links [44]; we did not modify the original scoring rubric for the analyses performed here and omitted items that were not applicable on a site by site basis.

The 16-item DISCERN instrument [24] assesses the quality of material as a source of information on treatment choices. The DISCERN is divided into 3 subsections. Section 1 (questions 1-8) addresses the reliability of the source and investigates whether the document achieves a clearly articulated aim, the information is relevant and provides appropriate options to patients, the material is evidence-based, dated, and has been externally reviewed or is from a variety of sources and is unbiased, the information covers areas of uncertainty and finally, other relevant resources are provided. Section 2 (questions 9-15) assesses treatment options and describes how the treatment acts on the condition and may affect the patient, the benefits and risks of treatment, how a condition might progress without medical management, the quality of life that can be expected depending on treatment choice, if it is clear that different options exist, and finally, how to proceed with shared decision making which may include family, friends, or other health care professionals. Section 3 is composed of one final question that provides an overall rating for the material. Each item is rated from 1 to 5, where higher scores reflect higher quality. The DISCERN tool does not specify how to present total scores, but a majority of researchers have presented sum scores of the first 15 items (ranging from 15 to 75), and categorized total scores as “excellent” (63-75), “good” (51-62), “fair” (39-50), “poor” (27-38), and “very poor” (15-26) [45-48]. The DISCERN tool has demonstrated high internal consistency (Cronbach alpha = .78) and interrater agreement in total scores (ICC = .82, P<.001), as well as convergent validity by being significantly related to total scores using a different measure of material quality (r=.53, P<.001) [49].

In the present study, interrater reliability (IRR) was calculated for both the SAM and the DISCERN based on a sample of 15 websites coded by both raters (17.6% of the total sample). IRR was assessed for mean subsection scores for both scales using two-way single-measures mixed effects intraclass correlation (ICC) for absolute agreement, where higher values reflect higher IRR. According to the interpretation guidelines summarized by Cicchetti [50], our ICC scores ranged from fair (ICC=.533) to excellent (ICC= .828). Disagreements in scores were resolved by consensus ratings. A summary of IRR statistics can be found in Table 1.

Other characteristics that were assessed included website affiliation (government, university/medical, non-profit organization, commercial or corporate, private practice), position in Google, year last updated, and whether or not sites contained information for men. We noted if sites contained advertising: announcements such as banner images, animations, text, or images that redirected to outside websites and which were designed to sell a commodity or service. The presence of audiovisual aids such as embedded videos or animation, podcasts, or audio recordings was also evaluated.

Data were analyzed using the Statistical Package for the Social Sciences (IBM SPSS Statistics for Windows version 19.0, Armonk, NY). Critical value of significance was determined at P=.05.

The mean readability score for the entire sample was grade level 14.19, meaning that the content was geared to readers with at least some university education. The French language readability tool (SATO) calculated 16.62 years of education were necessary for French sites to be easily understood, equivalent to the “proficient user” level (C1-C2) as defined by the Common European Framework of Reference for Languages [51]. Overall, readability scores ranged from grade level 10.19 to 20.40, demonstrating that none of the websites were appropriate for the recommended 8th grade level readability.

Only 1 (1%, 1/85) website met the criteria for “superior” suitability as established by the SAM, 61 (72%, 61/85) met the criteria for “adequate” suitability, and 23 (27%, 23/85) were deemed “not suitable”. The sample-wide mean SAM score was 45.37% (SD 11.21), indicating “adequate” suitability.

Descriptive statistics for individual items on the SAM can be found in Table 3. The highest suitability scores were for items related to the placement of context (2d), visual navigation aids (2e) typography (4c), and motivation (5c). Items that rated low in suitability included content about behavior (1b), summaries (1d), reading grade level (2a), the relevance of illustrations (3c), and cultural images (6b). Overall, many sites failed to provide illustrations or other graphics, as revealed by missing scores in section 3.

For the DISCERN instrument, only 3 websites met the criteria for “excellent” quality, 21 were “good”, 32 “fair”, 25 “poor”, and 4 “very poor”. The sample-wide mean DISCERN score was 43.19 (SD 10.46), which falls in the lower end of the “fair” range. Mean scores of DISCERN items are found in Table 4.

Items that rated highly in terms of quality included the material being suited to user needs and thus relevant (3), the provision of further sources of support or information (7), a description of how treatment acts on the condition and may affect the patient (9), and that there may be more than one choice of treatment (14). Items that scored lowest in terms of fulfilling the criterion were clarity of sources (4), description of risks of treatment (11), and what would happen without treatment (12).

Only 28 websites (33%, 28/85) reported the year of the last update.

Associations between different measures of readability were tested using Pearson correlation coefficient. The mean grade level readability score, SAM score, DISCERN total score, and DISCERN question 16 score were all significantly correlated. A summary of correlations can be found in Table 5. This demonstrates some convergent validity for the measures used to assess the websites.

Using one-way analysis of variance, we examined whether classification of website location (French Canada, English Canada or worldwide) or category (Canadian fertility clinic, major North American site, Google keyword search), were related to differences in our measures of content.

A significant between-group difference was found for average readability based on location, (F_2,82 = 12.87, P<.001). Post-hoc analyses using Tukey’s honest significant difference (HSD) demonstrated that the location “French Canada” had a significantly (P<.001) higher mean readability score of 16.66 (SD 2.14) than both “Canada” and “worldwide” locations with mean values of 13.91 (SD 1.60) and 13.73 (SD 1.88), respectively, indicating that French Canadian websites were less readable. Neither suitability nor quality ratings differed by search location.

Mean readability differed significantly across search categories (F_2,70 = 3.27, P=.05). In Tukey’s HSD post-hoc analyses, major North American websites had a mean grade reading level of 12.91 (SD 1.84), significantly lower than Canadian fertility clinics with a mean value of 14.32 (SD 1.36, P=.04). However, they were not different from Google keyword search sites (P=.28) with a mean value of 13.76 (SD 1.86).

There was also a marked difference in quality for sites from different categories (F_2,82 = 5.46, P=.006). Major North American sites had a significantly higher DISCERN mean score of 51.23 (SD 8.88) than clinic sites with a mean score of 40.36 (SD 9.30, P=.002) or Google keyword search websites with a mean score of 42.61(SD 10.59, P=.007). Search category was not related to suitability.

Sites were grouped based on their association with a private practice, government, a corporation or commercial enterprise, a university or medical society, or a non-profit organization. Readability, quality, and suitability did not differ based on affiliation.

The first 20 sites retrieved by Google were scored in the order they appeared, with the highest rank receiving the score of 1, or first. Google positioning was not associated to measures of readability or suitability with r (58) values of -.144 (P=.282) and -.151 (P=.257), respectively; however, higher positioning in Google was significantly correlated to the overall DISCERN score with an r (58) value of -.328 (P=.012).

We also investigated whether overall readability, SAM, and DISCERN scores differed according to presence of advertisements, or presence of audiovisual material. Sites that contained advertisements were not different in readability (t₁₀=0.798, P=.449), suitability (t₁₀=-0.021, P=.984) nor overall quality rating (t₈₃=1.56, P=.122), than those without. Websites that incorporated audiovisual aids were also not significantly different from those that did not in relation to readability (t₈₃=0.248, P=.805), suitability (t₈₃ =0.152, P=.880), and quality (t₈₃=-0.426, P=.671) scores