Figure 1 outlines the process of selecting target websites. We searched for websites using 2 search engines: Google Japan and Google USA (Google LLC). When conducting Web searches, we turned off both the positional information and history information reference functions of the personal computer. The search terms were (palliative care OR palliative medicine [in Japanese]) on Google Japan and (palliative care OR comfort care OR symptom management) on Google USA. We targeted the top 50 websites displayed in the search results. When selecting sites, we regarded webpages having the same domain as belonging to the same website. We excluded websites not containing palliative care–related information, such as those introducing books and magazines, and with PDF pages of research papers. We ascertained that the top 50 websites found on Google USA were US sites.

We evaluated the information available on the target websites by examining the owner information included in each website and by using 2 evaluation indices: the DISCERN (Textbox 1) and LIDA instruments (Textbox 2).

The DISCERN instrument [8] is an index to evaluate the quality of information regarding treatment choices, developed by Shepperd and colleagues at Oxford University, Oxford, UK. We evaluated the reliability of information (reliability-1) and the quality of information regarding treatment choices (information quality) on a 5-point scale from 1 (no) to 5 (yes). Subsequently, we evaluated overall aspects on a 5-point scale from 1 (serious or extensive shortcomings) to 5 (minimal shortcomings).

The LIDA instrument is an index to evaluate health-related information available on websites, created by Minervation, a health care consultancy [9]. It consists of 3 domains: accessibility, usefulness, and reliability (reliability-2). In this study, we rated the evaluation items on a 4-point scale from 0 (never) to 3 (always). Among the 3 domains, we evaluated only usefulness and reliability-2, as accessibility is not closely associated with reliability.

The evaluation was conducted by multiple raters independently. When interrater agreement was poor, we discussed the results with a new supervisor and used the agreed-upon results.

We divided owner information into 2 categories for evaluation: personal and organizational. We further divided organizational into medical, administrative (public), general (profit), and general (nonprofit) for comparison between Japan and the United States. We defined each category as outlined in Textbox 3.

We divided the categories of the website contents into dedicated palliative care sites, general information sites about cancer (including palliative care–related information), general medical sites (including palliative care–related information), and nonmedical sites (including palliative care–related information), and we assessed those. Textbox 4 outlines the definitions of the website content categories.

To evaluate their reliability, we also divided the target websites into 2 groups—high rank and low rank—based on their ranks when displayed in the search results. We then compared their scores for each item of the 2 evaluation indices. With regard to the reliability-related items (reliability-1 and reliability-2) of both evaluation indices, we examined the correlation between the scores for these items and ranks.

We conducted a chi-square test to compare owner information. For comparisons of reliability, quality, and usefulness of the information available on the target websites between Japan and the United States, we calculated the mean scores for the 2 evaluation indices. The t test was conducted (with the significance level set at .05%, and using Cohen d for effect size calculation) using the statistical software IBM SPSS Statistics 24 (IBM Corporation). We applied Cohen’s standard setting to measure effect size, where no effect size is 0≦ d ＜0.2, small effect size is 0.2≦ d ＜0.4, medium effect size is 0.4≦ d ＜0.6, and large effect size is 0.6≦ d. Furthermore, we examined the correlation between each score and ranks by calculating the Pearson product-moment correlation coefficient and conducting a test to confirm the absence of a correlation.

We identified approximately 9,000,000 websites using Google Japan (date of search: December 13, 2016) and 17,500,000 websites using Google USA (December 14, 2016). After excluding 1 website from the Google Japan search and 4 from the Google USA search, we targeted 49 Japanese sites and 46 US sites for evaluation.

None of the targeted websites from either Japan or the United States were personal websites. We categorized organizational websites as follows: (1) from Japan, medical: 33; general (profit): 7; general (nonprofit): 3; administrative (public): 4; and academic: 2; and (2) from the United States, medical: 24; general (profit): 7; general (nonprofit): 4; administrative (public): 7; and academic: 4. In both cases, medical websites were the most frequent (Table 1). The chi-square test revealed no significant differences in the categorization of owner information between the Japanese and US websites (P=.55).

Most dedicated palliative care sites from the United States, but none of those from Japan, were hospice service program websites. Furthermore, most general medical sites from the United States were informational sites about home visits, while most of those from Japan were general hospital introductory sites.

On comparing evaluation index scores, US websites had significantly higher scores related to reliability-1 and reliability-2 than did the Japanese websites (Table 2). Scores related to information quality and usefulness did not differ markedly.

This study compared and evaluated the reliability, quality, and usefulness of palliative care information available on websites from Japan and the United States. Although the usefulness was similar between the 2 countries, the reliability and quality were lower in Japanese websites, revealing insufficient measures to provide such information with sufficient reliability in Japan. In previous studies, websites not specifying information sources and those showing profit-seeking behavior were noted as problematic [10,11]. Similarly, in this study, information sources and dates of publication were not appropriately described on some Japanese websites, revealing their insufficient usability for patients and their families. It has been suggested that Japanese palliative care sites have problems with their update frequency, their update policies and procedures, and the scrutiny process that evidence must undergo, since the scores for the US websites showed a significantly large effect size in 2 categories: currency and content production procedures, especially in the LIDA subcategory. There was no significant difference in the overall usefulness of the websites, but in the LIDA subcategory engagability, US websites had a higher score and were slightly more user friendly because many sites had interactive content and pictograms as an alternative to text. The subcategory engagability includes queries about website integration with nonverbal expression and, in the realm of drug information, the usefulness of pictograms as a tool for information communication has been reported [12,13]. This suggests that using pictograms could enhance a website’s user friendliness for users whose mother tongue is different from the language of the website. Additionally, we do not know whether any Japanese palliative care–related websites have applied the HON Foundation code of conduct (HONcode), while many English-language websites do [14-16]. A certification system such as HONcode could be useful to help determine the reliability of a website because it would be possible to refine one’s search by whether sites are certified.

Owner information categorization did not significantly vary between the 2 countries, indicating that websites belonging to medical institutions tended to be the most frequent information sources for patients in both countries. However, regarding each content category, dedicated palliative care sites ranked highest in the United States, and general medical information sites ranked highest in Japan (especially in cases where palliative care information was included in the hospital informational sites). Therefore, it may be necessary to create official sites that are dedicated to palliative care service and to provide information that is regularly reviewed and updated. Additionally, proactively applying a third-party certification system such as the HONcode is desirable because even medical experts often vary in their subjective assessments [17].

The significant difference in the level of reliability between high- and low-rank websites in both countries and the weak correlation between such levels and ranks suggest an association between the rank of each website and its reliability. The order in which websites are displayed in Google search results is based on an algorithm called PageRank [18]. This algorithm calculates original scores for individual websites based on keyword matching, the number of backlinks, updating frequency, information volume and consistency, browsing frequency, and coding appropriateness. PageRank determines where a website ranks in the search results. Therefore, the rank of a website in the search results does not necessarily reflect its reliability. However, the possibility of a website appearing high in the order as a result of being regarded as highly reliable by users and achieving a large number of accesses should also be considered. In this respect, detailed search orders themselves may not accurately reflect the reliability of websites, although those providing more reliable information tend to appear higher in the search results. In fact, in previous studies, search orders were reported to be inappropriate as a measure to examine the reliability of websites [19], and we obtained similar findings in this study. Thus, when using internet search services, the following points listed by some researchers should be noted: information contained in webpages may be inaccurate; such information is based on providers’ intentions; and websites are not primary information sources [20]. Our study also suggested the necessity of exercising caution when using information available on the internet.

With the revision of Google’ search logic, the individualization of information based on locality and access history has been promoted, but reliability-related issues have yet to be resolved. At this point, information available on websites visited by patients has been reported to be less accurate than that provided through websites targeting medical professionals [21,22]. As it may be difficult for patients without expertise to judge whether such information is sufficiently reliable, medical professionals should support patients in obtaining information that they need.

This study had some considerable limitations. First, our searches may have missed some websites. Second, we might have obtained different results using other search engines. However, although search results can differ by adding keywords, we think that this difference would be negligible and unlikely to have a large influence on the results. Third, the results might not be applicable to palliative care websites from countries other than the United States and Japan. However, although there are some limitations to this study, few studies have assessed the reliability of website information in the palliative care realm. Therefore, our results can be considered to offer future possibilities for providing information.

While usefulness-related scores did not significantly vary, reliability-related scores were lower for Japanese websites. In 3 LIDA instrument subcategories, US scores were higher and had large effect sizes: engagability, currency, and content production procedure. This suggests that Japanese websites have problems in their frequency of being updated, update procedure and policy, and scrutiny process for evidence. We also clarified that the reliability of websites is weakly correlated with their ranks, but such ranks are not sufficient to judge whether the websites are sufficiently reliable. Based on these results, it may be necessary to evaluate information sources used by patients.