This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.
Websites are common sources of health information to stroke survivors and caregivers for continual management of stroke and its long-term sequelae. The presence of risk factors and mortality rates related to stroke are high in Korean Americans. A vast majority of this group are active Web users and rely on the Web-based information due to lack of insurance and, thus, limited access to long-term stroke care. Thus, it is critical to evaluate existing stroke websites for their trustworthiness, readability, and suitability.
The objective of our study was to provide a systematic evaluation of stroke-related websites regarding (1) trustworthiness, (2) readability, and (3) suitability for stroke prevention and self-management for Korean Americans.
We selected a total of 156 websites using search terms “stroke,” “CVA,” “중풍 (jungpung),” and “뇌졸증 (noejoljung)” on Google and Yahoo. After eliminating duplicates and irrelevant websites (n=116), we evaluated a total of 42 websites (15 in English and 27 in Korean) using the National Library of Medicine’s health website’s evaluation tool for trustworthiness; Simple Measure of Gobbledygook for readability; and Suitability Assessment of Materials for suitability. All three instruments used the 3-point Likert scale: superior (=2), adequate (=1), or not suitable (=0).
Of the 42 websites evaluated, we rated 62% (26/42) websites as “adequate” or above for trustworthiness. The information on 48% (20/42) websites had not been updated for more than a year, which indicates poor currency; 33% (14/42) websites failed to provide the publisher and contact information, which yields poor authority; 50% (21/42) websites did not cite sources of health information, which indicates lack of accuracy. Only 2 websites met the recommended readability (5th grade or lower reading level). The suitability was also suboptimal; only 1 website was rated as “superior”; 60% (25/42) websites were “adequate,” and 38% (16/42) were “not suitable.” Most websites were limited in graphical directions, interactive motivations for desired healthy behaviors, and multiple language translations.
The existing stroke-related websites in either English or Korean are trustworthy and suitable, yet precise citation of evidence-based information will improve trustworthiness. The contents requiring high reading level may set a barrier to the utilization of Web-based health information for Korean Americans with a lower level of education. In addition, supplementing graphical examples, interaction features, and culturally relevant information in multiple languages are the areas for improvement in suitability. The improved features can reduce the reading burden of stroke patients or caregivers and build their confidence when applying the information for stroke management in daily living. These strategies are especially crucial to Korean Americans, who inevitably seek Web-based information to fill the gap between their demand and access to health care for a long-term self-management after a stroke.
With advancements in the internet services and communication technologies, both release and seeking of health information through the Web have been exponentially growing. In the United States, 89% of adults use the internet, and among them, 72% lookup online for health information [
Use of Web-based health information is common among Korean Americans [
Universally, individuals with chronic or stigmatized diseases such as stroke are more likely to search for health information on the internet than those without the health conditions [
Stroke is the third leading cause of death among Korean Americans [
Obtaining Web-based health information can be beneficial for or harmful to people. The explosion and proliferation of health information available online are on the promise that these online resources can confirm or broaden patients or families’ understanding of diseases and treatment opinions that influence health care decisions and empower them to effectively self-manage health conditions [
Considering the high demand of Web-based health information in native or English language among Korean Americans, who have limited English proficiency, it is critical for the websites to be
The purpose of this study was to address the following question: Is the Web-based stroke information adequate for Korean Americans to read, understand, and engage in stroke care? Specifically, we aimed to evaluate stroke-related websites in terms of their (1) trustworthiness, (2) readability, and (3) suitability for stroke prevention and self-management in this group.
Initially, we considered a question that how Korean Americans would seek stroke information on the internet. One of the common behaviors exhibited was bilingual searches in their health information seeking as a technique for coping with limited English proficiency [
Based on a review of these studies about online health information-seeking behaviors of Korean Americans, we searched stroke-related websites in Google and Yahoo search engines using the following terms: “Stroke,” “CVA,” “중 (jungpung),” and “뇌졸증 (noejoljung).” The term, “Stroke” is health care consumers’ preferred term listed in the Consumer Health Vocabulary Initiative [
We initially evaluated each website for its trustworthiness using the following three criteria for health websites’ evaluation endorsed by the National Library of Medicine (NLM) [
We evaluated the readability of the information on selected websites using the Simple Measure of Gobbledygook (SMOG) Readability Test and the Reading Grade level of the Suitability Assessment of Materials (SAM) instrument. The SMOG has been validated as very easy to compute and provide a reasonably accurate measure of readability when evaluating consumer-oriented health care materials in many studies. We used the English version of the SMOG measure for evaluating the websites published in English [
Each website was evaluated by the SAM instrument [
When previous studies assessed the suitability of health information across multiple sources, the converted percent was also applied to the present the quality of the health information per SAM criterion [
Then, if the converted percent for a criterion was in the range of 70%-100%, the quality of the information regarding the criterion was considered as “superior” across websites. If the percentage was in the range of 40%-69%, the quality of the information per criterion was considered at an “adequate” level. If the percentage was in the range of 0%-39%, the information regarding a criterion was considered as “not suitable” across the websites.
Three content experts evaluated each website independently in the first round and collected the ratings. In the second round, the three raters reviewed the websites synchronously to ensure the accuracy of understanding of each criterion and, then, shared their rationales for the ratings. The frequencies and intraclass correlation coefficients (ICCs) per the evaluation criteria were obtained using SPSS 24. The final interrater agreement levels for the three evaluation criteria were as follows: NLM, ICC=.969-.987; SMOG, ICC=.810; and SAM, ICC=.626-.994.
Out of the 42 appraised websites, 15 stroke websites were in English and 27 were in Korean. Of all, 12 websites were those of the not-for-profit organizations, 7 were blogs on stroke, 5 were published by clinics or hospitals, 5 were Wikipedia pages, 4 were medical magazine sites, 3 were general magazine sites, 2 were postings on broadcasting websites, 1 was a pharmacological company website, 1 was an insurance company website; 1 was a general online forum site, and 1 was a medical information website.
Overall, 62% (26/42) websites received a rating of >3.0, which presents “adequate” level. Regarding the currency or timeline, 79% (33/42) of the selected websites indicated published date of the information on their websites, whereas 21% (9/42) did not indicate a published date of the information. The updated dates of the information on 31% (13/42) websites were <1 year old, while on 48% (20/42) websites these were older than 1 year. In terms of authority, 21% (9/42) websites did not present the publisher or contact information. Regarding accuracy, 33% (14/42) websites posted health information based on medical research evidence along with citations, whereas 50% (21/42) of the websites did not cite any source of information (
A total of 19 of 42 (45%) websites were presented at the 9th-grade level or above (0=not suitable). Twenty-one (50%) websites were presented at reading levels between 6th and 8th grade (1=adequate). Only 2 websites (5%) were presented at the 5th grade- or lower level (2=superior).
Under the Content category, the quality of purpose and scope was superior, with scores of 90.5% and 79.8%, respectively. Regarding the extent of the content topics, 26% (11/42) websites aimed at desirable behavior rather than at nonbehavioral facts. In addition, 60% (25/42) websites showed <40% of the content topics focusing on desirable behaviors or actions. However, 6 websites did not present such contents. Many of the appraised websites (23/42, 55%) did not present summaries or reviews well to convey key messages.
Regarding the Literacy Demands, the selected websites were suitable in writing style (81.0%), sentence construction (77.4%), vocabulary use (72.6%), and organization using road signs (82.1%). However, the reading grade level was not suitable across websites (29.8%).
Trustworthiness of stroke websites by the National Library of Medicine criteria.
Evaluation criteriaa | Frequency, |
ICCb | ||
.969 | ||||
Superior (2) | The published date of the information is indicated and is less than 1 year old (last year) | 13 (31) | ||
Adequate (1) | The published date of information is indicated, but is older than 1 year | 20 (48) | ||
Not suitable (0) | No indication | 9 (21) | ||
.980 | ||||
Superior (2) | The publisher’s information (individuals or organizations) and contact information can be easily found | 28 (67) | ||
Adequate (1) | The publisher’s information (individuals or organizations) can be found. But there is no contact information | 5 (12) | ||
Not suitable (0) | No indication | 9 (21) | ||
.987 | ||||
Superior (2) | The information is drawn based on sound medical research, and the information sources are cited | 14 (33) | ||
Adequate (1) | The information sources are cited but not based on medical research | 7 (17) | ||
Not suitable (0) | No indication | 21 (50) |
aScore shown in parentheses.
bICC: intraclass correlation coefficient.
The evaluation of the stroke websites based on Suitability Assessment of Materials.
Category and criteria | Frequency, n (%) | ICCa | Converted %b | |||
Not suitable (0) | Adequate (1) | Superior (2) | ||||
Purpose | 0 (0) | 8 (19) | 34 (81) | .626 | 90.5 | |
Scope | 2 (5) | 13 (31) | 27 (64) | .896 | 79.8 | |
Content topics | 6 (14) | 25 (60) | 11 (26) | .875 | 56.0 | |
Summary and review | 23 (55) | 8 (19) | 11 (26) | .993 | 35.7c | |
Average | 65.5 | |||||
Reading grade level | 19 (45) | 21 (50) | 2 (5) | .810 | 29.8c | |
Writing Style | 1 (2) | 14 (33) | 27 (64) | .859 | 81.0 | |
Sentence construction | 1 (2) | 17 (41) | 24 (57) | .727 | 77.4 | |
Vocabulary | 2 (5) | 19 (45) | 21 (50) | .746 | 72.6 | |
Learning enhancement by advance organizers | 6 (14) | 3 (7) | 33 (79) | .962 | 82.1 | |
Average | 68.6 | |||||
Cover graphic | 20 (48) | 13 (31) | 9 (21) | .950 | 36.9c | |
Type of illustrations | 14 (33) | 15 (36) | 13 (31) | .950 | 48.8 | |
Relevance of illustrations | 15 (36) | 11 (26) | 16 (38) | .975 | 51.2 | |
Graphical direction: lists, tables, charts, and forms | 25 (60) | 10 (24) | 7 (17) | .956 | 28.6c | |
Captions are used to announce or explain graphics | 14 (33) | 17 (41) | 11 (26) | .976 | 46.4 | |
Average | 42.4 | |||||
Typography | 1 (2) | 9 (21) | 32 (76) | .863 | 86.9 | |
Layout | 1 (2) | 12 (29) | 29 (69) | .813 | 83.3 | |
Subheadings and chunking | 4 (10) | 4 (10) | 34 (81) | .965 | 85.7 | |
Average | 85.3 | |||||
Interaction included in the text or graphics | 16 (38) | 15 (36) | 11 (26) | .967 | 44.0 | |
Desired behavior patterns are modeled | 14 (33) | 12 (29) | 16 (38) | .975 | 52.4 | |
Motivation | 16 (38) | 14 (33) | 12 (29) | .965 | 45.2 | |
Average | 47.2 | |||||
Cultural match: logic, language, experience | 24 (57) | 8 (19) | 10 (24) | .992 | 33.3c | |
Cultural image and examples | 29 (69) | 10 (24) | 3 (7) | .987 | 19.0c | |
Multiple languages translation | 36 (86) | 0 (0) | 6 (14) | .994 | 14.3c | |
Average | 22.2c | |||||
Overall average | 55.7 |
aICC: intraclass correlation coefficient.
bConverted Percent=([n of websites×0 point]+[n of websites×1 point]+[n of websites×2 points])/total possible best score (84=42 websites×2 points)×100. Interpretation: 70%-100%=superior; 40%-69%=adequate; 0%-39% =not suitable.
cInformation rated as “not suitable” per criterion across the evaluated websites.
The overall suitability of the Graphic Illustration, Lists, Tables, and Charts category was rated as “adequate” (42.4%). In this category, the type of illustrations (48.8%), relevance of illustrations (51.2%), and captions (46.4%) were “adequate” at the lower levels, but cover graphic (36.9%) and graphical direction (28.6%) were “not suitable.” In more detail, 48% (20/42) of the websites did not present cover graphic to attract attention and to clearly portray the purpose of the website.
In addition, 60% (25/42) of the websites presented graphics without explanation and 24% (10/42) showed too brief “how to” directions with graphics for readers; only 17% (7/42) websites provided step-by-step directions with an example that will build self-efficacy (confidence). The layouts and typography were superiorly presented in most appraised websites, with the converted suitability scores of 83.3%-86.9%.
In the Learning Stimulation and Motivation category, the suitability of each criterion was rated as “adequate,” even though the suitability percent scores were not high, ranging 44.0%-52.4%. In more details, 38% (16/42) websites did not provide interaction learning or stimulation and 36% (15/42) used passive interactions with the Questions & Answer format. Regarding the desired behavior pattern presentation, 38% (16/42) websites demonstrated instructions for specific behavior and skills by using specific, familiar instances with the rating of “superior.” In addition, 29% (12/42) websites were rated as “adequate” as they provided the information in a mix of technical and common language that the reader may not easily link with daily living activities. In terms of Motivation, only 29% (12/42) websites reached the “superior” level, which means that complex topics are subdivided so that readers may experience small successes in understanding or problem solving, leading to self-efficacy (confidence). Furthermore, 38% (16/42) websites did not have features of motivation.
The Cultural Appropriateness across websites was “not suitable”; the converted suitability percent scores ranged from 14.3% to 33.3%. Regarding cultural match, 57% (24/42) websites did not present the information in a culturally similar logic, language, and experience of the target population culture. The cultural image and examples were rarely shown in 69% (29/42) websites. Moreover, 86% (36/42) of the websites were presented in only one language and could not be translated into other languages.
This study is the first to provide a systematic evaluation of the stroke-related websites considering Korean Americans. The findings of this study present valuable information about the trustworthiness, readability, and suitability of stroke-related English and Korean websites, which were trustworthy and marginally suitable, but not easily readable. In addition, areas that need improvement were identified in each criterion. This information can be utilized by clinicians and researchers in improving or designing stroke websites as well as by people who seek stroke information in their self-assessment of the quality of the information on websites.
Patients and caregivers look up the websites as they are easy to access, not expensive, and, in general, believed to be timely [
In this study, the information on 20 websites was published more than 1 year ago and 9 websites did not indicate a published date of the information. Notably, half of the websites did not cite any source of information on their websites. Agarwal et al [
Reading level is an essential component of health literacy and is included as one of the Literacy Demand criteria in the SAM tool. Concerned about health literacy, the Joint Commission [
The health information is only useful if the consumer can comprehend the presented information. The consequences of not understanding health information can negatively affect both a person’s health and his or her utilization of health care services. The use of consumer health vocabulary may help lower the reading grade level on both English and Korean stroke websites. In the United States, the Open-Access, Collaborative Consumer Health Vocabulary provides 156,826 consumer-friendly health phrases and synonyms for professional clinical terms [
The overall suitability of all 42 websites was rated as “adequate” (55.7%); however, there are needs for improvement in many areas. Regarding the Content, the majority of the websites explicitly showed the purpose and the scope of the content. However, they did not present summaries or reviews well to convey key messages. In order to ensure the delivery of important messages and to assist the readers’ comprehension, it is necessary to retell key messages in different words or examples and provide their summaries.
The overall suitability of Graphic Illustration was marginally adequate (42.4%). Half of the websites did not present cover graphic to attract attention and clearly portray the purpose of the websites. In addition, 60% (25/42) of the websites presented graphics without explanation and 24% (10/42) showed too brief “how to” directions for readers. This finding is similar to the lack of graphics and other nontextual media covering patient education materials in Agarwal et al’s [
The Learning Stimulation and Motivation category also received the rating of “adequate,” but it was not high (47.2%). No interaction learning or stimulation was provided on 38% (16/42) websites. Regarding the desired behavior patterns presentation, the criterion of the “superior” rating probed if websites demonstrated instructions for specific behavior and skills by using specific, familiar instances; more than half of the websites failed to reach this level. Only 29% (12/42) websites ranked “superior” in the Motivation criterion. Mere access to Web-based health information does not necessarily empower consumers and patients. Providing actionable health information with accuracy is the top vision in the National Action Plan to Improve Health Literacy published by the US Department of Health and Human Services [
Of all, 86% (36/42) websites were presented in only one language and could not be translated into other languages. More than half of the websites in English or Korean did not present information matched to the Korean culture in texts, images, or examples. The cultural factors influence health information acquisition and access to social support for ethnic minorities [
We evaluated the websites that appeared on the first and second pages only on Google and Yahoo browsers. Thus, the results cannot be generalized to reflect the features of all stroke-related websites. However, these websites can be the most popular or important ones because studies have found that 73%-95% of Web users never view past the first search engine results page and have emphasized the value of the first page [
The quality of health information for patients with stroke is vital to ensure the good recovery and to improve the quality of life. Easily obtainable, user-friendly, accurate, and reliable online resources could help people make appropriate decisions about how best to maintain or improve their health condition.
We found that the existing stroke-related websites in either English or Korean are trustworthy and suitable. However, it was notable that the accuracy of information on these websites was low due to lack of citations and references. Providing accurate, reliable, high-quality, and evidence-based information is a critical responsibility of health professionals. In addition, the study findings signified the need for diverse features to reduce the high reading level of the information on the websites. Graphical examples, multimedia, and interactive features can reduce the reading burden of stroke patients and caregivers, as well as build more confidence or self-efficacy when applying the information for condition management and rehabilitation in daily living. Rather than posting the information in a nonspecific way, practical guidelines demonstrating specific behavior and skills applicable to daily living will motivate the reader to experience small success in stroke self-management, leading to self-efficacy.
These strategies to enhance readers’ understanding of Web-based information are especially important to Korean Americans, who inevitably seek online information due to lack of familiarity with the US health care system, inadequate health insurance coverage, and language barriers. Culturally sensitive, high-quality health information relevant to this group through websites may have great potential to increase the knowledge of stroke self-management and make significant contributions to promote the health of Korean American stroke survivors. In the future, the evaluation of stroke websites by Korean American laypersons using the evaluation tools used in this study might be an educational health literacy intervention to learn how to evaluate and consider the design and content aspects in the websites. The examination of how this population navigates stroke websites and apply the health information obtained from these websites for their daily health behaviors should be considered for future studies.
cardiovascular accident
intraclass correlation coefficient
National Library of Medicine
Suitability Assessment of Materials
Simple Measure of Gobbledygook
The processing of this manuscript was supported by Texas Woman’s University Libraries’ Open Access Fund.
None declared.