Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 17.04.15 in Vol 4, No 2 (2015): April-June

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/4090, first published Dec 02, 2014.

This paper is in the following e-collection/theme issue:

    Original Paper

    Web-Based and Telephone Surveys to Assess Public Perception Toward the National Health Insurance in Taiwan: A Comparison of Cost and Results

    Institute of Health Policy and Management, College of Public Health, National Taiwan University, Taipei City, Taiwan

    Corresponding Author:

    Ming-Chin Yang, DrPH

    Institute of Health Policy and Management

    College of Public Health

    National Taiwan University

    Room 637, No. 17. Xu-Zhou Road

    Taipei City,

    Taiwan

    Phone: 886 2 33668067

    Fax:886 2 23434200

    Email:


    ABSTRACT

    Background: Numerous studies have examined the impact of global budget payment systems of health insurance on patient access to medical care. In order to monitor the population’s accessibility to health services, a series of survey are often used to understand public perceptions of the health care provider. Taiwan implemented the single-payer National Health Insurance in 1995 and has been conducting a series of surveys to monitor public perception of the NHI after adopting a global budget payment system in 2002. Although telephone surveys are commonly used in obtaining public opinions on various public health issues, limitations such as higher cost and influence of interviewers do raise some concerns. Web-based surveys, one of the alternative methods, may be free from these problems.

    Objective: Our aim was to examine the difference of sociodemographic characteristics, satisfaction of NHI and NHI-contracted health care providers, attitude toward NHI-related issues, behavior in seeking medical advice and self-reported health status between those who completed Web-based surveys and those reached by telephone.

    Methods: This study compared the demographic factors of participants who took either a Web-based survey (1313 participants) or random digit dialing telephone survey (2411 participants) that contained identical questions.

    Results: Compared to telephone survey respondents, Web-based respondents tended to be younger (P<.001), unmarried (P<.001), non-smokers (P<.001), with a higher education (P<.001), and a higher monthly household income (P<.001) and worse self-reported health status (P=.008); however, they were less likely to report suffering from a chronic disease (P<.001). Despite these differences in background characteristics, no significant differences were observed in their answers related to the seeking of medical care or NHI-related issues. Telephone survey respondents reported greater satisfaction with NHI services. Web-based surveys were also shown to provide a lower average cost per sample (US$0.71) compared to telephone surveys (US$3.98).

    Conclusions: Web-based surveys provide a low-cost alternative method for the polling of public attitudes toward NHI-related issues. Despite general similarities between the two polling methods with regard to responses, respondents to telephone surveys reported a stronger agreement with regard to satisfaction with NHI services and a more positive self-reported health status.

    Interact J Med Res 2015;4(2):e9

    doi:10.2196/ijmr.4090

    KEYWORDS



    Introduction

    Background

    Health care systems often use the reform of payment system to create incentives for health care providers to provide only necessary service within a limited budget which could have influence on population’s accessibility to health care. Numerous studies have examined the impact of global budget system on access to health care. Iglehart et al found that after introducing a global budget, changed allocation of local physicians affected accessibility to medical care [1]. Chu et al also indicated that active suppression of medical expenditures curtailed access to medical care due to bed closures [2]. Many countries use surveys as a monitoring measure aimed at ensuring equity in access to medical care. The Agency for Healthcare Research and Quality (AHRQ) in the United States developed the Consumer Assessment of Healthcare Provider and Systems (CHAPS) with the aim of measuring, reporting on, and improving the quality of health care from the perspective of consumers and patients [3]. Similarly, the Department of Health in the United Kingdom (UK) has been conducting surveys regarding public perceptions of the National Health Service (NHS) and the GP Patient Survey [4,5].

    Taiwan implemented a National Health Insurance (NHI) system in 1995 and implemented a global budget payment system in 2002 to contain skyrocketing medical expenditures. The NHI system is a single-payer system which covers more than 99% of residents and contracts with more than 90% of the hospitals and clinics. Like other countries, Taiwan’s National Health Insurance Administration (NHIA) has been conducting a series of surveys on public perception toward the NHI since 2002. The aim of this series of surveys is to understand public perception regarding NHI and health care services and to track how perceptions have changed over time. Questions in this series of surveys comprise core and optional items. Core items are included in every iteration of the annual survey, while optional items are designed specifically for current topics of interest. Every year, telephone survey data are collected using a computer-assisted telephone interviewing (CATI) system, in which samples are selected using a random digit dialing (RDD) system.

    A number of problems have been identified in these telephone surveys [6]. Evidence has shown that the gender and race of the interviewer can influence the results [7,8]. The attitude and demeanor of the interviewers may also bias the responses [9,10]. Despite the high response rates, telephone surveys are becoming increasingly expensive, which may be due to the time required to complete surveys using these methods [11]. One alternative to telephone surveys is to use the World Wide Web. Web-based surveys are more likely to be free from interviewer bias due to the nature of being self-administered. They also create a data input file directly. Information obtained from Web-based surveys displays higher test-retest reliability than do telephone surveys [12]. Previous studies have suggested that respondents to Web-based surveys are more likely to describe their personal habits such as drinking [13,14], as well as their concerns regarding traffic safety [15]. This approach also raises concern about the comparability of information obtained using Web-based surveys with that obtained by telephone surveys. Telephone surveys generally obtain a representative national sample; however, Web-based surveys are restricted to respondents who have Internet access.

    Study Aims

    To the best of our knowledge, no previous study has examined the differences between Web-based surveys and telephone surveys regarding perceptions of NHI or NHI-contracted health care providers and self-reported health status. Our aim was to examine the difference of sociodemographic characteristics, satisfaction of NHI and NHI-contracted health care providers, attitude toward NHI-related issues, behavior in seeking medical advice and self-reported health status between those who completed Web-based surveys and those reached by telephone.


    Methods

    Development of Questionnaires

    To obtain the data necessary to answer the study questions, we formulated a cross-sectional data collection framework using the telephone and internet as communication media. Both survey methods were conducted in August-September 2013. All people aged 20 or older in any region of Taiwan were eligible to take the survey. Screening was conducted by asking respondents whether they were 20 years of age.

    The conceptual framework of questionnaires incorporated seven dimensions: satisfaction, health status, use of health service, personal health practice, need for health care, enabling resources, and predisposing characteristics. The draft version of the questionnaire was first reviewed by five experts to enhance content validity. Based on their suggestions, we modified the wording and items on the questionnaires. Then, a pre-test with small sample size was executed to examine the internal reliability of questionnaires. The Cronbach alpha coefficients of these seven dimensions were between .5 and .8. The final questionnaires included 43 items aimed at measuring satisfaction with the NHI and NHI-contracted health care providers, attitudes toward NHI-related issues, use of health service, health status (SF-8 health survey), health behaviors (eg, smoking, drinking, and chewing betel nuts), sociodemographic characteristics, and medical history.

    The contents of the Web- and telephone-based questionnaires were identical in terms of the questions asked, the wording, and the order of presentation. The study protocol was approved by the ethics committee at National Taiwan University Hospital (IRB number: 1000401).

    Telephone Interview Survey

    Individuals aged 20 years and older were eligible for interviews. Stratified random sampling procedure was used to draw respondents for telephone interviews. We contracted a company with experience in conducting telephone interviews; they used the RDD method with CATI to carry out the interview. All interviews were conducted from a central location, and the interviewing process was monitored by supervisors to control quality. All interviewers were well-trained individuals working on a regular basis for the survey company. The household telephone number list was obtained from Chunghwa Telecom, a major provider in Taiwan. To improve the coverage rate and increase the chances of contacting of households that were not registered in the phone book, we randomly replaced the last two digits of the drawn telephone numbers with two different digits to form a new telephone number. If there were missing items, the telephone interviewers would connect participants again to complete the questionnaires. At the end of the survey, the total number of usable responses was 2411 with a sampling error of 2.0%.

    Web-Based Survey

    The sampling lists used in the Web-based and telephone surveys were obtained from different suppliers to avoid overlap. The panel list used in the Web-based survey included 320,000 individuals who were also matched to the defined target population. The respondents were randomly drawn from panel members aged 20 and older and invited via email. In addition, interested participants could click on a hyperlink to access information related to the study. The participants were required to finish all the items on the questionnaires, and then they could join the lottery. Finally, we obtained 1313 usable responses with sample error of 2.7% in the Web-based survey.

    Statistical Analysis

    Data were analyzed using SAS version 9.3 for Windows (SAS Institute). Descriptive statistics were used to describe the difference between respondents in the telephone-based and Web-based groups. An independent sample t test and Pearson chi-square test were used to test whether the differences were statistically significant. A P value of less than .05 was considered statistically significant. Multivariate logistic regression and mixed linear regression models were used to test the differences in each dimension of the questionnaire by reporting their odds ratios, coefficients, and standard deviations.


    Results

    The demographic and clinical characteristics of all 3724 respondents are presented in Table 1. The respondents who filled out the Web-based questionnaire were generally younger (P=.0001), unmarried (P<.001), nonsmokers (P<.001) with a higher education level (P<.001) and higher monthly household income (P<.001), were less likely to suffer from chronic disease (P<.001), and were more likely to be living in the Taipei area (P<.001) when compared to respondents of the telephone survey.

    Table 1. Characteristics of respondents who participated in the Web-based and telephone surveys (n=3724).
    View this table

    Table 2 presents the satisfaction felt by participants toward NHI and NHI-contracted health care providers, as well as behavior patterns in use of health service. Telephone survey respondents were more likely than Web-based respondents to report a feeling of satisfaction with the NHI (58.4% vs 44.7%; adjusted odds ratio [AOR] 1.97, 95% CI 1.62-2.40) and NHI-contracted health care providers (63.4% vs 48.4%; AOR 2.39, 95% CI 1.96-2.92). No significant differences were observed between telephone and Web-based survey respondents with regard to their behavior in use of health service.

    Table 2. Comparison of Web-based and telephone survey respondents with regard to satisfaction with NHI and patterns in use of health service.
    View this table

    Table 3 illustrates the attitudes of respondents toward NHI-related issues. Results of logistic regression analysis indicate that, after accounting for sociodemographic factors, comorbidity history and self-reported health status, Web-based respondents were more likely than telephone survey respondents to be aware of the details of the NHI policy on the programs specifically designed to assist the disadvantaged (41.1% vs 25.3%; AOR 1.86, 95% CI 1.52-2.27).

    Table 3. Comparison of respondents to Web-based and telephone surveys regarding the degree to which they were concerned with NHI-related issues.
    View this table

    Table 4 presents the self-reported health statuses of respondents of the two surveys. Web-based respondents generally had poorer health status with regard to physical functioning (adjusted difference: −1.73; P<.001), role-physical (adjusted difference: −2.25; P<.001), body pain (adjusted difference: −1.94; P<.001), social functioning (adjusted difference: −3.89; P<.001), role-emotional (adjusted difference: −4.06; P<.001) and mental health (adjusted difference: −2.94; P<.001). However, Web-based respondents reported a higher degree of vitality (adjusted difference: 0.94; P=.015) than did the telephone survey respondents.

    Table 4. Self-reported health status of respondents to Web-based and telephone surveys.
    View this table

    Discussion

    Principal Findings

    This study sought to determine whether a Web-based survey could be used as an alternative to RDD telephone surveys. Our results indicate substantial demographic and socioeconomic differences between the individuals who participated in the two surveys. Our findings are consistent with previous studies, in which Web-based survey respondents are generally younger, healthier, and more highly educated than respondents to telephone surveys [16,17]. This was not unexpected, due to the fact that elderly persons still lag behind other age groups with regard to internet usage [18]. The bias did not disappear after weighting according to sociodemographic factors. This raises concerns about the accuracy of Web-based surveys and how representative the information gained is of a population.

    Comparison to Prior Work

    This is the first study to compare Web-based and RDD telephone surveys with regard to different dimensions. Previous studies have demonstrated that the answers provided by respondents to Web-based surveys often differ from those provided by respondents to telephone surveys [15,19]; however, other studies showed no significant differences. Zuidgeest et al investigated differences between postal surveys and mixed-model surveys (Internet questionnaires with paper follow-up). Their findings failed to indicate any differences in global ratings of health care between respondents to the two surveys [20]. Braunsberger et al compared telephone and Web-based surveys with regard to test-retest reliability in which participants filled out two-wave questionnaires. Their results indicate that the reliability of data from Web panels exceeds that obtained from telephone survey panels [21]. Our results show that despite fundamental differences between those who took the two surveys, behaviors with regard to use of health service and attitudes toward NHI-related issues were similar, except for the awareness of assisting the disadvantaged policy in NHI. This implies that the two data collection modes are able to obtain unbiased data related to these issues. However, if one would like to assess the public perceptions of health care system and population health status, the results from different survey modes could not be compared directly.

    Previous studies showed that many of the satisfaction polls done in other countries focused more on assessing consumer and patient satisfaction with the use of health services but rarely assessed the public perception on the health care system. In terms of survey modes, most of the satisfaction surveys collected data via telephone, mail, face-to-face, or mixed methods. Recently, Web-based surveys with advantages of lower cost and more time saving have been broadly used in many fields. In the UK, for example, the GP patient survey used three different methods to collect data: paper, telephone, and Web. Their results indicated that the proportion of patients using Web-based survey was increasing.

    Compared to previously mentioned polls done in other countries, this study demonstrated the feasibility of combining satisfaction toward use of health service with awareness and attitudes toward selected policies. Furthermore, we used both telephone and Web-based surveys to collect responses to the same questionnaire and examined whether the responses were comparable. The results indicate that, after adjusting for respondents’ socioeconomic characteristics, use of health service, personal health practice, and health status, the satisfaction toward various issues was significantly different between respondents from telephone surveys and Web-based surveys. Moreover, results of this study also showed that respondents to the telephone survey were more satisfied with NHI and NHI-contracted health care providers than those who took the Web-based survey. Previous studies have indicated that Web-based respondents tend to give more neutral or negative attitudinal evaluations [22]. Evidence also indicates that the techniques, gender, and race of the interviewer can bias the results [7,8]. Thus, it may be reasonable to expect that individuals engaged in a direct conversation with a telephone interviewer may show more positive attitude toward the issues discussed in the survey. Our findings also indicate that Web-based survey respondents are more aware of NHI policies. This is not unexpected considering the higher level of education held by these individuals and their greater access to information via the Internet.

    In terms of self-reported health status, Rivara et al investigated the model effect of health-related quality of life among parents and children aged birth to 17 years using Web-based versus telephone surveys [23]. They found that the differences caused by model effect were small but statistically significant in some of the measures. Our findings are consistent with these studies, indicating that the self-reported health status of respondents to telephone surveys was significant better than those responding to Web-based surveys, after controlling for demographic and socioeconomic characteristics, health behaviors, and comorbidity history. When matched by demographic and socioeconomic characteristics and health behaviors, the differences between the two groups remained significant.

    Previous studies have indicated that the cost of and time required to complete Web-based surveys (measured per respondent) are far lower than that of other methods [20,21,24]. In this study, the data obtained from the two survey methods was collected by the same commercial research firm. The overall cost of the telephone survey included the cost of acquiring and compiling the survey data followed by preparation of a database for analysis as well as overhead and profit. The overall cost of the Web-based survey included the cost of contacting panel members, directing them to the survey site, creating a platform on which to conduct the survey as well as an allocation for overhead and profit. The average cost for each usable telephone survey sample was US$3.98, whereas the average cost of obtaining each usable Web-based survey sample was US$0.71, 17.8% of the cost of the telephone survey samples.

    Nonetheless, several issues should be considered when interpreting these results. Any survey of perception and behavior is subject to various forms of self-selection and self-reported bias. As discussed earlier, the RDD telephone and Web-based surveys used in this study were not immune to these limitations. We attempted to minimize the effects of these issues by ensuring informed consent and standardized data collection procedures. Despite the fact that we employed an RDD contact method for the telephone survey, the contacted individuals were still able to choose whether to participate in the interview. Similarly, the Web-based survey, which required respondents to fill out the survey online, is prone to the issue of self-selection. Further research should address this issue by using a random assignment process to determine if the same population who agrees to be interviewed would choose a different format for the interview process.

    Conclusions

    Web-based surveys offer cost savings and efficiency in obtaining data, especially on awareness and attitudes toward NHI-related issues. But when assessing the public’s satisfaction with the health care system and population health status, survey results could not be compared directly because different methods of survey could yield different outcomes.

    Acknowledgments

    This study was supported by the National Health Insurance Administration in Taiwan. The authors wish to thank all the experts who reviewed the questionnaires. We also thank the reviewers and editor of this journal for providing constructive suggestions.

    Conflicts of Interest

    None declared.

    References

    1. Iglehart JK. Germany's health care system (1). N Engl J Med 1991 Feb 14;324(7):503-508. [CrossRef] [Medline]
    2. Chu DK. Global budgeting of hospitals in Hong Kong. Soc Sci Med 1992 Oct;35(7):857-868. [Medline]
    3. Consumer Assessment of Healthcare Providers and Systems (CAHPS). Agency for Healthcare Research and Quality. 2013.   URL: https://cahps.ahrq.gov/ [accessed 2015-04-01] [WebCite Cache]
    4. Ipsos MORI Social Research Institute. Public perceptions of the NHS and social care. In: An ongoing tracking study conducted for the Department of Health, December 2011 Wave. United Kingdom: Department of Health; Jun 8, 2012.
    5. Department of Health, United Kingdom. 2013. The GP Patient Survey 2013   URL: https://gp-patient.co.uk/surveys-and-reports [accessed 2014-12-01] [WebCite Cache]
    6. Link MW, Oldendick RW. Call screening: Is it really a problem for survey research? Public Opin 1999;63:577-589. [CrossRef]
    7. Johnson TP, Fendrich M, Shaligram C. An evaluation of the effects of interviewer characteristics in an RDD telephone survey of drug use. J Drug Issues 2000;30:77-101 [FREE Full text]
    8. Lueptow LB, Moser SL, Pendleton BF. Gender and response effects in telephone interviews about gender characteristics. Sex Roles 1990 Jan;22(1-2):29-42. [CrossRef]
    9. Beatty P. Understanding the standardized/non-standardized interviewing controversy. Journal of Official Statistics 1995;11:147-160 [FREE Full text]
    10. Keeter S, Miller C, Kohut A. Consequence of reducing nonresponse in a national telephone survey. Public Opin 2000;64:125-148 [FREE Full text] [CrossRef]
    11. Fahimi M, Link M, Mokdad A, Schwartz DA, Levy P. Tracking chronic disease and risk behavior prevalence as survey participation declines: statistics from the behavioral risk factor surveillance system and other national surveys. Prev Chronic Dis 2008 Jul;5(3):A80 [FREE Full text] [Medline]
    12. Rankin KM, Rauscher GH, McCarthy B, Erdal S, Lada P, Il'yasova D, et al. Comparing the reliability of responses to telephone-administered versus self-administered Web-based surveys in a case-control study of adult malignant brain cancer. Cancer Epidemiol Biomarkers Prev 2008 Oct;17(10):2639-2646 [FREE Full text] [CrossRef] [Medline]
    13. Link MW, Mokdad AH. Alternative modes for health surveillance surveys: an experiment with web, mail, and telephone. Epidemiology 2005 Sep;16(5):701-704. [Medline]
    14. Link MW, Mokdad AH. Effects of survey mode on self-reports of adult alcohol consumption: a comparison of mail, web and telephone approaches. J Stud Alcohol 2005 Mar;66(2):239-245. [Medline]
    15. Beck KH, Yan AF, Wang MQ. A comparison of web-based and telephone surveys for assessing traffic safety concerns, beliefs, and behaviors. J Safety Res 2009 Oct;40(5):377-381. [CrossRef] [Medline]
    16. Klovning A, Sandvik H, Hunskaar S. Web-based survey attracted age-biased sample with more severe illness than paper-based survey. J Clin Epidemiol 2009 Oct;62(10):1068-1074. [CrossRef] [Medline]
    17. Wade AG, Crawford GM, Pumford N, Koscielny V, Maycock S, McConnachie A. Baseline characteristics and patient reported outcome data of patients prescribed etanercept: web-based and telephone evaluation. BMC Med Res Methodol 2011;11:91 [FREE Full text] [CrossRef] [Medline]
    18. Research, Development and Evaluation Commission, Executive Yuan. Digital opportunity survey: households and individuals. In: Digital opportunity survey Households and Individuals. Taipei, Taiwan: National Development Council; 2011.
    19. Christian LM, Dillman DA, Smyth JD. The Effects of mode and format on answers to scalar questions in telephone and web surveys. In: Advances in Telephone Survey Methodology. Hoboken, NJ, USA: John Wiley & Sons, Inc; 2007:250-275.
    20. Zuidgeest M, Hendriks M, Koopman L, Spreeuwenberg P, Rademakers J. A comparison of a postal survey and mixed-mode survey using a questionnaire on patients' experiences with breast care. J Med Internet Res 2011;13(3):e68 [FREE Full text] [CrossRef] [Medline]
    21. Braunsberger K, Wybenga H, Gates R. A comparison of reliability between telephone and web-based surveys. J Bus Res 2007 Jul;60(7):758-764. [CrossRef]
    22. Kadner RJ, Bassford PJ. Relation of cell growth and colicin tolerance to vitamin B12 uptake in Escherichia coli. J Bacteriol 1977 Jan;129(1):254-264 [FREE Full text] [Medline]
    23. Rivara FP, Koepsell TD, Wang J, Durbin D, Jaffe KM, Vavilala M, et al. Comparison of telephone with World Wide Web-based responses by parents and teens to a follow-up survey after injury. Health Serv Res 2011 Jun;46(3):964-981 [FREE Full text] [CrossRef] [Medline]
    24. Cascade E, Marr P, Winslow M, Burgess A, Nixon M. Conducting research on the Internet: medical record data integration with patient-reported outcomes. J Med Internet Res 2012;14(5):e137 [FREE Full text] [CrossRef] [Medline]


    Abbreviations

    AOR: adjusted odds ratio
    AHRQ: Agency for Healthcare Research and Quality
    CATI: computer-assisted telephone interviewing
    CHAPS: Consumer Assessment of Healthcare Provider and Systems
    NHI: National Health Insurance
    NHIA: National Health Insurance Administration
    NHS: National Health System
    RDD: random digit dialing
    UK: United Kingdom


    Edited by G Eysenbach; submitted 02.12.14; peer-reviewed by YH Kao Yang; comments to author 29.12.14; revised version received 28.02.15; accepted 04.03.15; published 17.04.15

    ©Ming-Chin Yang, Elise Chia-Hui Tan. Originally published in the Interactive Journal of Medical Research (http://www.i-jmr.org/), 17.04.2015.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.