This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Interactive Journal of Medical Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.i-jmr.org/, as well as this copyright and license information must be included.
Engagement with digital health interventions (DHIs) may be regarded as a prerequisite for the intervention to achieve positive health or behavior change outcomes. One method employed to promote engagement is the use of prompts such as emails and text messages. However, little is known about the characteristics of prompts that promote engagement. This study explored the association between the content and delivery mode of prompts and the users’ engagement with HeLP-Diabetes (Healthy Living for People with type 2 Diabetes), a DHI that aimed to promote self-management in adults with type 2 diabetes.
The objective of this study was to identify the characteristics of prompts, specifically the content and delivery mode, which were associated with increased engagement.
This was a mixed-methods study. Email and text message prompts were sent to the registered users of HeLP-Diabetes. Use of the intervention was recorded and examined to identify which email and text message prompts were associated with subsequent visits to the DHI. Characteristics of prompts that were identified as particularly effective or ineffective were explored through think-aloud interviews with the participants.
Of a total of 39 email prompts, 49% (19/39) prompts showed a significant association with subsequent visits to the DHI. However, none of the text message prompts were associated with subsequent visits to the DHI. Furthermore, think-aloud interviews were carried out with 6 experienced participants with type 2 diabetes. The findings suggest that these participants preferred email prompts that were clear, relatively short, and empowering; used nondirective advice; included health professional references; were visually appealing; and contained news and updates.
The findings of this study contribute to the existing evidence supporting the role of email prompts in promoting and maintaining engagement with DHIs. This study described the content of prompts that may be engaging. However, the results should be interpreted with caution, as prompts may be context-specific interventions and the results may not be generalizable across other DHIs or other types of interventions targeting self-management of type 2 diabetes.
Digital health interventions (DHIs) are programs that provide emotional, decision, or behavior information and/or support for physical or mental health problems via digital platforms such as computers, mobile phones, or websites [
It has been suggested that engagement, defined here as users’ regular interaction with part or all of the DHI [
One suggested method of promoting engagement is the use of technology-based prompts such as emails and text messages [
There is relatively little work exploring whether the features of content and delivery are associated with enhancing engagement, and if so, identifying which features promote engagement. One study suggested that prompts with new content (eg, updated content on the DHI) were potentially associated with enhanced engagement [
HeLP-Diabetes (Healthy Living for People with type 2 Diabetes) is a DHI targeting self-management of type 2 diabetes developed by a research team from University College London (UCL). It was developed with a strong theoretical underpinning and following the principles of participatory design, where users were defined as patients with type 2 diabetes and health professionals’ caring for such patients. Focus groups conducted during the development process explored users’ views on engagement and the potential of prompts delivered via emails and text messages to promote engagement. In response to the focus group data [
HeLP-Diabetes was the subject of a research program that included two major studies undertaken in parallel: an individually randomized controlled trial (RCT) in primary care to determine effectiveness and cost-effectiveness [
In the eHealth (electronic health) and mHealth (mobile health) field, there has been a call to accelerate the pace of health research to correlate with the speed of technology development. One suggestion to accelerate eHealth research is to use studies with smaller samples that answer discrete, specific questions, as we aimed to do in this study, rather than conducting one major randomized controlled study [
This was a mixed-methods study. It consisted of two components (see
Flow diagram of study steps.
Objective 1 was to identify prompts that were associated with increased numbers of subsequent visits to HeLP-Diabetes, whereas objective 2 was to identity prompts that appeared to have no association with the number of visits.
Participants included all the users registered on HeLP-Diabetes between March 2013 and May 2015 who had not specifically “unsubscribed” from receiving emails from the HeLP-Diabetes team. Hence, users were people with type 2 diabetes aged 18 years or older who were living in England, the United Kingdom. They had either volunteered to be a part of the RCT evaluating the clinical effectiveness and cost-effectiveness of HeLP-Diabetes or had been offered HeLP-Diabetes as a part of their routine care.
The primary goal of the email and text message prompts was to encourage users to visit the HeLP-Diabetes program; the prompts were not intended to have a direct impact on recipients’ self-care behaviors. Our proposed mechanism of action was that the prompts would bring key areas of the HeLP-Diabetes program to the attention of users and hence encourage visits. The HeLP-Diabetes program was considered the “active ingredient” in terms of promoting self-management. Content for prompts was developed through discussions with a multidisciplinary panel that included patient representatives, general practitioners, diabetes nurses, psychologists, dietitians, and project managers. Prompts were written by the lead author (GA) every month, with content selected to reflect recent diabetes-related news or research, HeLP-Diabetes updates, seasonal events, and other contemporary topics. In general, prompts opened by greeting the users with their username, followed by an introduction to the topics covered in the prompt and links to the relevant parts of HeLP-Diabetes. They ended with the HeLP-Diabetes contact email and the option of unsubscribing from future emails. Due to word count limits, text messages were usually limited to an introduction to the topic of the prompts and the relevant link (see
The initial frequency of prompts was based on the advice from the participatory design panel and revised in line with subsequent feedback from patients’ representatives. Initially, the frequency for the prompts was 1 prompt per week; it was subsequently changed to 3 prompts per month. The prompts were not automated, hence a researcher was assigned to send them when they were finalized as per the abovementioned procedure; this resulted in differences in time periods between prompts. Any new user registering would only receive the prompts sent after their registration date. When users registered on HeLP-Diabetes, they were automatically subscribed to emails and given the option to subscribe to text messages. Email prompts were first sent in November 2013, whereas text message prompts started from October 2014. Recipients could unsubscribe from emails and/or text messages at any time.
Example of email prompt “World Diabetes Day.”.
Usage data (website metrics) refer to parameters that are constructed from digital traces left by users of digital interventions [
“HeLP-Diabetes user visits” referred to the user logging into HeLP-Diabetes in the period between prompts. The HeLP-Diabetes user visits measure was used for both email and text message prompts. This was a binary outcome and not a continuous measure—people either visited or they did not in each period, and multiple logins were not counted as multiple visits. For example, if a user visited HeLP-Diabetes after receiving an email prompt twice before the next prompt was sent, only 1 visit was counted for that user.
The “email opens” measure was used for email prompts only. An email was counted as opened if the user’s email client (eg, Yahoo Mail, Gmail, Outlook) downloaded the images embedded in the email, as this is the industry standard [
Statistical analysis was conducted using the Statistical Package for the Social Sciences (SPSS) version 22. To describe the time over which prompts appeared to be associated with subsequent visits, we analyzed the data to identify the median and interquartile range (IQR) for the number of days between sending an email prompt and subsequent visits. This analysis suggested that any association of the prompts was limited to N days, and all subsequent analyses were limited to the visits that occurred within N days of an email prompt being sent. Thereafter, the association between opening an email prompt and visiting HeLP-Diabetes for each prompt within N days was analyzed with the chi-square test, using an alpha of <.05 to indicate statistical significance. A group of email prompts that showed a significant association between opening a prompt and visiting HeLP-Diabetes had a mixture of low and high number of visits, and those that did not show an association were selected visually to be explored in the think-aloud interviews in stage 2.
Chi-square test was used to find any significant association between sending a text message prompt and visiting HeLP-Diabetes before the next prompt was sent (whether an email or a text message prompt). An alpha of <.05 was used to indicate statistical significance
Objective 3 was to explore the features of specific prompts to understand why they did or did not appear to lead to subsequent visits to HeLP-Diabetes.
The sample was a convenience sample, as participants were 6 patient representatives who had been involved in the development of HeLP-Diabetes and the prompts. They were invited to participate in the think-aloud interviews via email. The interviews took place at a time convenient to the participant and at the UCL eHealth unit. During the interview, participants were introduced to think-aloud interviews, and they were encouraged to say all their thoughts and opinions about each prompt, irrespective of whether their thoughts or opinions were negative or considered by them to be insignificant. A short practice was performed at the beginning of the interview to familiarize the participants with the think-aloud techniques (eg, speaking their first thoughts loudly). They were then asked to choose one of the email prompts that had not shown a significant association between opening it and visiting HeLP-Diabetes in stage 1 of the study to practice on. They viewed the prompts on a computer screen. After that, participants were randomly shown the email prompts that the stage 1 of the study had found to be significantly associated with visiting HeLP-Diabetes. They were asked to vocalize their thoughts and opinions while opening each email and describe what they liked or disliked about the content of each prompt and their first impressions or thoughts. After viewing all the email prompts, they were asked some questions based on what they expressed while viewing the emails, and what other participants had expressed in previous interviews (the interview guide evolved throughout the interviews). The interviewer (GA) took notes while the participants viewed the emails and prompted them when they forgot to vocalize their thoughts aloud. At the end of the interview, GA asked the participants for some basic demographic information, namely their age, sex, highest level of education achieved, how long they had had diabetes for, and how they rated their expertise with computers (basic, intermediate, or advanced), as these characteristics may influence participants’ perceptions of email prompts [
Interviews were recorded and anonymized. They were transcribed verbatim by a professional and discreet transcriber who had signed a confidentiality agreement. Each participant had an identification number to ensure their anonymity (eg, P1). NVivo 10 was used for data management and analysis. An inductive thematic analysis approach was used. This started with familiarization with the transcripts and any notes taken during the interviews, followed by identification of themes that were strongly linked to the data rather than using preconceptions or a preexisting coding frame. An open coding process was applied where the transcripts were coded line-by-line and paragraph-by-paragraph. In addition, an “in vivo” coding (ie, coding using participants’ words) and constant comparative method, where we constantly compared the data across all the interviews by moving back and forth between them, were used. Emerging codes and themes were discussed and presented within the HeLP-Diabetes team meetings to ensure rigor and thoroughness of the analysis and to include expert and multidisciplinary input in the interpretation of findings.
Patients with type 2 diabetes who were involved in the development of HeLP-Diabetes were also involved in the development of the prompts. These patient representatives provided feedback and suggestions on topics, content, and the frequency of prompts. They also participated in the think-aloud interviews.
The results are divided into two sections. The first section presents the results of the analysis of the email and text message prompts, whereas the second section presents the results of the think-aloud interviews with participants exploring the content of email prompts.
Objective 1 was to identify prompts that were associated with increased numbers of subsequent visits to HeLP-Diabetes, whereas objective 2 was to identify prompts that appeared to have no association with the numbers of visits.
Between February 2014 and May 2015, 49 prompts were sent to all registered HeLP-Diabetes users; of these, 42 were email prompts and 7 were text message prompts. The number of users who received the prompts ranged between 69 (for prompts sent early in the study) and 432 (for later prompts). The period between each prompt and the next ranged from 3 to 24 days (see
The number of patients registered to use HeLP-Diabetes increased steadily throughout the study period. Each of the prompt recipients had different percentages of characteristics. For example, of the 411 users with available characteristics information who were sent “HeLP Diabetes Newsletter 20-What can you eat?,” 60% (247/411) were male, 41.3% (170/411) were aged between 41 and 60 years, and 51.3% (211/411) were aged 61 years or older. Furthermore, 18% (74/411) participants had had diabetes for less than a year, 33.8% (139/411) for 1 to 5 years, 19.7% (81/411) for 5 to 10 years, 21.6% (89/411) for 10 to 20 years, and the rest for over 20 years 5.8% (24/411) or not stated 0.9% (4/411). Of the 200 people who answered the question about previous computer experience, 38% (76/200) described it as “basic,” 37% (74/200) as “intermediate,” and 25% (50/200) as “advanced.”
Examining all the user visits (N=918) that were recorded following the sending of each email prompt, from the first one in February 2014 until the last in May 2015, the time taken for users to visit HeLP-Diabetes after an email prompt was sent ranged from the same day to 23 days. The median time taken to visit HeLP-diabetes was 1 day after receiving an email prompt, with an IQR of 0 to 5 days (ie, 25% visited HeLP-diabetes on the same day of receiving an email prompt, and 75% did so up to 5 days of receiving a prompt). The percentage of users who opened or did not open an email and visited HeLP-Diabetes up to 5 days after a prompt was sent is shown in
Data for 3 email prompts (“Keeping your bones healthy,” “HeLP-Diabetes Newsletter 9-Anxiety,” and “HeLP-Diabetes Newsletter 10-Break a sweat this summer!”) were excluded because the time between sending the prompt and the next email prompt was less than 5 days.
The chi-square test identified 19 email prompts (out of the 39 analyzed ones) that showed a statistically significant association (
Email prompt title | Users who visited HeLP-Diabetes |
Users who opened an email prompt |
Users who opened an email prompt and visited HeLP-Diabetes |
Chi-square result |
How are your New Year's resolutions going?b | 1 (1/71) | 38 (27/71) | 0 (0/27) | 0.6 (1, 71), .43 |
HeLP-Diabetes Newsletter 6-Medicationb | 1 (1/69) | 27 (19/69) | 5 (1/19) | 2.7 (1, 69), .10 |
Boosting your health during winterb | 3 (3/79) | 40 (32/79) | 6 (2/32) | 0.9 (1, 79), .35 |
Best diet advice!b | 2 (2/81) | 44 (36/81) | 2 (1/36) | 0.02 (1, 81), .87 |
Share your personal experience with us!b | 2 (2/69) | 36 (25/69) | 8 (2/25) | 3.6 (1, 69), .06 |
How do I lose weight and feel better?b | 0 (0/83) | 38 (32/83) | 0 (0/32) | No user visited HeLP-Diabetes |
Designing your care planb | 0 (0/90) | 38 (35/90) | 0 (0/35) | No user visited HeLP-Diabetes |
HeLP-Diabetes Newsletter 7-Making changesb | 5 (5/98) | 31 (31/98) | 9 (3/31) | 2 (1, 98), .16 |
It's Springtimeb | 3 (3/99) | 39 (39/99) | 5 (2/39) | 1 (1, 99), .32 |
Happy Easter | 2 (2/102) | 37 (38/102) | 5 (2/38) | 3.4 (1, 102), .06 |
Shopping for food | 0 (0/103) | 22 (23/103) | 0 (0/23) | No user visited HeLP-Diabetes |
Achieving your goalsb | .9 (1/106) | 30.2 (32/106) | 3 (1/32) | 2.3 (1, 106), .13 |
HeLP-Diabetes Newsletter 8-Personal experiencesb | 2.8 (3/108) | 30.6 (33/108) | 6 (2/33) | 1.9 (1, 108), .17 |
How many meals do you eat per day?c | 10.7 (12/112) | 41.1 (46/112) | 21 (10/46) | 9.9 (1, 112), <.001 |
What you need to know about hypoglycemia!c | 8.5 (11/130) | 40.8 (53/130) | 18 (10/53) | 12.5 (1, 130), <.001 |
Are you a complementary therapy user?c | 6.6 (9/136) | 31.6 (43/136) | 18 (8/43) | 14.6 (1, 136), <.001 |
Sexual health-let's talk about it!c | 4.8 (7/145) | 27.6 (40/145) | 5 (2/40) | 0.004 (1,145), .95 |
Fasting during Ramadanc | 8.4 (14/167) | 28.7 (48/167) | 16 (8/48) | 6 (1, 167), .01 |
HeLP-Diabetes Newsletter 11-Holiday preparations | 8.8 (16/182) | 35.7 (65/182) | 18 (12/65) | 11.8 (1, 182), .001 |
How to handle the summer heat? | 10.8 (23/213) | 36.6 (78/213) | 23 (18/78) | 19.3 (1, 213), <.001 |
HeLP-Diabetes Newsletter 12-Your diabetes is in your hands | 12.4 (29/233) | 39.5 (92/233) | 27 (25/92) | 30.3 (1, 233), <.001 |
HeLP-Diabetes Newsletter 13-Get rid of your medication worries!c | 5.4 (13/242) | 36 (87/242) | 10 (9/87) | 6.6 (1, 242), .01 |
Smile - You're on Camera! | 6.8 (17/249) | 35.7 (89/249) | 14 (13/89) | 13.2 (1, 249), <.001 |
Autumn health reminderc | 4.5 (12/268) | 35.4 (95/268) | 5 (5/95) | 0.2 (1, 268), .64 |
HeLP-Diabetes Newsletter 14-What's happening this October? | 11.5 (33/286) | 38.8 (111/286) | 21.6 (24/111) | 18.1 (1, 286), <.001 |
World Diabetes Day | 14.6 (45/308) | 43.2 (133/308) | 28.6 (38/133) | 36.6 (1, 308), <.001 |
HeLP-Diabetes Newsletter 15-Shopping done the right way | 10.5 (35/333) | 38.1 (127/333) | 22.8 (29/127) | 33.2 (1, 333), <.001 |
HeLP-Diabetes Newsletter 16-Eye care | 8.9 (30/338) | 42.0 (142/338) | 17.6 (25/142) | 23.1 (1, 338), <.001 |
Happy Holidays | 6.9 (24/346) | 39.9 (138/346) | 15.9 (22/138) | 28.8 (1, 346), <.001 |
New Year Tips | 8.9 (31/348) | 43.1 (150/348) | 18.7 (28/150) | 30.9 (1, 348), <.001 |
HeLP-Diabetes Newsletter 17-Change for 2015 | 8.9 (32/358) | 34.9 (125/358) | 22.4 (28/125) | 42.8 (1, 358), <.001 |
How to manage your diabetes using the HeLP-Diabetes care plan? | 10.6 (40/376) | 42 (158/376) | 20.3 (32/158) | 26.5 (1, 376), <.001 |
HeLP-Diabetes Newsletter 18-Alcohol, love and activity in February | 9.5 (37/390) | 41.8 (163/390) | 20.2 (33/163 ) | 37.7 (1, 390), <.001 |
Get to know HeLP-Diabetes | 6.7 (27/404) | 44.1 (178/404) | 12.9 (23/178) | 19.9 (1, 404), <.001 |
HeLP-Diabetes Newsletter 19-Spring, delicious recipes and dark chocolate | 10.3 (42/407) | 36.9 (150/407) | 22.7 (34/150) | 39.1 (1, 407), <.001 |
What HeLP-Diabetes can do for you…. | 7.5 (31/416) | 38.5 (160/416) | 18.1 (29/160) | 42.9 (1, 416), <.001 |
HeLP-Diabetes Newsletter 20-What can you eat? | 6.3 (26/416) | 38.9 (162/416) | 12.3 (20/162) | 16.8 (1, 416), <.001 |
Making HeLP-Diabetes easier | 3.5 (15/428) | 40.2 (172/428) | 8.1 (14/172) | 18.3 (1, 428), <.001 |
HeLP-Diabetes Newsletter 21-Mindfulness, HeLP-Diabetes and fruit sugar | 6.7 (29/432) | 39.4 (170/432) | 15.3 (26/170) | 33 (1, 432), <.001 |
adf: degrees of freedom.
b2 cells (50.0%) have expected count less than 5.
c1 cell (25.0%) has expected count less than 5.
Percentage of users who opened or did not open an email prompt and visited HeLP-Diabetes up to 5 days after sending one.
There were 7 text message prompts sent between October 2014 and May 2015. None of these 7 text message prompts showed a statistically significant association between subscribing to receive text message prompts and visiting HeLP-Diabetes (see
Objective 3 was to explore features of specific prompts to understand why they did or did not appear to lead to subsequent visits to HeLP-Diabetes.
There were 6 patient representatives who agreed to participate. Five out of the 6 representatives had worked with us previously, and they provided regular feedback about prompts’ frequency, content, and timing from early 2014. The sixth participant started working with us in mid-2015 and only provided feedback on 1 prompt. All the participants were over the age of 50 years. Two of them were males, and 4 participants had degree-level qualifications. Most of the participants rated their computer experience as medium to high. The length of diabetes diagnosis ranged between 5 and 40 years (see
The main findings from the interviews related to participants’ likes and dislikes of the prompts are presented below, accompanied with illustrative quotes.
Participants preferred short and clear email prompt content with an overview at the beginning of an email summarizing the content. Participants suggested including as many links to HeLP-Diabetes pages and section as possible without overwhelming users. One participant liked it when a clear description of what the links included in a prompt would direct the users to what was provided:
So, someone might say, I haven’t got time to watch a video, and they look at their watch, and next the computer says, watch the 3-minute video, so that would be good, because 3 minutes is nothing, isn’t it? I like that.
Some participants liked it when the emails started by greeting them with their usernames, whereas others felt that personalizing emails in such a way should only be done in short emails and that newsletters should not include their usernames. One user felt that personalizing emails did not make a big difference to the content and how engaging it was.
Text message prompt topics | Subscribed users who visited HeLP-Diabetes |
Unsubscribed users who visited HeLP-Diabetes |
Chi-square result |
Flu jab reminder | 5.8 (10/172) | 10.9 (14/129) | 2.6 (1, 301), .11 |
Home exercises | 6.5 (12/185) | 2.1 (3/142) | 3.5 (1, 327), .06 |
Eating and drinking on holidays | 4.0 (8/199) | 5.3 (8/151) | 0.3 (1, 350), .57 |
January blues | 5.3 (10/190) | 4.8 (8/167) | 0.04 (1, 357), .83 |
Sharing problems and advices | 4.8 (10/209) | 5.6 (9/161) | 0.1 (1, 370), .72 |
National Health Service medical exemption certificate | 4.7 (11/234) | 3.5 (6/173) | 0.4 (1, 407), .53 |
Specialist and technical support | 2.8 (7/246) | 2.8 (5/180) | 0.002 (1, 426), .96 |
adf: degrees of freedom.
ID | Sex | Age, in years | Education level | Length of diabetes diagnosis, in years | Computer experience | When user joined the team, number of prompts the user provided feedback on |
P1 | Female | 55 | Degree | 15 | High | From mid-2014, 8 prompts |
P2 | Male | 50 | Postgraduate | 5 | High | From mid-2015, 1 prompt |
P3 | Female | 60 | Degree | 20 | Medium-high | From early 2014, 10 prompts |
P4 | Male | 58 | Grammar school | 12 | Medium | From early 2014, 9 prompts |
P5 | Female | 68 | Postgraduate | 40 | Medium-high | From early 2014, 12 prompts |
P6 | Female | 69 | A level | 10 | Medium | From early 2014, 12 prompts |
Popular content included the use of empowering statements in the titles or in the body of the email such as in the email prompt “HeLP-Diabetes Newsletter 12-Your diabetes is in your hands,” statements referencing health professionals or linking to their recommendations, email titles written as questions (eg, “How to handle the summer heat?” and “HeLP-Diabetes Newsletter 20-What can you eat?”), and including news or HeLP-Diabetes-related updates were seen as the most attention-grabbing type of content:
Well, obviously new developments and things like that, and research and obviously this October stuff. Things like that, topical things. Topical things to keep people engaged. Just anything new that’s coming out, and keeping people up to date with research and stuff like
As for the visual appeal of email prompts, all the participants favored bold colors for the text, bullet point use, short emails (specifically newsletters) not exceeding A4 in length, and the use of pictures.
The majority of participants did not like wordy email prompts with lots of information; they felt that prompts should have information written succinctly and concisely:
I don't like a big sheet where it’s all mangled up together.
Too much info. Too much info. I wouldn’t really be reading that.
Participants felt prompts should be easily understood and disliked the use of complicated medical terminology or language that was hard to understand.
Most of the participants disapproved of directive advice. They felt, as people living with a chronic condition, that they were told on a daily basis what they should or should not be doing when it comes to any aspect of their type 2 diabetes self-management. They preferred the use of the word “try” rather than “orders” when advice was given:
This is very directive - stay optimistic, stay happy. Rather than, try to stay optimistic is more of an empowering sort of thing.
Perceived irrelevancy of email prompt content was a major reason behind not clicking the links embedded in the prompt. Irrelevant content included specific content targeting small subgroups of participants (eg, content targeting only smokers). Participants preferred more general content that applied to the needs of different patients.
When it came to the visual aspects of email prompts, participants disliked faint colors for text or pictures and particularly disliked it when color of the included links was indistinguishable from the rest of the written content.
This mixed-method study provides an insight into the potential components or characteristics (specifically delivery modes and content) that can promote engagement with a DHI targeting self-management of type 2 diabetes (HeLP-Diabetes) and the factors that may influence their effectiveness.
Just under half of email prompts showed a significant association with visits to HeLP-Diabetes. Nineteen out of the 39 email prompts (49%) showed a significant association with visits to HeLP-Diabetes up to 5 days after an email prompt was sent, whereas none of the text message prompts showed a significant association. Furthermore, 75% of HeLP-Diabetes visits occurred in the 5 days after an email prompt was sent.
The think-aloud interviews suggested that an email prompt should be relevant to recipients, short and concise, easy to understand, with simple language and short sentences, contain links to the intervention, contain nondirective advice and aim for an empowering approach, and contain news and updates. Preferred visual aspects included the use of bullet points, pictures, and bold colors.
The email prompts’ results are consistent with the literature regarding engagement prompts; prompts may promote engagement but have a small to moderate effect as shown in a published meta-analysis [
The results of the study showed that, unlike other published studies [
Some of the preferred email content identified in the interviews was consistent with what is recommended in the literature; the use of nondirective language in face-to-face health or behavior change-related settings has an influence on behavior change [
There were a number of key strengths of this study, including the use of both quantitative and qualitative methods to achieve the research aim, as the combination of methods complemented and provided a clearer picture of the results. Quantitative data cleaning and validating was conducted and reviewed by 2 authors, along with reviewing the interview transcripts and coding to ensure rigor and transparency. The quantitative outcome measure (ie, HeLP-Diabetes user visits) used in this study was objective, meaningful, highly sensitive, and responsive to change, as opposed to subjective measures such as questionnaires. A final key strength was that this research was conducted in a real-life setting rather than a controlled setting where variables that might influence causation are removed.
The study had some limitations, which is to be expected in the emerging field of DHI engagement prompt research. The first main limitation concerned the prompts; the number of users who received the earlier email prompts was small and underpowered to detect an association between receiving a prompt and visiting HeLP-Diabetes; some email prompts’ opens might have not been accounted for if the embedded images were not downloaded; and only a small number of text messages were sent because of technical reasons. The second limitation concerned the think-aloud participants; 6 participants may not have been enough to identify a larger variety of HeLP-Diabetes users’ possible preference for prompt content, however, in the field of human-computer interaction, using 5 users in a usability study is enough to show 85% of design problems that need to be fixed [
This mixed-method study provides a means to explore and optimize the effect of different prompt characteristics on engagement with any DHI before conducting an RCT to determine the effectiveness of the prompts. This type of study can be conducted within any research evaluating a DHI that uses prompts to explore their characteristics.
By mostly selecting prompts that showed a significant association with visiting HeLP-Diabetes for the think-aloud interviews, the study results identified characteristics of email prompts that could be tested in a pilot RCT by the research team. Also, the quantitative stage of the study showed that some emails were associated with visits to HeLP-Diabetes, whereas none of the text messages showed any association. This led to a pilot RCT to assess the effect of different modalities.
One important characteristic that needs to be researched further in future studies is the frequency of prompts. For example, one systematic review that looked at frequency of prompts for DHIs targeting different health behaviors found that high-intensity and low-intensity prompts or irregular prompts both yielded positive engagement [
Usage data analysis is a challenging area of research, as it involves considering and balancing advantages and disadvantages of using specific measures. In this study, “email opens” was used instead of links clicked because users might be triggered to visit HeLP-Diabetes without clicking the links in the email. Visits to HeLP-Diabetes were used instead of Web page visited, which would have shown users’ interests and whether they visited the pages in the prompts or not. However, as with the earlier measure (ie, links clicked and email opens), visits to HeLP-Diabetes are more general and better able to catch users’ activity than a specific measure. Future studies can compare the results of different measures and investigate which ones can capture engagement with DHI because of receiving prompts accurately.
The relationship between engagement with DHI and health outcome improvement was not the focus of this study. The HeLP-Diabetes RCT [
The data from stage 2 was not rich enough to facilitate deeper interpretation of how the reactions to the prompts led to subsequent behaviors. This may have been because of the method of data collection—“think-aloud interviews” are a method emanating from human-computer interaction science, mostly used to test usability issues rather than look for deeper meaning. Alternatively, it may be that our participants in stage 2, who had all worked with the HeLP-Diabetes team and were used to contributing feedback and ideas for the program, had become used to providing actionable feedback rather than deeper reflections. It would be useful to recruit naïve users in future studies and perhaps undertake additional interviews after the “think-aloud” data collection to obtain richer data.
This study showed that specific email prompts were associated with greater engagement with a DHI targeting type 2 diabetes self-management (HeLP-Diabetes), but that was not the case for text messages. Participants tended to open an email prompt and visit the DHI up to 5 days of receiving the prompt. The prompts explored in think-aloud interviews led to the identification of the prompt content features that users liked (eg, new content) or disliked (eg, directive advice). The study results can be explored further in future RCTs evaluating characteristics of engagement prompts.
Prompts date, delivery mode, content, number of recipients.
List of prompts used in the think aloud interviews.
Interview schedule guide.
clinical commissioning group
degrees of freedom
digital health intervention
Healthy Living for People with type 2 Diabetes
interquartile range
randomized controlled trial
relative risk
University College London
The authors would like to thank the HeLP-Diabetes team, patient representatives, and developers who have helped tremendously with the email and text message prompts. This paper presents independent research funded by the National Institute for Health Research (NIHR) under its Programme Grants for Applied Research Programme (Grant Reference Number RP-PG-0609-10135). The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR, or the Department of Health. The funder had no role in the study design, data collection, data analysis, data interpretation, or writing of the report. All authors had full access to all the data in the study and can take responsibility for the integrity of the data and data analysis. The lead author, who had final responsibility for the decision to submit for publication, affirms that the manuscript is an honest, accurate, and transparent account of the study being reported; that no important aspects of the study have been omitted; and that there were no significant discrepancies from the published protocol.
Ghadah Alkhaldi was funded by a PhD studentship from the Saudi Cultural Bureau.
EM is the managing director of a not-for-profit community interest company, which aims to disseminate HeLP-Diabetes across the English NHS. She does not, and will not, take any remuneration for this role.