This observational, analytic study incorporated both qualitative and quantitative research components. Experts in questionnaire design reviewed the original version of the questionnaire and in-depth cognitive interviews were conducted with students to evaluate the comprehensibility and acceptability of the instrument. The revised version of the questionnaire was administered to a sample of undergraduate students at UW-Madison twice, with an interval of 3 to 4 weeks. This timeframe was selected to reduce the likelihood of participants remembering their previous responses while avoiding a gap long enough to span different semesters, which could impact the comparability of recall. Other validated instruments were administered alongside the first administration of the Color of Drinking questionnaire: (1) the Alcohol Use Disorders Identification Test (AUDIT), the microaggressions subscale from the Racial and Ethnic Microaggression Scale (REMS) [9], and the Sense of Belonging Index (SBI). The following properties were evaluated: internal consistency, test-retest reliability, and construct validity.

The study was conducted in a sample of undergraduate students aged ≥18 years at UW-Madison.For in-depth cognitive interviews, an opportunistic sample of students (n=8), ensuring variability in age, gender, and ethnicity or race among students with freshman, sophomore, junior, and senior standing, was selected. For the online survey the minimum sample size was estimated in 100 participants. This sample size was determined to assess the construct validity by applying a 2-sample t test, based on data previously published from the REMS subscale in a sample of undergraduate students, considering a mean difference of 0.5, an SD of 0.89, an error α of .05 and a statistical power (1-β) of .80 [10]. This sample size is also adequate for assessing test-retest reliability (n=63 participants completing the survey on 2 occasions), based on an anticipated intraclass correlation of 0.7 (with a minimum acceptably threshold of 0.5 [11]), a significance level (α) of .05 and β of .20 [12,13].

Student identity data was obtained from the UW-Madison Office of the Registrar and sampled self-identified White and censused self-identified undergraduate students of color or international students. Based on the recruitment rate from the study previously conducted in the same population [7], a random sample of 300 self-identified White undergraduate students and 200 self-identified undergraduate students of color or international students were invited to participate. Each student received an email with the invitation to participate, and 2 reminder emails were sent. It was explained that there were 2 opportunities to be part of the validation of the Color of Drinking Survey (a qualitative Zoom [Zoom Communications, Inc] interview or an online survey administered twice). Participation was voluntary, and invitations were sent via listservs. Thus, students had the option not to respond to the task. Participants who had completed the first questionnaire were contacted by email with the invitation to complete the second questionnaire.

The original instrument was developed by the University Health Services at the UW-Madison [6]. The dimensions of interest were selected, and a list of questions was developed by an interdisciplinary team composed of a Subject Matter Expert on Alcohol (RW) and a Physician, Epidemiologist specialist in Public Health (AM), based on information gathered during interviews with students and a framework based on previous research [5]. The instrument explored the following dimensions: (1) personal alcohol consumption, (2) academics and alcohol drinking culture, (3) locations (avoiding areas at campus due to concerns of the alcohol use of others or where feeling unsafe), (4) the impact of others’ alcohol use, (5) experiencing microaggressions (targeting students of color only), (6) bystander witnessing and intervention in situations of microaggressions or alcohol intoxications), (7) impact on personal health and well-being, and (8) overall experience and sense of belonging at the campus. The tool also included questions about sociodemographic characteristics such as age, gender, race or ethnicity, and financial situation. The questionnaire included both open-ended and closed-ended questions. Questions about experiencing microaggressions were asked only to students of color, under the definition of microaggressions used by the authors: “racial microaggressions are an everyday manifestation of oppression that brings psychological consequences to target groups” [14]. The well-being section included a validated scale, the Diener Flourishing Scale, consisting of 8 items [15]. The questionnaire was previously administered in 2015, 2017, and 2018 to more than 1100 students; thus, the research team has previously analyzed useful information from both closed-ended and open-ended questions.

During the first phase of the validation study, the original version was revised by a group of researchers with expertise in questionnaire design, obtaining a preliminary draft version. Later, a series of cognitive interviews were conducted with a sample of students. They consisted of the flexible administration of the draft questionnaire and relied heavily on verbal probing to improve the instrument [16,17]. An “inspect-and-repair” model was used to inspect the items, detect flaws, and repair them. If the interviewer identified any problem with 1 or more questions (about comprehension, recall, difficulty to answer, response, or other), they would rephrase the items to test a potential solution [18]. The interviews were focused on the questions developed for the questionnaire and did not include the already validated Diener Flourishing Scale or the section on sociodemographic characteristics. After the series of interviews, a new version of the questionnaire was produced.

For a section of questions showing low reliability, the answers to the open-ended questions and other series of in-depth interviews were used to generate a revised set of questions. Then, online surveys were conducted with another sample of undergraduate students to evaluate reliability after changes.

Descriptive statistics were calculated for the population and each item. Hypothesis tests were selected according to the variable distribution. Wilcoxon rank sum test with continuity correction and Kendall correlation coefficients were used to test the hypotheses related to construct validity.

The internal consistency was evaluated by calculating the Cronbach α coefficient by dimension. Additionally, we calculated the variation in the α coefficient after excluding each item from the analysis to assess the effect of dropping a particular item. A Cronbach α value of at least 0.7 is usually interpreted as adequate internal consistency [23]. Low values of α represent poor interrelatedness between items.

To assess test-retest reliability, the Cohen κ coefficient (dichotomous data), the Cohen weighted κ coefficient (ordinal data), and the intraclass correlation coefficient (for continuous data) and their 95% CI were calculated. The higher the coefficient, the stronger the test-retest reliability [13]. κ values between 0.41 and 0.6 were considered to be of moderate reliability, between 0.61 and 0.8 as substantial reliability and between 0.81 and 1 very high reliability [19]. κ is influenced by the trait prevalence and basal rates. κ may be low even though there are high levels of agreement and even though individual ratings are accurate, thus we have also calculated the proportion of observed and expected agreement only by chance. For the interpretation of the intraclass correlation coefficient, values lower than 0.5 were considered indicative of poor reliability, values between 0.5 and 0.75 indicate moderate reliability, values between 0.75 and 0.9 good reliability, and values higher than 0.9 excellent reliability [11].

All the analyses considered α <.05 as statistically significant. Analyses were performed using R Statistical Software (version 4.1.3; R Foundation) and RStudio statistical software (version 2022.02.3 for Windows).

This study was approved by the Institutional Review Board of the University of Wisconsin (Protocol Institutional Review Board ID 2017‐0930). This study was conducted according to the guidelines established in the Declaration of Helsinki for studies involving human subjects. Written informed consent was obtained from all participants in the in-depth interviews, and online consent before the access to the survey in the Qualtrics platform. Students who completed the qualitative interview received US $20 on their campus card. Participants who completed both sets of surveys received US $10 on their student campus card. Data was de-identified for anaylsis.

Table S1 in Multimedia Appendix 1 presents the characteristics of the students who participated in the interviews. The sample included 8 students, aged 19 to 22 years, representing diverse ethnicities, genders, and undergraduate years. The interview duration ranged from 20 minutes to 1 hour.

Overall, participants found most questions easy to answer and acceptable, with their interpretations “in their own words” aligning with the intended purpose of each question. However, for a few questions, 1 or more participants suggested that the interpretation could be improved by incorporating definitions, clarifications, or more specific terminology. Revisions based on these interviews are detailed in Table S2 in Multimedia Appendix 1.

The first version to assess the psychometric properties consists of 31 items (19 core questions+12 skip questions). Additionally, the survey includes 12 questions about sociodemographic characteristics.

A total of 181 students completed the first administration of the questionnaire between June and November 2022. Of these, 116 also completed the second administration. Six observations were excluded from the analysis due to missing answers in >50% of the questions (4 from the first administration and 2 from the second administration). All of them were from the group of students of color. Thus, 177 responses were available for the analysis of internal consistency, 115 for test-retest reliability assessment and 98 for construct validity.

Participants’ characteristics for each substudy are shown below in Table 1. Students of color represented 48.6% (86/177), 43.5% (50/115), and 57.1% (56/98) in substudies of assessment of internal consistency, test-retest assessment, and construct validity, respectively. Most of the students of color were Asian or Asian American, followed by those of Hispanic, Latinx, or Spanish origin. There were students representing all stages of academic studies. A larger number of students lived off campus, and the most frequent current financial situation was described as “tight but doing fine.”

The four dimensions evaluated presented good internal consistency, with Cronbach α coefficients ranging from 0.723 to 0.898 (Table 2).

Approximately 33% (38/115) and 30% (35/115) of the students reported that they do not drink alcohol or did not drink during the last 30 days in test and retest surveys, respectively. The mean number of drinking days within the last 2 weeks was 1.4 and 1.8 for test and retest, respectively. In summary, the section “alcohol use” showed moderate to substantial reliability (Table S3 in Multimedia Appendix 1).

On test and retest surveys, 9.6% (11/114) of the participants reported that the alcohol culture at UW-Madison had impacted their academics (Table S4 in Multimedia Appendix 1). Most items on the section “alcohol culture and academics” showed moderate to substantial test-retest reliability, but 1 of them showed low test-retest reliability.

On test and retest surveys, 51.8% (59/114) of the participants reported that during the semester they had avoided specific areas on or off campus due to concerns about alcohol use by others. The proportion in which participants were negatively impacted by other students’ alcohol consumption was 35% (40/113) for the test and 28% (32/113) for the retest. Items from these 2 sections, “areas avoided” and “impact of by other students’ alcohol consumption” also showed mostly moderate to substantial test-retest reliability (Table S2 in Multimedia Appendix 1).

Among the students of color, 47% (29/62) and 40% (25/62) reported having experienced a microaggression on campus or surrounding areas on test and retest surveys. On test and retest surveys, 71% (81/114) and 73% (83/114) of the students had witnessed at least once a microaggression on campus or surrounding areas during their time at UW-Madison. A summary of test-retest reliability assessment for experiencing, witnessing, and bystander intervention of microaggressions and alcohol intoxications is presented in Table S4 in Multimedia Appendix 1. Most of the items from these dimensions showed moderate to substantial test-retest reliability.

From test and retest surveys, 33% (37/112) and 25% (28/112) of the participants answered that their own or alcohol consumption by other students impacted their health during the time they attended UW-Madison. At test and retest, 43% (49/113) and 42% (47/113) of the students reported that the alcohol culture had impacted their overall sense of belonging at UW-Madison. Most of the items from the section “impact on health and sense of belonging” showed moderate to substantial test-retest reliability, except for 1 related to the impact on health (Table S4 in Multimedia Appendix 1).

Students with AUDIT scores ≥7 reported a higher number of drinking days within the last 2 weeks than those with lower AUDIT scores. A similar association was found regarding the maximum number of drinks in a day and the AUDIT score with a cutoff point of 7 (Table 3). Correlations between the number of drinking days, the maximum number of drinks in a day and the AUDIT score as a continuous variable were moderate to high, r=0.630 (95% CI 0.533-0.719) and r=0.647 (95% CI 0.548-0.741), respectively (Table 3).

A few students reported that they received a poor grade because they had chosen to drink instead of study. Their GPA was lower than the other students’ (Table 4).

Additionally, 27 of 67 students of color answered that in their time at UW-Madison they had experienced a microaggression on campus or surrounding areas. They scored higher on the REMS than the rest of the students.

Lastly, 41 of 98 students answered that the alcohol culture impacted their overall sense of belonging at UW-Madison. These students scored lower in the SBI in comparison with the rest of the students (Table 4).

The “health impacts” section showed a low test-retest reliability. When the qualitative responses were evaluated, students brought up different interpretations but also very profound answers and concepts, such as mental health struggles or experiencing sexual assault. Based on these results, the decision was made to rewrite and retest the validity of this question grounded on the evaluation of content validity, and other series of in-depth interviews were used to generate a revised set of questions. Eight items were developed for the personal consumption of alcohol on diverse aspects of health, and the same number of items were included for the consumption of others.

Online surveys were conducted including the 16 new items with another sample of undergraduate students selected following the same procedures. The sample included 118 students who completed the first questionnaire. Most participants were women (84, 71%), and 66 participants were self-identified as White. Sixty-seven participants also completed the second questionnaire.

The results for internal consistency and test-retest reliability were Cronbach α values of .735 and .855, for each set of questions, respectively, and most of the items presented κ coefficients from 0.476 to 0.877 (Tables S5 and S6 in Multimedia Appendix 1).