Evaluation of a Questionnaire Measuring University Students’ Sense of Belonging to and Involvement in a Biology Department

INTRODUCTION

Being part of a group and having a feeling of social belonging are basic human needs (Maslow, 1962; Ryan and Deci, 2019; Strayhorn 2019). Both Maslow (1962) and Strayhorn (2019) further state that “a college student’s need for belonging must be satisfied before any other higher-order needs such as knowledge and self-actualization” (Strayhorn, 2019, p. 31). Thus, acquiring knowledge or proficiency is not possible without having a sense of belonging to the learning space.

Sense of belonging is included in a number of different theories concerning student achievement or student motivation (e.g., Appleton et al., 2006; Ryan and Deci, 2019; Strayhorn, 2019). A positive relationship with student success has also been empirically established. For example, there is evidence that sense of belonging is associated with academic achievement (Goodenow, 1993a,b; Pittman and Richmond, 2007), retention (Morrow and Ackermann, 2012), and persistence (Hausmann et al., 2007). Sense of belonging has been shown to be particularly important for motivation, achievement, and retention of minority students (Walton and Cohen, 2007; Strayhorn, 2008); and campus interventions designed to eliminate the belonging uncertainty have been positively correlated with increases in grade point average for marginalized first-year-students (Walton and Cohen, 2011). Thus, understanding sense of belonging seems to be one important factor in increasing students’ achievement and retention.

High-quality instruments are needed to be able to study sense of belonging. Because sense of belonging is context dependent, that is, sense of belonging in a certain context (e.g., campus, department, classroom) has the greatest influence on outcome in that particular context (Maslow, 1962; Strayhorn, 2019), instruments used to measure sense of belonging need to be adapted and evaluated for the particular context of interest.

Aim

The aim of the current study was to develop and evaluate a questionnaire measuring university student sense of belonging to and involvement in a university department and to collect initial validity evidence supporting the proposed use of the instrument. We adapted the existing Psychological Sense of School Membership scale (Goodenow, 1993b) into a departmental context and added newly designed involvement items. We collected initial validity evidence concerning response processes, internal structure, and relationships to external variables by performing cognitive interviews and factor analysis and by testing differences in sense of belonging and involvement between different groups of students.

BACKGROUND

We found it relevant to develop and evaluate a questionnaire specifically focused at the departmental level, because most of the previous studies on sense of belonging have occurred at the institutional level or classroom level (e.g., Goodenow, 1993b; Hausmann et al., 2007; Lam et al., 2015). Findings regarding students’ sense of belonging to their university departments, a type of “middle” ground, remain limited (however, see Walton and Cohen, 2007). Further, we have found no previous study presenting validity evidence for a questionnaire adapted to the departmental level (information regarding existing questionnaires is presented later). While we agree that understanding sense of belonging at institution or class level is important, we also argue that studies focusing on departments could be valuable.

The department level is the most likely place for students to interact with a wide range of faculty and staff as they move through the required class sequence, seek guidance on course selection or career advice, or become involved in research. Through these interactions, departments have several opportunities to increase the extent of students’ sense of belonging. Further, most departments have some autonomy to enact changes directly, and because of this, some educational change efforts have been developed to focus specifically on the departmental level (e.g., Partnership for Undergraduate Life Sciences Education fellows, PULSE, [n.d.]). Departments are also involved in student retention and often have to collect and report graduation metrics, including the number of students graduating and the time it took those students to graduate. Most likely, students’ sense of belonging varies more within a given institution if they are in different departments than between institutions (Kuh et al., 2006), and therefore sense of belonging studies focused at the departmental level instead of the general institution level will provide more relevant information.

Further, we chose to include not only items referring to sense of belonging but also items referring to involvement in the department in our questionnaire. While there is disagreement in the literature on whether or not involvement and sense of belonging can be used interchangeably or should be considered separate entities (Strayhorn, 2019), there seems to be a consensus that they are closely related factors. Involvement has a central part in Strayhorn’s model of college students’ sense of belonging: A desire to belong may lead students to become involved, and greater involvement may increase belongingness (Strayhorn, 2019). Strayhorn (2019) describes a range of studies that have shown statistically significant positive correlations between students’ involvement in various campus activities and perceived sense of belonging on campus. The relationship between sense of belonging and involvement seems to be a positive feedback loop, as students who have a sense of belonging are more likely to participate in different activities, and students who engage in different activities enhance their sense of belonging (Astin, 1984; Appleton et al., 2006; You et al., 2011). Further, departments have the opportunity to offer students a wide range of activities to get involved in, and thus involvement could be one main aspect departments need to understand in order to facilitate students’ sense of belonging.

In our early interview studies on students’ relationships to a university department, university students talked about being accepted, respected, included, and supported by others; however, they did not consider themselves to be an essential part of the department (Knekta and McCartney, 2018). Specifically, most students described themselves as “taking” from the department, that is, doing course work, instead of “giving” to the department in the form of direct participation (Knekta and McCartney, 2018). This lack of involvement in, or “giving” to the biology department, was the most common reason students gave for not feeling a sense of belonging. Conversely, students who did feel as though they were “giving” to the department expressed a sense of belonging, specifically two students who stated “I think (I feel like I belong) more so now that I’ve joined the research lab” (Knekta and McCartney, 2018). We also discovered several “missed” chances for students to become involved in the department that they were unaware existed, including volunteering in a biology-related field, volunteering in nature preserves, or attending research seminars and poster sessions. Students stated that had they known about these opportunities, they would have been willing to get involved and would have felt more like members of the department (Knekta and McCartney, 2018). Thus, in our experience, examining sense of belonging and involvement together seems relevant, and evaluating these factors together might also contribute to the understanding of whether or not they should be seen as separate entities.

In summary, departments have a wide range of connection to students that provides them with several different change levers, all of which can be used to promote student sense of belonging. Further, sense of belonging to and involvement in a department seem to have a positive reciprocal relationship with each other. To get a more general understanding of different groups’ sense of belonging to and involvement in a department, or to evaluate the success of different interventions aiming to increase students’ sense of belonging to or involvement in a department, it is crucial to have a high-quality instrument measuring belonging at the departmental level.

THEORETICAL FRAMEWORK

There are many different definitions of sense of belonging (e.g., Rosenberg and McCullough, 1981; Goodenow, 1993b; Osterman, 2000; Tovar and Simon, 2010). The many different definitions make it particularly important to present a definition and theoretical basis of sense of belonging in order to clearly convey the intended meaning of sense of belonging used in our instrument.

In our work, Strayhorn’s model of college students’ sense of belonging was used to give a broader understanding of sense of belonging (Strayhorn, 2019). According to Strayhorn (2019), sense of belonging has seven important core elements: 1) it is a basic human need; 2) it is fundamental motive sufficient to drive behavior, needing to belong makes people act, and acting might increase belongingness (we interpret this as the model supporting the importance of understanding involvement in relationship to sense of belonging); 3) context, time, and factors determine importance, for example, sense of belonging in a certain context (e.g., the biology department) has the greatest influence on outcome in that particular context (e.g., courses taken at the department); 4) it is related to the feeling that one matters, is valued, or appreciated by others; 5) it is influenced by one’s identities; 6) it leads to positive outcomes and success such as achievement, engagement, and happiness; and 7) it must be satisfied continuously and changes as circumstances and conditions change. Strayhorn is a leading expert in the field of college students’ sense of belonging, and his model is closely connected to Maslow’s (1962) model of basic human needs. As students enter various contexts during their time at college, their fundamental needs emerge in the same order as articulated by Maslow (1962). First the basic human physiological needs, such as food, shelter, and sleep have to be satisfied; then social motives or goals, such as belongingness and esteem emerge. At the top of the hierarchy, the higher-most human need is self-actualization, which includes creativity, innovation, and morality. This hierarchical emergence of human needs drives students’ behaviors and perceptions, for example, where they choose to live or which activities they participate in. When different needs are satisfied, the whole cycle will repeat itself with a new center. The absence of a sense of belonging impedes development of higher-order needs, such as creativity, innovation, or desire for knowledge. It is thus of primary importance that the need of sense of belonging is satisfied before any interest in involvement is developed (Strayhorn, 2019).

Inspired by Strayhorn’s (2019, p. 4) working definition of sense of belonging, we define students’ sense of belonging to the department as students’ perceived social support at the department, a feeling or sensation of connectedness, and the experience of mattering or feeling cared about, accepted, respected, valued by, and important to the department community or others in the community, such as faculty, staff, and peers.

We have adapted Astin’s definition of involvement: “the amount of physical and psychological energy that the students devote to the academic experience” (Astin, 1984, p. 5). This definition allows involvement to be seen as a behavioral construct, “what people do” (Astin, 1984), and Astin describes a student as highly involved when he/she devotes considerable energy to studying, spends ample time on campus, participates actively in student organizations, and interacts frequently with faculty members and other students (Astin, 1984). We expand further on Astin’s definition, using data from our previous research, to also include students who “give” to the department, most often in the form of “what people do,” such as contributing to a research lab, serving as a teaching assistant, or participating in a volunteer program.

EXISTING MEASURES OF SENSE OF BELONGING

Most studies on sense of belonging involving questionnaires have used only three or four items to conceptualize sense of belonging (e.g., Hausmann et al., 2007; Maestas et al., 2007; Strayhorn, 2008). While using only a few items to measure sense of belonging is not bad practice, it will provide a limited understanding of the this complex construct. According to most definitions, sense of belonging includes aspects such as perceived social support, experience of being accepted, and feeling respected. Asking specifically about these different aspects will give a more nuanced understanding of the construct, for example, how different aspects relate to one another as well as to other external variables, such as achievement and involvement. Further, using only general items, such as “I feel a sense of belonging to the biology department,” might introduce bias in the response patterns, because different respondents might hold slightly different ideas of what sense of belonging is and therefore apply different understandings of sense of belonging to inform their responses.

Although few in number, comprehensive sense of belonging questionnaires with published validity evidence are available (Goodenow, 1993b; Walton and Cohen, 2007; Trujillo and Tanner, 2014). The Psychological Sense of School Membership (PSSM) scale is a questionnaire that has been used extensively in various studies investigating student sense of belonging at the class or institution level (You et al., 2011). The PSSM was originally developed by Goodenow (1993b, p. 80) to measure middle school students’ sense of school membership and included 18 items concerning “the extent to which students feel personally accepted, respected, included, and supported by others in the school environment.” Hoffman et al., (2002) developed a questionnaire measuring college students’ sense of belonging, based on focus groups interviews and the theoretical frameworks provided by Tinto (1975) and Astin (1984). The final questionnaire included 26 items representing five factors: perceived peer support, perceived faculty support/comfort, perceived classroom comfort, perceived isolation, and empathetic faculty understanding (Hoffman et al., 2002). In 2007, Walton and Cohen presented the Sense of Social Fit scale including 17 Likert-type items targeting students’ sense of social fit in a department (Walton and Cohen, 2007).¹ This scale has been used in several different studies since then (e.g., Walton et al., 2012; Stephens et al., 2014; Devers et al., 2016); however, no validation studies or theoretical framework for the questionnaire has been published so far.

Thus, there are a few existing comprehensive questionnaires for sense of belonging, but to the best of our knowledge there is no existing study presenting and evaluating a questionnaire targeting students’ sense of belonging to and involvement in university departments. Because the PSSM scale has best conformity with Strayhorn’s theory and definition of sense of belonging, and there are several published studies evaluating the quality of the scale, we chose to adapt and evaluate the PSSM scale for the purpose of our study.

The PSSM Scale

Originally, Goodenow (1993b, p. 80) stated that the PSSM assesses a single construct (unidimensional) defined as the “extent to which students feel personally accepted, respected, included, and supported by others in the school environment.” However, this definition of school membership offered by Goodenow, the face validity of the PSSM, as well as later empirical studies, suggest that the PSSM scale has a multidimensional structure (Hagborg, 1994; Cheung and Hui 2003; You et al., 2011). Hagborg (1994) concluded that the PSSM scale had a multidimensional structure but found all but the general belonging factor to be of limited application. Cheung and Hui (2003) concluded that the PSSM consisted of two factors, “school belonging” (13 items) and “feelings of rejection” (5 items). You et al. (2011) administered the PSSM and found a multidimensional scale structure consisting of three factors “caring relationships,” “acceptance,” and “rejection,” with four, five, and three items respectively. Freeman et al. (2007) were the first to implement the PSSM with university students and developed two adaptations of the PSSM: one for students’ sense of belonging within a specific class and another for students’ sense of belonging within the general university. Interestingly, at the class level, they found a single, general measure of belonging. However, at the university level, results were similar to what previous studies had found: three separate factors, including “general sense of university belonging,” “perceived pedagogical caring from professors,” and “social acceptance by others on campus” (Freeman et al., 2007). Thus, although we deemed the PSSM scale to be a suitable questionnaire to adapt for our purposes, previous studies confirm the need to further understand the dimensionality of the sense of belonging construct. Further, because both sense of belonging (Strayhorn, 2019) and instrument validation are context dependent (Knekta et al., 2019), at least some new validity evidence has to be collected each time a sense of belonging instrument is used in a new context.

METHODS

Participants and Procedures

The questionnaire was developed and validated in an iterative process over three data collections (Table 1). All data were collected at a large, minority-serving, R1 institution in the southeastern United States, and the questionnaire was administered to students enrolled in biology courses. We sampled from biology courses that contained both biology majors and non–biology majors. First, 10 cognitive interviews (Willis, 2005) were performed in Fall 2017 to evaluate the wording of the items. Based on the results from the cognitive interviews, the questionnaire was refined (see Supplemental Material 1), and a second version of the survey (Table 2) was administered in Spring 2018 to biology students enrolled in an introductory biology course. Students took the questionnaire 2–3 weeks before the end of the Spring semester. In total, 201 biology majors completed the questionnaire. Exploratory factor analysis (EFA) was conducted on data from the second data collection, and results were used to design a third version of the questionnaire that was administered in Fall 2018 across all biology classes at all levels (Table 2). Three groups of students who we hypothesized having different sense of belonging to and involvement in the biology department were asked to complete the questionnaire during the third data collection, that is, biology majors, students participating in a special biology program called QBIC (see more information about the QBIC program in the next section), and non–biology majors. In total, 737 students completed the questionnaire in Fall 2018: 596 biology majors, 73 QBIC students, and 68 non–biology majors. On all occasions the questionnaire was distributed through Qualtrics (an online survey tool).

The Biology Department

The biology department investigated in this study is composed of 41 tenure-track and 15 instructor-level professors. Degree programs include bachelor of arts (BA) and bachelor of science (BS) in biological sciences, BS in biological sciences: biology education, BS in marine biology, master of science (MS) in biology, PhD in biology, and a combined MS in forensic science/PhD in biology. Required courses include general biology 1 and 2, ecology, evolution, genetics, and cell biology. Several of these required courses are also required for entrance into U.S. medical schools; therefore, a large number of pre-med students enroll in the biology major with the intention of attending medical school instead of doing additional graduate work in biology. Students who take biology courses but are not biology majors may include psychology majors or chemistry/physics majors who want a background in biology. Other non–biology majors may take a biology course purely out of interest. In the academic year 2016–2017, 624 BS degrees in biology were conferred. In the academic year 2017–2018, 723 BS degrees in biology were conferred. The department website lists opportunities for internships, jobs, scholarships, seminars, and clubs, as well as opportunities to get involved in community events and K–12 programs.

The QBIC Program

Quantifying Biology in the Classroom (QBIC) is a 4-year program within the biology department (Weeks et al., 2011). The overall objective of the program is “to establish an integrated program of biology, physics, chemistry, statistics, and mathematics to prepare undergraduates to understand how things fit together and to provide them with the skills to do research in biological sciences” (Weeks and Koptur, 2013, p. 1). The program also has the goal of developing a community of learners: students/students, professors/students, and professors/professors, and to enhance a sense of community among participants (Weeks and Koptur, 2013). The QBIC program uses a teaching design that aligns with previous research suggesting a strong sense of belonging and high involvement, that is, small classes, courses are integrated with one another, active- and cooperative-learning techniques are employed, and both problem-based and peer-led team learning is used (Appleton et al., 2006; Tinto, 2012). Thus, if the QBIC program’s intentions are successful, these groups of students are likely to have higher sense of belonging and involvement than other biology majors.

The Questionnaire

The questionnaire aimed to measure students’ sense of belonging to and involvement in the biology department. Three different versions of the questionnaire were administered (version 1 is shown in the Supplemental Material and versions 2 and 3 are shown in Table 2). First, the sense of belonging scale published by Goodenow (1993b), composed of 18 items, was adapted to a departmental context. For example, items were reworded from “The teachers here respect me” to “The instructors here respect me” and from “I can really be myself at this school” to “I can really be myself at the [name of the university] biology department.” Further, 15 involvement items describing different possible activities connected to the biology department that the students could become involved in were created based on possible opportunities for involvement at the biology department. The involvement items were discussed with faculty members at the biology department. All items were rated on a six-point Likert-type scale (1 = strongly disagree, 2 = disagree, 3 = slightly disagree, 4 = slightly agree, 5 = agree, and 6 = strongly agree). Students also had the option to choose “prefer not to respond.”

First, cognitive interviews (n = 10) were performed with participating students to collect validity evidence based on response processes, and the questionnaire was revised based on the results from the student interviews (Supplemental Material 1; Knekta and McCartney, 2018). For example, on several occasions, students mentioned a non-faculty staff being interested in them, which contributed to their sense of belonging; therefore, “faculty member” was changed to “faculty and staff” for several items. Two sense of belonging items and two involvement items were removed, because students found them irrelevant or had problems interpreting key terms. For example, several students found the item “I feel very different from most other students here” difficult to interpret. Being different could be something positive (everyone is unique) or something negative (not being like everyone else in a negative sense). Based on the cognitive interviews, five additional sense of belonging items were developed. The second version of the questionnaire included 21 sense of belonging items and 14 involvement items (Table 2). The second version was revised based on results from factor analysis, resulting in a third version of the questionnaire including 16 sense of belonging items and 13 involvement items (Table 2).

Data Analysis

All data analyses were run in R v. 3.5.1 (R Core Team, 2018). For the third data collection, data were broken down into biology majors, QBIC students, and non–biology majors. Before factor analysis, descriptive statistics and correlations between items were examined. For the biology majors, missing values ranged between 1% and 10% for the second data collection and were < 4% for the third data collection. Skewness and kurtosis did not indicate severe nonnormality (univariate skewness < |1.6| and kurtosis < |4.0|). Deletion of all cases with missing values would cause a substantial loss of information for both data sets (>21% of the cases), and therefore multiple imputation using logistic regression based on all variables included in the factor analysis was used to estimate missing values (implemented with the MICE package; van Buuren and Groothuis-Oudshoorn, 2011). The first imputation was used for subsequent factorial analysis for both data collections. The same analysis was later run using the four other imputations; no substantial differences in the results were found for any of the data collections. Since missingness for the second data collection was >5% for some variables; all analysis was also run on a data set including only complete cases, without substantial differences in results. Analysis of Mahalanobis distance was used to identify potential outliers. Cases with high Mahalanobis distance (p < 0.001) were inspected in detail, and we found no justification for removing any of them for any of the data sets.

Dimensionality of the Questionnaire

To collect validity evidence concerning internal structure, factor analysis was performed on the data from the second data collection. Considering that these data were collected with a newly developed instrument, EFA was performed. We hypothesized that the sense of belonging items would separate from the involvement items and that the involvement would form a unidimensional scale. Because of existing research on the sense of belonging scale varied widely, we had no hypothesis about the dimensionality of the sense of belonging scale.

Based on the findings from EFA on data from the second data collection, confirmatory factor analysis (CFA) was performed on data from biology majors from the third data collection to confirm the results found from the EFA on data from the second data collection. Because results from that CFA indicated that the data did not support the suggested structure, the original sample from the third data collection was split in half and EFA was performed one half followed by a CFA on the other half. EFA was run using the R package psych (Revelle, 2017). CFA was run using the R package lavaan (Rosseel, 2012).

EFA.

A weighted least-square estimator, an estimator suitable for ordinal and nonnormally distributed data, was used to extract the variances from the data. Because we hypothesized a correlation between the sense of belonging and involvement scales, an oblique rotation (oblimin rotation) was chosen. Visual inspection of the scree plot, parallel analysis based on eigenvalues from the principal components, and factor analysis in combination with theoretical considerations were used to decide on the appropriate number of factors to retain (psych package; Revelle, 2017). Total variance explained, communalities, pattern coefficients, and factor correlations were used to evaluate the fit of the data to the model as well as the fit of individual items to the scales. For this initial evaluation of the instrument, we used a pattern coefficient cutoff >0.40 for retention of items, as we wanted to keep as many items as possible from the original scale. For the initial evaluation based on the second data collection, pattern coefficients >0.30 on a second factor (cross-loadings) were considered problematic. This rather generous cutoff value was used, as we did not want to exclude items too early in the process. For the EFA performed on data from the third data collection, the stricter guideline >0.20 for cross-loadings was used.

CFA.

In consideration of the ordinal and nonnormal nature of the data, the robust maximum-likelihood estimation (MLR) was used to extract the variances from the data. Multiple fit indices (chi-square value from robust MLR [MLR χ²], comparative fit index [CFI], root-mean-square error of approximation [RMSEA], and standardized root-mean-square residual [SRMR]) were consulted to evaluate model fit. The fit indices were chosen to represent absolute, parsimony-adjusted, and incremental fit indices (Bandalos and Finney, 2010). Consistent with the recommendations by Hu and Bentler (1999), SRMR < 0.08 was considered as indicative of adequate model fit. For RMSEA and CFI, T-size measures of model fit for equivalence testing, RMSEA_t and CFI_t, for α = 0.05 were calculated using the technique described by Marcoulides and Yuan (2017) to evaluate the goodness of fit of the models.

Differences in Sense of Belonging and Involvement between Different Groups of Students

Finally, to collect validity evidence concerning relationships to external variables, analysis of variance (ANOVA) was applied to test mean differences in mean scale scores for both the sense of belonging scale and the involvement scale between QBIC students, biology majors, and non–biology majors. Tukey’s honestly significant difference (HSD) test, was performed to identify the differences in mean scale scores. We hypothesized that QBIC students would report the highest sense of belonging and involvement and that non–biology majors would report the lowest sense of belonging and involvement.

RESULTS

In this section, descriptive statistics and results from the EFA applied to data from the second data collection are presented. Next, descriptive statistics and results from EFA and CFA applied to data from biology majors from the third data collection are presented. Finally, results from ANOVA testing mean differences in sense of belonging and involvement between biology majors, QBIC students, and non–biology majors are presented.

Second Data Collection

Descriptive Statistics.

The items had a mean between 3.3 and 5.0 (on a Likert scale ranging from 1 to 6), a univariate skewness < |1.7| and kurtosis < |3.1|, and the SD ranged between 1.0 and 1.7 (Supplemental Material 2). Mardia’s multivariate normality test (implemented with the psych package; Revelle 2017) showed significant multivariate skewness and kurtosis, which indicated multivariate nonnormality. The interitem correlation matrix showed that the correlations ranged, from r = −0.05 to r = 0.72 within the sense of belonging scale, from r = 0.41 to r = 0.86 within the involvement scale, and from r = −0.03 to r = 0.43 between sense of belonging and involvement items (Supplemental Material 2). The Kaiser’s measure of sampling adequacy value was >0.92, which indicated good factorability. Multicollinearity was investigated by examining interitem correlations and tolerance values from multiple regressions implemented with the olsrr package (Hebbali, 2018). The highest interitem correlation was 0.86. Item I6 was removed, because it had a tolerance = 0.01, which could cause problems due to multicollinearity. Looking at the wording on that item, we found that it is theoretically very close to item I1. Thus, we concluded that it would not be a big theoretical loss to remove item I6.

EFA.

In our study, the total sample size of biology majors was 201, which can be considered as a rather small but still sufficient sample for performing factor analysis if number of items per factor, as well as item correlations, are high (Gagne and Hancock, 2006; Wolf et al., 2013). Parallel analysis based on all items indicated three components and four factors. The scree plot leveled out at three factors. Consequently, EFAs with three and four factors were performed.

The pattern matrix for a four-factor solution showed that the first factor consisted of items with pattern coefficient >0.40 for all involvement items (Table 3). The three other factors consisted of items with pattern coefficients for different sense of belonging items. The second factor had factor coefficients >0.40 for some of the sense of belonging items (S3, S5, S8, S9, S10, S12, S13, S14, S15, S19, S20, S21) and the third factor had a pattern coefficient > 0.40 for the negatively worded sense of belonging items (S4, S7, S17). The last factor explained only 5% of the variance in the data and had pattern coefficients >0.40 for the sense of belonging items S1, S2, S3, S6, and S11. The sense of belonging items S3, S8, S16, and S18 had pattern coefficients >0.30 on factors 2 and 4. For the three-factor solutions, again the involvement items had high pattern coefficients for one factor and low pattern coefficients for the other factors. The sense of belonging items ended up on two factors with all the negatively worded items (S4, S7, S17) and S1 having a pattern coefficient >0.40 on one factor. Items S1, S2, S3, S5, S6, S8, S9, S10, S12, S13, S14, S15, S16, S18, S19, S20, and S21 had a pattern coefficient >0.40 on the other factor.

Thus, the involvement scale seems to be a unidimensional scale with high pattern coefficients for all items. The sense of belonging scale did not exhibit good psychometric properties. The negatively worded items are highly correlated to one another but not to the other items at the scale, and we do not see any theoretical reasons for them to form their own scale, leading us to believe that they correlate due to their format (i.e., negatively worded) rather than representing a meaningful underlying subconstruct. Further, item S11 had low correlations in the correlation matrix and low communalities.

As a first step, we decided to remove items S11, as well as the negatively worded items, in a stepwise manner. As soon as one negatively worded item was removed, parallel analysis indicated one less factor, and the remaining two negatively worded items got pattern coefficients <0.40 on all factors. After all the negatively worded items and item S11 were removed, parallel analysis indicated three components and three factors. The sense of belonging items still divided into two factors, with items having pattern coefficients >0.30 on both factors (cross-loading items). To create factors without cross-loadings, one solution would be to remove all items that cross-load in a stepwise manner or to remove the items from the sense of belonging scale explaining the least variance until a unidimensional scale is retrieved. Removing items that cross-loaded (i.e., S5, S8, S18) gave a three-factor solution with sense of belonging items S1, S2, S3, and S6 on one scale and items S9, S10, S12, S13, S14, S15, S16, S19, S20, and S21 on the other scale. The first factor did not really make theoretical sense (I feel like a real part of the biology department, People here notice when I’m good at something, Faculty and staff at the biology department value my opinions, Most faculty and staff at the biology department are interested in me). Instead, we brought back the cross-loading items and removed items S1, S2, S3, and S6 in a stepwise manner. Upon removal of item S1, parallel analysis indicated three components and two factors. Also, after removal of the other items, parallel analysis indicated three components and two factors. Item S1 had the lowest communality and was therefore deemed as less useful to keep.

In summary, the results from the EFAs suggest that the questionnaire consists of two subscales, one sense of belonging scale including 16 items from the second version (excluding S1, S11, and the negatively worded items: S4, S7, S17) and one involvement scale including all items from the second version, except for item I6. In total, we have 13 items (Table 3). The sense of belonging factor explained 29% of the variance in the data, and the involvement factor explained 30%.

Third Data Collection

Descriptive Statistics.

The items had a mean between 3.5 and 5.1 (on a Likert scale ranging from 1 to 6), a univariate skewness < |1.7| and kurtosis < |4.0|, and the SD ranged between 0.95 and 1.6 (Supplemental Material 3). Mardia’s multivariate normality test (implemented with the psych package; Revelle 2017) showed significant multivariate skewness and kurtosis values, which indicated multivariate nonnormality. The interitem correlation matrix showed that the correlations ranged from r = 0.20 to r = 0.76 within the sense of belonging scale, from r = 0.20 to r = 0.73 within the involvement scales, and from r = 0.07 to r = 0.49 between sense of belonging and involvement items (Supplemental Material 3). The Kaiser’s measure of sampling adequacy value was 0.93, which indicated good factorability. The highest interitem correlation was 0.76 and the lowest tolerance 0.30, thus no problem with multicollinearity was indicated.

CFA, First Round.

First, CFA was performed on data from biology majors (n = 596) from the third data collection to confirm the two-factor structure suggested by the EFA based on the second data collection. Items S2, S3, S5, S6, S8, S9, S10, S12, S13, S14, S15, S16, S18, S19, S20, and S21 represented the sense of belonging factor, and items I1 to I5 and I7 to I14 represented the involvement factor. Correlation between the two factors was allowed. Results from the CFA indicated that the data did not support the hypothesized model. The chi-square test of model fit was significant (χ² = 1916, df = 376, p < 0.000), with model fit indices SRMR = 0.070, CFI = 0.80, and RMSEA = 0.083. According to equivalence testing, with a probability of 0.95, CFIt was > 0.77, which corresponds to poor model fit² according to the cutoff values determined by the equivalence testing calculation. With a probability of 0.95, the size of the misspecification in the model as measured by RMSEA is <0.087, which corresponds to mediocre model fit³ according the cutoff values determined by the equivalence testing calculation. Factor loading, correlational residuals, original interitem correlation, and modification indices did not point out any single item or special item pair as problematic. Thus, we decided to split the sample in half and perform EFA on one half of the data set to determine the dimensionality of questionnaire and detect possible problematic items. The EFA was followed by CFA on the other half of the data set.

EFA.

An EFA was conducted on half the data set (n = 298). Parallel analysis based on all items indicated four factors and three components. The scree plot leveled out at four factors. Consequently, EFAs with three and four factors were performed. As in the first EFA for data from the second data collection, the pattern matrix for a four-factor solution showed that the first factor consisted of items with a high pattern coefficient for all involvement items, and the three next factors consisted of different sets of the sense of belonging items. The last factor had only two items with a pattern coefficient >0.4 and explained only 5% of the variance in the data. The pattern matrix for a three-factor solution showed that the first factor consisted of items with a pattern coefficient >0.4 for all involvement items, except item I14 (Table 4). Items I3, I4, and I13 had pattern coefficients >0.2 on factor 1 and/or factor 3. The two other factors consisted of items with a high pattern coefficient for different sense of belonging items. The second factor had factor coefficients >0.4 for the items S9, S10, S12, S13, S14, S15, S19, and S20, and the third factor had pattern coefficient >0.4 for the items S2, S3, S6, S8, S9, S15, S16, S18, and S21. Items S5, S8, S9, S15, and S21 had pattern coefficients >0.2 on both factors 2 and 3.

Thus, as was also seen in the EFA based on the second data collection, the involvement items seem to represent one separate factor. However, a few items had a low pattern coefficients or pattern coefficients >0.2 on more than one factor. We decided to, in a stepwise manner, remove items I14, I3, and I4. I13 was kept in the model, although it had a pattern coefficient > 0.2 on factors 2 and 3, because of its unique wording and, after removing the other involvement items, the pattern coefficient on the nonfocal factor was <0.2. The sense of belonging items seemed to split into two factors, with five items having pattern coefficients >0.20 on both factors 2 and 3. The splitting was not equal to but similar to the splitting in the four-factor EFA based on data from the second data collection. We decided to, in a stepwise manner, remove items S9, S15, S21, S5, and S8. After doing so, parallel analysis based on all items indicated three factors and three components. The scree plot indicated a level out at three factors. All items had a pattern coefficient >0.4 on one factor and <0.2 on the other factors (Table 4). The involvement factor explained 24% of the variance in the data, the first sense of belonging factor explained 16% of the variance, and the second sense of belonging scale explained 15% of the variance.

In conclusion, the initial CFA and the EFA based on data from the third data collection did not support the two-factor structured achieved by the EFA based on data from the second data collection. Instead, the sense of belonging scale seems to be multidimensional. To achieve a good factor structure, eight items had to be removed. The final three factors were labeled involvement, sense of belonging: valued competence, and sense of belonging: social acceptance. Theoretically speaking, understanding and labeling the two sense of belonging factors was not straightforward and will be addressed further in the Discussion.

CFA, Second Round.

Based on the results from the EFA, a three-factor CFA including 21 items was specified. Correlations between all three factors were allowed. The chi-square test of model fit was significant (χ² = 450, df = 186, p < 0.000), with model fit indices SRMR = 0.054, CFI = 0.91, and RMSEA = 0.069. According to equivalence testing, with a probability of 0.95, CFIt was > 0.87, which corresponds to mediocre model fit⁴ according to the cutoff values as determined by the equivalence testing calculation. With a probability of 0.95, the size of the misspecification in the model as measured by RMSEA was <0.077, which corresponds to fair model fit⁵ according the cutoff values as determined by the equivalence testing calculation. Thus, analysis suggested a better fit to the data than in the first round of CFA, but still not a good fit to the data. Factor loadings, correlational residuals, original interitem correlation, and modification indices suggested that removing item I5 could substantially improve the model fit. After item I5 was removed, the model fit improved. The chi-square test of model fit was still significant (χ² = 337, df = 167, p < 0.000), but the model fit indices (SRMR = 0.050, CFI = 0.94, and RMSEA = 0.058) improved. According to equivalence testing, with a probability of 0.95, CFIt > 0.90, which corresponds to fair model fit⁶ according the cutoff values as determined by the equivalence testing calculation. With a probability of 0.95, the size of the misspecification in the model as measured by RMSEA was <0.068, which corresponds to fair model fit⁷ according to the cutoff values as determined by the equivalence testing calculation. Factor loadings were > 0.7 for all sense of belonging items and for most of the involvement items, meaning that, for most of the items, the factor explained most of the items well (Figure 1). The factor correlation between the two sense of belonging factors was 0.57, indicating that they do represent different but related subconstructs (Figure 1). The factor correlation to the involvement scale was slightly higher for the sense of being valued due to competency (0.52) compared with the sense of social acceptance (0.40). Items included in the final version of the DeSBI are listed in Table 5. Coefficient ω for the three different subscales were: ω = 0.90 for sense of belonging: valued competence, ω = 0.89 for sense of belonging: social acceptance, and ω = 0.89 for involvement. Thus, although there still is room for improvement, our analysis altogether showed good psychometric properties for the DeSBI questionnaire.

Differences in Sense of Belonging and Involvement between Different Groups of Students

We tested the DeSBI questionnaire among three different groups of students: QBIC students, who are enrolled in a special biology program designed to promote a sense of belonging and increase involvement; biology majors; and non–biology majors. Significant differences were found in all three subscales’ mean scores among the three groups of students tested (sense of belonging: valued competence F(2.66) = 12, p < 0.001; sense of belonging: social acceptance F(2.67) = 4.4, p = 0.012; involvement F(2.69) = 30, p < 0.001; Figure 2). The largest differences were found for the involvement scale, where significant differences were found between all three groups. QBIC students reported the highest involvement in the biology department, while non–biology majors reported the lowest involvement (Tukey’s HSD p < 0.001 for all three comparisons; biology majors to QBIC Cohen’s d = 0.46⁸; biology majors to non–biology majors Cohen’s d = 0.86 and Hedges’ g = 0.81; and QBIC to non–biology majors Cohen’s d = 1.4). Regarding the factor sense of belonging: valued competence, QBIC students scored significantly higher than biology majors (Tukey’s HSD p < 0.001, Cohen’s d = 0.50) and non–biology majors (Tukey’s HSD p < 0.001, Cohen’s d = 0.85). A relatively small difference was found between biology majors and non–biology majors (Tukey’s HSD p < 0.001, Cohen’s d = 0.32, Hedges’ g = 0.30). For the sense of belonging: social acceptance, the only significant difference found was between QBIC students and the non–biology majors (Tukey’s HSD = 0.009, Cohen’s d = 0.51).

DISCUSSION

Previous research has shown that sense of belonging is positively associated with academic achievement, retention, and persistence (Pittman and Richmond, 2007; Hausmann et al., 2007; Morrow and Ackermann, 2012) and that increasing students’ involvement is one way to increase their sense of belonging (Strayhorn, 2019). Departments provide a wide range of change levers, all of which can be used to promote involvement, student sense of belonging, and, in the long run, retention. To explore different groups’ sense of belonging and involvement or to evaluate the success of different interventions aiming to increase students’ sense of belonging or involvement, high-quality measures adapted to the specific level of interest are crucial. Because we found no existing study presenting and evaluating a questionnaire targeting students’ sense of belonging to and involvement in university departments, we aimed to create and evaluate a questionnaire in this study. In summary, cognitive interviews showed that students overall understood the items in the questionnaire in the way we intended. The psychometric properties of the instrument were fair, and expected differences in sense of belonging and involvement between different groups of students were found.

Internal Structure of the DeSBI Questionnaire

Factor analysis showed that the DeSBI consisted of three subscales: 1) sense of belonging: valued competence, 2) sense of belonging: social acceptance, and 3) involvement. Previous studies concerning the factor structure of the PSSM scale have shown varying results (e.g., Cheung and Hui, 2003; Freeman et al., 2007; You et al., 2011), consistent with our analysis presented here of two data collections showing inconsistent results regarding the factor structure of the PSSM. Although these inconsistencies could depend on the specific sample of students, analysis used, criteria for interpretation used, and a slight variation in the survey items between the different studies, it is clear that further studies are needed in order to understand the dimensionality of the sense of belonging construct. While the different studies on the PSSM scale showed inconsistent results with regard to the factor structure, most conclude that, if all items are included, the PSSM scale is multidimensional (e.g., Cheung and Hui, 2003; Freeman et al., 2007; You et al., 2011). Results from our study agree with this.

Our two sense of belonging factors do not clearly correspond to any of the previous factors found for the PSSM scale. As in three of the abovementioned studies, the negatively worded items formed their own scale in our analysis. In contrast to the previous studies, we chose not to keep that factor, as we do not see any theoretical reasons for these items to form their own scale. We believe that they correlate due to their format (i.e., negatively worded) rather than representing a meaningful underlying construct.

Several previous studies, including ours, report on a factor called acceptance or social acceptance. Our factor “sense of belonging: social acceptance” seems to contain fairly similar items as the social acceptance factor reported by Freeman et al. (2007), but different items than the acceptance factor reported by Hagborg (1994) and You et al. (2011). Thus, although having similar names, the acceptance factors do not measure the same thing. No previous studies have found a subscale including a similar set of items as the sense of belonging: valued competence scale in our study.

Our factor “sense of belonging: valued competence” mainly contains items concerning faculty and staff, and our factor “sense of belonging: social acceptance” mainly contains items concerning peers (Table 5). However, looking more closely at the two factors and the items within them, we interpret the first factor to be more connected to feelings of competence or being valued and the second factor to be more closely connected to feeling accepted or included. While faculty and staff and peers are part of these two factors, each of these factors also includes more general items not referring to faculty, staff, or peers specifically. We had items concerning valued competence connecting to peers and items concerning social acceptance connecting to faculty and staff in versions 2 and 3 of our questionnaire, but they did cross-load in both factors and therefore cannot be used when creating mean or scale scores.

Our questionnaire differs from the original PSSM scale because of our addition of involvement questions. The involvement items separated clearly from the sense of belonging items and showed unidimensional properties that were in agreement with Astin’s (1984) singular definition of involvement. Our results indicate that while involvement is closely related to sense of belonging, it is still a separate construct. The involvement items were thought to represent not only a good selection of the activities the students discussed with us during the cognitive interviews, but also a good selection of activities that most departments would encourage their students to participate in (Knekta and McCartney, 2018). As we have only tested our questionnaire at one department at one university, it is still unknown whether we overlooked some important aspects of student involvement in other departments.

External Validity Evidence

Within our university, we found expected differences in both sense of belonging and involvement between three groups: QBIC students, biology majors, and non–biology majors. This can be considered as validity evidence relating to external variables. QBIC students, who are part of a special group designed to promote a sense of belonging and involvement, scored highest on all three factors, while non–biology majors consistently scored the lowest.

Suggestions on Possible Use of the Questionnaire

According to the model of college students’ sense of belonging, belonging is a basic human need that drives students’ behavior and is necessary for students to succeed in higher education (Strayhorn, 2019). A department that understands students’ sense of belonging to and involvement in the department is more likely to be successful in having students who will want to persist to completion of their studies (Strayhorn, 2019). Our questionnaire provides university departments a tool for investigating students’ perspectives on sense of belonging and involvement. It could provide interested educators with valuable information and also show students that their departments are actively seeking out their opinions.

The DeSBI questionnaire can be used for a general understanding of whether departments are successfully connecting with students and can help target areas for improvement. In our study, we can conclude that the biology department has been successful with the QBIC program with regard to creating a high sense of belonging and involvement. While biology majors also report high scores, it is clear from the QBIC group data that there is room for improvement. As expected, non–biology majors scored the lowest on all factors. What does this mean for our department? Is this an area we want to improve in? Is it important for us, as a department, to make all students feel like a part of the department, or is it more important to spend our resources in increasing the scores of the biology majors? Given the particular interests of a department, the DeSBI questionnaire could be expanded to include open-ended responses following particular statements, so that students could elaborate further when they rate a statement low (or high), which may provide additional information about what a department needs to improve on (or is being successful in).

While a total sum or mean scores based on all items in the final involvement scale can be used to get an overall indication of students’ involvement in the biology department, specific items can be evaluated individually as well. For example, responses on the involvement items can let departments know whether they have been successful in providing students with information on how to participate; for example, items I12 (join a student group or club) and I13 (participate in volunteer work). Too often, departments place announcements on the department website or post flyers outside the departmental office. Do students notice these announcements? Using DeSBI in this way may help departments better target their advertising to ensure that students are receiving the information.

Limitations and Further Research

Our work provides departments with a questionnaire specifically adapted for evaluating students’ perceptions of sense of belonging to the department as well as the activities the students may be involved in. Our work has also contributed to increased theoretical understanding of sense of belonging and involvement.

To advance the theoretical understanding of sense of belonging even more, additional items could be evaluated. Based on our cognitive interviews, we complemented the PSSM scale with a few items. Future development of the questionnaire could consider complementing version 3 of the DeSBI with more items concerning peers and valued competence and more items concerning faculty and staff and social acceptance, followed by a re-evaluation of the factor structure. Furthermore, more items based on the theoretical definition and understanding of sense of belonging could be added to better cover the whole construct; for example, by adding items more clearly asking about feeling connected to others.

Validation of an instrument is an iterative and continuous process, and we recommend more studies collecting validity evidence for the DeSBI questionnaire. Considering the diverse conclusions about the dimensionality of the sense of belonging construct in previous research, as well as in our research, we encourage repeated studies collecting validity evidence regarding the factor structure of the DeSBI questionnaire. This could preferably be done using items in the third version of DeSBI instead of the final version in order to re-evaluate the excluded items once more. We would also recommend collecting additional external validity evidence, such as comparison of results from DeSBI with other existing sense of belonging scales. Additionally, studies collecting evidence on the relationship to other closely related constructs, such as identity (e.g., Carlone and Johnson, 2007) and self-efficacy (e.g., Pajares, 1996), would also be valuable.

The DeSBI questionnaire was validated for use with biology students, and thus we do not know whether the same relationships among survey items would have been found with non–biology majors or for other departments. Measurement invariance studies with regard to demographic factors such as gender and ethnicity should be done if differences between different groups are of interest.

While collecting data only from undergraduates in this study is not a limitation, it would be interesting to conduct the same study with graduate students. We predict that our factors and items would not look the same, as items that were removed from the current DeSBI, such as I3, “talk about my career plans with biology faculty or staff,” may behave entirely different within a graduate student population. In this way, departments may need to consider the sense of belonging and involvement needs of varying student populations in different ways.

FOOTNOTES

ACKNOWLEDGMENTS

REFERENCES