The Dissertation House Model: Doctoral Student Experiences Coping and Writing in a Shared Knowledge Community

INTRODUCTION

The United States’ commitment to the advancement of research in science, technology, engineering, and mathematics (STEM) fields and broadening participation of people from underrepresented groups in those areas is evident in the number of initiatives developed by federal agencies such as the National Science Foundation (NSF) and the National Institutes of Health (NIH) over the past several years. In a recent press release from the Council of Graduate Schools (CGS, 2015) on the findings from the Doctoral Initiative of Minority Attrition and Completion (DIMAC), the president of CGS remarked, “One of the striking lessons from this study is that the dissertation phase is particularly a critical time for students. Our country’s STEM workforce will lose a great deal of potential talent if we don’t help underrepresented doctoral students cross the finish line” (p. 2). As with the PhD Completion Project (CGS, 2009) and the DIMAC Project (Sowell et al., 2015), the CGS website boasts “CGS Best Practice initiatives address common challenges in graduate education by supporting institutional innovations and sharing effective practices with the graduate community. Our programs have provided millions of dollars of support for improvement and innovation projects at member institutions” (CGS, 2016).

Although CGS has labeled these practices “effective,” without rigorous theory-driven evaluations, these practices can simply be labeled exploratory, descriptive, and explanatory case studies. Evaluation of the impact of these supplemental professional development graduate degree–completion programs most often focuses on enrollment and graduation data of students who are directly funded by a particular grant-funded program, in comparison with overall student performance or performance before implementation of a given program. Many of the initiatives, such as establishing a doctoral student writing room, offering summer dissertation-writing residency fellowships, and hosting dissertation boot camps for students at the dissertation stage, have been described in the CGS Ph.D. Completion and Attrition: Policies and Practices to Promote Student Success (2010). While programs can present information on the successes of their student participants, it is challenging to address the counterfactual question that asks about levels of success without such programs or the levels of success of similar students who did not participate in the program.

According to Simpson (2012), graduate education relies heavily on mentoring as the “engine” for teaching, especially in the sciences. Still, many agree that mentoring and thereby student-learning experiences vary by field and even within graduate programs. Where some mentors are extremely hands-on, others are hands-off or extremely busy. The challenge, as Simpson points out, is to create sources of inputs that strengthen (but do not compete with) the mentoring relationship and connect students to existing resources in their fields. We believe that mentoring is not the sole responsibility of the research advisor. We have previously addressed the concept of the “it takes a village” approach to mentoring in The University as Mentor: Lessons Learned from UMBC Inclusiveness Initiatives (Bass et al., 2007).

The objectives of this study are threefold. First, we provide a description of the application of a model of collaborative learning and mentoring that is distinguishable from both the traditional dyadic mentor–protégé (master–apprentice and advisor–student) relationship (Yeatman, 1995) and the collaborative cohort model (CCM) of doctoral supervision (Burnett, 1999) that have been applied within a single discipline. Second, we employ a mixed-methods approach to evaluate the effectiveness of the intervention on retention and PhD completion. Third, we provide a qualitative look at the impact of the intervention on students’ perceptions of the collaborative-learning and mentoring experience. Although dissertation advisors maintain primary responsibility for the supervision of doctoral students’ research, the findings are suggestive of the advantages of using a collaborative, interdisciplinary approach as a supplement to—not a replacement for—the traditional independent dyadic advisor–student supervision model.

Moreover, for policy makers within NSF and the federal government, state and local officials, and decision makers in the educational community, our mixed-methods approach provides empirical evidence of the effectiveness of the Dissertation House model (DHM) for collaborative learning and mentoring during the doctoral dissertation–writing stage.

The Dissertation House (DH) is a program of NSF’s PROMISE–Maryland’s Alliance for Graduate Education and the Professoriate (AGEP) developed to help underrepresented minority (URM) graduate students in STEM transition from PhD candidacy to completion. The NSF’s PROMISE AGEP is a University System of Maryland–wide program that includes a large number of students at the master’s and doctoral levels from the life sciences and research areas that link to the biomedical sciences in other STEM disciplines (e.g., bioinformatics in computer science, biomechanics in engineering, and mathematical biology). This study looks at the effectiveness of the DH program’s primary goal of achieving greater retention and PhD completion. As a research partner of CGS PhD Completion Project, we established the DH as a promising practice in the area of mentoring and advising for URM students in STEM doctoral programs who have reached the dissertation stage.

Historically, the success of the cohort model of doctoral supervision over the master–apprentice model (AMM; Yeatman, 1995) has been well documented (Samuel and Vithal, 2011; Bista and Cox, 2014). Therefore, we will not explore the merits of the AMM in depth. Instead, we address the question of collaborative supervision and the development of communities of practice without the use of a predefined structured cohort. Moreover, in this study, we use the terms “knowledge community” and “communities of practice” interchangeably. According to Hoadley (2012), a knowledge community will have a learning goal at the outset. A community of practice occurs naturally and typically will not have a learning goal; learning will emerge depending on the community’s function and role in society (p. 292).

Researchers have documented the use of cohort models for supervision of doctoral degrees within non-STEM disciplines such as education (Norris and Barnett, 1994; Burnett, 1999; Mather and Hanley, 1999; Graduate Institute, 2006; Bista and Cox, 2014) and counseling education (Burnett, 1999). Lewis et al. (2010) provide a comprehensive review of the literature and identify three well-documented trends in PhD attrition that may lead a university to implement a cohort model: 1) poor completion rates of doctoral studies (Lovitts, 2001; CGS, 2004; Smallwood, 2004; Academy of Science of South Africa, 2010); 2) lack of support and feelings of isolation among doctoral students (Ali and Kohun, 2006, 2007; Unzueta et al., 2008); and 3) pressure on students, faculty, and administrators to meet academic expectations (Unzueta et al., 2008; Lewis et al., 2010; Bista and Cox, 2014). Most cohort models describe an organizational structure in which groups of students are bound together by a program of study within a single academic department. An inherent characteristic of cohort models is that students take the majority of their course work together (Miller and Irby, 1999; Barnett et al., 2000; Potthoff et al., 2001). Cohort models that are built on the Huey eight-factor framework (as cited in Potthoff et al., 2001) suggest that that there are eight dimensions to cohorts: 1) social interaction, 2) common mission, 3) group and individual learning, 4) cohesiveness, 5) collaboration, 6) academic success, 7) interaction with professors, and 8) retention. Bista and Cox (2014) already provided a comprehensive review of the literature on the advantages and disadvantages of cohort-based doctoral programs, and we refer the reader to that article for a more in-depth view. Burnett (1999) introduced the use of the CCM to the supervision of doctoral dissertations within one academic discipline, counseling education. In the CCM model, graduate students in counseling education who had completed their comprehensive exams enrolled in a semester-long faculty-guided support group. Results from the students’ evaluations showed that the structure and regular communication with faculty through meetings, emails, and a cohort newsletter were beneficial to their success (Burnett, 1999). Rather than restructuring an entire discipline or university system, a new model based on the DH program offers a hybrid approach by suggesting that the supervision of doctoral dissertations can be accomplished by capitalizing on the positive aspects of cohort-based models (Bista and Cox, 2014; Hartmann et al., 2015). We introduce here a more organic, self-selected, and diverse multidisciplinary model of doctoral dissertation supervision, the DHM, which expands the single-mentor and single-department AMM and CCM approaches to develop shared learning communities with multiple mentors across several academic disciplines. This approach focuses on the social shared learning experiences of doctoral students in a broader context, beyond their specific academic discipline.

Bowen and Rudenstine (1992) and Liechty et al. (2009) provide a comprehensive literature review of nationwide trends of doctoral attrition and focus on both the barriers and facilitators that affect dissertation completion. They note that much of the attrition (17%) from doctoral programs occurs when students are at the “all but dissertation (ABD)” stage, described as the time after the course work is finished but before and during the dissertation-writing process (Di Pierro, 2007). Challenges in the area of doctoral writing range from the unstructured nature of the dissertation stage (Davis, 2000; Hockey, 1994) and social isolation (Ali and Kohun, 2007; Jones, 2013), to the inability to make intellectual progress when the student becomes stuck (Johnson, 2015). From the faculty advisor’s perspective, the greatest challenge to successfully completing a dissertation rests on students’ lack of knowledge about how to plan, implement, and write up a large-scale independent project (D’Andrea, 2002). Liechty and colleagues suggest that “the dissertation phase of a doctoral program is a high-risk period for attrition and that targeted interventions at this juncture are warranted” (Liechty et al., 2009, p. 482). The same authors document some of the interventions instituted in universities in the United States, Australia (Burnett, 1999), and New Zealand (Johnson and Conyers, 2001). They provide a description of the CCM model of supervision by Burnett (1999) and support group programs with counseling services (Johnson and Conyers, 2001), and they even include institutional-level support programs such as the “campus-wide workshops for students on motivational strategies on time-management relevant to the dissertation” (Liechty et al., 2009, p. 490) offered at the University of Maryland, Baltimore County (UMBC). Despite a comprehensive review of both the challenges and the interventions, the authors did not document any empirical evidence regarding the effectiveness of the interventions mentioned. At the end of the review, they find that, despite the growing body of research on the factors that affect overall PhD completion, few studies focus on the dissertation process. In addition, the authors directly called for “rigorous theory-driven evaluations, including experimental designs, to determine the effectiveness and “active ingredients” of specific individual, relational, and departmental/institutional interventions to promote dissertation completion” (Liechty et al., 2009, p. 494). An empirical evaluation of the effectiveness of a dissertation-writing initiative, the DHM of doctoral dissertation supervision that we introduce here, answers that call and addresses the paucity of literature on the dissertation process.

Much of the literature on the dissertation process has provided case study explorations of dissertation-writing interventions that include dissertation-writing boot camps (CGS, 2010; Simpson, 2012; Peters et al., 2015; Sowell et al., 2015). At best, the authors provide examples of promising practices to identify the institutional patterns and support structures that should enhance the dissertation process. This study extends the existing literature on this topic by providing a rigorous, theory-driven evaluation of one institutional-level intervention at the UMBC. The DH promotes dissertation completion with underrepresented students, many of whom are conducting research in the biomedical sciences, across academic departments. We begin with a detailed description of the DHM, followed by a propensity score analysis of the effectiveness of the DHM, and conclude with a qualitative look at the students’ feedback on their experiences with the DHM.

THE DHM

In an attempt to maximize social support for doctoral students, PROMISE AGEP introduced the DHM in 2006 to a group of African-American and Hispanic graduate students from several academic disciplines who were working on either their master’s theses, PhD proposals, or dissertations. A faculty member/administrator, who is also a member of an underrepresented group, served as a primary PhD facilitator, coordinator, and mentor (hereafter referred to as the “dissertation coach”) to the group of underrepresented students. The first DH pilot was held in 2006 during a 3-day weekend retreat in a rustic, remote, and rural mountain conference center in Berkeley Springs, West Virginia, with limited access to the Internet and cell phone reception. The name “Dissertation House” was based on the students’ cabin-style living quarters, with students working on their dissertations grouped together in selected cabins.

The DH project used as a foundation the successful Scholar’s Retreat at the University of Colorado at Denver (described in Smallwood’s 2004 article in the Chronicle of Higher Education titled “A Week at Camp Dissertation”), which provides the opportunity for intensive, focused, distraction-free, supervised writing time so that writers (with or without a PhD) can make significant progress toward the completion of their dissertations, theses, or writing projects. Although the DH began as a full-weekend experience in a remote area, current implementations take on different formats. These formats include DH sessions that are concurrent with multiday conferences at hotels, sessions designed for 2–4 days on campuses, and/or sessions that take place online. All versions of the DH facilitate students’ progression through the dissertation-writing process by providing the professional consultation, guidance, and support necessary for scholarly research and writing. The program focuses on reducing isolation by having students work as a cohort toward shared goals. Since its inception, more than 200 graduate students from three Maryland universities have participated in the DH; the majority (n = 154) of participants attended UMBC.

Currently, the most common format is a 4-day session on a university campus (DH on campus). Students come to campus for consecutive days and stay from 9:00 am until 5:00 pm. Orientation on the morning of the first day includes introductions and goal-setting activities. Students share their goals in three different formats: written on posters that are affixed to the wall and viewable by all, orally through a shared goals session, and virtually via the DH website. In their introductions to the other members of their DH cohort, students must describe the project that will be tackled during the session and their goals for the full DH session. The facilitator provides feedback on the feasibility of each participant’s listed goals. Each student then logs into the DH website to post the revised, measurable goals for the day. The public declaration of the goals introduces positive peer pressure and accountability to the group and to the dissertation coach.

Each day, DH sessions include goal setting, two minilectures/professional development exercises, one-on-one coaching, and at least 5 h of uninterrupted writing (2 h in the morning and 3 h in the afternoon). Writing activities in the DH include writing sections of the proposal or dissertation, working on data analysis or data coding, reading and summarizing journal articles, writing up experimental results, deciphering feedback from their advisors, and addressing necessary revisions for resubmission. Overall, students are either working on their own while seated together in one room, or they are in one-on-one meetings with the dissertation coach, who serves as the facilitator for the DH.

The one-on-one meetings with the dissertation coach are confidential discussions focused on anything that prevents a student from writing. Discussions can involve personal issues at home, situations that occur in the laboratory, difficulties with graduate school, or lack of vision for managing the entire project. It is rare that a student is looking for an editor or feedback on the writing itself. The dissertation coach focuses on big picture items: moving the student from a stagnant position, slow or nonexistent writing, lack of clarity, or indecision. Since these DH discussions occur outside the student’s academic department, with the dissertation coach serving as an external mentor, discussions can gravitate from research or writing to honest issues about what might be stalling work on the dissertation.

Students receive additional support via 30-min minilectures after lunch and in the morning. These minilectures provide resources, tips, and strategies for PhD completion and are delivered by the director of the DH program, the dean of the graduate school, the university counseling services, the writing center, or others. Guest DH alumni, whose past circumstances mirrored current participants’ challenges, return to the DH to support current students. Topics may include managing stress, preparing for a dissertation/proposal defense, managing the relationship with your advisor, writing a literature review, updates on new policies, guidelines and graduation requirements, time management, and public speaking. These additional supports or minilectures provide students with information that may not be readily available within their academic departments.

The DH provides breakfast, lunch, and snacks at the same time every day and adheres to a strict schedule for the duration. This schedule helps students understand the integrity of a writing schedule and the value of time management. Providing food helps create ideal conditions for working on the document without the distractions of responsibilities such as preparing meals, cleaning, and washing dishes. The last DH day ends with a roundtable discussion/assessment of what students have learned and what tools they will use in the coming weeks when they are not writing under these ideal conditions.

The DH occurs during winter or summer breaks when the students most likely will not have other academic or teaching responsibilities. Because they are on campus, the participants can check in on their experiments either before the 9:00 am start of the DH, at lunch hour, or after the DH is finished at 5:00 pm. When time permits during the DH, students are able to take time away to meet with mentors on campus. After the workshop ends, some students elect to continue communicating online with the dissertation coach or continue to set goals and keep daily accountability via the DH website.

Using the framework established by Hoadley (2012), the DHM could be considered a shared knowledge community using his description of a “model of learning—learning in which people, through a process of legitimate peripheral participation, take up membership and identity with a community which serves as the home of the shared practices” (p. 299). In an attempt to maximize support for its doctoral students without changing the overall structure of any graduate programs at UMBC, the DHM embraced the traditional AMM and supplemented the supervisor’s role with a concurrent voluntary support mechanism for students based on a CCM. The traditional learning AMM model characterizes the master–apprentice relationship as the central component of the learning process. Specifically, for graduate students who are developing a dissertation proposal or writing a dissertation, the dissertation chair/advisor/supervisor’s involvement in the dyadic process is essential. During the writing process, the document is closely monitored to its completion. Student experiences with the DH are not in competition with the role of the advisor, but act as other positive components in the learning community available for student success.

Eligibility

The DH is free and open to graduate students from all disciplines. The applicant must be an advanced PhD graduate student enrolled either full-time or part-time at the institution where the campus event is being hosted. Graduate students must have already selected a research topic before they can participate in the DH (a requirement that was instituted early on, because we found that, without it, students could not fully benefit from the experience). Thus, each applicant submits a written application, documenting his or her current stumbling blocks along with a brief synopsis (maximum two pages) of the research project he or she wishes to work on during the DH period.

Selection Process

Up to 18 students are selected to participate in each DH. Applicants are preferentially selected if they 1) plan to finish the PhD within the next 6 months; 2) are working on a dissertation rather than a proposal; and 3) meet the application requirements and deadline. Early on, we found that students who already had a dissertation topic, had started writing the proposal or dissertation, were willing to commit to attend the entire 4 days of the DH, and were willing to write the two-page application proved most able to benefit from the services offered at the DH.

ASSESSING THE DHM

This research draws on two sources of data: 1) institutional data from UMBC (N = 1890) for PhD cohorts from 2000 to 2012 (see Supplemental Material Table A2), some of whom have participated in the DH (n = 154) and others who have not (n = 1736); and 2) written evaluations from DH participants who were students enrolled in three AGEP Alliance institutions from 2007 to 2013 (N = 267). The UMBC PeopleSoft database contains information on the demographic and academic data for each participant, including race, gender, citizenship, graduate program, cumulative grade point average, date entered, and date departed.

Study 1: Retention and Graduation

Study 1 addresses the primary research question as to whether an intervention such as the DH was able to meet its main objective of helping students persist until they received their PhDs. As with many graduate program interventions, the challenge has been to answer the following question: Would individuals who did not participate in the broadening participation programs also likely achieve similar results?

We did not randomly assign students to the DH, so we are unable to address this question without using some statistical techniques. In other words, in a true randomized experiment, students who participated in the DH would be considered the treatment group and would have been randomly assigned to the DH. The students who were not assigned to attend would be considered the control group. The voluntary nature of program participation complicates the issue of identifying the true “treatment effect,” that is, the effect of participation. Because students apply and self-select to participate in the DH, a failure to adequately control for preexisting differences between retained and graduating students leaves open the possibility that preexisting characteristics rather than DH may be the cause of the postintervention outcomes. Simply comparing the percentages of graduates in each group or comparing the mean difference in time to degree does not adequately tell us much about the effectiveness of the program. Table 1 shows that 42% of the non-DH group graduated compared with 76% of the DH group. Likewise, the mean time to degree was calculated to be M = 5.4 (SD = 1.89) and M = 5.8 (SD = 1.90), respectively.

Social scientists prefer regression methods for estimating intervention impacts using comparison group data. However, the use of propensity score matching (PSM) as originally proposed by Rosenbaum and Rubin (1983, 1985) has become increasingly popular over the past few decades (see Thoemmes and Kim, 2011), notably in the areas of education (Hong and Raudenbush, 2005; Wu et al., 2008; Hughes et al., 2010) and evaluation research (Hong and Raudenbush, 2005; Hughes et al., 2010). As in the medical literature, PSM is a tool for approximation of a randomized trial and for reducing selection bias in observed studies. PSM is a nonexperimental evaluation method used to compare what would have happened to similar graduate students who did not participate in the DH. In this case, PSM techniques are designed to measure the impact of participation among similar individuals for whom the only difference between them is the treatment outcome, which is retention or graduation.

Heinrich et al. (2010) note that, to determine whether matching is likely to reduce selection bias, it is crucial to understand under what conditions it is most likely to work. First, the variables for both treated and untreated must be observable to the researcher (p. 15). This assumption is known as the conditional independence assumption, meaning that “the potential outcomes are independent of the treatment status, given X. Or in other words, after controlling for X, the treatment assignment is as good as random” (p. 16). Second, the common support condition must be met, meaning that, to calculate the difference in mean outcomes for each value of X, there must be a positive probability of finding both a treated and untreated unit to ensure that each treated unit can be matched with an untreated unit (pp. 15–16). As with many causal inference methodologies, PSM also requires a large sample size to gain statistically reliable results. When the treatment group in relatively small, this is particularly true for PSM due to the tendency to discard many observations that do fall under the common support.

PSM balances out the covariates across the treatment and control group based on a single dimension. The single dimension is the probability that a unit in the combined sample of treated and untreated units receives the treatment, given a set of observed variables. If all information relevant to participation and outcomes is observable to the researcher, the propensity score (or probability of participation) will produce valid matches for estimating the impact of an intervention. Therefore, rather than attempting to match on all values of the variables, cases can be compared on the basis of propensity scores alone (Thoemmes, 2012, p. 7). Zanutto (2006) suggests that, when compared with regression analysis, “this ability to easily check that the data can support comparisons between the two groups is one of the advantages of a propensity score analysis over a regression analysis” (p. 81). According to Titus (2007), unlike the more popular ordinary least-squares regression, PSM addresses the issue of self-selection bias and allows for a decomposition of treatment effects into three different components: 1) the average treatment effect (ATE), which measures the mean impact of the program across all individuals in the population; 2) the average treatment effect on the treated (ATT), which measures the impact of the program on those who participated in the treatment; and 3) the average treatment effect for the untreated, which evaluates the impact that the program would have had on those who did not participate.

This study focuses on a quantitative analysis of the institutional data provided to us by the Office of Institutional Research, Analysis and Decision Support at UMBC. The data set contains demographic PhD graduate student information tied to measures of PhD completion and attrition for cohorts of graduate student enrolled from 2000 to 2012. DH participation/attendance lists matched to these data confirmed the number of distinct UMBC DH participants (n = 154).

Table 1 presents a distribution of the doctoral student cohorts entering UMBC from 2000 to 2012. More importantly, it presents a comparison of the baseline characteristics between DH and non-DH participants at UMBC. To be clear, students starting in 2000 would have been in their seventh year when the on-campus DH program started in 2007 at UMBC. The study population (N = 1890) consists of doctoral students, a majority (87%) of whom were enrolled in STEM programs. Each student was defined as either a PhD graduate (n = 853), a PhD continuing student (n = 451), or a student who left his or her graduate program and/or the university (n = 586). These categories are mutually exclusive; students who transferred from one program to another within the university were counted as continuing. The majority of students were well distributed across STEM programs (17.5% engineering, 10.8% life sciences, 40.7% physical sciences, 24.5% social sciences). The physical sciences category includes computer science, information science, and human-centered computing. A small percentage of doctoral students (6.5%) were in the humanities (only one of the 24 doctoral programs at UMBC is considered non-STEM based on NSF classification). Approximately 14% of the students belonged to a URM group (9.7% black/African American, 1.3% Native American, 2.5% Hispanic). Males constituted a slight majority at 52%.

Our first task was to calculate propensity scores and establish a control group. We used binary logistic regression to generate propensity scores predicting the likelihood of participating in the DH (see Supplemental Material Table A.1). Afterward, to reduce the effects of confounding factors, we matched each DH participant to a nonparticipant based on the probability or likelihood that the non-DH student would have participated in the program. The specified logistic model is designed to capture the propensity to participate in the DH program; DH participation was the dependent variable coded 1 for DH participation and 0 for nonparticipation. The included covariates were based on the selection criteria, which gave priority to advanced graduate students (cohort year = categorical), STEM (dichotomous; 1 = STEM, 0 = non-STEM), gender (dichotomous; 1 = female, 0 = male), and URM students (dichotomous; 1 = underrepresented, 0 = not underrepresented). Also included among the covariates was a series of dummy variables (life sciences, humanities, physical sciences, engineering, and social sciences) to account for differences in completion rates by broad fields. Note that, when an outcome is measured using propensity scoring, the covariates must be common across both groups (participants and nonparticipants). Using student information data, we provide information on the observed characteristics for both groups. Matching by propensity scores assumes that all differences between individuals affecting treatment and outcome can be captured by observable pretreatment characteristics.

After calculating the predicted probabilities, we conducted a matched analysis to estimate the program effect. Researchers have several balancing techniques (conditional, stratified, matching) to choose from. Herbert and Yao (2009) provide a review of each technique, along with the advantages and disadvantages of each option. We selected matching because of the interest in comparing the likelihood of graduating between doctoral students who participated in the DH and those who did not. Without going into the subtleties of each technique, we chose the simplest case of matching (1:1), wherein one DH student is matched to one non-DH student with similar characteristics. While there are many possible matching techniques that could have been selected, we used Dattalo’s (2010) on-line syntax for 1:1 matching (nearest-neighbor matching) in SPSS. The large sample size of the nontreatment group allowed for matching without replacement, which means that one score in the treatment group could match one person in the nontreatment group with the nearest matched score. For a more detailed discussion on PSM using SPSS, see Thoemmes (2012) or Dattalo (2010).

To assess our matching results (see Supplemental Material Appendix 3), we calculated standardized differences for each variable in the logistic model (Austin, 2011). Tables 1 and 2 present a comparison of the distribution covariates for both the control group (non-DH participants) and the treatment group (DH participants) before and after being matched, respectively. In both tables, the absolute standardized difference (ASD) column provides a measure of the balance of covariates. Researchers suggest that ASD values greater than 10% indicate the presence of imbalance (Linden et al., 2005; Austin, 2011; Rankin, 2014). Table 1 provides a comparison of the distribution of DH participants to the entire distribution of doctoral students in the same cohorts. For example, 14.9% of the DH participants majored in life sciences compared with 10.54% of non-DH participants. Thus, the standardized difference is 0.11 (or 11%), which suggests that the distribution is imbalanced. After matching, the results in Table 2 show much lower absolute values on each covariate. For example, the same covariate for life sciences now has an ASD value of 0.08 (or 8%), reducing the selection bias by 32%. The last columns in Table 2 show what percentage of the selection bias is being reduced by introducing matched data.

Once the covariates’ balance had been established, we measured what researchers in the PSM literature refer to as the ATT. As noted earlier, the ATT is the average gain from treatment for those who actually were treated. The results in Table 2 show that ∼76% of DH participants graduated with a PhD compared with 40% of non-DH participants. To measure the ATT, we used the relative risk (RR) ratio, which assesses the difference in outcomes for DH participants and non-DH participants (see results marked with an asterisk in Table 2). For our purposes, the RR tells how much more likely a student is to graduate based on whether he or she participated in the DH. An RR of 1 indicates that the outcomes did not differ in the two groups. Our calculated RR of 1.92 indicates that the DH participants had a probability of graduating at least 1.92 times that of nonparticipants (the control group). In other words, participating in the DH increases the likelihood of graduating by 92%. Similarly, we calculated a RR for retention to be 1.64, which increases the likelihood of retention by 64%.

Whereas RR reflects a ratio between two conditions, we wanted to know what the RR increase is for students who participated. In other words, what is the effect of the treatment? To find this answer, we look to the attributable risk (AR), calculated as the ratio or percentage of an event in one group minus the same within a comparison group. Thus, the AR for this study was 0.479 for graduation and 0.392 for retention. In sum, 47.9% of graduation and 39.2% of retention is attributable to the DH experience.

Limitations of PSM

As with any evaluation using observational data, or “ex post facto data” (after the fact data), we highlight some common limitations. First, in the absence of experimental data, the researcher presumes that all biases and confounding variables have been adjusted for in the model. However, this assumption cannot be truly tested absent a randomized study; biased estimates are the norm for observational data. Although random assignment is considered the gold standard research design tool for evaluation, it is not always feasible, politically expedient, or ethical to implement.

Similar to omitted variable bias in regression analysis, one limitation specific to PSM is that the researcher cannot balance the groups based on unobserved (unmeasured) factors. For example, we do not have measures of household living arrangements or writing anxiety, nor do we have measures of other responsibilities such as employment, children, or teaching. These factors could influence likelihood of participation in the DH. Nonetheless, these unobserved factors could occur randomly in both groups.

According to the CGS DIMAC Report, women and URM students have high levels of attrition from STEM doctoral programs. Some might argue that students with greater motivation to finish are more likely to participate in the DH. Thus, unmeasurable variables such as motivation and grit might be the “hidden bias” missing from this study. From experience, we find that the most motivated URM students do not rush to sign up to participate in the DH. The opposite is true; application essays and summative program evaluations indicate that those who are “stuck” or struggling with a lack of motivation tend to participate in the DH. All the same, Heinrich et al. (2010), among others, suggest that the programs’ more explicit eligibility criteria should be used as variables under consideration for matching purposes.

Second, for researchers using ex post facto data, the analysis could be hindered by missing data. The deletion of cases with at least one missing response (e.g., listwise deletion) could result in a further reduction of available cases. Other issues of residual bias, precision, and lack of independence across units are beyond the scope of this study and have been addressed by Hill (2008).

Study 2: Student Qualitative Perspectives on Social Support and Coping Assistance

To understand the role of social support in completing a doctoral degree, the authors culled information from open-ended questions on 267 evaluations completed by students who participated in the DH program. We analyzed answers of three open-ended questions to explore the shared experiences of doctoral students involved in DH and to investigate whether these successful experiences could be understood using the coping assistance conceptual framework (Thoits, 1986). Beyond gathering basic demographic and programmatic information, the evaluation had a series of yes/no questions followed by open-ended questions that asked graduate students about their experiences in the DH:

1. Did the Dissertation House experience help you to progress with your dissertation? If yes, how significant was your progress? If no, what could have been done differently?

2. Was having the “Dissertation House” a good use of funding? If yes, please explain. If no, please provide ideas for better ways to use the money that was spent on this experience.

3. What are suggestions for improvement that will reduce your “time to degree”?

The participants in the study were graduate students from three University of Maryland campuses: University of Maryland, College Park (n = 87), University of Maryland, Baltimore (n = 19), and UMBC (n = 157). Based on the number of evaluations returned, Table 3 presents a profile of the DH participants that includes cohorts who participated in DH sessions across Maryland between 2007 and 2013. Whereas the other institutions in the PROMISE AGEP sponsored a DH sporadically or once a year, UMBC sponsors the event twice or sometimes three times a year based on funding availability. At UMBC, beyond the regular twice-yearly DH, we introduced in October of 2010 the Employee DH for employees of the university and nontraditional students who were working on their dissertations and seeking help and time to write. This explains the variation in number of participants from each campus. The majority of participants who returned evaluations were women (79%) and members of an underrepresented group (55%). These percentages reflect a convenience sample to provide contextual information from those who experienced the DH. That the majority are women and come from underrepresented groups reflects targeted groups the DH was designed to attract. For the qualitative portion of the study, our findings reflect the experiences of DH participants and do not necessarily generalize to the experiences of nonparticipants. It is important to note that the numbers in Table 3 are based on the number of evaluations returned, not necessarily the number of students who participated. Moreover, because students are allowed to participate more than once, and evaluations are anonymous, these numbers may not reflect unique individuals.

These evaluations represented doctoral student assessments from a variety of academic disciplines that could generally be classified under STEM (n = 133, 49.8%) and non-STEM (n = 59, 22.1%) for those who specified a discipline on the evaluation form. Twenty-seven percent of students did not account for their academic disciplines on the evaluation forms. The respondents were primarily female (n = 209) with a small group of male (n = 56) participants. The majority (55%) of the respondents were members of underrepresented racial minority groups (black/African American = 121, Hispanic = 24). Results from evaluations show that nearly all the students affirmed that the DH was helpful in making progress with the dissertation (98.1%), that it was a good use of funding (98.9%), and that they would recommend having the DH again (99.2%). (See Table 3.)

The first question, on how the DH experience helped progress with the dissertation, resulted in the largest number of responses. One of the most important themes to emerge from the evaluations was that DH helped students work through a state of writing paralysis, lack of direction, or lack of any significant progress. Progress means different things to different students; however, most of the focus is either on time management, project management, or writing. To answer the question, one student wrote, “I had nothing on paper, I now have an overall outline and I feel that I will be able to get finished now. This workshop has helped me become ‘unblocked.’ I have been at UMBC for 4 ½ years and was in danger of just quitting.” An important outcome of the daily goal-setting exercise is learning how to set measurable goals and celebrate small daily successes. One student who had been stuck for 2 months said, “It was good to see others struggling and I started making more realistic expectations for myself. The advice was invaluable.” In response to this same question, another student shared a similar sentiment about making significant progress: “I have achieved more this week than I would have in a typical month. Having professionals on-hand to assist me, and having structured time to do the work has been invaluable.” Commenting on the shared experience, one student noted that “the opportunity to share our progress with other students” helped a lot and singled out the on-site coach as a valuable aspect: “It helps a lot for guidance for sure. The positive encouragement while we are working together brings a lot of benefit.”

Although issues such as writer’s block might seem to have an obvious solution, graduate students are sometimes reluctant to reach out and ask for help. Issues with writer’s block might be research related, related to the anxiety of writing itself (John-Steiner and Mahn, 2003) or perfectionism, or related to the fear of being wrong. The DH provides coping assistance by providing a safe place and a shared knowledge community of doctoral students all facing the similar challenge of making progress on the dissertation. In AMM, the struggling graduate student might be forced to acknowledge to his or her advisor that he or she is stuck or uncertain about what to write. AMM relies on the advisor’s ability to motivate and encourage the student to write. Coping assistance theory (Thoits, 1986) argues that the student might be unwilling to ask for help under this circumstance for a number of reasons, including fear that the advisor might be an unsympathetic helper. That advisors had to write their own dissertations in the past does not provide a shared situational experience whereby graduate students might willingly seek help with dissertation writing. The advisor is seen as an expert in the student’s research area, not an expert in writing or dealing with emotional issues like stress, family troubles, or burnout. Moreover, the student might not be able to identify the underlying shared stressor unless the advisor had revealed his/her own struggle with writing the dissertation. Although students are from different disciplines, they identify with one another.

Coping assistance theory (Thoits, 1986) suggests that misery loves company. Thus, the second theme to emerge was social interaction and social support, especially when students were able to meet students from other campuses in the PROMISE AGEP at a weekend retreat. Sometimes students in STEM from URM groups are “the only one” in their departments or colleges. In addition to coping assistance based on a shared experience, students suggested that DH made plain what is otherwise a tacit understanding between the faculty and the student about what he or she should be doing to complete the degree. A student commented by writing, “Dissertation House provides students with access to information, resources, tips etc. that is usually not readily available to a student. There seems to be an assumption among faculty, across disciplines and campuses that student should just know ‘these things.’ … And because most participants are so productive during the program [sic] they are more likely to continue to utilize these techniques in the DH to achieve or accelerate their progress.”

Many students commented that the program provided them with valuable tips about organizing their dissertations. While students may have presented parts of their dissertation topic at a conference or prepared chapters for submission, perceiving it as one cohesive document can become overwhelming without some form of organization, be it for the references or the document itself. A student wrote, “I did not have my thesis organized into one location or structured into a format resembling a thesis, now I have a solid first draft.” Further assistance came in areas beyond the nuts-and-bolts writing phase. Another student said DH taught him useful tips “about defending, [and] interacting with the committee.” One part-time student wrote, “I am now more motivated to write a contract with myself for finishing and creating a schedule.”

Much of a STEM student’s time in graduate school is spent in the lab conducting experiments, and many advisors might think that conducting successful experiments is the most time-consuming and challenging aspect of graduate research. The underlying assumption is that once the student is able to get successful results, the student will be able to write up the results quickly and turn in a comprehensible document in the form of a publishable paper or doctoral dissertation. The challenge of writing itself is rarely considered a hindrance to getting results. Nonetheless, students in the DH provide feedback about the writing process and the lack of time to write. The DH provides a space to write, but support from the student’s advisor/supervisor provides the student with the opportunity to write. Several students discuss the benefits of getting time away from the lab to concentrate on writing. One student wrote, “This program was very significant to me in helping me realize how important writing every day is to the completion of my degree.”

Responses to the question concerning whether the DH was a good use of funding echoed many of the positive benefits that have already been discussed earlier. There were no specific suggestions on better use of the funding; however; the main theme emerging to justify the use of funding is that DH helps students complete the degree. One student in particular wrote, “Providing funding for a dissertation house was a great idea. The outcome of ensuring all PhD students that participated in the program will complete their dissertation is priceless.” A subtheme is the time the DH is estimated to save in completing the degree. Another student concurs: “I was able to do in one day of DH what it would have taken me at least a whole week to accomplish.… it helped us to speed up and clarify our thinking so we can finish ASAP ‘an investment in our progress.’” Others commented on the personal value of a program that helps them achieve their goals. A student who is working in another state while completing her dissertation summed up her experience by writing: “It has helped to know that I am part of a ‘group.’ Knowing that there are other students going through the exact same experience has made me feel that this is doable! I know I can get that PhD!! I strongly recommend Diss House to everyone! I even flew 3000 mile and paid a lot of money to be here, flight, hotel, rent-a-car, food etc) and it was absolutely worth it.”

Information on what students think could reduce their time to degree provides the staff with new ideas that can be incorporated into future DHs. Some students wanted the same services offered at the department level (Golde, 2005), and others wanted more of the program, from extended time each day to offering the program on a more frequent basis. We have since provided the opportunity for students who want to stay in the room after the mentored program ends at 5:00 pm. We found that students established social and collegial networks that extend beyond the 4 days of the DH. Small groups of three to four students often continue to meet regularly on their own, (on or off campus) to work on their dissertations until completion.

CONCLUSION

The challenges that students face in completing the dissertation often extend beyond department and disciplinary boundaries. Whereas the CCM introduced doctoral dissertation supervision in a collaborative-learning environment with several faculty mentors in a single non-STEM discipline, the DHM extends this model across several disciplines by introducing a model of doctoral dissertation supervision that involves an external dissertation coach and multiple mentors. Unlike the CCM, this multidisciplinary approach of doctoral dissertation supervision preserves the traditional master–apprentice relationship between faculty and students within academic departments while providing an additional support mechanism through interdisciplinary collaborative cohorts. Social isolation is common at the dissertation-writing stage (Golde, 1998; Burnett, 1999; Lovitts, 2001; Ali and Kohun, 2007). However, for underrepresented students working in laboratories, social isolation might be the norm for their entire graduate careers. Hortulanus et al. (2006) described social isolation as a “lack of meaningful relationship” that negatively impacts an individual’s quality of life. Ali and Kohun (2007) suggest that, in graduate school, “meaningful relationship” might refer to a social contract among students as well as with faculty members. The DHM builds on lessons learned through the PROMISE program (Tull et al., 2012) on the importance of creating an environment of inclusiveness that promotes social support and connections. Moreover, the ongoing support of the dissertation coach and the online DH website blog help to reduce the sense of isolation well after the DH event ends.

Given a dearth of scholarship on both PhD completion at the dissertation phase and the lack of rigorous evaluation of programs designed to enhance STEM education for URM doctoral students who are at the dissertation phase, this article seeks to provide some insights into the potential for mixed-methods approaches. First, using a counterfactual analytical framework, this investigation revealed how the use of PSM techniques can be used by institutional and other investigators to help address growing concerns about the lack of “evidence-based” policies and practices in higher education. The study described here provides support for the DHM as an effective intervention that combines one-to-one doctoral dissertation supervision with an interdisciplinary support learning community for students from underrepresented groups.

Second, growing concerns about constrained resources and the lack of empirical evidence to justify institutional support to sustain some policies and practices in higher education that might have been established with grant funding require a mixed-methods approach. This approach provides researchers with a wider array of analytical tools for conducting comparison studies with observational data. Whereas the results of propensity score analysis indicate that those who attended the DH had higher rates of graduation, the qualitative results provide further support for the effectiveness of the DHM, the value that students place on the DH experience, and the impact it has on both their progression and satisfaction.

ACKNOWLEDGMENTS

REFERENCES