Examination of Admissions Criteria in Higher Education Graduate Preparatory Programs| Tolman, O’Halloran, Meert

written by:

Steven Tolman
Georgia Southern University

Kimberly O’Halloran
Widener University

Abby Meert
Grinnell College

Introduction

The field of student affairs in higher education highlights a noble commitment to the growth and development of the whole person (Gansemer-Topf, Ross, & Johnson, 2006; Komives & Woodard, 1996). This commitment is not limited to student development, although in a helping-profession it can often take precedence. For aspiring and current student affairs professionals, a commitment to holistic individual development can be just as intrinsically and extrinsically beneficial. One avenue for stimulating whole person growth is the pursuit of a graduate degree; a transformational experience also has the ability to increase an individual’s potential earnings (Baum, Ma, & Payea, 2013; Zhang, 2005).

As of fall 2019, approximately 19.9 million students enrolled at an institution of higher education in the United States; 16.9 million of which enrolled in an undergraduate program while 3 million pursued a graduate degree (National Center for Education Statistics [NCES], n.d.). In an effort to attract more students to higher education, institutions have continued to expand their graduate program offerings both in number and subject matter. As a result, the number of graduate degrees conferred has increased substantially since the late twentieth century, prompting many to call them “the new bachelor’s degree” (Blagg, 2018, p. 1). For example, in the academic year 2000-01, postsecondary institutions conferred approximately 474,000 master’s degrees and 120,00 doctoral degrees; of those, 134,300 were related to education (NCES, 2019). The most recently published data indicate that in the 2016-17 academic year, postsecondary institutions awarded 805,000 master’s degrees and 181,000 doctoral degrees; of those, 158,700 were related to education (NCES, 2019). Applications to graduate programs increased steadily at a 4% average annual rate from Fall 2007 to Fall 2017. Graduate enrollment is projected to increase annually by 3% through 2028 (McFarland et al., 2019).

The American College Personnel Association’s (ACPA) online “Graduate Prep Program Directory” lists 127 higher education graduate programs across the United States in educational leadership, higher education administration, student affairs, and counseling, among others (n.d.). These programs in particular face the challenge of recruiting from a diverse pool of prospective students in the absence of true undergraduate feeder programs. For example, while many biology graduate students will possess undergraduate biology backgrounds, higher education graduate students come from diverse educational backgrounds as it is not a typical undergraduate degree option. The same can be said of prospective Ph.D. or Ed.D. candidates; although most professionals in higher education have a master’s degree related to education, these degrees often vary in focus. For example, a candidate may have a master’s degree in “Higher Education Administration,” “Student Affairs,” “College Student Development and Administration,” “College Student Personnel,” “Educational Leadership,” “Global Higher Education,” or “College Counseling,” to name a few (ACPA, n.d.).

As graduate education continues to increase in popularity (Blagg, 2018; Darolia et al., 2014), admissions committees for degree programs in higher education have been tasked with identifying candidates who are more qualified for success with relatively limited applicant information. In good humor, Litwiller (2017) posed a valid question: “. . . what criteria are they using, anyway? Grades? Good looks? Tea Leaves?” (p 43). While this process may seem mysterious to those outside of the decision-making process, admissions committees generally use the following as common indicators of graduate success: undergraduate grade point average (UGPA), undergraduate institution, degree field, standardized test scores, letters of recommendation, personal statements, academic writing, interviews, and research experience, although this list is not exhaustive (Chari & Potvin, 2019; Darolia et al., 2014; Kuncel et al., 2014; Posselt, 2016). However, these admissions criteria may or may not predict the extent to which a student will excel or struggle in graduate school (Darolia et al., 2014).

Review of the Literature

For many years, test scores such as those from the Graduate Record Examinations (GRE) have been widely influential in the selection process of potential students for graduate programs. First introduced by the Educational Testing Service (ETS) in 1949, the GRE has become a mainstay in the evaluation of applicants to many graduate school programs (Educational Testing Service, 2014; Miller & Stassun, 2014). According to ETS, “GRE tests are best known for their role in helping faculty committees select applications for admission to graduate programs” (Educational Testing Service, 2018, p. 4). ETS notes that their assessment has been shaped by over 70 years of data to create an examination that is a fair and reliable measurement of a person’s knowledge (Educational Testing Service, 2018). However, data suggests that disagreements have developed regarding the GRE’s ability to predict the success of future graduate students leading some to suggest it may be best suited as one measure amongst many (Orlando, 2005).

Kuncel et al. (2001) evaluated both GRE scores and UGPA as predictors of graduate school success for over 80,000 graduate students, and they found that both measures were generally valid predictors of graduate GPA. Moneta-Koehler et al. (2017) also considered the GRE as a tool in predicting success of doctoral students in the bio-medical field, and they found that the exam featured several limitations. While Moneta-Koehler et al. (2017) provided evidence that the GRE was moderately predictive of student grades, the data showed a failure to predict student research productivity and overall student progress in the program as measured by qualifying exam results, graduation and time to defense, and the number of presentations and publications.

Both Kuncel et al. (2001) and Moneta-Koehler et al. (2017) found the GRE to be a valid predictor of first semester GPA, which could be due to the GRE’s assessment of “characteristics such as test taking skills, attention, time management, stress management, test question comprehension, and reviewing one’s work” (Moneta-Koehler et al, 2017, p. 14), skills that do not encompass the critical thinking, research, and writing skills necessary for continued success in many graduate programs (Kuncel et al., 2001; Moneta-Koehler et al., 2017). Sternberg and Williams (1997) produced similar results in their study of graduate psychology students, for which they found the GRE to be useful in predicting first semester GPA but fairly limited in its effectiveness at predicting other measures of progress or success. In general, a majority of studies seem to advise admission committees to limit the value they place on the GRE in favor of a more wholesome admissions process (Fedynich, 2017).

ETS also makes a couple of cautious claims regarding the application of GRE scores. They state that GRE scores, “do not and cannot offer insight about all of the qualities that are important in predicting academic success,” and, “The scores need to be interpreted carefully because… they are not exact measures” (Educational Testing Services, 2018, p. 5). In fact, there are other arguments that the GRE may even have bias against certain demographics of people that may be statistically more likely to score lower on the GRE due to factors such as race, gender, or socioeconomic status (Educational Testing Service, 2012; Moneta-Koehler et al., 2017; Pennock-Roman, 1994). For example, data exists that suggests that men systematically score higher on the GRE than women, and white people score higher than African Americans in the field of physical science (Miller & Stassun, 2014; Scott & Shaw, 1985).

Milner et al. (1984) suggested that not only does the GRE fail to predict future school performance, but they also argued that its elimination would not result in the matriculation of a lower quality population of students. Removal of the GRE as an admission requirement may also attract students that would not otherwise apply to graduate school. However, one study of health professions programs showed that removing the GRE will not necessarily increase the enrollment of underrepresented populations (Cahn, 2015). Rather, intentional recruitment practices seemed to promote more diverse student populations. Additionally, studies that consider GRE scores as a predictor of graduate school success often only offer data on matriculated students, which may skew results through the exclusion of this possible data (Kuncel et al., 2001; Moneta-Koehler et al., 2017; Ryan et al., 1998). Although they found that GRE scores slightly predicted graduate performance in a Master of Public Administration (MPA) program, Darolia et al. (2014) ultimately suggest that admissions officers consider the benefits and costs of using an exam that research has suggested has implicit race, gender, and age bias.

Darolia et al. (2014) also concluded that GRE scores and UGPA are the two most common variables considered by committees in the graduate admissions process. Various studies have reinforced the importance of UGPA by concluding that as an “objective [measure] of previous academic performance,” it is predictive of an individual’s graduate GPA (Darolia et al., 2014; Halberstam and Redstone, 2005, p. 267; Leavitt et al., 2011). However, Chari and Potvin (2019) found that in addition to UGPA and standardized test scores, letters of recommendation were perceived by undergraduate students and faculty as an important part of the graduate admissions process for programs in physics. This perception was mirrored in a study of graduate admissions files for students in speech-language pathology, where Halberstam and Redstone (2005) concluded that letters of recommendation are “a powerful source of information for how well students will be able to perform academically” and “whether students will be rated as strong or weak” by academic staff members (p. 268).

Several studies identified standardized test scores, undergraduate GPA, and letters of recommendation as the three most important elements that demonstrate a prospective student’s ability to succeed in graduate study (Khaydarov et al., 2019). These elements collectively represent a prospective student’s experience, skills and interests in the specific field of study.

In their study of the graduate school admissions process for students of psychology, Appleby & Appleby (2006) identified five common “kisses of death” (KOD), or the “characteristics of graduate school candidates that decrease their chances for acceptance” (Appleby & Appleby, 2006, p. 19). The study found that admissions committees gave strong consideration to an applicant’s letters of recommendation and academic writing, among other criteria, in the decision-making process (Appleby & Appleby, 2006). Although the KODs were not viewed as a reflection of the applicant’s intelligence, they were indicative of what admissions committees view as a potentially successful candidate: someone with strong, reliable letters of recommendation and scholarly writing mechanics at the very least (Appleby & Appleby, 2006). Affirming the latter in their study of graduates from a specific Master of Public Administration (MPA) program were Leavitt et al. (2011), who together found that strong writing mechanics positively contributed to an individual’s academic performance. However, GRE scores and letters of recommendation were not found to be strongly predictive of the same measure (Leavitt et al., 2011).

Evans (2017) conducted a study of admission criteria for graduate teacher education programs. The study found that overall GRE score did not have a significant impact on graduate GPA. However, undergraduate GPA had a positive relationship with graduate GPA, and the effect of undergraduate GPA on graduate GPA was less for older students. Overall, the study concluded that undergraduate GPA was the best predictor of graduate student success, as defined by graduate GPA.

In their study of admissions criteria for Speech Language Pathology graduate programs, Forrest and Naremore (1998) examine the relationship of undergraduate GPA and GRE scores to graduate student success. They defined this success based on final graduate GPA and scores on a required PRAXIS test. The study concluded that undergraduate GPA had the greatest predictive effect on student success, and it did so with 93% accuracy. When the researchers added the GRE score to the formula, accuracy dropped to just 63%.

In a study of graduate business program admission criteria, Lizares et al. (2015) similarly found that undergraduate GPA was a strong predictor of graduate student success. However, they also found that quantitative test score, as opposed to overall test score, also served as a predictor of success.  Verbal and writing test scores, on the other hand, did not have the same effect on success. While there are areas of agreement across several studies, the differences illustrate the importance of examining the effect of admission criteria on graduate student success by discipline or field.

Discussion and Implications for Research & Practice

There is a growing movement nationally for HE/SA programs (and similar graduate programs) to remove GRE and ETS requirements. The catalyst for this change is in part due to biased nature of these exams towards certain groups (Educational Testing Services, 2012; Darolia et al., 2014; Moneta-Koehler, 2017; Pennock-Roman, 1994; Miller & Stassun, 2014), higher financial cost associated with taking these exams (Darolia et al., 2014), and their limited ability to predict academic success at the graduate level (Evans, 2017; Leavitt et al., 2011; Milner et al., 1984).  While there is growing support for this removal, it poses a challenge to the graduate admission’s process as it removes a central variable to consider when reviewing the student’s candidacy into admission into the program.

Its removal obviously benefits students who have strong undergraduate GPAs (as there would not be another metric to compare their GPA to), but it can inadvertently harm students with lower GPAs. A good number of students had life hardships and/or just lacked the maturity to be successful during their undergraduate experience. However, these students do not lack the academic ability to be successful at the graduate level and could apply themselves to study for the GRE/MAT and receive a score that could offset graduate admission’s committees’ red flags about undergraduate GPA. It’s plausible that a student with a weak UGPA could be denied admission outright or given provisional status, while coupling that same low UGPA with a solid GRE/MAT score could yield that same student full admittance into the graduate program (especially when coupled with an admission essay articulating their academic shortcomings but solid test scores). To this end, it is critical for future research studies to examine the other components of the graduate admission application to help identify the students who will be most successful in the program without the use of GRE/MAT test scores.

In the absence of having GRE/MAT test scores as an admission criterion, there is an inherent benefit and/or predictive nature of undergraduate GPA (Darolia et al., 2014; Leavitt et al., 2011; Khaydarov et al., 2019; Evans, 2017). Similarly, it has been found there are stark differences between the recommendation letters between the caliber of students within the program (Chari & Povtin, 2019; Halberstam & Redstone, 2005; Khaydarov et al., 2019). Anecdotally from our experiences, it can be posited that recommenders of less qualified students will have greater variance in the rankings from their recommender, who also would tend to write lackluster letters that are vague and/or unclear if they have developed a meaningful relationship with the student. This is particularly troubling, as Halberstam and Redstone (2005) assert that recommendations are “a powerful source of information” (p. 268). Recognizing the elevated writing requirements at the graduate level, it is not surprising that others before us have found this to a be a predictor of future academic success (Leavitt et al., 2011).

Recognizing that HE/SA programs are apprenticeship models where graduates immediately move into student-facing positions on college campuses holding great responsibility for student well-being, it is important for studies like this one to examine these programs. Many of these programs are only able to admit a finite number of students, so there is great benefit in identifying the students who will be the most successful if admitted. Conversely, there is great power in understanding which students are likely to struggle academically in the program.

On one-hand, this knowledge could arguably be used in admission’s decisions which may result not favorably in these students’ admission. However, a better use may be to identify students who showed promise in their admission application (i.e. held UG leadership positions such as a Resident Assistant or voice a clear desire to work in higher education) and to identify them as “at-risk” students once admitted into the program. Doing so would enable the faculty to provide additional layers of support to these students to help them be successful. To that end, there should be transparency with these students to share their concerns proactively to help them understand the need for services being offered to them. In addition, providing them the information to make an informed decision on whether they believe they can be academically successful in the program while taking on debt/loans/tuition to do so.

Implications For Practice

This article can serve as a catalyst to help inform HE/SA faculty and program directors on admission decisions. For highly selective programs that seek to take the most academically accomplished students, who in turn should perform at the highest level academically once matriculated into the program, these insights into the components of the admission application can be another criterion to evaluate from a critical lens. This could be especially helpful as programs begin to eliminate the GRE requirement from their admission practices and thus are seeking other data points to help make their decisions.

Conversely, programs can use the findings of this study not as a gatekeeper to graduate school, but rather as a mechanism to identify students who are potentially at-risk for not being academically successful in the HE/SA program. Having this information would allow the HE/SA program and its faculty to be proactive with these identified students and to offer interventions. These interventions can include support for writing at the graduate level, navigating scholarly engagement with classmates in the classroom setting, developing an identity of a scholarly-practitioner, communication with faculty, and time management.

Considerations for Future Research

This article should serve as a catalyst for further inquiry into the admission practices of graduate preparatory programs in higher education and the correlation with student success once matriculated. Considerations for future research should include:

  • The assumption of this study and the academy is that academic success correlates with professional success upon graduation. Is there evidence to support academically stronger students are better student affairs and higher education administrators? The profession would benefit from empirical studies that examine if this correlation exists.
  • Are academically stronger HE/SA students more likely to pursue their doctorate? Is their academic success at the graduate level beyond GPA (i.e. perception of their faculty) an indicator of their potential success at the doctoral level?

Conclusion

Regardless of whether HE/SA programs desire to be highly selective or open access, faculty and program directors should seek additional insight into those applying to their programs. Recognizing the national movement to remove the GRE/ETS requirements from HE/SA graduate admission, it’s likely that faculty will need to focus on undergraduate GPA, quality of writing, and recommendation forms/letters. This reinforces there is clearly a need to empirically study these admission criterions and their correlation to student success once matriculated into the program. Faculty should now be positioned to systematically examine their admission decisions and/or provide interventions to at-risk students based on deficiencies found in their admission application.

References

Appleby, D. C., & Appleby, K. M. (2006). Kisses of death in the graduate school application process. Teaching of Psychology, 33(1), 19–24.

Baum, S., Ma, J., & Payea, K. (2013). Education pays 2013: The benefits of higher education for individuals and society (Trends in Higher Education Series).  New York, NY: College Board.

Cahn, P.S. (2015). Do health professions graduate programs increase diversity by not requiring the graduate record examination for admission? Journal of Allied Health, 44(1), 51-56.

Chari, D., & Potvin, G. (2019). Understanding the importance of graduate admissions criteria according to prospective graduate students. Physical Review Physics Education Research, 15, 1-6.

Cohen, L., Manion, L., & Morrison, K. (2018).  Research methods in education (8th edition).  New York, NY: Routledge.

Creswell, J. W. (2005). Educational research: Planning, conducting, and evaluating quantitative (2nd edition). Upper Saddle River, NJ: Prentice Hall.

Darolia, R., Potochnick, S., & Menifield, C. E. (2014). Assessing admission criteria for early and mid-career students: Evidence from a U.S. MPA program.  Education Policy Analysis Archives, 22(101), 1-21.

Educational Testing Service. (2018). GRE guide to the use of scores 2018-19. Retrieved from https://www.ets.org/s/gre/pdf/gre_guide.pdf

Educational Testing Service. (2014). The research foundation for the GRE revised general test: A compendium of studies. Retrieved from https://www.ets.org/s/research/pdf/gre_compendium.pdf

Educational Testing Service (2012). GRE general test score information by ethnicity/racial groups, 2009-2010. Retrieved from https://www.ets.org/s/gre/pdf/gre_general_test_score_information_by_ethnicity_2009_2010.pdf

Evans, C. M. (2017). Predictive validity and impact of CAEP Standard 3.2: Results from one master’s-level teacher preparation program. Journal of Teacher Education, 68(4), 363–376.

Fedynich, L. (2017). The grand question: do entrance examinations determine graduate student academic success? Research in Higher Education Journal, 33, 1-8.

Flick, U. (1998). Sampling strategies. An introduction to qualitative research.  London, United Kingdom: Sage.

Forrest K, & Naremore RC. (1998). Analysis of criteria for graduate admissions in speech-language pathology: Predictive utility of application materials. American Journal of Speech-Language Pathology, 7(4), 57–61.

Halberstam, B., & Redstone, F. (2005). The predictive value of admissions materials on objective and subjective measures of graduate school performance in speech-language pathology. Journal of Higher Education Policy and Management, 27(2), 261–272.

Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277-1288.

Komives, S.R., & Woodard, D.B. (1996). Student services: A handbook for the profession (3rd edition).  San Francisco, CA: Jossey-Bass Inc.

Kuncel, N. R., Hezlett, S. A., & Ones, D. S. (2001). A comprehensive meta-analysis of the predictive validity of the Graduate Record Examinations: Implications for graduate student selection  and performance. Psychological Bulletin, 127(1), 162-181.

Kuncel, N. R., Kochevar, R. J., & Ones, D. S. (2014). A meta-analysis of letters of recommendation in college and graduate admissions: Reasons for hope. International Journal of Selection & Assessment, 22(1), 101–107.

Leavitt, W. M., Lombard, J. R., & Morris, J. C. (2011). Examining admission factors in an MPA program. Journal of Public Affairs Education, 17(3), 447-460.

Litwiller, H. (2017). So you want to go to graduate school: A step-by-step guide to getting accepted. Ocala, Florida: Atlantic Publishing Group, Inc.

Lizares, R., Rahnema, L., Pang-Rey, M., Suan, I., & Bautista, C. C. (2016).  Graduate business program admission criteria and student graduate academic performance.  Philippine Management Review, 24, 83-98.

Miller, C., & Stassun, K. (2014). A test that fails. Nature, 510, 303-304.

Milner, M., McNeil, J. & King, S.W. (1984). The GRE: A question of validity in predicting performance in professional schools of social work. Educational and Psychological Measurement, 44, 945-950.

Moneta-Koehler, L., Brown, A. M., Petrie, K. A., Evans, B. J., & Chalkley, R. (2017). The limitations of the GRE in predicting success in biomedical graduate school. PLoS ONE, 12(1), 1-17.

Orlando, J. (2005). The reliability of GRE Scores in predicting graduate school success: A meta-analytic, cross-functional, regressive, unilateral, post-Kantian, hyper-empirical, quadruple blind, verbiage-intensive and hemorrhoid-inducing study. Ubiquity, (6)21, 1-15.

Patton, M. Q. (2002). Qualitative research and evaluation methods.  Thousand Oaks, CA: Sage.

Pennock-Roman, M. (1994). Background characteristics and future plans of high-scoring GRE and general test examinees. (ETS Research Rep. No. RR-94-12.) Princeton, NJ: Educational Testing Service.

Ravitch, S. M., & Carl, N. M. (2015). Qualitative research: Bridging the conceptual, theoretical, and methodological. Thousand Oaks, CA: Sage.

Rose, S., Spinks, N., & Canhoto, A. I. (2014). Management research: Applying the principles. New York, NY: Routledge. Ryan, W. J., Morgan, M., & Wacker-Mundy, R. (1998). Pre-Admission criteria as predictors of selected  outcome measures for SLP graduate students. CICSD, 5, 54-61.

Scott, R.R., & Shaw, M.E. (1985). Black and white performance in graduate school and policy implications for using GRE scores in admission. Journal of Negro Education, 54(1), 14-23.

Sternberg, R. J., & Williams, W. M. (1997). Does the Graduate Record Examination predict meaningful success in the graduate training of psychologists? American Psychologist, 52,630-641.

Tolman, S., & Calhoun, D.W. (2019). Pedagogical approach to developing the hiring practices of higher education administrators. Georgia Journal of College Student Affairs, 35(1), 61-81.

Steven Tolman, Ed.D is an Assistant Professor of Higher Education Administration at Georgia Southern University. His previous roles included serving as a Higher Education Administration program director and 12 years as a student affairs administrator in Residence Life, Student Conduct, and Student Life. He holds a Doctorate from Rutgers University, Master’s from Texas Tech University, and Bachelor’s from Central Michigan University. His research is theoretically informed and guided by the tenets of student development theory.  In particular, he explores the application of Maslow’s Hierarchy of Needs, Kolb’s Experiential Learning, Sanford’s Model of Challenge and Support, and Astin’s Theory of Involvement.  This theoretical framework is intertwined with the two streams of his scholarly agenda: 1) The profession of student affairs and 2) The residential and co-curricular experience of college students.

Kimberly O’Halloran, Ph.D is Associate Provost and Dean of Graduate Studies and Extended Learning at Widener University, where she also serves as an Associate Professor in the Center for Education. She has spent the last 25 years working in higher education, including posts in both academic affairs and student affairs at Rutgers University, Cornell University, New York University and Montclair State University.  Her research has focused on organizational issues in higher education and the factors that impact persistence and success for non-traditional undergraduate and graduate students.  Dr. O’Halloran received a B.A. in English and an M.Ed. in Education Administration, both from Rutgers University – New Brunswick and Ph.D. in Higher Education Administration from New York University.

Abigail Meert is a Residence Life Coordinator at Grinnell College.  She holds a Master’s from Georgia Southern University and Bachelor’s from Armstrong State University, now named the Armstrong Campus of Georgia Southern University.  Her previous roles included being a Graduate Hall Director, Resident Advisor, First-Year Peer Mentor, and Honors Ambassador. Abigail’s research interests include higher education law and policy, university systems, and the consolidation of post-secondary institutions.