The Returns to Colleges: Estimating Value Added and Match Effects in Higher Education
- Date Posted:
- Is Database:
- Database
College value add seems unrelated to selectivity, while STEM degree attainment proves to be valuable.
Jack Mountjoy and Brent Hickman, "The Returns to College(s): Estimating Value Added and Match Effects in Higher Education," University Of Chicago, January 2020, https://bfi.uchicago.edu/working-paper/the-returns-to-colleges-estimating-value-added-and-match-effects-in-higher-education/
Overall the paper does look at admitted “elite” students choosing less selective schools however I don’t see a carveout for those students and STEM selection, "...While Figure 2 shows that admission portfolio controls nearly eliminate imbalances in observable student potential acrosscollege treatments, a lingering concern with the matched applicant approach is that it relies on “the small share of students who make what is a very odd choice” (Hoxby, 2009): enrolling in a less selective college when admitted to a more selective alternative. Table 2 directly investigates the nature of this identifying variation in our sample, and finds that such choices are actually far more common than the concern supposes. The first section of Table 2 shows that among students in our identifying sample, i.e. those admitted to at least two colleges, only 55% end up enrolling in their most selective option, with the average admission portfolio spanning a selectivity range of 116 SAT points (SD 78). The second section of Table 2 shows that this phenomenon is not confined to students choosing solely among less selective institutions. Of the 27% of students in our identifying sample who are admitted to UT-Austin, the most selective campus in our students’ choice sets, only 58% choose to enroll there. And the remainder are not simply sidestepping en masse over to the other flagship: conditional on being admitted to UT-Austin and not being admitted to Texas A&M, still only 60% of students choose UT-Austin, meaning 40% forgo it in favor of a less selective, non-flagship school. Very similar patterns also arise among Top Ten Percent students in our sample, who are guaranteed admission to every Texas public university they apply to. Only 30% of them end up enrolling at UT-Austin, and 25% at Texas A&M, leaving fully 45% who decline to exercise their flagship entitlement and instead fan out across the 28 non-flagship campuses in our sample...."
Ed Comment:I read the paper. I would start by asking WTF? The paper is straightforward. Conceptually, it does exactly what I thought it should do: regress outcomes against pre-college characteristics, most importantly test score, and then analyzes the residuals. For all intents and purposes, it finds that colleges add very little value beyond what is predicted by a student’s demonstrated pre-college success. Beyond that, the insights are pretty thin. Schools that make the effort to get a higher share of their students to graduate produce higher earning students relative to the earnings their pre-college success would predict. And students that earn stem degrees earn more than students who don’t. No surprise, learning valuable things, like STEM, is valuable. So, no surprise, schools that produce a greater share of students who earn stem degrees add more value than schools that don’t. The signally value of college selectivity fades quickly. There also doesn’t seem to be much effect from “isms” in college or the 8 years after, even in Texas, since the results are essentially the same for black vs whites, men vs women, and rich vs poor. Had you written this summary in response to my questions, I wouldn’t have had any questions. And I wouldn’t have needed to read the paper. Scratching my head why it isn’t that easy.
Ed Comment:Are you sure they have individual scores and not just a pool of students who turned down school X? With the latter, they can compare that turn-down pool's average earnings to the average earnings of the pool of students from the school they turned down? Is 14 based on the pool of people who turned down x for x- or a larger pool? I don’t understand how completion factors into the above. That seems like different data. If a student starts in STEM and ends up graduating in not-STEM, how are they measured and what does it mean? If you have the scores of individuals, why all the rigmarole? Wouldn't you start with how much more does pedigree of school add over and above test score, and how much more does degree (STEM vs not-STEM) add on top of that? Then you would incrementalism from there and explain any differences. My guess, not havening read the paper, is that they are solving for the fact that they don’t have scores.
Ben Comment:I don’t really know what you mean by rigmarole. That said - I think they are doing what you’re suggesting, I think it’s just more complicated a procedure than you’re suggesting it would be. The first they they do in the paper (section 3) is calculate the value add of the colleges. In section 3 they discus selection bias (3.1 and 3.2) and the assumptions that need to hold for their model of value added to be correct. 3.3 Looks at matching, sorting, and support to try to get at whether or not the value added at different schools depends on the student as well as the school (going to Michigan might mean something else for me and for one of my classmates). So once they’ve solved for these things they have something they’re calling the schools value added - it’s a kind of residual from taking out the characteristics of the students at the school. They do other analysis first but then get to section 7 where we find Fig 14. Fig 14 uses all the schools and students and looks at how the value added (of earnings) of the schools compares to the value added of completing certain degrees from particular schools. If a student starts in stem but ends up an English major, they’ve completed their degree as an English major and are non-stem completion. I don’t think they’re tracking every major switch through each person’s entire college career. From my reading, non-completion has to do with drop outs, not people who switched majors. I am sure they have the scores.
Ed Comment:By rigmarole I mean.: Regressing earning against test scores and other pre-college credentials, and then regressing the residuals against some measure of the colleges' prestige/selectiveness is an obvious and straightforward starting point that goes a long way toward assesses the value added of a college and college_degree combination over and above the student's pre-college credentials. If they did that, report it, I'd like to see it, because it's the obvious starting point. If they didn't, explain why they didn't. Then explain to me what additional information/refinement they gained (or thought they gained) by looking at the sub pool of people who turned down a more prestigious/selective college. If they didn't have scores and other pre-college credentials, they would have gain a lot. But remember, they claim colleges don't add much value so the turned down pool shouldn't be much different than the not-turned down pool. So if you have the analysis in the first paragraph (and why wouldn't you if you had the data?....which is my question.) Then you haven't learned much more. You're just confirming what you learned in the first paragraph. You only learn that colleges don't add much value IF you don't have the analysis in the first paragraph. Then explain what we are learning in the rest of the paper (presumably fig 14). It doesn't seem to build on the analysis above. Rather it goes off in another direction. And I don't understand the insight. If they have the analysis in paragraph 1, then the analysis in paragrpah 2 seems like rigmarol if it's doing more than just confirming 1. And not doing the analysis in 1 if they have the data to do it seems like rigmarol. And going off on a tangent to fig 14 seems like rigmarol if it doesn't yield a valuable and unambiguous insight --an insight so far I don't understand. The goal here is to 1) teach me the insight while saving me the time of having to read the paper myself, which so far you haven't done, 2) don't make me pull teeth to achieve my objective, which is to save my time 3) do it in a way that I enjoy.
They find that things like lots of STEM majors and less selective schools really helps value add (Figure 14), or supporting students in a way that helps them get towards graduation (this one is a little more complicated and I don’t buy what they’ve done). “
They go on to test this assumption (and it’s fairly good). Looking back at the graph I showed you, any time a horizontal red line crosses the vertical lighter-red line, that is a college making no difference - they’re using 95% confidence intervals. A school like Texas Southern is a no results because they fail to reject the null, but it’s close. After seeing that many colleges don’t provide value add (but some do!), they try to look at what the colleges that do add value do to add value. Presumably, if you believed this analysis, you could use the next sections of the paper to improve colleges’ value add by adopting the characteristics of schools that do add value.
This graph shows the earnings increase a student would get from going to a college relative to what that student would earn relative to UT Austin. The red dots are the point estimate and the red lines are the confidence intervals. The triangles show the same calculation using a different estimator (I am unfamiliar with Bayes Shurnk Forecasts but can look them up if you would like). The main point of the using the admissions portfolio as a control is econometric. In order to discover something about the colleges, the authors need to net out the effects of the students. Typically you would do this with a fixed effect specification. They’ve found that they cannot use student fixed effects this way because so they assume the admissions portfolio is a decent proxy for the individual.
Ben Comment:They wouldn’t have done it stepwise as you’re suggesting because of the econometric issues they discuss in section 3 of the paper (I can give you more info here, but I’m not sure I can do that without “jargon”). Instead, they came up with the estimation strategy they’re employing. This paper is about the colleges themselves so, other than some summary statistics about the sample, they do not present any data about the students. In the data appendix they do show the graph of the value added by college:
Ed Comment:It’s still basically impossible to figure out what insights the paper has from your description of it. I believe that technical expertise (STEM as a surrogate here) is valuable for smart people to learn. So if the conclusion is schools that have more technical grads add more value, ok but I need more. For example, how have they removed selection bias from students earning technical degrees? If they haven’t accounted for it, they haven’t really accounted for anything. So if your summary doesn’t explain that, it doesn’t explain their insight (if they have one). I explained my concern up front. That students who couldn’t get into to Michigan STEM, went to Michigan State STEM rather than Michigan Non-STEM. They were smarter on average than Michigan State grads and as smart as Michigan no-STEM grads. So, they ended earning (how do they measure results?) like Michigan not-STEM grads. What does that prove? I don’t think it proves that Michigan doesn’t add value even though I don’t believe it does. If they’re going to make points about STEM, they have to factor out selection bias. By degrees. Further, statements like “This paper is about the colleges themselves so, other than some summary statistics about the sample, they do not present any data about the students” leave me scratching my head. I’m not asking about students. I’m asking about colleges. At the end of the day, aren’t the results of the college the summation of the results of its students? I’m asking about the summation of the results of the colleges (and degrees). I have no clue what they could tell me about students that would be of interest other than the effects colleges have on them on average. I’m not saying a step wise regression is necessarily the way to do the analysis. I’m trying to explain what it is about your summary of the paper that confuses me by explaining my perspective on the paper taken from your summary, so that you can write a summary that addresses my concerns. I often feel that economists start with their assumptions so lose that they end up way over complicate things from the get-go. That’s fine for an ending point. It’s not so great as a stating point. So I look at things and say what additional insight/refinement/precision did the complications (i.e. rigmarole) add relative to a more straightforward approach. Here, as a practical businessman, I would sort students into STEM and not-STEM graduates and regress earning against test scores and then see if the exclusiveness of the school as measured by the school’s (divided into STEM and non-STEM average test scores (or average earnings of their students) adds any additional predictive value. My guess is surely they did that and compared their results to that even if they didn’t report it. I specifically asked you: why do they need all the complications? I think your answer was: read section three—not an answer to my question. The summary should start with the bottom line. Do colleges add a lot of value or not? I thought your summary said no. usually you don’t get published for not finding something statistically significant, so then explain what adds value/is statistically significant. I also don’t know what “fixed effects” are unless you explain them. if you mean test score fro example, say “test score.”
Steve Comment: The paper doesn’t address individuals degrees, but goes carve out the impact of STEM majors. "....We next unpack BA completion by STEM versus non-STEM degrees to explore whether value-added on completing different majors has different predictions for earnings effects. The top two panels of Figure 14 shows that a college’s value-added on completing a STEM degree (left) has a strong correlation with its earnings VA, while value-added on non-STEM completion (right), in contrast, has almost no bivariate relationship with earnings VA. The middle panel shows that STEM and non-STEM VA are negatively correlated, however, suggesting that colleges may face tradeoffs across fields in boosting degree completion. Exemplifying this to the extreme, UT-Austin at the (0,0) origin has the lowest value-added on STEM completion, as we also saw in Section 6, but simultaneously has the highest VA on non-STEM degree completion. Simple bivariate correlations like the top right panel of Figure 14 may thus underestimate the earnings gains of producing more non-STEM degrees, since this is correlated with fewer STEM degrees in the cross-section. The bottom right panel of Figure 14, consistent with this hypothesis, shows that the relationship between earnings VA and non-STEM VA becomes significantly more positive when controlling for STEM VA. On the other hand,STEM VA becomes an even stronger predictor of earnings VA when controlling for non-STEM VA (bottom left panel of Figure 14). The partial slope coefficients imply that a 10 percentage point controlled increase in STEM VA predicts roughly twice as much earnings VA ($3,550) as the same controlled increase in non-STEM VA ($1,741)....”