To Whom It May Concern: Demographic Differences in Letters of Recommendation

Beverly Hirtle and Anna Kovner

Letters of recommendation from faculty advisors play a critical role in the job market for Ph.D. economists. At their best, they can convey important qualitative information about a candidate, including the candidate’s potential to generate impactful research. But at their worst, these letters offer a subjective view of the candidate that can be susceptible to conscious or unconscious bias. There may also be similarity or affinity bias, a particularly difficult issue for the economics profession, where most faculty members are white men. In this post, we draw on our recent working paper to describe how recommendation letters differ by the gender, race, or ethnicity of the job candidate and how these differences are related to early career outcomes.

Methodology

We analyze the text of 6,365 recommendation letters received by a large U.S.-based research institution for 2,227 new Ph.D. job candidates during four recent annual recruiting cycles (2018 to 2021). We pair the recommendation letters with information supplied by the candidates about their primary and secondary fields of research interest, their Ph.D. granting institution, and confidential information about their self-identified gender, race, and ethnicity. Information on gender and race/ethnicity was collected for statistical purposes on a voluntary basis from all job applicants to the organization, not just for economists. The information candidates submitted was not used in the hiring process and was not provided to hiring managers or those reviewing or interviewing job candidates.

We identify key characteristics of each letter to measure the quality of the recommendation. These measures include overall word count and the number of words associated with “standout” and “grindstone” characteristics. Standout characteristics are related to how candidates might excel relative to others and are captured by words such as “innovative,” “extraordinary,” and “exceptional.” Grindstone characteristics are related to effort and are captured by words such as “dependable,” “hard-working,” and “dedicated.” In focusing on standout and grindstone words, we mirror much of the earlier literature both for economics and for other fields (see, for example, studies involving applicants for jobs in biochemistry, orthopedic surgery, and general surgery). While standout words are unambiguously positive, grindstone words can have a more mixed connotation and are sometimes considered as “damning with faint praise.”

We also develop a novel measure of letter quality based on language contained in many of the letters that provides the letter writer’s recommendation for the caliber of hiring institution appropriate for the candidate—in particular, whether the letter writer recommends the candidate to a “top” department. Overall, about 10 percent of the letters in our sample contain such a recommendation, so these recommendations are comparatively rare.

Demographic Differences in Letter Quality

Much of the earlier research on letters of recommendation has focused on differences in letter characteristics by candidate gender (see here and here for two earlier examples in economics from European universities). Unlike some of this earlier work, we do not find statistically significant differences in letter length or the share of standout words in letters for female job candidates. We do, however, find that letters for female candidates contain higher shares of grindstone words, which as noted, have a potentially ambiguous interpretation.

One novel result is the relationship between letters and race and ethnicity. Letters for candidates who self-identify as Asian are significantly shorter and contain fewer standout words and more grindstone words—findings that remain when we limit the sample to candidates from top 10 U.S. economics and finance programs, one way to address potential selection bias in the candidate pool. We also find some differences in letters written for candidates who self-identify as Hispanic or Black; letters for these candidates contain a lower share of standout words. While the role of race and ethnicity in letters of recommendation in economics has not been studied to our knowledge, these results are consistent with those in some other fields (for instance, standout words are more likely to be used for white surgical residents).

Macroeconomics and Finance Are Harsh Graders

We also find significant differences in the length and substance of letters by subdiscipline within economics. Letters written for candidates who identify “finance” or “macroeconomics” as their primary field of interest are shorter and use fewer grindstone words. Letters for candidates focusing on macroeconomics also use fewer standout words. Candidates focusing on finance are more likely to be receiving their degree from a business school than candidates in other fields, so the differences could reflect the type of school rather than the field per se. We also run specifications containing a control for business school as the Ph.D.-granting institution and the differential results continue to hold in these specifications.

Underrepresented Candidates Are Less Likely to Be Recommended to the Top Departments

We find meaningful differences by gender, race, and ethnicity in whether a letter recommends a candidate to a “top” economics department. Letters for female, Asian, and Black or Hispanic candidates are all significantly less likely to include such a recommendation. These differences persist when we control for the characteristics of the Ph.D.-granting institution, when we control for the characteristics of the letter writer (both female and Asian letters writers are less likely to make such recommendations), and when we limit our sample to candidates graduating from top 10 economics and finance departments. These differences are both statistically and economically important. Letters for female candidates are 18 percent less likely to contain a “top” recommendation than letters for male candidates, a result that holds even when the letter writer is female. Letters for Black or Hispanic candidates are 30 percent less likely to contain this recommendation than letters for white candidates, while letters for Asian candidates are 45 percent less likely to have a “top” recommendation.

Early Career Outcomes

But do these differences in letter quality matter? To address this question, we examine the impact of letter characteristics on early career outcomes for the job candidates in our sample. In particular, we examine initial job placements (whether a candidate’s initial job is at a top 20 economics or finance department) and publications (the number of top journal publications a candidate has within two years of receiving their Ph.D.).

Controlling for candidate characteristics, field of interest, Ph.D.-granting institution characteristics, and letter writer characteristics, we find that stronger letters are indeed associated with better early career outcomes. Longer letters and a “top” recommendation are both positively associated with the probability of having a top 20 initial job and with the number of top journal publications. A higher share of standout words is associated with more top journal publications while a higher share of grindstone words is negatively associated with early career publications. We find evidence that early career outcomes are stronger for candidates from top 10 economics and finance programs and that some outcomes are weaker for female, Asian, and Black or Hispanic candidates, even after controlling for letter characteristics.

Summing Up

Taken together, our findings suggest that there are meaningful differences in the content of recommendation letters correlated with the gender, race, and ethnicity of the candidate, as well as with the candidate’s field of interest, and that these differences matter in predicting early career outcomes. A key open question from our work is to understand the reasons for these findings. Is this a natural outcome of preferences for similarity whereby underrepresented candidates are less similar to their letter writers? Does this represent differences in the types of topics that different types of candidates choose to study? What leads to the differences in candidates being recommended to “top” departments? Understanding the associations in the data is only the first step in thinking about how to weigh the qualitative information contained in letters of recommendation.

How to cite this post:
Beverly Hirtle and Anna Kovner, “To Whom It May Concern: Demographic Differences in Letters of Recommendation,” Federal Reserve Bank of New York Liberty Street Economics, November 15, 2024, https://libertystreeteconomics.newyorkfed.org/2024/11/to-whom-it-may-concern-demographic-differences-in-letters-of-recommendation/.