Aligning Questions and Methods: Discussing “In the Shadow of War.”
In the last days of 2023, our team at the Cohen Center for Modern Jewish Studies published a report, “In the Shadow of War: Hotspots of Antisemitism on US College Campuses,” based on a data collected after Hamas’ October, 7, 2023 attack on Israel. The report was discussed in the January, 2024, CASJE Research Digest and has been the subject of discussion both in the popular press and among researchers, Jewish professionals, and university administrators. The commentary about our study illustrates some broader issues that are at the core of CASJE’s goal to build more effective bridges between researchers and practitioners in Jewish education. In service of that goal, as well as a broader objective of enhancing how the Jewish community responds to antisemitism, we would like to offer a number of comments about the methodology we utilized and its relationship to what can be learned from the study.
An epistemological foundation of our work (not just in this study) is a principle, famously articulated by that Donald T. Campbell, that “validity is not a property of methods, but of inferences.” That is, data and methods per se are neither valid nor invalid, neither “good” nor “bad,” in the abstract. Instead, data and methods need to be evaluated with respect to their ability to provide plausible answers to particular research questions, and to help rule out plausible alternative explanations. No single piece of research can answer all questions, and a researcher’s task is to manage the inevitable trade-offs that arise and choose the best methodological approach for the task at hand.
What Makes a Good Sample?
Our 2023 report, and the subsequent commentary, provide an opportunity to see how these considerations play out in practice. Perhaps the best illustration of these issues involves our choice of sample. The study was designed to answer questions about variations in Jewish students’ perceptions of anti-Jewish and anti-Israel hostility on their campus: the extent to which perceptions of antisemitism on campus by Jewish students were higher at some schools and lower at others. Understanding the extent to which perceptions of antisemitism vary across campuses is important because it opens the door for future work exploring the predictors of antisemitism on campus, and the identification of effective strategies to respond to it.
In service of this goal, we collected data from a sample of Jewish students at around 50 US campuses who had applied to Birthright Israel. This sample was well suited to answering questions about comparisons between campuses because all individuals were selected to the sample through the same procedure, regardless of campus.
As we noted in the report, and as the CASJE Research Digest as well as the popular press also reported, this sample of Birthright applicants is not necessarily representative of all Jewish college students. Indeed, our sample of Birthright applicants is likely to overrepresent individuals who are more likely to report, or be the target of, antisemitism related to criticism of Israel. But the implications of this limitation for this study need to be evaluated in regard to the questions that the report was trying to answer.
Indeed, despite its limitations, a sample of Birthright Israel applicants is well-suited to answering the questions posed by the report – about the variation in perceptions of hostility between schools – because it allows for an apples-to-apples comparison of Jewish students at different schools. This is true even if Birthright applicants as a whole are more likely to perceive or experience antisemitism than other Jewish students. It is because of this limitation that the report never discusses or presents overall estimates of antisemitic hostility for the sample as a whole, and instead focuses on comparisons between students at different school (or comparisons between these results and our 2016 study, which was also limited to Birthright applicants).
To Weight or not to Weight?
Like the choice of a sample, analytic decisions also need to be made in light of particular research questions. An analytic choice that makes sense for one question may not make sense for another. We can see this dynamic play out in discussions about our decision to not apply any statistical weights in our reported analyses. In its discussions of the limitations of the study, the CASJE Research Digest expressed concern that our unweighted estimates for the characteristics of respondents in each of the four quartiles effectively “[give] much greater weight to schools with a larger number of respondents than smaller schools.” To address this concern, it was recommended that we “treat each school equally…by weighting responses so that the total weight coming from each school is equal.”
As with the choice of a sample, the question of whether or not schools should be treated “equally” in an analysis depends on the question that the analysis is trying to answer. The first question that the report aims to answer concerns variations in the antisemitic climate at different schools. For this question, schools need to be treated equally, and they were: we simply calculate the average hostility score for students at each school, regardless of how many respondents exist for that school. That said, to avoid the average score of schools with smaller numbers of respondents were “pulled” more towards the overall average through empirical Bayes estimation, in acknowledgement of the fact that estimates for schools with smaller numbers of respondents are likely to be less reliable.
Once schools have been ranked, however, and then divided into quartiles on the basis of that ranking, the population of interest for all other analyses presented in the report is “Jewish students at schools with different levels of hostility.” For this population, we did not want to “treat all schools equally” because Jewish students themselves are unequally distributed among schools. Thus, for example, Hillel International estimates that around five times as many Jewish students attend the University of Florida (UF) as attend California Polytechnic State University ([Cal Poly] 6,400 vs 1,200). If these numbers are approximately accurate then it makes sense for an analysis of “Jewish students at universities with low levels of antisemitic hostility" (a group that includes both of these schools) to include a larger number of respondents from UF. Conversely, calculating weights that sum to the same value for both schools would produce a highly biased estimates of the population of “Jewish students who attend UF and Cal Poly.”
As the research digest suggests, the best solution to this problem would be to weight respondents at each school to the total Jewish population at that school. Unfortunately, there are no reliable estimates of that number for most of these schools. The Hillel estimates cited above are not derived from systematic data collection and represent a judgment of Hillel professionals at each campus.
Absent this data we made the decision to report unweighted data. This effectively treats the number of Birthright Israel applicants who responded at each school as a proxy for the total Jewish population at each school and should produce less bias than weighting up respondents at schools with smaller Jewish populations. This is obviously an imperfect approach (although, perhaps by coincidence, our achieved sample also includes approximately five times as many respondents from UF as from Cal Poly:113 vs 27). But given our research goals and the limited data available, presenting unweighted data seems to represent the best available option.
Thinking with Research
Studying American Jews, a population for whom census-like population data does not exist, is always a challenge. This challenge is magnified for studies that attempt to provide quick answers to pressing questions in the middle of a crisis. We hope this discussion has illuminated some general principles that can help guide the creation and interpretation of high-quality research to advance Jewish education. First, while no sample is perfect, some are better than others for a particular purpose. A sample of Birthright Israel applicants would not be well suited for a study that attempts to estimate the overall prevalence of antisemitism on campus but can still provide reliable data on how antisemitism varies from one campus to the next. Second, analytic choices require clarity: both with respect to research questions and the population of interest. It makes sense to treat different sized schools equally in an analysis of schools, but perhaps not in an analysis of students. Finally, attempts to “adjust” data (e.g., through weighting, or the imputation of missing values) runs the risk of exacerbating, rather than reducing, bias. Deployment of these methods should always consider the risks as well as the potential rewards. As researchers, we have a responsibility to not only choose the right methods for the job, but to allow the strengths and weaknesses of our methods to guide how we share our findings with practitioners and the wider world.