An Automated Method to Estimate Survey Question Bias

Aaron Kaufman

Abstract: Many survey researchers are interested in gauging public support for government policy, but there is strong evidence that a question’s wording affects responses to it. I develop the first automated and scalable method to predict the magnitude and direction of the partisan bias a question’s wording may impose on survey responses, and show using a series of survey experiments that it outperforms public opinion scholars in predicting that bias. Using a novel data set of almost one million survey questions from 1997 to 2017, I then examine trends in partisan survey question biases over time. I find that while questions related to economic issues are relatively unbiased, questions related to Barack Obama become steadily more conservatively biased from 2008 to 2017. Questions related to abortion and immigration are generally conservative, while questions related to healthcare and education are consistently liberal. Substantively, my results suggest that measurements of American public opinion are systematically biased; I discuss the implications of this result for democratic representation. Methodologically, this paper opens up new opportunities for studying ideology from text, and for improving survey methodology and measurement in public opinion.

Draft Here

Implementing Novel, Flexible, and Powerful Survey Designs in R Shiny

Aaron Kaufman Survey research in the social sciences is ubiquitous as a cost-effective and time-efficient means of collecting data. However, the available software for implementing and disseminating such surveys lacks flexibility, stifling researcher creativity and severely limiting the scope of questions that survey research can address. In this paper, I introduce the use ofR Shiny, an open source web application and scripting language, for implementing powerful, innovative, and fully customizable surveys. Through five applications and accompanying software, I show that R Shiny allows for (1) randomized question selection, (2) programmatic treatments, (3) programmatic survey flow, (4) adaptive question batteries, and (5) sequentially block-randomized designs, expanding the scope, ease, and cost effectiveness of online survey research. Draft Here

Matching for Inference on Text Data with an Application to Measuring Media Bias

Reagan Mozer, Luke Miratrix, Aaron Kaufman, and Jason Anastasopoulos
Forthcoming at Political Analysis

Abstract: There is little existing methodology for drawing potentially causal conclusions when pre-treatment confounders are represented by text data. Even more unclear is how to approach inference in the setting where both the pre-treatment covariates and the outcome of interest are de ned by different summary measures of the same observed text. We summarize the challenges and limitations for principled analysis in this domain and propose a framework for estimating effects in studies where both the covariates and outcomes are summary measures built from text. First, we extend recent work on matching documents on features generated using text analysis methods. After matching, we estimate differential word use and sentiment using other text analysis tools. We demonstrate our procedure by comparing partisan bias across US news sources, as measured by their rates of coverage of issues and, given the same coverage, their different representation of topics. Here both the covariates (i.e., topics covered) and the outcome (i.e., language used and sentiment of covered content) are measured from the text. Our approach allows for investigation of two questions: are news sources systematically selecting different content to cover, and furthermore, when covering the same topics, are news sources presenting content using different language or sentiment?

Draft Here

Can Violent Protest Change Local Policy Support? Evidence from the Aftermath of the 1992 Los Angeles Riot

Ryan Enos, Aaron Kaufman, and Melissa Sands
Forthcoming at American Political Science Review

Abstract: Violent protests are dramatic political events often credited with causing significant changes in public policy.
Scholarly research usually treats violent protests as deliberate acts, undertaken in pursuit of specific policy goals. However, due to a lack of appropriate data and difficulty in causal identification, there is little evidence of whether riots accomplish these goals. We collect unique electoral measures of policy support before and after the 1992 Los Angeles Riot—one of the most high-profile events of political violence in recent American history—which occurred just prior to an election. Contrary to some expectations from the academic literature and the popular press, we find that the riot caused a liberal shift in policy support at the polls. Investigating the sources of this shift, we find that it was likely the result of increased mobilization of both African American and white voters. Remarkably, this mobilization endures over a decade later.

Draft Here

Supplemental Appendix

How to Measure Legislative District Compactness if You Only Know it When You See it

Aaron Kaufman, Gary King, and Mayya Komisarchik
Forthcoming at American Journal of Political Science

Abstract: The US Supreme Court, many state constitutions, and numerous judicial opinions require that legislative districts be “compact,” a concept assumed so simple that the only definition given in the law is “you know it when you see it.” Academics, in contrast, have concluded that the concept is so complex that it has multiple theoretical dimensions requiring large numbers of conflicting empirical measures. We hypothesize that both are correct — that the concept is complex and multidimensional, but one particular unidimensional ordering represents a common understanding of compactness in the law and across people. We develop a survey method designed to elicit this understanding with high levels of intracoder and intercoder reliability (even though the standard paired comparison approach fails). We then create a statistical model that predicts, with high accuracy and solely from the geometric features of the district, compactness evaluations by judges and other public officials from many jurisdictions, as well as redistricting consultants and expert witnesses, law professors, law students, graduate students, undergraduates, ordinary citizens, and Mechanical Turk workers. As a companion to this paper, we offer data on compactness from our validated measure for 18,215 US state legislative and congressional districts, as well as software to compute this measure from any district shape. We also discuss what may be the wider applicability of our general methodological approach to measuring important concepts that you only know when you see.

Draft Here

The Unilateral Presidency, 1933-2017

Aaron Kaufman and Jon Rogowski

Abstract: Unilateral action is a defining characteristic of the modern presidency. Existing scholarship on unilateral action, however, has important empirical and theoretical limitations. Empirically, though scholars recognize the range of unilateral tools presidents may deploy, including executive orders, memoranda, proclamations, and other directives, these tools are generally considered in isolation and researchers focus most often solely on executive orders. Moreover, existing approaches provide no basis for comparing the substantive significance of unilateral action across directives and over time. Theoretically, scholars have focused on inter-institutional conflict as a constraint on unilateral power but have mostly neglected the role of public opinion. In this paper, we address both limitations and use new data and text analysis to characterize the significance of unilateral directives issued between 1933 and 2017. We present new findings about patterns of unilateral action over the last 85 years and show that public opinion may constrain presidents’ exercise of unilateral powers.

Draft Here

And Yet They Move: Candidates’ Ideological Repositioning During Primary and General Election Campaigns

Pablo Barberá and Aaron Kaufman

Abstract: A rich literature in formal modeling makes predictions regarding candidate behavior, and how incumbents may be ideologically constrained by a primary or general election challenger. However, no empirical evidence has ever been brought to bear on the question of whether candidates conform to these theories. In this paper, we apply a novel method of ideal point estimation to empirically test predictions of these models. Using follower networks on Twitter, we estimate the ideal points for every candidate running for United States Federal office in 2016, for every day from April 2015 until election day. In doing so, we create the first temporally fine-grained data set of candidate ideology over time for more than 1,000 candidates for the Presidency, US House, and US Senate. We show that candidates do ideologically reposition over the course of the campaign, but with heterogeneity conditional upon electoral contexts and challenger-based incentives. In particular, we examine whether candidates adopt more extreme ideological positions during primary election season when they are challenged, to later converge to the ideological center in contested general elections, and how these changes vary across parties. In doing so, we shed new light on well-developed but largely untested theories of electoral politics.

Improving Supreme Court Forecasting Using Boosted Decision Trees

Aaron Kaufman, Peter Kraft, and Maya Sen
Forthcoming at Political Analysis

Abstract: Though used frequently in machine learning, AdaBoosted decision trees (ADTs) are rarely used in political science, despite having many properties that are useful for social science inquiries. In this paper, we explain how to use ADTs for social science predictions. We illustrate their use by examining a well-known political prediction problem, predicting U.S. Supreme Court rulings. We nd that our AdaBoosted approach out-performs existing predictive models. We also provide two additional examples of the approach, one predicting the onset of civil wars and the other predicting county-level vote shares in U.S Presidential elections.

Draft Here

A Sequential Modeling Approach for Predicting Mortality of High Risk Critically Ill Patients

Rodrigo Octavio Deliberato, Stephanie Ko, Tejas Sundaresan, Aaron Russell Kaufman, and Leo Anthony Celi

Severity of illness scores are used for risk adjustment when comparing cohorts of critically ill patients in intensive care units (ICUs). Although these models have good discrimination, they are typically poorly calibrated, and over-predict mortality for low-risk patients and under-predict mortality for high-risk patients. ITherefore, clinicians have are skeptical of their accuracy for real-time patient prognostication. We propose a sequential modeling approach to improve these prediction models.  We hypothesized that by first stratifying patients into high (mortality prediction ≥ 10%) and low-risk cohorts, then applying four standard machine learning tools on a much larger set of candidate variables on only on the high-risk cohort, we could improve discrimination and calibration of mortality risk prediction in critically ill patients.

Testing Theories of Judicial Behavior Using Networks of Text

Aaron Kaufman and Maya Sen

Abstract: How open are Supreme Court Justices to being persuaded by attorneys at oral argument? We use novel network analyses to explore a novel data set of nearly six thousand Supreme Court oral argument transcripts to understand how and why the Justices speak. In so doing, we little to no evidence that Justices are primarily using oral arguments primarily as an information finding mechanism, and limited evidence that they use it to lobby or to try to influence vulnerable colleagues. Instead, consistent with theories that Justices behave more like political actors with well-formed policy preferences, we find that Justices take opportunities at oral argument to stake out positions. That is, Justices use oral arguments to delineate their potential vote and to indicate to their colleagues the strength of their beliefs. Ultimately, our contributions here challenge the idea that Justices approach oral arguments as information-fathering missions, in turn providing evidence for a decision-making model rooted in more firmly fixed attitudes.