Designing Social Inquiry

Robert Keohane recounts the origins of KKV as follows, Designing Social Inquiry was not generated by puzzles of world politics. Instead, it was the result of serendipity. Sid Verba and I were friends, and when I joined the Harvard Government Department in 1985, he said that we should teach a course together. I regarded this remark as a welcoming pleasantry, typical of Sid's grace and warmth. Three years later I became chair of the department and in my first year as chair was forced to listen to 24 job talks. Most of these talks were dead on arrival, since the speaker had made fundamental mistakes in research design. I complained to colleagues, including Sid, and Gary King. Gary said the three of us should teach a course on research design together... I agreed, and we taught the course the following year... After the semester was over, Gary said: “We should teach the course again. And this time, we should write a book on this subject.” The next year we met regularly for a bag lunch, discussing not only themes of the course but drafts that one of us—most often Gary, which is why his name appears first on the book—had produced. == Contents ==

The goal of the book is guide researchers in producing valid causal inferences in social science research. The book primarily applies lessons from regression-oriented analysis to qualitative research, arguing that the same logics of causal inference can be used in both types of research. KKV criticize Harry H. Eckstein's notion of "crucial case studies", warning that a single observation makes it harder to estimate multiple causal effects, more likely that there is measurement error, and risks that an event in a single case was caused by random error. According to the authors, a strong research design requires both qualitative and quantitative research, a research question that poses an important and real question that will contribute to the base of knowledge about this particular subject, and a comprehensive literature review from which hypotheses (theory-driven) are then drawn. Data that are collected should be operationalized so that other researchers could replicate the study and achieve similar results. For the same reason, the reasoning process behind the analysis needs to be explicit. While gathering data the researcher should consider the observable implications of the theory in an effort to explain as much of the data as possible. This is in addition to examining the causal mechanisms that connect one variable to another. While qualitative methods cannot produce precise measurements of uncertainty about the conclusions (unlike quantitative methods), qualitative scholars should give indications about the uncertainty of their inferences. KKV argue that "the single most serious problem with qualitative research in political science is the pervasive failure to provide reasonable estimates of the uncertainty of the investigator’s inferences." According to KKV, the rules for good causal theories are that they need to: • be falsifiable • have internal consistency (generate hypotheses that do not contradict each other) • have variation (explanatory variables should be exogenous and dependent variables should be endogenous) • have "concrete" concepts (concepts should be observable) • have "leverage" (the theory should explain much by little). KKV sees process-tracing and qualitative research as being "unable to yield strong causal inference" due to the fact that qualitative scholars would struggle with determining which of many intervening variables truly links the independent variable with a dependent variable. The primary problem is that qualitative research lacks a sufficient number of observations to properly estimate the effects of an independent variable. They write that the number of observations could be increased through various means, but that would simultaneously lead to another problem: that the number of variables would increase and thus reduce degrees of freedom. In terms of case selection, KKV warn against "selecting on the dependent variable". For example, researchers cannot make valid causal inferences about wars outbreak by only looking at instances where war did happen (the researcher should also look at cases where war did not happen). There is methodological problem in selecting on the explanatory variable, however. They do warn about multicollinearity (choosing two or more explanatory variables that perfectly correlate with each other). They argue that random selection of cases is a valid case selection strategy in large-N research, but warn against it in small-N research. KKV reject the notion of "quasi-experiments", arguing that either all the key causal variables can be controlled (an experiment) or not (a non-experiment). == Reception ==

In his 2010 review, James Mahoney writes that the field of social science methodology has "benefited from KKV even as it has also moved beyond it." There are different logics to the manner in which qualitative research is conducted and what qualitative scholars seek and can do with their data. John J. Mearsheimer and Stephen M. Walt argue that International Relations scholarship has shifted away from crafting and refining IR theory to "simplistic hypothesis-testing", in part due to the influence of KKV in political science graduate programs. Alexander George and Andrew Bennett say there is "much to agree with" in KKV, but they argue that the book has several flaws in its guidance on qualitative research: • Causal mechanisms: KKV suggest that "causal mechanisms" are less important that "causal effects" in causal explanations – George and Bennett argue that they are equally important • Hypothesis-testing: KKV overly emphasize the role of hypothesis-testing in theory development – George and Bennett argue that the formation of new hypotheses and the choosing of new questions are also important aspects of theory development • Causal complexity: KKV fail to consider problems of causal complexity, such as equifinality, multiple interaction effects, feedback loops, path dependency, tipping points, selection effects, expectations effects and sequential interactions – George and Bennet argue that case studies, process tracing and typological theories can clarify causality in situations of causal complexity • Increasing N: KKV argue that scholars should always seek to increase the number of cases – George and Bennett argue that KKV fail to consider the costs to increasing the number of cases (such as conceptual stretching and unintentional comparisons of dissimilar cases). George and Bennett note that much value can be derived from single-case studies. • Process-tracing: KKV characterize process-tracing as a way to increase the number of observable implications – George and Bennett argue that the logic of process-tracing is entirely different. The logic behind using process-tracing is to focus on the sequences and timings within a case, not to correlate data across cases. Thus, if one piece of evidence in the sequence is inconsistent with the theoretical expectations, then the theory has been shown to be flawed. • "Degrees of freedom" problem: KKV argue that a single case cannot evaluate competing explanations due problems that arise from degrees of freedom – George and Bennett argue that it is flawed to apply this statistical logic to qualitative research. George and Bennett say that while quantitative scholars try to aggregate variables to reduce the number of variables and thus increase the degrees of freedom, qualitative scholars intentionally want their variables to have many different attributes and complexity. ==Further reading==