Integrating statistics into economic theory to make causal claims leads to disagreements within the discipline, prompting criticism of econometrics. Most of these criticisms have been resolved as a result of the
credibility revolution and the improved rigor of the
potential outcomes framework, which is used today by applied economists, microeconomists, and econometricians to generate causal inferences. While econometricians began developing methods in the mid-1960s to improve statistical measures, the 2009 publication of
Mostly Harmless Econometrics by economists
Joshua D. Angrist and
Jörn-Steffen Pischke has summarized the advances in econometric modeling.
Structural causal modeling, which attempts to formalize the limitations of quasi-experimental methods from a causal perspective and enables experimenters to quantify the risks of quasi-experimental research precisely, is the primary academic response to this critique. Like other forms of statistical analysis, badly specified econometric models may show a
spurious relationship where two variables are correlated but causally unrelated. In a study of the use of econometrics in major economics journals,
McCloskey concluded that some economists report
p-values (following the
Fisherian tradition of
tests of significance of point
null-hypotheses) and neglect concerns of
type II errors; some economists fail to report estimates of the size of effects (apart from
statistical significance) and to discuss their economic importance. She also argues that some economists fail to use economic reasoning in
model selection, especially when deciding which variables to include in a regression. In some cases, economic variables cannot be experimentally manipulated as treatments randomly assigned to subjects. In such cases, economists rely on
observational studies, often using data sets with many strongly associated
covariates, resulting in enormous numbers of models with similar explanatory ability but different covariates and regression estimates. Regarding the plurality of models compatible with observational data-sets,
Edward Leamer urged that "professionals ... properly withhold belief until an inference can be shown to be adequately insensitive to the choice of assumptions".
Deirdre McCloskey argues that in published econometric work, economists often fail to use economic reasoning for including or excluding variables, equivocate statistical significance with substantial significance, and fail to report the
power of their findings. Economic variables are observed in reality, and therefore are not readily isolated for experimental testing.
Edward Leamer argued there was no essential difference between econometric analysis and
randomized trials or
controlled trials, provided the use of statistical techniques reduces the specification bias, the effects of collinearity between the variables, to the same order as the uncertainty due to the sample size. Today, this critique is unbinding, as advances in identification are stronger. Identification today may report the average treatment effect (ATE), the average treatment effect
on the treated (ATT), or the
local average treatment effect (LATE).
Specification bias, or
selection bias can be easily removed, through advances in sampling techniques and the ability to sample much larger populations through improved communications, data storage, and
randomization techniques. Secondly, collinearity can be easily controlled for using instrumental variables. By reporting either ATT or LATE, we can control for or eliminate heterogeneous error, reporting only the effects within the defined group. Economists, when using data, may have many explanatory
variables they want to use that are highly collinear, such that researcher bias may be important in variable selection. Leamer argues that economists can mitigate this by running statistical tests with different specified models and discarding any inferences that prove to be "fragile", concluding that "professionals ... properly withhold belief until an inference can be shown to be adequately insensitive to the choice of assumptions." P-hacking is not accepted in economics, and the requirement to disclose original data and the code to perform statistical analysis. However
Sala-I-Martin argued, it's possible to specify two models suggesting contrary relation between two variables. The phenomenon was labeled
emerging recalcitrant result phenomenon by
Robert Goldfarb. This is known as
two-way causality, and should be discussed with respect to the underlying theory that the mechanism is attempting to capture.
Kennedy (1998, p 1-2) reports econometricians as being accused of
using sledgehammers to crack open peanuts. That is they use a wide range of complex statistical techniques
while turning a blind eye to data deficiencies and the many questionable assumptions required for the application of these techniques. Kennedy quotes
Stefan Valavanis's 1959 Econometrics textbook's critique of practice:Econometric theory is like an exquisitely balanced French recipe, spelling out precisely with how many turns to mix the sauce, how many carats of spice to add, and for how many milliseconds to bake the mixture at exactly 474 degrees of temperature. But when the statistical cook turns to raw materials, he finds that hearts of cactus fruit are unavailable, so he substitutes chunks of cantaloupe; where the recipe calls for vermicelli he used shredded wheat; and he substitutes green garment die for curry, ping-pong balls for turtles eggs, and for Chalifougnac vintage 1883, a can of turpentine. (1959, p.83)
Macroeconomic critiques Looking primarily at macroeconomics,
Lawrence Summers has criticized econometric formalism, arguing that "the empirical facts of which we are most confident and which provide the most secure basis for theory are those that require the least sophisticated statistical analysis to perceive."
Summers is not critiquing the methodology itself but instead its usefulness in developing macroeconomic theory. He looks at two well-cited macroeconometric studies (
Hansen &
Singleton (1982, 1983), and
Bernanke (1986)), and argues that while both make brilliant use of econometric methods, neither paper speaks to formal theoretical proof. Noting that in the natural sciences, "investigators rush to check out the validity of claims made by rival laboratories and then build on them," Summers points out that this rarely happen in economics, which to him is a result of the fact that "the results [of econometric studies] are rarely an important input to theory creation or the evolution of professional opinion more generally." To Summers:
Lucas critique Robert Lucas criticised the use of overly simplistic econometric models of the macroeconomy to predict the implications of
economic policy, arguing that the structural relationships observed in historical models break down when decision-makers adjust their preferences in response to policy changes. Lucas argued that policy conclusions drawn from contemporary
large-scale macroeconometric models were invalid, as economic actors would revise their expectations about the future and adjust their behaviour accordingly. A good macroeconometric model should incorporate
microfoundations to model the effects of policy change, with equations representing economic
representative agents responding to economic changes based on
rational expectations of the future; implying their pattern of behaviour might be quite different if economic policy changed.
Austrian School critique The current-day
Austrian School of Economics typically rejects much of econometric modeling. The historical data used to make econometric models, they claim, represent behavior under circumstances idiosyncratic to the past; thus, econometric models show correlational rather than causal relationships. Econometricians have addressed this criticism by adopting quasi-experimental methodologies. Austrian school economists remain skeptical of these corrected models, maintaining their belief that statistical methods are unsuited to the social sciences. The Austrian School holds that the counterfactual must be known for a causal relationship to be established. The counterfactual changes could then be extracted from the observed changes, leaving only the changes attributable to the variable. Meeting this critique is very challenging since "there is no dependable method for ascertaining the uniquely correct counterfactual" for historical data. For non-historical data, the Austrian critique is met with
randomized controlled trials. Randomized controlled trials must be purposefully prepared, which historical data is not. The use of randomized controlled trials is becoming more common in social science research. In the United States, for example, the Education Sciences Reform Act of 2002 made funding for education research contingent on scientific validity defined in part as "experimental designs using random assignment, when feasible." In answering questions of causation, parametric statistics only address the Austrian critique in randomized controlled trials. If the data is not from a randomized controlled trial, econometricians meet the Austrian critique with
quasi-experimental methodologies, including identifying and exploiting
natural experiments. These methodologies attempt to extract the counterfactual post hoc, thereby justifying the use of parametric statistical tools. Since parametric statistics depends on any observation following a Gaussian distribution, which is only guaranteed by the
central limit theorem in a randomization methodology, the use of tools such as the confidence interval will be outside of their specification: the amount of selection bias will always be unknown. ==See also==