Bayesian information criterion

The BIC is formally defined as : \mathrm{BIC} = k\ln(n) - 2\ln(\widehat L). \ where • \hat L = the maximized value of the likelihood function of the model M, i.e. \hat L=p(x\mid\widehat\theta,M), where \{\widehat\theta\} are the parameter values that maximize the likelihood function and x is the observed data; • n = the number of data points in x, the number of observations, or equivalently, the sample size; • k = the number of parameters estimated by the model. For example, in multiple linear regression, the estimated parameters are the intercept, the q slope parameters, and the constant variance of the errors; thus, k = q + 2. == Derivation ==

The BIC can be derived by integrating out the parameters of the model using Laplace's method, starting with the following model evidence:) Define \ell_n(\theta)=-\ln(p(x\mid\theta,M)\pi(\theta\mid M))/n. If the sequence of gradients \{\nabla\ell_n(\theta)\} is Lipschitz continuous, uniformly in n, and every \ell_n(\theta) is strongly convex with a common parameter m>0 (independent of n), then as n\rightarrow\infty : \ln p(x\mid M)=\ln(p(x\mid\bar\theta_n,M)\pi(\bar\theta_n\mid M))-\frac{k}{2}\ln(n)+O(1), where \bar\theta_n=\arg\min_{\theta}\ell_n(\theta) is the maximum a posteriori (MAP) estimate of \theta. Note that it is always possible to select a prior density \pi(\theta\mid M) such that every \ell_n(\theta) is strongly convex (with the same parameter m). Such a prior ensures that the MAP estimate \bar\theta_n exists, even in cases where the MLE estimate \hat\theta itself does not exist. == Use ==

When picking from several models, ones with lower BIC values are generally preferred. The BIC is an increasing function of the error variance \sigma_e^2 and an increasing function of k. That is, unexplained variation in the dependent variable and the number of explanatory variables increase the value of BIC. However, a lower BIC does not necessarily indicate one model is better than another. Because it involves approximations, the BIC is merely a heuristic. In particular, differences in BIC should never be treated like transformed Bayes factors. It is important to keep in mind that the BIC can be used to compare estimated models only when the numerical values of the dependent variable are identical for all models being compared. The models being compared need not be nested, unlike the case when models are being compared using an F-test or a likelihood ratio test. To compare two different models, simply compute the BIC for each model and compare according to the table below: == Limitations ==

The BIC suffers from two main limitations: • the above approximation is only valid for sample size n much larger than the number k of parameters in the model. • the BIC cannot handle complex collections of models as in the variable selection (or feature selection) problem in high-dimension. == Gaussian special case ==

Under the assumption that the model errors or disturbances are independent and identically distributed according to a normal distribution and the boundary condition that the derivative of the log likelihood with respect to the true variance is zero, this becomes (up to an additive constant, which depends only on n and not on the model): : \mathrm{BIC} = n \ln(\widehat{\sigma_e^2}) + k \ln(n) \ where \widehat{\sigma_e^2} is the error variance. The error variance in this case is defined as : \widehat{\sigma_e^2} = \frac{1}{n} \sum_{i=1}^n (x_i-\widehat{x}_i)^2. which is a biased estimator for the true variance. In terms of the residual sum of squares (RSS) the BIC is : \mathrm{BIC} = n \ln(\text{RSS}/n) + k \ln(n) \ When testing multiple linear models against a saturated model, the BIC can be rewritten in terms of the deviance \chi^2 as: : \mathrm{BIC}= \chi^2 + k \ln(n) where k is the number of model parameters in the test. == See also ==