Reliability (statistics)

There are several general classes of reliability estimates: • Inter-rater reliability assesses the degree of agreement between two or more raters in their appraisals. For example, a person gets a stomach ache and different doctors all give the same diagnosis. • Test-retest reliability assesses the degree to which test scores are consistent from one test administration to the next. Measurements are gathered from a single rater who uses the same methods or instruments and the same testing conditions. • Internal consistency reliability, assesses the consistency of results across items within a test. == Difference from validity ==

Reliability does not imply validity. That is, a reliable measure that is measuring something consistently is not necessarily measuring what is supposed to be measured. For example, while there are many reliable tests of specific abilities, not all of them would be valid for predicting, say, job performance. While reliability does not imply validity, reliability does place a limit on the overall validity of a test. A test that is not perfectly reliable cannot be perfectly valid, either as a means of measuring attributes of a person or as a means of predicting scores on a criterion. While a reliable test may provide useful valid information, a test that is not reliable cannot possibly be valid. For example, if a set of weighing scales consistently measured the weight of an object as 500 grams over the true weight, then the scale would be very reliable, but it would not be valid (as the returned weight is not the true weight). For the scale to be valid, it should return the true weight of an object. This example demonstrates that a perfectly reliable measure is not necessarily valid, but that a valid measure necessarily must be reliable. ==General model==

In practice, testing measures are never perfectly consistent. Theories of test reliability have been developed to estimate the effects of inconsistency on the accuracy of measurement. The basic starting point for almost all theories of test reliability is the idea that test scores reflect the influence of two sorts of factors: • Consistency factors: stable characteristics of the individual or the attribute that one is trying to measure. • Inconsistency factors: features of the individual or the situation that can affect test scores but have nothing to do with the attribute being measured. These factors include: • Temporary but general characteristics of the individual: health, fatigue, motivation, emotional strain • Temporary and specific characteristics of individual: comprehension of the specific test task, specific tricks or techniques of dealing with the particular test materials, fluctuations of memory, attention or accuracy • Aspects of the testing situation: freedom from distractions, clarity of instructions, interaction of personality, etc. • Chance factors: luck in selection of answers by sheer guessing, momentary distractions The goal of estimating reliability is to determine how much of the variability in test scores is due to measurement errors and how much is due to variability in true scores (true value). A true score is the replicable feature of the concept being measured. It is the part of the observed score that would recur across different measurement occasions in the absence of error. Errors of measurement are composed of both random error and systematic error. It represents the discrepancies between scores obtained on tests and the corresponding true scores. This conceptual breakdown is typically represented by the simple equation: X=T+E where X is the observed test score, T is the true score, and E is the measurement error ==Classical test theory==

The goal of reliability theory is to estimate errors in measurement and to suggest ways of improving tests so that errors are minimized. The central assumption of reliability theory is that measurement errors are essentially random. This does not mean that errors arise from random processes. For any individual, an error in measurement is not a completely random event. However, across a large number of individuals, the causes of measurement error are assumed to be so varied that measure errors act as random variables. • Mean error of measurement = 0 • True scores and errors are uncorrelated • Errors on different measures are uncorrelated Reliability theory shows that the variance of obtained scores is simply the sum of the variance of true scores plus the variance of errors of measurement. : \sigma^2_X = \sigma^2_T + \sigma^2_E This equation suggests that test scores vary as the result of two factors: • Variability in true scores • Variability due to errors of measurement. The reliability coefficient \rho_{xx'} provides an index of the relative influence of true and error scores on attained test scores. In its general form, the reliability coefficient is defined as the ratio of true score variance to the total variance of test scores. Or, equivalently, one minus the ratio of the variation of the error score and the variation of the observed score: : \rho_{xx'} = \frac{\sigma^2_T}{\sigma^2_X} = 1 - \frac{ \sigma^2_E }{ \sigma^2_X } = 1 - \frac{ \operatorname{Var}(E) }{ \operatorname{Var}(X) } Unfortunately, there is no way to directly observe or calculate the true score, so a variety of methods are used to estimate the reliability of a test. Some examples of the methods to estimate reliability include test-retest reliability, internal consistency reliability, and parallel-test reliability. Each method comes at the problem of figuring out the source of error in the test somewhat differently. ==Item response theory==

It was well known to classical test theorists that measurement precision is not uniform across the scale of measurement. Tests tend to distinguish better for test-takers with moderate trait levels and worse among high- and low-scoring test-takers. Item response theory extends the concept of reliability from a single index to a function called the information function. The IRT information function is the inverse of the conditional observed score standard error at any given test score. ==Estimation==

The goal of estimating reliability is to determine how much of the variability in test scores is due to errors in measurement and how much is due to variability in true scores. Four practical strategies have been developed that provide workable methods of estimating test reliability: Cronbach's alpha is a generalization of an earlier form of estimating internal consistency, Kuder–Richardson Formula 20. These measures of reliability differ in their sensitivity to different sources of error and so need not be equal. Also, reliability is a property of the scores of a measure rather than the measure itself and are thus said to be sample dependent. Reliability estimates from one sample might differ from those of a second sample (beyond what might be expected due to sampling variations) if the second sample is drawn from a different population because the true variability is different in this second population. (This is true of measures of all types—yardsticks might measure houses well yet have poor reliability when used to measure the lengths of insects.) Reliability may be improved by clarity of expression (for written assessments), lengthening the measure, and other informal means. However, formal psychometric analysis, called item analysis, is considered the most effective way to increase reliability. This analysis consists of computation of item difficulties and item discrimination indices, the latter index involving computation of correlations between the items and sum of the item scores of the entire test. If items that are too difficult, too easy, and/or have near-zero or negative discrimination are replaced with better items, the reliability of the measure will increase. • R(t) = 1 - F(t). • R(t) = \exp(-\lambda t), where \lambda is the failure rate. ==Ideal reliability level and how to increase reliability==

This section discusses recommendations for reliability of scale scores, relationship between reliability and validity, and strategies for increasing the reliability of scales scores. Standards for the level of reliability A widely cited hueristic for minimum reliability is 0.70 and Nunnally's (1978) textbook is often mentioned as the primary source for this standard. However, Nunnally actually said something rather different. He made a distinction between the early stages of basic research (where he thought that a reliability of 0.70 or higher was sufficient) and the situation where important decisions were being made about candidates (where "a reliability of 0.90 is the minimum ... and a reliability of 0.95 should be considered the desirable standard. (p. 246)"). Nunnally's recommendation of {\rho}_{xx'}=0.95 for important decision-making was based on a concern that the cost of false negative errors in decision-making fall disproportionately on the candidates. Given sufficient applicants, the organization just hires a different candidate or admits a different student, but the candidate suffers the loss of an opportunity. Nunnally's recommendations may therefore be cast as a recommendation for ethical decision-making via assessment. But he did not explicitly address any "costs" associated with attaining a high reliability. For example, if a 50-item test has reliability {\rho}_{xx'}=0.80, then the test length required for a reliability of {\rho}^*_{xx'}=0.95 is about 238 items (and the additional items must be comparable to the existing items). Nunnally did not address how an organization would fund an exam of 238 items nor how examinees would feel about sitting an exam with 238 items. He did not discuss whether fatigue effects might negatively impact candidate scores, nor whether extremely long test administration windows might disadvantage some classes of candidates. Trade-off with validity A high value of reliability can conflict with content validity if a psychometrician removes items to maximize an estimate like coefficient alpha without regard to the content of the remaining items. Repeatedly measuring essentially the same question in different ways is often used solely to increase reliability while damaging content validity. Trade-off with efficiency When the other conditions are equal, reliability increases as the number of items increases. However, the increase in the number of items hinders the efficiency of measurements. Methods to increase reliability The following methods can be considered to increase reliability. Before data collection: • Eliminate the ambiguity of the measurement item. • Do not measure what the respondents do not know. • Increase the number of items. • Use a scale that is known to be highly reliable. After data collection: • Use item-analysis to identify and remove problematic items (carefully avoiding damage to content validity). ==See also==