There exist several conceptually distinct ways of thinking about, designing, and conducting evaluation efforts. A number of the evaluation approaches in use today make unique contributions to solving important problems, while others refine existing approaches in some way.
Classification of approaches Two classifications of evaluation approaches by House and Stufflebeam and Webster can be combined into a manageable number of approaches in terms of their unique and important underlying principles. House considers all major evaluation approaches to be based on a common
ideology entitled
liberal democracy. Important principles of this ideology include freedom of choice, the uniqueness of the
individual and
empirical inquiry grounded in
objectivity. He also contends that they are all based on
subjectivist ethics, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is
utilitarian, in which "
the good" is determined by what maximizes a single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is
intuitionist/
pluralist, in which no single interpretation of "the good" is assumed and such interpretations need not be explicitly stated nor justified. These ethical positions have corresponding
epistemologies—
philosophies for obtaining
knowledge. The objectivist epistemology is associated with the utilitarian ethic; in general, it is used to acquire knowledge that can be externally verified (intersubjective agreement) through publicly exposed
methods and
data. The subjectivist epistemology is associated with the intuitionist/pluralist ethic and is used to acquire new knowledge based on existing personal knowledge, as well as experiences that are (explicit) or are not (tacit) available for public inspection. House then divides each epistemological approach into two main political perspectives. Firstly, approaches can take an elite perspective, focusing on the interests of managers and professionals; or they also can take a mass perspective, focusing on
consumers and
participatory approaches. Stufflebeam and Webster place approaches into one of three groups, according to their orientation toward the role of
values and ethical consideration. The political orientation promotes a positive or negative view of an object regardless of what its value actually is and might be—they call this
pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object—they call this
quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of an object—they call this true evaluation. When the above concepts are considered simultaneously, fifteen evaluation approaches can be identified in terms of epistemology, major perspective (from House), and orientation. Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective. Six quasi-evaluation approaches use an objectivist epistemology. Five of them—
experimental research, management
information systems, testing programs, objectives-based studies, and
content analysis—take an elite perspective. Accountability takes a mass perspective. Seven true evaluation approaches are included. Two approaches, decision-oriented and
policy studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and
connoisseur studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and
client-centered studies are based on a subjectivist epistemology from a mass perspective.
Summary of approaches The following table is used to summarize each approach in terms of four
attributes—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The purpose represents the desired
outcome for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped together.
Pseudo-evaluation Politically controlled and
public relations studies are based on an
objectivist epistemology from an
elite perspective. Although both of these approaches seek to misrepresent value interpretations about an object, they function differently from each other.
Information obtained through politically controlled studies is released or withheld to meet the special interests of the holder, whereas public relations information creates a positive image of an object regardless of the actual situation. Despite the application of both studies in
real scenarios, neither of these approaches is acceptable evaluation practice.
Objectivist, elite, quasi-evaluation As a group, these five approaches represent a highly respected collection of disciplined inquiry approaches. They are considered quasi-evaluation approaches because particular studies legitimately can focus only on questions of knowledge without addressing any questions of value. Such studies are, by definition, not evaluations. These approaches can produce characterizations without producing appraisals, although specific studies can produce both. Each of these approaches serves its intended purpose well. They are discussed roughly in order of the extent to which they approach the objectivist ideal. •
Experimental research is the best approach for determining
causal relationships between
variables. The potential problem with using this as an evaluation approach is that its highly controlled and stylized
methodology may not be sufficiently responsive to the dynamically changing needs of most human service programs. •
Management information systems (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily
quantifiable data usually available at regular intervals. •
Testing programs are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected. •
Objectives-based approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. The objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object. •
Content analysis is a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations.
Objectivist, mass, quasi-evaluation •
Accountability is popular with
constituents because it is intended to provide an accurate accounting of results that can improve the
quality of
products and
services. However, this approach quickly can turn practitioners and
consumers into adversaries when implemented in a heavy-handed fashion.
Objectivist, elite, true evaluation •
Decision-oriented studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close
collaboration between an evaluator and decision-maker, allowing it to be susceptible to
corruption and
bias. •
Policy studies provide general guidance and direction on broad issues by identifying and assessing potential
costs and benefits of competing
policies. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants.
Objectivist, mass, true evaluation •
Consumer-oriented studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a good and credible evaluator to do it well.
Subjectivist, elite, true evaluation •
Accreditation / certification programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects. •
Connoisseur studies use the highly refined skills of individuals intimately familiar with the subject of the evaluation to critically characterize and appraise it. This approach can help others see programs in a new light, but it is difficult to find a qualified and unbiased connoisseur.
Subject, mass, true evaluation • The
adversary approach focuses on drawing out the pros and cons of
controversial issues through quasi-
legal proceedings. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later
cooperation and heighten animosities between contesting parties if "winners" and "losers" emerge.
Client-centered •
Client-centered studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external
credibility and a favorable bias toward those who participated in the study. ==Methods and techniques==