Evaluation
Encyclopedia
Evaluation is systematic determination of merit, worth, and significance of something or someone using criteria against a set of standards.
Evaluation often is used to characterize and appraise subjects of interest in a wide range of human enterprises, including the arts
, criminal justice
, foundations
and non-profit organization
s, government
, health care
, and other human services.
, that is taking place during the life of a project or organisation, with the intention of improving the strategy or way of functioning of the project or organisation. It can also be summative
, drawing lessons from a completed project or an organisation that is no longer functioning.
Evaluation is inherently a theoretically informed approach (whether explicitly or not), and consequently a definition of evaluation would have be tailored to the theory, approach, needs, purpose and methodology of the evaluation itself. Having said this, evaluation has been defined as:
An alternative view is that "projects, evaluators and other stakeholders (including funders) will all have potentially different ideas about how best to evaluate a project since each may have a different definition of ‘merit’. The core of the problem is thus about defining what is of value."
From this perspective, evaluation "is a contested term", as "evaluators" use the term evaluation to describe an assessment, or investigation of a program whilst others simply understand evaluation as being synonymous with applied research.
Not all evaluations serve the same purpose some evaluations serve a monitoring function rather than focusing solely on measurable program outcomes or evaluation findings and a full list of types of evaluations would be difficult to compile. This is because evaluation is not part of a unified theoretical framework, drawing on a number of disciplines, which include management
and organisational theory, policy analysis
, education
, sociology
, social anthropology
, and social change
.
Despite its progress, there are still many fundamental problems faced by this field as "unlike medicine, evaluation is not a discipline that has been developed by practicing professionals over thousands of years, so we are not yet at the stage where we have huge encyclopaedias that will walk us through any evaluation step-by-step", or provide a clear definition of what evaluation entails (Davidson, 2005).
It could therefore be argued that a key problem that evaluators face is the lack of a clear definition of evaluation, which may "underline why program evaluation is periodically called into question as an original process, whose primary function is the production of legitimate and justified judgments which serve as the bases for relevant recommendations."
However, the strict adherence to a set of methodological assumptions may make the field of evaluation more acceptable to a mainstream audience but this adherence will work towards preventing evaluators from developing new strategies for dealing with the myriad problems that programs face.
It is claimed that only a minority of evaluation reports are used by the evaluand (client) (Datta, 2006). One justification of this is that "when evaluation findings are challenged or utilization has failed, it was because stakeholders and clients found the inferences weak or the warrants unconvincing" (Fournier and Smith, 1993). Some reasons for this situation may be the failure of the evaluator to establish a set of shared aims with the evaluand, or creating overly ambitious aims, as well as failing to compromise and incorporate the cultural differences of individuals and programs within the evaluation aims and process.
None of these problems are due to a lack of a definition of evaluation but are rather due to evaluators attempting to impose predisposed notions and definitions of evaluations on clients. The central reason for the poor utilization of evaluations is arguably due to the lack of tailoring of evaluations to suit the needs of the client, due to a predefined idea (or definition) of what an evaluation is rather than what the client needs are (House, 1980).
Evaluating programs and projects, as to their value and impact within the context they are implemented, can be ethically challenging. Evaluators may encounter complex, culturally specific systems resistant to external evaluation. Furthermore, the project organization or other stakeholders may be invested in a particular evaluation outcome. Finally, evaluators themselves may encounter conflict of interests issues, or experience interference or pressure to present findings supporting a particular assessment.
General professional codes of conduct, as determined by the organization worked for, usually cover three broad aspects of standards of behaviour including inter-collegial relations (such as respect for diversity and privacy), operational issues (due competence, documentation accuracy and appropriate use of resources) and conflicts of interest (nepotism, accepting gifts) Evaluation Office, Global Environment Facility (2007). However, specific guidelines particular to the evaluator’s role and unique ethical challenges are required.
The Joint Committee on Standards for Educational Evaluation
has developed standards for program, personnel, and student evaluation. The Joint Committee standards are broken into four sections: Utility, Feasibility, Propriety, and Accuracy. Various European institutions have also prepared their own standards, more or less related to those produced by the Joint Committee. They provide guidelines about basing value judgments on systematic inquiry, evaluator competence and integrity, respect for people, and regard for the general and public welfare.
The American Evaluation Association has created a set of Guiding Principle
s for evaluators. The order of these principles does not imply priority among them; priority will vary by situation and evaluator role. The principles run as follows:
Furthermore, the international organizations such as the I.M.F. and the World Bank have independent evaluation functions. The various funds, programmes, and agencies of the United Nations has a mix of independent, semi-independent and self-evaluation functions, which have organized themselves as a system-wide UN Evaluation Group (UNEG), that works together to strengthen the function, and to establish UN norms and standards for evaluation. There is also an evaluation group within the OECD-DAC, which endeavors to improve development evaluation standards.
are conceptually distinct ways of thinking about, designing and conducting evaluation efforts. Many of the evaluation approaches in use today make truly unique contributions to solving important problems, while others refine existing approaches in some way.
House considers all major evaluation approaches to be based on a common ideology
, liberal democracy
. Important principles of this ideology include freedom of choice, the uniqueness of the individual
, and empirical
inquiry
grounded in objectivity
. He also contends they are all based on subjectivist ethics
, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is utilitarian
, in which “the good” is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is intuitionist
/ pluralist, in which no single interpretation of “the good” is assumed and these interpretations need not be explicitly stated nor justified.
These ethical positions have corresponding epistemologies—philosophies
of obtaining knowledge
. The objectivist epistemology is associated with the utilitarian ethic. In general, it is used to acquire knowledge capable of external verification (intersubjective agreement) through publicly inspectable methods
and data
. The subjectivist epistemology is associated with the intuitionist/pluralist ethic. It is used to acquire new knowledge based on existing personal knowledge and experiences that are (explicit) or are not (tacit) available for public
inspection.
House further divides each epistemological approach by two main political
perspectives. Approaches can take an elite
perspective, focusing on the interests of managers and professionals. They also can take a mass perspective, focusing on consumer
s and participatory
approaches.
Stufflebeam and Webster place approaches into one of three groups according to their orientation toward the role of values
, an ethical
consideration. The political orientation promotes a positive or negative view of an object regardless of what its value actually and might be. They call this pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object. They call this quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of some object. They call this true evaluation.
When the above concepts are considered simultaneously, fifteen evaluation approaches can be identified in terms of epistemology, major perspective (from House), and orientation. Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective. Six quasi-evaluation approaches use an objectivist epistemology. Five of them—experimental research, management information systems, testing programs, objectives-based studies, and content analysis—take an elite perspective. Accountability takes a mass perspective. Seven true evaluation approaches are included. Two approaches, decision-oriented and policy studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and connoisseur studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective.
—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The purpose represents the desired outcome
for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped together.
Pseudo-evaluation
Politically
controlled and public relations studies are based on an objectivist epistemology from an elite perspective. Although both of these approaches seek to misrepresent value interpretations about some object, they go about it a bit differently. Information
obtained through politically controlled studies is released or withheld to meet the special interests of the holder.
Public relations
information is used to paint a positive image of an object regardless of the actual situation. Neither of these approaches is acceptable evaluation practice, although the seasoned reader can surely think of a few examples where they have been used.
Objectivist, elite, quasi-evaluation
As a group, these five approaches represent a highly respected collection of disciplined inquiry approaches. They are considered quasi-evaluation approaches because particular studies legitimately can focus only on questions of knowledge without addressing any questions of value. Such studies are, by definition, not evaluations. These approaches can produce characterizations without producing appraisals, although specific studies can produce both. Each of these approaches serves its intended purpose well. They are discussed roughly in order of the extent to which they approach the objectivist ideal.
Experimental research
is the best approach for determining causal relationships
between variables. The potential problem with using this as an evaluation approach is that its highly controlled and stylized methodology
may not be sufficiently responsive to the dynamically changing needs of most human service programs.
Management information system
s (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily quantifiable
data usually available at regular intervals.
Testing programs are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected.
Objectives-based approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. Unfortunately, the objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object.
Content analysis
is a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations.
Objectivist, mass, quasi-evaluation
Accountability
is popular with constituents because it is intended to provide an accurate accounting of results that can improve the quality of products and services. However, this approach quickly can turn practitioners and consumer
s into adversaries when implemented in a heavy-handed fashion.
Objectivist, elite, true evaluation
Decision
-oriented studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close collaboration
between an evaluator and decision-maker, allowing it to be susceptible to corruption
and bias
.
Policy studies
provide general guidance and direction on broad issues by identifying and assessing potential costs and benefits
of competing policies
. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants.
Objectivist, mass, true evaluation
Consumer
-oriented studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a very good and credible evaluator to do it well.
Subjectivist, elite, true evaluation
Accreditation / certification
programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects.
Connoisseur
studies use the highly refined skills of individuals intimately familiar with the subject of the evaluation to critically characterize and appraise it. This approach can help others see programs in a new light, but it is difficult to find a qualified and unbiased connoisseur.
Subjectivist, mass, true evaluation
The adversary approach
focuses on drawing out the pros and cons of controversial
issues through quasi-legal proceedings
. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later cooperation and heighten animosities between contesting parties if “winners” and “losers” emerge.
Client-centered studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external credibility
and a favorable bias toward those who participated in the study.
, statistical analysis, and model building among others. A more detailed list of methods, techniques and approaches for conducting evaluations would include the following:
Evaluation often is used to characterize and appraise subjects of interest in a wide range of human enterprises, including the arts
ARts
aRts, which stands for analog Real time synthesizer, is an audio framework that is no longer under development. It is best known for previously being used in KDE to simulate an analog synthesizer....
, criminal justice
Criminal justice
Criminal Justice is the system of practices and institutions of governments directed at upholding social control, deterring and mitigating crime, or sanctioning those who violate laws with criminal penalties and rehabilitation efforts...
, foundations
Foundation (charity)
A foundation is a legal categorization of nonprofit organizations that will typically either donate funds and support to other organizations, or provide the source of funding for its own charitable purposes....
and non-profit organization
Non-profit organization
Nonprofit organization is neither a legal nor technical definition but generally refers to an organization that uses surplus revenues to achieve its goals, rather than distributing them as profit or dividends...
s, government
Government
Government refers to the legislators, administrators, and arbitrators in the administrative bureaucracy who control a state at a given time, and to the system of government by which they are organized...
, health care
Health care
Health care is the diagnosis, treatment, and prevention of disease, illness, injury, and other physical and mental impairments in humans. Health care is delivered by practitioners in medicine, chiropractic, dentistry, nursing, pharmacy, allied health, and other care providers...
, and other human services.
Definition
Evaluation is the comparison of actual impacts against strategic plans. It looks at original objectives, at what was accomplished and how it was accomplished. It can be formativeFormative assessment
Formative assessment is a range of formal and informal assessment procedures employed by teachers during the learning process in order to modify teaching and learning activities to improve student attainment. It typically involves qualitative feedback for both student and teacher that focuses on...
, that is taking place during the life of a project or organisation, with the intention of improving the strategy or way of functioning of the project or organisation. It can also be summative
Summative Assessment
Summative assessment refers to the assessment of the learning and summarizes the development of learners at a particular time. After a period of work, e.g. a unit for two weeks, the learner sits for a test and then the teacher marks the test and assigns a score. The test aims to summarize learning...
, drawing lessons from a completed project or an organisation that is no longer functioning.
Evaluation is inherently a theoretically informed approach (whether explicitly or not), and consequently a definition of evaluation would have be tailored to the theory, approach, needs, purpose and methodology of the evaluation itself. Having said this, evaluation has been defined as:
- A systematic, rigorous, and meticulous application of scientific methods to assess the design, implementation, improvement or outcomes of a program. It is a resource-intensive process, frequently requiring resources, such as, evaluator expertise, labour, time and a sizeable budget
- 'The critical assessment, in as objective a manner as possible, of the degree to which a service or its component parts fulfils stated goals' (St Leger and Walsworth-Bell). The focus of this definition is on attaining objective knowledge, and scientifically or quantitatively measuring predetermined and external concepts.
- 'A study designed to assist some audience to assess an object’s merit and worth' (Shufflebeam). In this definition the focus is on facts as well as value laden judgements of the programs outcomes and worth.
Purpose
The main purpose of a program evaluation can be to "determine the quality of a program by formulating a judgment" Marthe Hurteau,Sylvain Houle,Stéphanie Mongiat (2009).An alternative view is that "projects, evaluators and other stakeholders (including funders) will all have potentially different ideas about how best to evaluate a project since each may have a different definition of ‘merit’. The core of the problem is thus about defining what is of value."
From this perspective, evaluation "is a contested term", as "evaluators" use the term evaluation to describe an assessment, or investigation of a program whilst others simply understand evaluation as being synonymous with applied research.
Not all evaluations serve the same purpose some evaluations serve a monitoring function rather than focusing solely on measurable program outcomes or evaluation findings and a full list of types of evaluations would be difficult to compile. This is because evaluation is not part of a unified theoretical framework, drawing on a number of disciplines, which include management
Management
Management in all business and organizational activities is the act of getting people together to accomplish desired goals and objectives using available resources efficiently and effectively...
and organisational theory, policy analysis
Policy analysis
Policy analysis is "determining which of various alternative policies will most achieve a given set of goals in light of the relations between the policies and the goals". However, policy analysis can be divided into two major fields. Analysis of policy is analytical and descriptive—i.e., it...
, education
Education
Education in its broadest, general sense is the means through which the aims and habits of a group of people lives on from one generation to the next. Generally, it occurs through any experience that has a formative effect on the way one thinks, feels, or acts...
, sociology
Sociology
Sociology is the study of society. It is a social science—a term with which it is sometimes synonymous—which uses various methods of empirical investigation and critical analysis to develop a body of knowledge about human social activity...
, social anthropology
Social anthropology
Social Anthropology is one of the four or five branches of anthropology that studies how contemporary human beings behave in social groups. Practitioners of social anthropology investigate, often through long-term, intensive field studies , the social organization of a particular person: customs,...
, and social change
Social change
Social change refers to an alteration in the social order of a society. It may refer to the notion of social progress or sociocultural evolution, the philosophical idea that society moves forward by dialectical or evolutionary means. It may refer to a paradigmatic change in the socio-economic...
.
Discussion
Within the last three decades there have been tremendous theoretical and methodological developments within the field of evaluation.Despite its progress, there are still many fundamental problems faced by this field as "unlike medicine, evaluation is not a discipline that has been developed by practicing professionals over thousands of years, so we are not yet at the stage where we have huge encyclopaedias that will walk us through any evaluation step-by-step", or provide a clear definition of what evaluation entails (Davidson, 2005).
It could therefore be argued that a key problem that evaluators face is the lack of a clear definition of evaluation, which may "underline why program evaluation is periodically called into question as an original process, whose primary function is the production of legitimate and justified judgments which serve as the bases for relevant recommendations."
However, the strict adherence to a set of methodological assumptions may make the field of evaluation more acceptable to a mainstream audience but this adherence will work towards preventing evaluators from developing new strategies for dealing with the myriad problems that programs face.
It is claimed that only a minority of evaluation reports are used by the evaluand (client) (Datta, 2006). One justification of this is that "when evaluation findings are challenged or utilization has failed, it was because stakeholders and clients found the inferences weak or the warrants unconvincing" (Fournier and Smith, 1993). Some reasons for this situation may be the failure of the evaluator to establish a set of shared aims with the evaluand, or creating overly ambitious aims, as well as failing to compromise and incorporate the cultural differences of individuals and programs within the evaluation aims and process.
None of these problems are due to a lack of a definition of evaluation but are rather due to evaluators attempting to impose predisposed notions and definitions of evaluations on clients. The central reason for the poor utilization of evaluations is arguably due to the lack of tailoring of evaluations to suit the needs of the client, due to a predefined idea (or definition) of what an evaluation is rather than what the client needs are (House, 1980).
Standards
Depending on the topic of interest, there are professional groups which look to the quality and rigor of the evaluation process.Evaluating programs and projects, as to their value and impact within the context they are implemented, can be ethically challenging. Evaluators may encounter complex, culturally specific systems resistant to external evaluation. Furthermore, the project organization or other stakeholders may be invested in a particular evaluation outcome. Finally, evaluators themselves may encounter conflict of interests issues, or experience interference or pressure to present findings supporting a particular assessment.
General professional codes of conduct, as determined by the organization worked for, usually cover three broad aspects of standards of behaviour including inter-collegial relations (such as respect for diversity and privacy), operational issues (due competence, documentation accuracy and appropriate use of resources) and conflicts of interest (nepotism, accepting gifts) Evaluation Office, Global Environment Facility (2007). However, specific guidelines particular to the evaluator’s role and unique ethical challenges are required.
The Joint Committee on Standards for Educational Evaluation
Joint Committee on Standards for Educational Evaluation
The Joint Committee on Standards for Educational Evaluation is an American/Canadian based Standards Developer Organization . The Joint Committee represents a coalition of major professional associations formed in 1975 to help improve the quality of standardized evaluation. The Committee has thus...
has developed standards for program, personnel, and student evaluation. The Joint Committee standards are broken into four sections: Utility, Feasibility, Propriety, and Accuracy. Various European institutions have also prepared their own standards, more or less related to those produced by the Joint Committee. They provide guidelines about basing value judgments on systematic inquiry, evaluator competence and integrity, respect for people, and regard for the general and public welfare.
The American Evaluation Association has created a set of Guiding Principle
Principle
A principle is a law or rule that has to be, or usually is to be followed, or can be desirably followed, or is an inevitable consequence of something, such as the laws observed in nature or the way that a system is constructed...
s for evaluators. The order of these principles does not imply priority among them; priority will vary by situation and evaluator role. The principles run as follows:
- Systematic InquiryInquiryAn inquiry is any process that has the aim of augmenting knowledge, resolving doubt, or solving a problem. A theory of inquiry is an account of the various types of inquiry and a treatment of the ways that each type of inquiry achieves its aim.-Deduction:...
: Evaluators conduct systematic, dataDataThe term data refers to qualitative or quantitative attributes of a variable or set of variables. Data are typically the results of measurements and can be the basis of graphs, images, or observations of a set of variables. Data are often viewed as the lowest level of abstraction from which...
-based inquiries about whatever is being evaluated. This requires quality data collection, including a defensible choice of indicators, which lends credibility to findings. Findings are credible when they are demonstrably evidence-based, reliable and valid. This also pertains to the choice of methodology employed, such that it is consistent with the aims of the evaluation and provides dependable data. Furthermore, utility of findings is critical such that the information obtained by evaluation is comprehensive and timely, and thus serves to provide maximal benefit and use to stakeholders.
- CompetenceCompetence (human resources)Competence is the ability of an individual to perform a job properly. A competency is a set of defined behaviors that provide a structured guide enabling the identification, evaluation and development of the behaviors in individual employees. As defined, the term "competence" first appeared in...
: Evaluators provide competent performance to stakeholders. This requires that evaluation teams comprise an appropriate combination of competencies, such that varied and appropriate expertise is available for the evaluation process, and that evaluators work within their scope of capability.
- IntegrityIntegrityIntegrity is a concept of consistency of actions, values, methods, measures, principles, expectations, and outcomes. In ethics, integrity is regarded as the honesty and truthfulness or accuracy of one's actions...
/ HonestyHonestyHonesty refers to a facet of moral character and denotes positive, virtuous attributes such as integrity, truthfulness, and straightforwardness along with the absence of lying, cheating, or theft....
: Evaluators ensure the honesty and integrity of the entire evaluation process. A key element of this principle is freedom from bias in evaluation and this is underscored by three principles: impartiality, independence and transparency.
-
- Independence
- Independence is attained through ensuring independence of judgment is upheld such that evaluation
- conclusions are not influenced or pressured by another party, and avoidance of conflict of
- interest, such that the evaluator does not have a stake in a particular conclusion. Conflict of
- interest is at issue particularly where funding of evaluations is provided by particular bodies
- with a stake in conclusions of the evaluation, and this is seen as potentially compromising the
- independence of the evaluator. Whilst it is acknowledged that evaluators may be familiar with
- agencies or projects that they are required to evaluate, independence requires that they not have
- been involved in the planning or implementation of the project. A declaration of interest should
- be made where any benefits or association with project are stated. Independence of judgment is
- required to be maintained against any pressures brought to bear on evaluators, for example, by
- project funders wishing to modify evaluations such that the project appears more effective than
- findings can verify.
-
- Impartiality
- Impartiality pertains to findings being a fair and thorough assessment of strengths and
- weaknesses of a project or program. This requires taking due input from all stakeholders involved
- and findings presented without bias and with a transparent, proportionate and persuasive link
- between findings and recommendations. Thus evaluators are required to delimit their findings to
- evidence. A mechanism to ensure impartiality is external and internal review. Such review is
- required of significant (determined in terms of cost or sensitivity) evaluations. The review is
- based on quality of work and the degree to which a demonstrable link is provided between findings
- and recommendations.
-
- Transparency
- Transparency requires that stakeholders are aware of the reason for the evaluation, the criteria
- by which evaluation occurs and the purposes to which the findings will be applied. Access to the
- evaluation document should be facilitated through findings being easily readable, with clear
- explanations of evaluation methodologies, approaches, sources of information and costs
- incurred.
- RespectRespectRespect denotes both a positive feeling of esteem for a person or other entity , and also specific actions and conduct representative of that esteem. Respect can be a specific feeling of regard for the actual qualities of the one respected...
for PeoplePeoplePeople is a plurality of human beings or other beings possessing enough qualities constituting personhood. It has two usages:* as the plural of person or a group of people People is a plurality of human beings or other beings possessing enough qualities constituting personhood. It has two usages:*...
: Evaluators respect the securitySecuritySecurity is the degree of protection against danger, damage, loss, and crime. Security as a form of protection are structures and processes that provide or improve security as a condition. The Institute for Security and Open Methodologies in the OSSTMM 3 defines security as "a form of protection...
, dignityDignityDignity is a term used in moral, ethical, and political discussions to signify that a being has an innate right to respect and ethical treatment. It is an extension of the Enlightenment-era concepts of inherent, inalienable rights...
and self-worthSelf-esteemSelf-esteem is a term in psychology to reflect a person's overall evaluation or appraisal of his or her own worth. Self-esteem encompasses beliefs and emotions such as triumph, despair, pride and shame: some would distinguish how 'the self-concept is what we think about the self; self-esteem, the...
of the respondents, program participantsParticipation (decision making)Participation in social science refers to different mechanisms for the public to express opinions - and ideally exert influence - regarding political, economic, management or other social decisions. Participatory decision making can take place along any realm of human social activity, including...
, clientsConsumerConsumer is a broad label for any individuals or households that use goods generated within the economy. The concept of a consumer occurs in different contexts, so that the usage and significance of the term may vary.-Economics and marketing:...
, and other stakeholders with whom they interact.This is particularly pertinent with regards to those who will be impacted upon by the evaluation findings. Protection of people includes ensuring informed consent from those involved in the evaluation, upholding confidentiality, and ensuring that the identity of those who may provide sensitive information towards the program evaluation is protected. Evaluators are ethically required to respect the customs and beliefs of those who are impacted upon by the evaluation or program activities. Examples of how such respect is demonstrated is through respecting local customs e.g. dress codes, respecting peoples privacy, and minimizing demands on others' time. Where stakeholders wish to place objections to evaluation findings, such a process should be facilitated through the local office of the evaluation organization, and procedures for lodging complaints or queries should be accessible and clear.
- ResponsibilitiesSocial responsibilitySocial responsibility is an ethical ideology or theory that an entity, be it an organization or individual, has an obligation to act to benefit society at large. Social responsibility is a duty every individual or organization has to perform so as to maintain a balance between the economy and the...
for General and PublicPublicIn public relations and communication science, publics are groups of individuals, and the public is the totality of such groupings. This is a different concept to the sociological concept of the Öffentlichkeit or public sphere. The concept of a public has also been defined in political science,...
WelfareQuality of lifeThe term quality of life is used to evaluate the general well-being of individuals and societies. The term is used in a wide range of contexts, including the fields of international development, healthcare, and politics. Quality of life should not be confused with the concept of standard of...
: Evaluators articulate and take into account the diversity of interestInterestInterest is a fee paid by a borrower of assets to the owner as a form of compensation for the use of the assets. It is most commonly the price paid for the use of borrowed money, or money earned by deposited funds....
s and valueValue (personal and cultural)A personal or cultural value is an absolute or relative ethical value, the assumption of which can be the basis for ethical action. A value system is a set of consistent values and measures. A principle value is a foundation upon which other values and measures of integrity are based...
s that may be related to the general and public welfare. Access to evaluation documents by the wider public should be facilitated such that discussion and feedback is enabled.
Furthermore, the international organizations such as the I.M.F. and the World Bank have independent evaluation functions. The various funds, programmes, and agencies of the United Nations has a mix of independent, semi-independent and self-evaluation functions, which have organized themselves as a system-wide UN Evaluation Group (UNEG), that works together to strengthen the function, and to establish UN norms and standards for evaluation. There is also an evaluation group within the OECD-DAC, which endeavors to improve development evaluation standards.
Approaches
Evaluation approachesEvaluation approaches
Evaluation approaches are conceptually distinct ways of thinking about, designing and conducting evaluation efforts. Many of the evaluation approaches in use today make unique contributions to solving important problems, while others refine existing approaches in some way...
are conceptually distinct ways of thinking about, designing and conducting evaluation efforts. Many of the evaluation approaches in use today make truly unique contributions to solving important problems, while others refine existing approaches in some way.
Classification of approaches
Two classifications of evaluation approaches by House and Stufflebeam & Webster can be combined into a manageable number of approaches in terms of their unique and important underlying principles.House considers all major evaluation approaches to be based on a common ideology
Ideology
An ideology is a set of ideas that constitutes one's goals, expectations, and actions. An ideology can be thought of as a comprehensive vision, as a way of looking at things , as in common sense and several philosophical tendencies , or a set of ideas proposed by the dominant class of a society to...
, liberal democracy
Liberal democracy
Liberal democracy, also known as constitutional democracy, is a common form of representative democracy. According to the principles of liberal democracy, elections should be free and fair, and the political process should be competitive...
. Important principles of this ideology include freedom of choice, the uniqueness of the individual
Individual
An individual is a person or any specific object or thing in a collection. Individuality is the state or quality of being an individual; a person separate from other persons and possessing his or her own needs, goals, and desires. Being self expressive...
, and empirical
Empiricism
Empiricism is a theory of knowledge that asserts that knowledge comes only or primarily via sensory experience. One of several views of epistemology, the study of human knowledge, along with rationalism, idealism and historicism, empiricism emphasizes the role of experience and evidence,...
inquiry
Inquiry
An inquiry is any process that has the aim of augmenting knowledge, resolving doubt, or solving a problem. A theory of inquiry is an account of the various types of inquiry and a treatment of the ways that each type of inquiry achieves its aim.-Deduction:...
grounded in objectivity
Objectivity (philosophy)
Objectivity is a central philosophical concept which has been variously defined by sources. A proposition is generally considered to be objectively true when its truth conditions are met and are "mind-independent"—that is, not met by the judgment of a conscious entity or subject.- Objectivism...
. He also contends they are all based on subjectivist ethics
Ethics
Ethics, also known as moral philosophy, is a branch of philosophy that addresses questions about morality—that is, concepts such as good and evil, right and wrong, virtue and vice, justice and crime, etc.Major branches of ethics include:...
, in which ethical conduct is based on the subjective or intuitive experience of an individual or group. One form of subjectivist ethics is utilitarian
Utilitarianism
Utilitarianism is an ethical theory holding that the proper course of action is the one that maximizes the overall "happiness", by whatever means necessary. It is thus a form of consequentialism, meaning that the moral worth of an action is determined only by its resulting outcome, and that one can...
, in which “the good” is determined by what maximizes some single, explicit interpretation of happiness for society as a whole. Another form of subjectivist ethics is intuitionist
Ethical intuitionism
Ethical intuitionism is usually understood as a meta-ethical theory that embraces the following theses:# Moral realism, the view that there are objective facts of morality,...
/ pluralist, in which no single interpretation of “the good” is assumed and these interpretations need not be explicitly stated nor justified.
These ethical positions have corresponding epistemologies—philosophies
Philosophy
Philosophy is the study of general and fundamental problems, such as those connected with existence, knowledge, values, reason, mind, and language. Philosophy is distinguished from other ways of addressing such problems by its critical, generally systematic approach and its reliance on rational...
of obtaining knowledge
Knowledge
Knowledge is a familiarity with someone or something unknown, which can include information, facts, descriptions, or skills acquired through experience or education. It can refer to the theoretical or practical understanding of a subject...
. The objectivist epistemology is associated with the utilitarian ethic. In general, it is used to acquire knowledge capable of external verification (intersubjective agreement) through publicly inspectable methods
Methodology
Methodology is generally a guideline for solving a problem, with specificcomponents such as phases, tasks, methods, techniques and tools . It can be defined also as follows:...
and data
Data
The term data refers to qualitative or quantitative attributes of a variable or set of variables. Data are typically the results of measurements and can be the basis of graphs, images, or observations of a set of variables. Data are often viewed as the lowest level of abstraction from which...
. The subjectivist epistemology is associated with the intuitionist/pluralist ethic. It is used to acquire new knowledge based on existing personal knowledge and experiences that are (explicit) or are not (tacit) available for public
Public
In public relations and communication science, publics are groups of individuals, and the public is the totality of such groupings. This is a different concept to the sociological concept of the Öffentlichkeit or public sphere. The concept of a public has also been defined in political science,...
inspection.
House further divides each epistemological approach by two main political
Politics
Politics is a process by which groups of people make collective decisions. The term is generally applied to the art or science of running governmental or state affairs, including behavior within civil governments, but also applies to institutions, fields, and special interest groups such as the...
perspectives. Approaches can take an elite
Elite
Elite refers to an exceptional or privileged group that wields considerable power within its sphere of influence...
perspective, focusing on the interests of managers and professionals. They also can take a mass perspective, focusing on consumer
Consumer
Consumer is a broad label for any individuals or households that use goods generated within the economy. The concept of a consumer occurs in different contexts, so that the usage and significance of the term may vary.-Economics and marketing:...
s and participatory
Participation (decision making)
Participation in social science refers to different mechanisms for the public to express opinions - and ideally exert influence - regarding political, economic, management or other social decisions. Participatory decision making can take place along any realm of human social activity, including...
approaches.
Stufflebeam and Webster place approaches into one of three groups according to their orientation toward the role of values
Value (personal and cultural)
A personal or cultural value is an absolute or relative ethical value, the assumption of which can be the basis for ethical action. A value system is a set of consistent values and measures. A principle value is a foundation upon which other values and measures of integrity are based...
, an ethical
Ethics
Ethics, also known as moral philosophy, is a branch of philosophy that addresses questions about morality—that is, concepts such as good and evil, right and wrong, virtue and vice, justice and crime, etc.Major branches of ethics include:...
consideration. The political orientation promotes a positive or negative view of an object regardless of what its value actually and might be. They call this pseudo-evaluation. The questions orientation includes approaches that might or might not provide answers specifically related to the value of an object. They call this quasi-evaluation. The values orientation includes approaches primarily intended to determine the value of some object. They call this true evaluation.
When the above concepts are considered simultaneously, fifteen evaluation approaches can be identified in terms of epistemology, major perspective (from House), and orientation. Two pseudo-evaluation approaches, politically controlled and public relations studies, are represented. They are based on an objectivist epistemology from an elite perspective. Six quasi-evaluation approaches use an objectivist epistemology. Five of them—experimental research, management information systems, testing programs, objectives-based studies, and content analysis—take an elite perspective. Accountability takes a mass perspective. Seven true evaluation approaches are included. Two approaches, decision-oriented and policy studies, are based on an objectivist epistemology from an elite perspective. Consumer-oriented studies are based on an objectivist epistemology from a mass perspective. Two approaches—accreditation/certification and connoisseur studies—are based on a subjectivist epistemology from an elite perspective. Finally, adversary and client-centered studies are based on a subjectivist epistemology from a mass perspective.
Summary of approaches
The following table is used to summarize each approach in terms of four attributesAbstraction
Abstraction is a process by which higher concepts are derived from the usage and classification of literal concepts, first principles, or other methods....
—organizer, purpose, strengths, and weaknesses. The organizer represents the main considerations or cues practitioners use to organize a study. The purpose represents the desired outcome
Outcome (Game theory)
In game theory, an outcome is a set of moves or strategies taken by the players, or their payoffs resulting from the actions or strategies taken by all players. The two are complementary in that, given knowledge of the set of strategies of all players, the final state of the game is known, as are...
for a study at a very general level. Strengths and weaknesses represent other attributes that should be considered when deciding whether to use the approach for a particular study. The following narrative highlights differences between approaches grouped together.
| Summary of approaches for conducting evaluations | ||||
|---|---|---|---|---|
| Approach | Attribute | |||
| Organizer | Purpose | Key strengths | Key weaknesses | |
| Politically Politics Politics is a process by which groups of people make collective decisions. The term is generally applied to the art or science of running governmental or state affairs, including behavior within civil governments, but also applies to institutions, fields, and special interest groups such as the... controlled |
Threats | Get, keep or increase influence, power or money. | Secure evidence advantageous to the client in a conflict. | Violates the principle of full & frank disclosure. |
| Public relations Public relations Public relations is the actions of a corporation, store, government, individual, etc., in promoting goodwill between itself and the public, the community, employees, customers, etc.... |
Propaganda needs | Create positive public image. | Secure evidence most likely to bolster public support. | Violates the principles of balanced reporting, justified conclusions, & objectivity. |
| Experimental research Experiment An experiment is a methodical procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. Experiments vary greatly in their goal and scale, but always rely on repeatable procedure and logical analysis of the results... |
Causal relationships | Determine causal relationships between variables. | Strongest paradigm for determining causal relationships. | Requires controlled setting, limits range of evidence, focuses primarily on results. |
| Management information system Management information system A management information system provides information needed to manage organizations efficiently and effectively. Management information systems involve three primary resources: people, technology, and information. Management information systems are distinct from other information systems in that... s |
Scientific efficiency | Continuously supply evidence needed to fund, direct, & control programs. | Gives managers detailed evidence about complex programs. | Human service variables are rarely amenable to the narrow, quantitative definitions needed. |
| Testing programs | Individual differences | Compare test scores of individuals & groups to selected norms. | Produces valid & reliable evidence in many performance areas. Very familiar to public. | Data usually only on testee performance, overemphasizes test-taking skills, can be poor sample of what is taught or expected. |
| Objectives-based | Objectives | Relates outcomes to objectives. | Common sense appeal, widely used, uses behavioral objectives & testing technologies. | Leads to terminal evidence often too narrow to provide basis for judging the value of a program. |
| Content analysis Content analysis Content analysis or textual analysis is a methodology in the social sciences for studying the content of communication. Earl Babbie defines it as "the study of recorded human communications, such as books, websites, paintings and laws."According to Dr... |
Content of a communication | Describe & draw conclusion about a communication. | Allows for unobtrusive analysis of large volumes of unstructured, symbolic materials. | Sample may be unrepresentative yet overwhelming in volume. Analysis design often overly simplistic for question. |
| Accountability Accountability Accountability is a concept in ethics and governance with several meanings. It is often used synonymously with such concepts as responsibility, answerability, blameworthiness, liability, and other terms associated with the expectation of account-giving... |
Performance expectations | Provide constituents with an accurate accounting of results. | Popular with constituents. Aimed at improving quality of products and services. | Creates unrest between practitioners & consumers. Politics often forces premature studies. |
| Decision Decision making Decision making can be regarded as the mental processes resulting in the selection of a course of action among several alternative scenarios. Every decision making process produces a final choice. The output can be an action or an opinion of choice.- Overview :Human performance in decision terms... -oriented |
Decisions | Provide a knowledge & value base for making & defending decisions. | Encourages use of evaluation to plan & implement needed programs. Helps justify decisions about plans & actions. | Necessary collaboration between evaluator & decision-maker provides opportunity to bias results. |
| Policy studies Policy analysis Policy analysis is "determining which of various alternative policies will most achieve a given set of goals in light of the relations between the policies and the goals". However, policy analysis can be divided into two major fields. Analysis of policy is analytical and descriptive—i.e., it... |
Broad issues | Identify and assess potential costs & benefits of competing policies. | Provide general direction for broadly focused actions. | Often corrupted or subverted by politically motivated actions of participants. |
| Consumer Customer A customer is usually used to refer to a current or potential buyer or user of the products of an individual or organization, called the supplier, seller, or vendor. This is typically through purchasing or renting goods or services... -oriented |
Generalized needs & values, effects | Judge the relative merits of alternative goods & services. | Independent appraisal to protect practitioners & consumers from shoddy products & services. High public credibility. | Might not help practitioners do a better job. Requires credible & competent evaluators. |
| Accreditation / certification Professional certification Professional certification, trade certification, or professional designation, often called simply certification or qualification, is a designation earned by a person to assure qualification to perform a job or task... |
Standards & guidelines | Determine if institutions, programs, & personnel should be approved to perform specified functions. | Helps public make informed decisions about quality of organizations & qualifications of personnel. | Standards & guidelines typically emphasize intrinsic criteria to the exclusion of outcome measures. |
| Connoisseur Connoisseur A connoisseur is a person who has a great deal of knowledge about the fine arts, cuisines, or an expert judge in matters of taste.Modern connoisseurship must be seen along with museums, art galleries and "the cult of originality"... |
Critical guideposts | Critically describe, appraise, & illuminate an object. | Exploits highly developed expertise on subject of interest. Can inspire others to more insightful efforts. | Dependent on small number of experts, making evaluation susceptible to subjectivity, bias, and corruption. |
| Adversary Evaluation Adversary evaluation According to Alkin and Christie, an adversary approach is one which reflects a valuing orientation. This approach developed in response to the dominant objectifying approaches in policy evaluation and is based on the notions that: 1) no evaluator can be truly objective, and, 2) no evaluation can... |
“Hot” issues | Present the pro & cons of an issue. | Ensures balances presentations of represented perspectives. | Can discourage cooperation, heighten animosities. |
| Client-centered | Specific concerns & issues | Foster understanding of activities & how they are valued in a given setting & from a variety of perspectives. | Practitioners are helped to conduct their own evaluation. | Low external credibility, susceptible to bias in favor of participants. |
| Note. Adapted and condensed primarily from House (1978) and Stufflebeam & Webster (1980). | ||||
Pseudo-evaluation
Politically
Politics
Politics is a process by which groups of people make collective decisions. The term is generally applied to the art or science of running governmental or state affairs, including behavior within civil governments, but also applies to institutions, fields, and special interest groups such as the...
controlled and public relations studies are based on an objectivist epistemology from an elite perspective. Although both of these approaches seek to misrepresent value interpretations about some object, they go about it a bit differently. Information
Information
Information in its most restricted technical sense is a message or collection of messages that consists of an ordered sequence of symbols, or it is the meaning that can be interpreted from such a message or collection of messages. Information can be recorded or transmitted. It can be recorded as...
obtained through politically controlled studies is released or withheld to meet the special interests of the holder.
Public relations
Public relations
Public relations is the actions of a corporation, store, government, individual, etc., in promoting goodwill between itself and the public, the community, employees, customers, etc....
information is used to paint a positive image of an object regardless of the actual situation. Neither of these approaches is acceptable evaluation practice, although the seasoned reader can surely think of a few examples where they have been used.
Objectivist, elite, quasi-evaluation
As a group, these five approaches represent a highly respected collection of disciplined inquiry approaches. They are considered quasi-evaluation approaches because particular studies legitimately can focus only on questions of knowledge without addressing any questions of value. Such studies are, by definition, not evaluations. These approaches can produce characterizations without producing appraisals, although specific studies can produce both. Each of these approaches serves its intended purpose well. They are discussed roughly in order of the extent to which they approach the objectivist ideal.
Experimental research
Experiment
An experiment is a methodical procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. Experiments vary greatly in their goal and scale, but always rely on repeatable procedure and logical analysis of the results...
is the best approach for determining causal relationships
Causality
Causality is the relationship between an event and a second event , where the second event is understood as a consequence of the first....
between variables. The potential problem with using this as an evaluation approach is that its highly controlled and stylized methodology
Methodology
Methodology is generally a guideline for solving a problem, with specificcomponents such as phases, tasks, methods, techniques and tools . It can be defined also as follows:...
may not be sufficiently responsive to the dynamically changing needs of most human service programs.
Management information system
Management information system
A management information system provides information needed to manage organizations efficiently and effectively. Management information systems involve three primary resources: people, technology, and information. Management information systems are distinct from other information systems in that...
s (MISs) can give detailed information about the dynamic operations of complex programs. However, this information is restricted to readily quantifiable
Numerical data
Numerical data is data measured or identified on a numerical scale. Numerical data can be analyzed using statistical methods, and results can be displayed using tables, charts, histograms and graphs. For example, a researcher will ask a questions to a participant that include words how often, how...
data usually available at regular intervals.
Testing programs are familiar to just about anyone who has attended school, served in the military, or worked for a large company. These programs are good at comparing individuals or groups to selected norms in a number of subject areas or to a set of standards of performance. However, they only focus on testee performance and they might not adequately sample what is taught or expected.
Objectives-based approaches relate outcomes to prespecified objectives, allowing judgments to be made about their level of attainment. Unfortunately, the objectives are often not proven to be important or they focus on outcomes too narrow to provide the basis for determining the value of an object.
Content analysis
Content analysis
Content analysis or textual analysis is a methodology in the social sciences for studying the content of communication. Earl Babbie defines it as "the study of recorded human communications, such as books, websites, paintings and laws."According to Dr...
is a quasi-evaluation approach because content analysis judgments need not be based on value statements. Instead, they can be based on knowledge. Such content analyses are not evaluations. On the other hand, when content analysis judgments are based on values, such studies are evaluations.
Objectivist, mass, quasi-evaluation
Accountability
Accountability
Accountability is a concept in ethics and governance with several meanings. It is often used synonymously with such concepts as responsibility, answerability, blameworthiness, liability, and other terms associated with the expectation of account-giving...
is popular with constituents because it is intended to provide an accurate accounting of results that can improve the quality of products and services. However, this approach quickly can turn practitioners and consumer
Consumer
Consumer is a broad label for any individuals or households that use goods generated within the economy. The concept of a consumer occurs in different contexts, so that the usage and significance of the term may vary.-Economics and marketing:...
s into adversaries when implemented in a heavy-handed fashion.
Objectivist, elite, true evaluation
Decision
Decision making
Decision making can be regarded as the mental processes resulting in the selection of a course of action among several alternative scenarios. Every decision making process produces a final choice. The output can be an action or an opinion of choice.- Overview :Human performance in decision terms...
-oriented studies are designed to provide a knowledge base for making and defending decisions. This approach usually requires the close collaboration
Collaboration
Collaboration is working together to achieve a goal. It is a recursive process where two or more people or organizations work together to realize shared goals, — for example, an intriguing endeavor that is creative in nature—by sharing...
between an evaluator and decision-maker, allowing it to be susceptible to corruption
Political corruption
Political corruption is the use of legislated powers by government officials for illegitimate private gain. Misuse of government power for other purposes, such as repression of political opponents and general police brutality, is not considered political corruption. Neither are illegal acts by...
and bias
Bias
Bias is an inclination to present or hold a partial perspective at the expense of alternatives. Bias can come in many forms.-In judgement and decision making:...
.
Policy studies
Policy analysis
Policy analysis is "determining which of various alternative policies will most achieve a given set of goals in light of the relations between the policies and the goals". However, policy analysis can be divided into two major fields. Analysis of policy is analytical and descriptive—i.e., it...
provide general guidance and direction on broad issues by identifying and assessing potential costs and benefits
Cost-benefit analysis
Cost–benefit analysis , sometimes called benefit–cost analysis , is a systematic process for calculating and comparing benefits and costs of a project for two purposes: to determine if it is a sound investment , to see how it compares with alternate projects...
of competing policies
Policy
A policy is typically described as a principle or rule to guide decisions and achieve rational outcome. The term is not normally used to denote what is actually done, this is normally referred to as either procedure or protocol...
. The drawback is these studies can be corrupted or subverted by the politically motivated actions of the participants.
Objectivist, mass, true evaluation
Consumer
Customer
A customer is usually used to refer to a current or potential buyer or user of the products of an individual or organization, called the supplier, seller, or vendor. This is typically through purchasing or renting goods or services...
-oriented studies are used to judge the relative merits of goods and services based on generalized needs and values, along with a comprehensive range of effects. However, this approach does not necessarily help practitioners improve their work, and it requires a very good and credible evaluator to do it well.
Subjectivist, elite, true evaluation
Accreditation / certification
Professional certification
Professional certification, trade certification, or professional designation, often called simply certification or qualification, is a designation earned by a person to assure qualification to perform a job or task...
programs are based on self-study and peer review of organizations, programs, and personnel. They draw on the insights, experience, and expertise of qualified individuals who use established guidelines to determine if the applicant should be approved to perform specified functions. However, unless performance-based standards are used, attributes of applicants and the processes they perform often are overemphasized in relation to measures of outcomes or effects.
Connoisseur
Connoisseur
A connoisseur is a person who has a great deal of knowledge about the fine arts, cuisines, or an expert judge in matters of taste.Modern connoisseurship must be seen along with museums, art galleries and "the cult of originality"...
studies use the highly refined skills of individuals intimately familiar with the subject of the evaluation to critically characterize and appraise it. This approach can help others see programs in a new light, but it is difficult to find a qualified and unbiased connoisseur.
Subjectivist, mass, true evaluation
The adversary approach
Adversary evaluation
According to Alkin and Christie, an adversary approach is one which reflects a valuing orientation. This approach developed in response to the dominant objectifying approaches in policy evaluation and is based on the notions that: 1) no evaluator can be truly objective, and, 2) no evaluation can...
focuses on drawing out the pros and cons of controversial
Controversy
Controversy is a state of prolonged public dispute or debate, usually concerning a matter of opinion. The word was coined from the Latin controversia, as a composite of controversus – "turned in an opposite direction," from contra – "against" – and vertere – to turn, or versus , hence, "to turn...
issues through quasi-legal proceedings
Lawsuit
A lawsuit or "suit in law" is a civil action brought in a court of law in which a plaintiff, a party who claims to have incurred loss as a result of a defendant's actions, demands a legal or equitable remedy. The defendant is required to respond to the plaintiff's complaint...
. This helps ensure a balanced presentation of different perspectives on the issues, but it is also likely to discourage later cooperation and heighten animosities between contesting parties if “winners” and “losers” emerge.
Client-centered studies address specific concerns and issues of practitioners and other clients of the study in a particular setting. These studies help people understand the activities and values involved from a variety of perspectives. However, this responsive approach can lead to low external credibility
Credibility
Credibility refers to the objective and subjective components of the believability of a source or message.Traditionally, modern, credibility has two key components: trustworthiness and expertise, which both have objective and subjective components. Trustworthiness is based more on subjective...
and a favorable bias toward those who participated in the study.
Methods and techniques
Evaluation is methodologically diverse using both qualitative methods and quantitative methods, including case studies, survey researchSurvey research
Survey research a research method involving the use of questionnaires and/or statistical surveys to gather data about people and their thoughts and behaviours. This method was pioneered in the 1930s and 1940s by sociologist Paul Lazarsfeld. The initial use of the method was to examine the effects...
, statistical analysis, and model building among others. A more detailed list of methods, techniques and approaches for conducting evaluations would include the following:
- Accelerated agingAccelerated agingAccelerated aging is testing that uses aggravated conditions of heat, oxygen, sunlight, vibration, etc. to speed up the normal aging processes of items. It is used to help determine the long term effects of expected levels of stress within a shorter time, usually in a laboratory by controlled...
- Action researchAction researchAction research or participatory action research – is a reflective process of progressive problem solving led by individuals working with others in teams or as part of a "community of practice" to improve the way they address issues and solve problems. Action research is done simply by action,...
- Advanced product quality planningAdvanced Product Quality PlanningAdvanced product quality planning is a framework of procedures and techniques used to develop products in industry, particularly the automotive industry. It is quite similar to the concept of Design For Six Sigma ....
- Alternative assessmentAlternative assessmentIn the education industry, alternative assessment or portfolio assessment is in direct contrast to what is known as performance evaluation, traditional assessment, standardized assessment or summative assessment...
- Appreciative InquiryAppreciative inquiryAppreciative Inquiry is primarily an organisational development method which seeks to engage all levels of an organisation to renew, change and improved performance. Its exponents view it as being applicable to organisations facing rapid change or growth...
- Assessment
- Axiomatic designAxiomatic designAxiomatic design is a systems design methodology using matrix methods to systematically analyze the transformation of customer needs into functional requirements, design parameters, and process variables....
- BenchmarkingBenchmarkingBenchmarking is the process of comparing one's business processes and performance metrics to industry bests and/or best practices from other industries. Dimensions typically measured are quality, time and cost...
- Case studyCase studyA case study is an intensive analysis of an individual unit stressing developmental factors in relation to context. The case study is common in social sciences and life sciences. Case studies may be descriptive or explanatory. The latter type is used to explore causation in order to find...
- Change managementChange managementChange management is a structured approach to shifting/transitioning individuals, teams, and organizations from a current state to a desired future state. It is an organizational process aimed at helping employees to accept and embrace changes in their current business environment....
- Clinical trialClinical trialClinical trials are a set of procedures in medical research and drug development that are conducted to allow safety and efficacy data to be collected for health interventions...
- Cohort studyCohort studyA cohort study or panel study is a form of longitudinal study used in medicine, social science, actuarial science, and ecology. It is an analysis of risk factors and follows a group of people who do not have the disease, and uses correlations to determine the absolute risk of subject contraction...
- Competitor analysisCompetitor analysisCompetitor analysis in marketing and strategic management is an assessment of the strengths and weaknesses of current and potential competitors. This analysis provides both an offensive and defensive strategic context to identify opportunities and threats...
- Consensus decision-makingConsensus decision-makingConsensus decision-making is a group decision making process that seeks the consent, not necessarily the agreement, of participants and the resolution of objections. Consensus is defined by Merriam-Webster as, first, general agreement, and second, group solidarity of belief or sentiment. It has its...
- Consensus-seeking decision-makingConsensus-seeking decision-makingConsensus-seeking decision-making is a term sometimes used to describe a formal decision process similar to the consensus decision-making variant known as but with the additional option of a fallback voting procedure if consensus appears unattainable during the consensus-seeking phase of the...
- Content analysisContent analysisContent analysis or textual analysis is a methodology in the social sciences for studying the content of communication. Earl Babbie defines it as "the study of recorded human communications, such as books, websites, paintings and laws."According to Dr...
- Conversation analysisConversation analysisConversation analysis is the study of talk in interaction . CA generally attempts to describe the orderliness, structure and sequential patterns of interaction, whether institutional or in casual conversation.Inspired by ethnomethodology Conversation analysis (commonly abbreviated as CA) is the...
- Cost-benefit analysisCost-benefit analysisCost–benefit analysis , sometimes called benefit–cost analysis , is a systematic process for calculating and comparing benefits and costs of a project for two purposes: to determine if it is a sound investment , to see how it compares with alternate projects...
- Data miningData miningData mining , a relatively young and interdisciplinary field of computer science is the process of discovering new patterns from large data sets involving methods at the intersection of artificial intelligence, machine learning, statistics and database systems...
- Delphi Technique
- Design Focused Evaluation
- Discourse analysisDiscourse analysisDiscourse analysis , or discourse studies, is a general term for a number of approaches to analyzing written, spoken, signed language use or any significant semiotic event....
- Educational accreditationEducational accreditationEducational accreditation is a type of quality assurance process under which services and operations of educational institutions or programs are evaluated by an external body to determine if applicable standards are met...
- Electronic portfolioElectronic portfolioAn electronic portfolio, also known as an e-portfolio or digital portfolio, is a collection of electronic evidence assembled and managed by a user, usually on the Web. Such electronic evidence may include inputted text, electronic files, images, multimedia, blog entries, and hyperlinks...
- Environmental scanningEnvironmental scanningEnvironmental scanning is one component of the global environmental analysis. Environmental monitoring, environmental forecasting and environmental assessment complete the global environmental analysis. Environmental scanning refers to the macro environment.The global environment refers to the...
- EthnographyEthnographyEthnography is a qualitative method aimed to learn and understand cultural phenomena which reflect the knowledge and system of meanings guiding the life of a cultural group...
- ExperimentExperimentAn experiment is a methodical procedure carried out with the goal of verifying, falsifying, or establishing the validity of a hypothesis. Experiments vary greatly in their goal and scale, but always rely on repeatable procedure and logical analysis of the results...
- Experimental techniquesExperimental techniquesExperimental research designs are used for the controlled testing of causal processes.The general procedure is one or more independent variables are manipulated to determine their effect on a dependent variable...
- Factor analysisFactor analysisFactor analysis is a statistical method used to describe variability among observed, correlated variables in terms of a potentially lower number of unobserved, uncorrelated variables called factors. In other words, it is possible, for example, that variations in three or four observed variables...
- Factorial experimentFactorial experimentIn statistics, a full factorial experiment is an experiment whose design consists of two or more factors, each with discrete possible values or "levels", and whose experimental units take on all possible combinations of these levels across all such factors. A full factorial design may also be...
- Feasibility studyFeasibility studyFeasibility studies aim to objectively and rationally uncover the strengths and weaknesses of the existing business or proposed venture, opportunities and threats as presented by the environment, the resources required to carry through, and ultimately the prospects for success. In its simplest...
- Field experimentField experimentA field experiment applies the scientific method to experimentally examine an intervention in the real world rather than in the laboratory...
- Fixtureless in-circuit testFixtureless In-Circuit TestFixtureless in-circuit test is a cost-effective alternative to a "bed of nails" tester for in-circuit testing of low to medium volumes of printed circuit board assemblies. It relies on a computerized optical inspection of the circuit assembly and positionable test probes...
- Focus groupFocus groupA focus group is a form of qualitative research in which a group of people are asked about their perceptions, opinions, beliefs and attitudes towards a product, service, concept, advertisement, idea, or packaging...
- Force field analysisForce field analysisForce field analysis is an influential development in the field of social science. It provides a framework for looking at the factors that influence a situation, originally social situations. It looks at forces that are either driving movement toward a goal or blocking movement toward a goal...
- Game theoryGame theoryGame theory is a mathematical method for analyzing calculated circumstances, such as in games, where a person’s success is based upon the choices of others...
- GradingGrade (education)Grades are standardized measurements of varying levels of comprehension within a subject area. Grades can be assigned in letters , as a range , as a number out of a possible total , as descriptors , in percentages, or, as is common in some post-secondary...
- Historical methodHistorical methodHistorical method comprises the techniques and guidelines by which historians use primary sources and other evidence to research and then to write histories in the form of accounts of the past. The question of the nature, and even the possibility, of a sound historical method is raised in the...
- InquiryInquiryAn inquiry is any process that has the aim of augmenting knowledge, resolving doubt, or solving a problem. A theory of inquiry is an account of the various types of inquiry and a treatment of the ways that each type of inquiry achieves its aim.-Deduction:...
- InterviewInterviewAn interview is a conversation between two people where questions are asked by the interviewer to obtain information from the interviewee.- Interview as a Method for Qualitative Research:"Definition" -...
- Marketing researchMarketing researchMarketing research is "the function that links the consumer, customer, and public to the marketer through information — information used to identify and define marketing opportunities and problems; generate, refine, and evaluate marketing actions; monitor marketing performance; and improve...
- Meta-analysisMeta-analysisIn statistics, a meta-analysis combines the results of several studies that address a set of related research hypotheses. In its simplest form, this is normally by identification of a common measure of effect size, for which a weighted average might be the output of a meta-analyses. Here the...
- Metrics
- Multivariate statisticsMultivariate statisticsMultivariate statistics is a form of statistics encompassing the simultaneous observation and analysis of more than one statistical variable. The application of multivariate statistics is multivariate analysis...
- Naturalistic observationNaturalistic observationNaturalistic observation is a research tool in which a subject is observed in its natural habitat without any manipulation by the observer. During naturalistic observation researchers take great care to avoid interfering with the behavior they are observing by using unobtrusive methods...
- Observational techniquesObservational techniquesIn marketing and the social sciences, observational research is a social research technique that involves the direct observation of phenomena in their natural setting...
- Opinion pollOpinion pollAn opinion poll, sometimes simply referred to as a poll is a survey of public opinion from a particular sample. Opinion polls are usually designed to represent the opinions of a population by conducting a series of questions and then extrapolating generalities in ratio or within confidence...
ing - Organizational learningOrganizational learningOrganizational learning is an area of knowledge within organizational theory that studies models and theories about the way an organization learns and adapts....
- Outcome mappingOutcome mappingOutcome mapping is a project progress measurement system that was designed by the grant-making organisation IDRC. It differs from traditional metrics in that it does not focus on measuring deliverables and its effects on primary beneficiaries but on behavioural change exhibited by secondary...
- Participant observationParticipant observationParticipant observation is a type of research strategy. It is a widely used methodology in many disciplines, particularly, cultural anthropology, but also sociology, communication studies, and social psychology...
- Participatory impact pathways analysisParticipatory Impact Pathways AnalysisParticipatory impact pathways analysis is a project management approach in which the participants in a project , including project staff, key stakeholders and the ultimate beneficiaries, together co-construct their program theory.- Overview :The PIPA theory describes plausible impact pathways by...
- Policy analysisPolicy analysisPolicy analysis is "determining which of various alternative policies will most achieve a given set of goals in light of the relations between the policies and the goals". However, policy analysis can be divided into two major fields. Analysis of policy is analytical and descriptive—i.e., it...
- Post occupancy evaluationPost occupancy evaluationPost Occupancy Evaluation has its origins in the United States and has been used in one form or another since the 1960s. Preiser and colleagues define POE as...
- Process improvementProcess improvementIn organizational development , process improvement is a series of actions taken by a process owner to identify, analyze and improve existing business processes within an organization to meet new goals and objectives. These actions often follow a specific methodology or strategy to create...
- Project managementProject managementProject management is the discipline of planning, organizing, securing, and managing resources to achieve specific goals. A project is a temporary endeavor with a defined beginning and end , undertaken to meet unique goals and objectives, typically to bring about beneficial change or added value...
- Qualitative researchQualitative researchQualitative research is a method of inquiry employed in many different academic disciplines, traditionally in the social sciences, but also in market research and further contexts. Qualitative researchers aim to gather an in-depth understanding of human behavior and the reasons that govern such...
- Quality auditQuality auditQuality audit is the process of systematic examination of a quality system carried out by an internal or external quality auditor or an audit team...
- Quality circleQuality circleA quality circle is a volunteer group composed of workers , usually under the leadership of their supervisor , who are trained to identify, analyze and solve work-related problems and present their solutions to management in order to improve the performance of the organization, and motivate and...
- Quality controlQuality controlQuality control, or QC for short, is a process by which entities review the quality of all factors involved in production. This approach places an emphasis on three aspects:...
- Quality managementQuality managementThe term Quality management has a specific meaning within many business sectors. This specific definition, which does not aim to assure 'good quality' by the more general definition , can be considered to have four main components: quality planning, quality control, quality assurance and quality...
- Quality management systemQuality management systemA quality management system can be expressed as the organizational structure, procedures, processes and resources needed to implement quality management.-Elements of a Quality Management System:# Organizational structure# Responsibilities# Methods...
- Quantitative researchQuantitative researchIn the social sciences, quantitative research refers to the systematic empirical investigation of social phenomena via statistical, mathematical or computational techniques. The objective of quantitative research is to develop and employ mathematical models, theories and/or hypotheses pertaining to...
- QuestionnaireQuestionnaireA questionnaire is a research instrument consisting of a series of questions and other prompts for the purpose of gathering information from respondents. Although they are often designed for statistical analysis of the responses, this is not always the case...
- Questionnaire constructionQuestionnaire constructionA questionnaire is a series of questions asked to individuals to obtain statistically useful information about a given topic. When properly constructed and responsibly administered, questionnaires become a vital instrument by which statements can be made about specific groups or people or entire...
- Root cause analysisRoot cause analysisRoot cause analysis is a class of problem solving methods aimed at identifying the root causes of problems or events.Root Cause Analysis is any structured approach to identifying the factors that resulted in the nature, the magnitude, the location, and the timing of the harmful outcomes of one...
- RubricsRubric (academic)A rubric is an assessment tool for communicating expectations of quality. Rubrics support student self-reflection and self-assessment as well as communication between assessor and assessees...
- SamplingSampling (statistics)In statistics and survey methodology, sampling is concerned with the selection of a subset of individuals from within a population to estimate characteristics of the whole population....
- Self-assessmentSelf-assessmentIn social psychology, self-assessment is the process of looking at oneself in order to assess aspects that are important to one's identity. It is one of the motives that drive self-evaluation, along with self-verification and self-enhancement...
- Six SigmaSix SigmaSix Sigma is a business management strategy originally developed by Motorola, USA in 1986. , it is widely used in many sectors of industry.Six Sigma seeks to improve the quality of process outputs by identifying and removing the causes of defects and minimizing variability in manufacturing and...
- Standardized testing
- Statistical process controlStatistical process controlStatistical process control is the application of statistical methods to the monitoring and control of a process to ensure that it operates at its full potential to produce conforming product. Under SPC, a process behaves predictably to produce as much conforming product as possible with the least...
- Statistical surveyStatistical surveySurvey methodology is the field that studies surveys, that is, the sample of individuals from a population with a view towards making statistical inferences about the population using the sample. Polls about public opinion, such as political beliefs, are reported in the news media in democracies....
- StatisticsStatisticsStatistics is the study of the collection, organization, analysis, and interpretation of data. It deals with all aspects of this, including the planning of data collection in terms of the design of surveys and experiments....
- Strategic planningStrategic planningStrategic planning is an organization's process of defining its strategy, or direction, and making decisions on allocating its resources to pursue this strategy. In order to determine the direction of the organization, it is necessary to understand its current position and the possible avenues...
- Structured interviewing
- Systems theorySystems theorySystems theory is the transdisciplinary study of systems in general, with the goal of elucidating principles that can be applied to all types of systems at all nesting levels in all fields of research...
- Student testing
- Total quality managementTotal Quality ManagementTotal quality management or TQM is an integrative philosophy of management for continuously improving the quality of products and processes....
- TriangulationTriangulation (social science)In the social sciences, triangulation is often used to indicate that more than two methods are used in a study with a view to double checking results. This is also called "cross examination"....
- Wizard of Oz experimentWizard of Oz experimentIn the field of human-computer interaction, a Wizard of Oz experiment is a research experiment in which subjects interact with a computer system that subjects believe to be autonomous, but which is actually being operated or partially operated by an unseen human being.-Concept:The term Wizard of Oz...
See also
- Assessment is the process of gathering and analyzing specific information as part of an evaluation.
- Competency evaluationCompetency evaluationIn applied linguistics and educational psychology, competency evaluation is a means for teachers to determine the ability of their students in other ways besides the standardized test.Usually this includes portfolio assessment...
is a means for teachers to determine the ability of their students in other ways besides the standardized test. - Educational evaluationEducational evaluationEducational evaluation is the evaluation process of characterizing and appraising some aspect/s of an educational process.Q. 3 Discuss the role of standards and criteria in educational evaluation...
is evaluation that is conducted specifically in an educational setting. - Immanent evaluationImmanent evaluationImmanent evaluation is a philosophical concept used by Gilles Deleuze in Nietzsche and Philosophy , opposed to transcendent judgment....
, opposed by Gilles DeleuzeGilles DeleuzeGilles Deleuze , was a French philosopher who, from the early 1960s until his death, wrote influentially on philosophy, literature, film, and fine art. His most popular works were the two volumes of Capitalism and Schizophrenia: Anti-Oedipus and A Thousand Plateaus , both co-written with Félix...
to value judgmentValue judgmentA value judgment is a judgment of the rightness or wrongness of something, or of the usefulness of something, based on a comparison or other relativity. As a generalization, a value judgment can refer to a judgment based upon a particular set of values or on a particular value system... - Performance evaluationPerformance EvaluationPerformance Evaluation is an international journal published by Elsevier. The current Editor-in-chief is Philippe Nain. The journal was previously published by North-Holland Publisher.-Editors:*1981–1986 Hisashi Kobayashi*1987–1990 Martin Reiser...
is a term from the field of language testing. It stands in contrast to competence evaluation. - Program evaluationProgram evaluationProject evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency...
is essentially a set of philosophies and techniques to determine if a program 'works'. - Donald Kirkpatrick'sDonald KirkpatrickDonald Kirkpatrick is Professor Emeritus of the University of Wisconsin in North America and a past president of the American Society for Training and Development . He is best known for creating a highly influential model for training evaluation, consisting of four levels of learning evaluation...
Evaluation Model for training evaluation
External links
- Links to Assessment and Evaluation Resources - List of links to resources on several topics
- Glossaries
- Evaluation Portal Link Collection Evaluation link collection with information about evaluation journals, dissemination, projects, societies, how-to texts, books, and much more
- Free Resources for Methods in Evaluation and Social Research