As noted by Ebel [], validity is considered the most important feature of a testing program. Its status should not be surprising given that yet, it often does not receive the attention it deserves. As a result, tests can end up being misaligned or unrelated to what they are intended to measure, with scores that have limited meaning or usefulness.
Consider studying diligently for an introductory measurement exam that assesses topics like validity only superficially with recall questions rather than essays that require a deep evaluation of competing ideas. You might Or consider a screening test for a job in customer service that measures the extraversion but not agreeableness of candidates. In each of these examples, the results may not support
Validity encompasses all design considerations, administration procedures, and relating to the testing process that makes score inferences useful and meaningful. Test results that are consistent and based on items written according to specified content standards with appropriate levels of difficulty and discrimination are more useful and meaningful than scores that do not have these qualities. Correct scaling, sound test construction, and rigorous statistical analysis are thus all prerequisites for validity.
This chapter begins with an overview of validity, including definitions of key terms and concepts as well as some historical perspective. Common sources of validity evidence are then discussed in detail, including:
Note that much of our discussion around these five sources will incorporate information presented in previous chapters. Test content and response processes will draw from Chapters and . Dimensionality and internal structure will elaborate on information from Chapters through . Relationships with other variables and consequences of test use are more unique to this chapter and involve some new concepts.
The three sources of validity evidence are discussed within what is referred to as a unified view of validity.
R analysis in this chapter is minimal. We’ll run correlations and make adjustments to them using the base R functions, and we’ll simulate scores using epmr.
According to Haynes, Richard, and Kubany [], content validity is “the degree to which elements of an assessment instrument are relevant to and representative of the targeted construct for a particular assessment purpose.” Note that this definition of content validity is very similar to our original definition of validity. The difference is that content validity focuses on elements of the construct and how well they are represented in our test. Thus, content validity assumes the target construct can be broken down into elements, and that we can obtain a representative sample of these elements.
Having defined the purpose of our test and the construct we are measuring, there are three main steps to establishing content validity evidence:
- Define the content domain based on relevant standards, skills, tasks, behaviors, facets, factors, etc. that represent the construct. The idea here is that our construct can be represented in terms of specific identifiable dimensions or components, some of which may be more relevant to the construct than others.
- Use the defined content domain to create a blueprint or outline for our test. The blueprint organizes the test based on the relevant components of the content domain, and describes how each of these components will be represented within the test.
- Subject matter experts evaluate the extent to which our test blueprint adequately captures the content domain, and the extent to which our test items will adequately sample from the content domain.
Here is an overview of how content validity could be established for the IGDI measures of early literacy. Again, the purpose of the test is to identify preschoolers in need of additional support in developing early literacy skills.
1. Define the content domain
The early literacy content domain is broken down into a variety of content areas, including alphabet principles [e.g., knowledge of the names and sounds of letters], phonemic awareness [e.g., awareness of the sounds that make up words], and oral language [e.g., definitional vocabulary]. The literature on early literacy has identified other important skills, but we’ll focus here on these three. Note that the content domain for a construct should be established both by research and practice.
2. Outline the test
Next, we map the portions of our test that will address each area of the content domain. The test outline can include information about the type of items used, the cognitive skills required, and the difficulty levels that are targeted, among other things. Review Chapter for additional details on test outlines or blueprints.
Table contains an example of a test outline for the IGDI measures. The three content areas listed above are shown in the first column. These are then broken down further into cognitive processes or skills. Theory and practical constraints determine reasonable numbers and types of test items or tasks devoted to each cognitive process in the test itself. The final column shows the percentage of the total test that is devoted to each area.
Table 9.1: Example Test Outline for a Measure of Early LiteracyAlphabet principlesLetter naming2013%Sound identification2013%Phonological awarenessRhyming1510%Alliteration1510%Sound blending107%Oral languagePicture naming3020%Which one doesn’t belong2013%Sentence completion2013%3. Evaluate
Validity evidence requires that the test outline be representative of the content domain and appropriate for the construct and test purpose. The appropriateness of an outline is typically evaluated by content experts. In the case of the IGDI measures, these experts could be researchers in the area of early literacy, and teachers who work directly with students from the target population.
Licensure testing
Here is an example of content validity from the area of licensure/certification testing. I have consulted with an organization that develops and administers tests of medical imaging, including knowledge assessments taken by candidates for certification in radiography. This area provides a unique example of content validity, because the test itself measures a construct that is directly tied to professional practice. If practicing radiographers utilize a certain procedure, that procedure, or the knowledge required to perform it, should be included in the test.
The domain for a licensure/certification test such as this is defined using what is referred to as a job analysis or practice analysis [Raymond ]. A job analysis is a research study, the central feature of which is a survey sent to practitioners that lists a wide range of procedures and skills potentially used in the field. Respondents indicate how often they perform each procedure or use each skill on the survey. Procedures and skills performed by a high percentage of professionals are then included in the test outline. As in the previous examples, the final step in establishing content validity is having a select group of experts review the procedures and skills and their distribution across the test, as organized in the test outline.
Psychological measures
Content validity is relevant in non-cognitive psychological testing as well. Suppose the purpose of a test is to measure client experience with panic attacks so as to determine the efficacy of treatment. The domain for this test could be defined using criteria listed in the DSM-V [www.dsm5.org], reports about panic attack frequency, and secondary effects of panic attacks. The test outline would organize the number and types of items written to address all relevant criteria from the DSM-V. Finally, experts who work directly in clinical settings would evaluate the test outline to determine its quality, and their evaluation would provide evidence supporting the content validity of the test for this purpose.
Threats to content validity
When considering the appropriateness of our test content, we must also be aware of how content validity evidence can be compromised. What does content invalidity look like? For example, if our panic attack scores were not valid for a particular use, how would this lack of validity manifest itself in the process of establishing content validity?
Here are two main sources of content invalidity. First, if items reflecting domain elements that are important to the construct are omitted from our test outline, the construct will be underrepresented in the test. In our panic attack example, if the test does not include items addressing “nausea or abdominal distress,” other criteria, such as “fear of dying,” may have too much sway in determining an individual’s score. Second, if unnecessary items measuring irrelevant or tangential material are included, the construct will be misrepresented in the test. For example, if items measuring depression are included in the scoring process, the score itself is less valid as a measure of the target construct.
Together, these two threats to content validity lead to unsupported score inferences. Some worst-case-scenario consequences include misdiagnoses, failure to provide needed treatment, or the provision of treatment that is not needed. In licensure testing, the result can be the licensing of candidates who lack the knowledge, skills, and abilities required for safe and effective practice.