Which type of validity occurs when investigators researchers use adequate definitions and measures of variables?

Criterion Validity

Criterion validity is the comparison of a measure against a single measure that is supposed to be a direct measure of the concept under study. Perhaps the simplest example of the use of the term validity is found in efforts of the American National Election Study (ANES) to validate the responses of respondents to the voting question on the post-election survey. Surveys, including the ANES, consistently estimate a measure of the turnout rate that is unreliable and biased upwards. A greater percentage of people respond that they voted than official government statistics of the number of ballots cast indicate.

To explore the reliability of the measure of turnout, ANES compared a respondent's answer to the voting question against actual voting records. A respondent's registration was also validated. While this may sound like the ideal case of validating a fallible human response to an infallible record of voting, the actual records are not without measurement error. Some people refuse to provide names or give incorrect names, either on registration files or to the ANES. Votes may be improperly recorded. Some people live outside the area where surveyed and records were left unchecked. In 1984, ANES even discovered voting records in a garbage dump. The ANES consistently could not find voting records for 12–14% of self-reported voters. In 1991, the ANES revalidated the 1988 survey and found 13.7% of the revalidated cases produced different results than the cases initially validated in 1989. These discrepancies reduced the confidence in the reliability of the ANES validation effort and, given the high costs of validation, the ANES decided to drop validation efforts on the 1992 survey.

The proceeding example is of criterion validity, where the measure to be validated is correlated with another measure that is a direct measure of the phenomenon of concern. Positive correlation between the measure and the measure it is compared against is all that is needed for evidence that a measure is valid. In some sense, criterion validity is without theory. “If it were found that accuracy in horseshoe pitching correlated highly with success in college, horseshoe pitching would be a valid measure of predicting success in college” (Nunnally, as quoted in the work of Carmines and Zeller). Conversely, no correlation, or worse negative correlation, would be evidence that a measure is not a valid measure of the same concept.

As the example of ANES vote validation demonstrates, criterion validity is only as good as the validity of the reference measure to which one is making a comparison. If the reference measure is biased, then valid measures tested against it may fail to find criterion validity. Ironically, two similarly biased measures will corroborate one another, so a finding of criterion validity is no guarantee that a measure is indeed valid.

Carmines and Zeller argue that criterion validation has limited use in the social sciences because often there exists no direct measure to validate against. That does not mean that criterion validation may be useful in certain contexts. For example, Schrodt and Gerner compared machine coding of event data against that of human coding to determine the validity of the coding by computer. The validity of the machine coding is important to these researchers, who identify conflict events by automatically culling through large volumes of newspaper articles. As similar large-scale data projects emerge in the information age, criterion validation may play an important role in refining the automated coding process.

6.5.1 Criterion Validity

Criterion validity relates to the ability of a method to correspond with other measurements that are collected in order to study the same concept. LDA topics are not necessarily intuitive ideas, concepts, or topics. Therefore, results from LDA may not correspond with results from topic labeling performed by humans.

A typical erroneous assumption frequently made by LDA users is that an LDA topic will represent a more traditional topic that humans write about such as sports, computers, or Africa. It is important to remember that LDA topics may not correspond to an intuitive domain concept. This problem was explored in Hindle et al. [20]. Thus, working with LDA-produced topics has some hazards: for example, even if LDA produces a recognizable sports topic, it may be combined with other topics or there may be other sports topics.

Validity

Evidence of three traditional types of validity—content, construct, and criterion evidence—were evaluated in order to demonstrate that the WIAT measures what it is intended to measure. Even though the content validity was assessed by a panel of national curriculum experts and deemed representative, school curricula vary. Users should review achievement test items to determine how closely the items match what is taught in their school.

Evidence of construct validity includes the expected intercorrelations among subtests reported by age, intercorrelations with the Wechsler scales (see Table D.6 of the WIAT manual), studies of group differences between the standardization sample and various clinical groups as well as differences between the various age/grade groups, and a multitrait-multimethod study of the WIAT and other achievement tests (Roid, Twing, O'Brien, & Williams, 1992). In summary, there was a striking consistency in the correlations among scores on the reading, mathematics, and spelling subtests of the WIAT and those of the corresponding subtests on the other achievement measures.

Since the majority of school psychologists’ assessment time is spent with students with learning disabilities (Smith, Clifford, Hesley, & Leifgren, 1992), WIAT scores were correlated with school grades, group-administered achievement tests, and clinical study groups. Flanagan (1997) notes that a strength of the WIAT is the demonstrated treatment validity because “data are reported that indicate that the WIAT effectively aids in diagnosis of educational/clinical concerns” (p. 84). Special study groups included children classified as gifted and children with mental retardation, emotional disturbance, learning disabilities, attention-deficit/hyperactivity disorder (ADHD), or hearing impairment. Mean composite scores ranged from 112.1 (SD = 9.9) to 117.8 (SD = 9.5) for gifted children, from 58.0 (SD = 10.2) to 66.3 (SD = 10.3) for children with mental retardation, and from 74.6 (SD = 12.0) to 92.8 (SD = 12.6) for children with learning disabilities. These results confirmed predicted expectations for achievement scores in each group.

Independent studies (Slate, 1994; Martelle & Smith, 1994; Saklofske, Schwean, & O'Donnell, 1996; Michalko & Saklofske, 1996) have provided additional evidence of WIAT validity. Saklofske, Schwean, and O'Donnell (1996) studied a sample of 21 children on Ritalin and diagnosed with ADHD and obtained subtest and composite means quite similar to those reported in the WIAT manual. Gentry, Sapp, and Daw (1995) compared subtest scores on the WIAT and the Kaufman Test of Educational Achievement (K-TEA) for 27 emotionally conflicted adolescents and found higher correlations between pairs of subtests (range of .79 to .91) than those reported in the WIAT manual. Because comparisons are often made between the WIAT and the WJPB-R, Martelle and Smith (1994) compared composite and cluster scores for the two tests in a sample of 48 students referred for evaluation of learning disabilities. WIAT composite score means were reported as 83.38 (SD = 10.31) on Reading, 89.32 (SD = 10.60) on Mathematics, 99.24 (SD = 11.84) on Language, and 80.32 on Writing. WJPB-R cluster score means were 87.67 (SD = 11.80) on Broad Reading, 92.09 (SD = 11.62) on Broad Mathematics, and 83.88 (SD = 8.24) on Broad Written Language. Although global scales of the WIAT and WJPB-R relate strongly to each other, mean WIAT composites were 3 to 6 points lower than mean scores on the WJPB-R clusters. Subtest analysis indicated some differences in the way skills are measured; for example, the two reading comprehension subtests (WIAT Reading Comprehension and WJPB-R Passage Comprehension) are essentially unrelated (r = .06). Study authors suggest that “for students with learning disabilities, the two subtests measure reading comprehension in different ways, resulting in scores that may vary greatly from test to test” (p. 7). In addition to a strong relationship between the WIAT Mathematics Composite and the WJPB-R Broad Mathematics cluster, WIAT Numerical Operations correlated equally well with Applied Problems (r = .63) and with Calculation (r = .57) on the WJPB-R, suggesting that WIAT Numerical Operations incorporates into one subtest those skills measured by the two WJPB-R subtests. At the same time, the WJPB-R Quantitative Concepts subtest does not have a counterpart on the WIAT. The Language Composite of the WIAT, however, is a unique feature of that test.

11.4.3.1 Validity

Validity is a very important concept in qualitative HCI research in that it measures the accuracy of the findings we derive from a study. There are three primary approaches to validity: face validity, criterion validity, and construct validity (Cronbach and Meehl, 1955; Wrench et al., 2013).

Face validity is also called content validity. It is a subjective validity criterion that usually requires a human researcher to examine the content of the data to assess whether on its “face” it appears to be related to what the researcher intends to measure. Due to its high subjectivity, face validity is more susceptible to bias and is a weaker criterion compared to construct validity and criterion validity. Although face validity should be viewed with a critical eye, it can serve as a helpful technique to detect suspicious data in the findings that need further investigation (Blandford et al., 2016).

Criterion validity tries to assess how accurate a new measure can predict a previously validated concept or criterion. For example, if we developed a new tool for measuring workload, we might want participants to complete a set of tasks, using the new tool to measure the participants’ workload. We also ask the participants to complete the well-established NASA Task Load Index (NASA-TLX) to assess their perceived workload. We can then calculate the correlation between the two measures to find out how the new tool can effectively predict the NASA-TLX results. A higher correlation coefficient would suggest higher criterion validity. There are three subtypes of criterion validity, namely predictive validity, concurrent validity, and retrospective validity. For more details regarding each subtype—see Chapter 9 “Reliability and Validity” in Wrench et al. (2013).

Construct or factorial validity is usually adopted when a researcher believes that no valid criterion is available for the research topic under investigation. Construct validity is a validity test of a theoretical construct and examines “What constructs account for variance in test performance?” (Cronbach and Meehl, 1955). In Section 11.4.1.1 we discussed the development of potential theoretical constructs using the grounded theory approach. The last stage of the grounded theory method is the formation of a theory. The theory construct derived from a study needs to be validated through construct validity. From the technical perspective, construct or factorial validity is based on the statistical technique of “factor analysis” that allows researchers to identify the groups of items or factors in a measurement instrument. In a recent study, Suh and her colleagues developed a model for user burden that consists of six constructs and, on top of the model, a User Burden Scale. They used both criterion validity and construct validity to measure the efficacy of the model and the scale (Suh et al., 2016).

In HCI research, establishing validity implies constructing a multifaceted argument in favor of your interpretation of the data. If you can show that your interpretation is firmly grounded in the data, you go a long way towards establishing validity. The first step in this process is often the construction of a database (Yin, 2014) that includes all the materials that you collect and create during the course of the study, including notes, documents, photos, and tables. Procedures and products of your analysis, including summaries, explanations, and tabular presentations of data can be included in the database as well.

If your raw data is well organized in your database, you can trace the analytic results back to the raw data, verifying that relevant details behind the cases and the circumstances of data collection are similar enough to warrant comparisons between observations. This linkage forms a chain of evidence, indicating how the data supports your conclusions (Yin, 2014). Analytic results and descriptions of this chain of evidence can be included in your database, providing a roadmap for further analysis.

A database can also provide increased reliability. If you decide to repeat your experiment, clear documentation of the procedures is crucial and careful repetition of both the original protocol and the analytic steps can be a convincing approach for documenting the consistency of the approaches.

Well-documented data and procedures are necessary, but not sufficient for establishing validity. A very real validity concern involves the question of the confidence that you might have in any given interpretive result. If you can only find one piece of evidence for a given conclusion, you might be somewhat wary. However, if you begin to see multiple, independent pieces of data that all point in a common direction, your confidence in the resulting conclusion might increase. The use of multiple data sources to support an interpretation is known as data source triangulation (Stake, 1995). The data sources may be different instances of the same type of data (for example, multiple participants in interview research) or completely different sources of data (for example, observation and time diaries).

Interpretations that account for all—or as much as possible—of the observed data are easier to defend as being valid. It may be very tempting to stress observations that support your pet theory, while downplaying those that may be more consistent with alternative explanations. Although some amount of subjectivity in your analysis is unavoidable, you should try to minimize your bias as much as possible by giving every data point the attention and scrutiny it deserves, and keeping an open mind for alternative explanations that may explain your observations as well as (or better than) your pet theories.

You might even develop some alternative explanations as you go along. These alternatives provide a useful reality check: if you are constantly re-evaluating both your theory and some possible alternatives to see which best match the data, you know when your theory starts to look less compelling (Yin, 2014). This may not be a bad thing—rival explanations that you might never find if you cherry-picked your data to fit your theory may actually be more interesting than your original theory. Whichever explanations best match your data, you can always present them alongside the less successful alternatives. A discussion that shows not only how a given model fits the data but how it is a better fit than plausible alternatives can be particularly compelling.

Well-documented analyses, triangulation, and consideration of alternative explanations are recommended practices for increasing analytic validity, but they have their limits. As qualitative studies are interpretations of complex datasets, they do not claim to have any single, “right” answer. Different observers (or participants) may have different interpretations of the same set of raw data, each of which may be equally valid. Returning to the study of palliative care depicted in Figure 11.2, we might imagine alternative interpretations of the raw data that might have been equally valid: comments about temporal onset of pain and events might have been described by a code “event sequences,” triage and assessment might have been combined into a single code, etc. Researchers working on qualitative data should take appropriate measures to ensure validity, all the while understanding that their interpretation is not definitive.

2.4.4 Threats to validity

As our research design is nonexperimental and we cannot make cause-effect statements, internal validity is not contemplated (Mitchell, 2004).

Face validity: The questions were shown to two researchers who were not involved in this research. They indicated that the terms efficiency and productivity, which are often used in TAM questions, are not easy to understand. As a result, the terms were explained in the introduction of the questionnaire.

Content validity: The questionnaire used is based on the established model of TAM for measuring usefulness and ease of use. The items in the questionnaire are similar to the questions used in several studies that have followed TAM.

Criterion validity: We checked whether the results behave according to the theoretical model (TAM). In this case, the criterion validity can be checked by the Spearman's ρ correlation. The correlations among the variables behave in the theoretical expected way. In addition, other TAM studies have also found similar correlations (Davis, 1989).

Construct validity: The internal consistency of the questions was verified with the Cronbach’s α. For minimizing bias errors, the researchers did not express to the participants opinions nor have any expectation. The surveys were collected anonymously. In addition, the researchers are not related to the creation of the ADD, and the results of the study do not affect them directly.

Conclusion validity: The main threat is the small sample used. However, in order to have more meaningful results, we used nonparametric tests instead of parametric tests. Because this is an exploratory study, the hypotheses built into this study can be used in future studies to be validated with a richer sample. In respect to the random heterogeneity of subjects, the participants have more or less the same design experience and have received the same training about software architecture design.

External validity: The results can be generalized to novel software architects who have received formal training in software architecture design and in the ADD method. We have repeated the experiment in order to confirm our initial findings with students. The domain of the project was changed in the two experiments, but both of them are Web applications with similar characteristics. Untrained architects and experienced architects in practice may have different perceptions than the ones found in this study. However, we believe there are practices common to all business-related (not critical or real time) domains. There is a threat that the academic context is not similar to industrial. In our case, we did not restrict the teams to work in specific hours and times such as in a lab. The development of the tasks was flexible. They were also given a deadline as in the real world to deliver the architecture documentation.

Ethical validity: The questionnaire questions and the study method were approved by The Research Ethics Committee of the University of Limerick.

3 Properties of Indicators

Properties of the indicators are useful to both current and future researchers who plan to use them. Among the two most important properties are the validity and the reliability of the indicators. Validity and reliability are properties that have received their greatest attention in the case of measurement models with continuous latent variables and approximately continuous effect indicators. Indicator validity concerns whether the indicator really measures the latent variable it is supposed to measure. Construct validity, criterion validity, and content validity are types of validity that researchers sometimes examine. Construct validity, for instance, assesses whether the indicator is associated with other constructs that it is supposed to relate to and not associated with those that it should not. Criterion validity compares the indicator to some standard variable that it should be associated with if it is valid. Content validity examines whether the indicators are capturing the concept for which the latent variable stands. See Nunnally and Bernstein (1994) for further discussion.

Reliability focuses on the consistency or ‘stability’ of an indicator in its ability to capture the latent variable. It is distinct from validity in that you can have a reliable indicator that does not really measure the latent variable. A general definition of the reliability of an indicator is the variance of the ‘true’ (latent variable) variance divided by the total indicator variance. A variety of statistics to estimate reliability exist. One of the most popular to measure the reliability of several combined effect indicators is Cronbach's (1951) alpha. It also makes a number of assumptions that might be difficult to satisfy in practice. Alternative measures of reliability built from less restrictive assumptions also are available (Bollen 1989).

IRT assumes a continuous latent trait and a categorical effect indicator, usually dichotomous or ordinal. IRT focuses on other properties of categorical items or indicators such as item discrimination and item difficulty (Hambleton and Swaminathan 1985). IRT is similar to the traditional treatments of reliability and validity in that it too focuses on effect indicators. Latent class or latent structure analysis (Lazarsfeld and Henry 1968) also deals with effect indicators. It applies when we have latent categorical variables with categorical indicators. The degree of classification error of the observed categorical variables provides information on the accuracy of the indicator. The combination of a latent categorical variable with continuous effect indicators are less extensively developed than are the cases of continuous latent variables with continuous or categorical effect indicators.

The measurement properties of causal indicators are less discussed. An important point is that use of the causal indicator assumes that it is the causal indicator that directly influences the latent variable. If the causal indicator itself contains measurement error, then this needs to be part of the measurement model.

3.3.7 Normalized Mutual Information

Normalized mutual information (NMI) (Vinh et al., 2009) is proposed to measure the consistency between any two partitions, which indicates the amount of information (common structured objects) shared between two partitions. Given a set of partitions {Pt}t=1 T obtained from a target data set, the NMI-based clustering validity criteria of assessed partition Pa are determined by summation of the NMI between the assessed partition Pa and each individual partition Pm. Therefore, the high-valued NMI represents a well-accepted partition and indicates the intrinsic structure of the target data set. However, this approach always shows bias toward highly correlated partitions and favors the balanced structure of the data set. The NMI is calculated as following

(3.19)NMI(Pa,Pb)=∑i=1Ka∑j=1 KbNijablog(NN ijabNiaNjb)∑i =1KaNialog(Nia N)+∑j=1KbNjblog(NjbN)

(3.20)NMI(P)=∑t=1TNMI(P,Pt)

Here Pa and Pb are labelings for two partitions that divide a data set of N objects into Ka and Kb clusters, respectively. Nijab is the number of shared objects between clusters Cia∈Pa and Cjb∈Pb, where there are Nia and Njb objects in Cia and Cjb.

19.4 Reliability and performance

Reliability of measurement. Reliability in the context of AFC refers to the extent to which labels from different sources (but of the same images or videos) are consistent. A useful distinction can be made between inter-observer reliability and inter-system reliability.

Inter-observer reliability refers to the extent to which labels assigned by different human annotators are consistent with one another. For example, inter-observer reliability is high if the annotators tended to assign images or videos the same labels (e.g., AUs). Note that, for inter-observer reliability, the “true” label of the image or video is often not knowable, so we are primarily interested in how much the annotators agreed with one another. Because such labels are used to train and evaluate supervised learning systems, inter-observer reliability matters.

Inter-system reliability refers to the extent to which labels assigned by AFC systems are consistent with labels assigned by human annotators. Inter-system reliability is also called “criterion validity” as the human labels are taken to be the gold standard or criterion measure. Inter-system reliability is the primary measure for the performance of an AFC system.

Inter-observer reliability of training data likely serves as an upper bound for what inter-system reliability is possible, and inter-observer reliability often exceeds inter-system reliability by a considerable margin [27–30]. To fulfill the goal of creating an AFC system that is interchangeable with (or perhaps even more accurate and consistent than) a trained human annotator, both forms of reliability must be maximized. Furthermore, the generalizability of the system (i.e., its inter-system reliability in novel domains) must be maximized.

Level of measurement. AFC systems typically analyze behaviors in single images or video frames, and reliability is calculated on this level of measurement. In the studies reviewed below, frame-level performance is almost always the focus. However, other levels of measurement are also possible and evaluating reliability on these levels may be appropriate for certain tasks or applications. For instance, behavioral events, which span multiple continguous frames, may be the focus when the dynamic unfolding of behavior is of interest [31–34]. Alternatively, when behavioral tendencies over longer periods of time are of interest, a more molar approach that aggregates many behaviors (e.g., into counts, means, or proportions) may be appropriate [35]. Note that reliability may differ between levels of measurement. Aggregated annotations are often more reliable than frame-level annotations [27], but they are also less detailed. It is important to match the analyzed level of measurement to the particular use-case of the system.

Metrics for quantifying reliability. Many metrics have been proposed for estimating reliability. When categorical labels are used, percentage agreement or accuracy (i.e., the proportion of objects that were assigned the same label) is an intuitive and popular option. However, accuracy is a poor choice when the categories are highly imbalanced, such as when a facial behavior has a very high (or very low) occurrence rate and the algorithm is trying to predict when the behavior did and did not occur. In such cases, a naïve algorithm that simply guessed that every image or video contained (or did not contain) the behavior would have a high accuracy.

To attempt to resolve this issue, a number of alternative metrics have been developed including the F-score, receiver operator characteristic (ROC) curve analyses, and various chance-adjusted agreement measures. In this paper, we focus on the three most popular metrics: accuracy, the F1 score, and 2AFC. Accuracy, as stated earlier, is the percentage of agreement. The F1 score or balanced F-score is the harmonic mean of precision and recall. Finally, 2AFC is resampling-based estimate of the area under the receiver operating characteristic (ROC) curve.

When dimensional labels are used, correlation coefficients (i.e., standardized covariances) are popular options [36]. The Pearson correlation coefficient (PCC) is a linearity index that quantifies how well two vectors can be equated using a linear transformation (i.e., with the addition of a constant and scalar multiplication). The consistency intra-class correlation coefficients (also known as ICC-C) are additivity indices that quantify how well two vectors can be equated with only the addition of a constant. Finally, the agreement intra-class correlation coefficients (also known as ICC-A) are identity indices that quantify how well two vectors can be equated without transformation. The choice of correlation type should depend on how measurements are obtained and how they will be used.

The existence and use of so many different metrics makes comparison between studies and approaches quite difficult. Different metrics are not similarly interpretable and may behave differently in response to imbalanced categories (Fig. 19.2) [37]. As such, we compare performance scores within metrics but never across them, and we acknowledge that differences in occurrence rates between studies may unavoidably confound some comparisons.

Reliability and Validity in Studies of Television Content