Child attachment Checklist scoring

The Relationship Problems Questionnaire [RPQ] was developed to screen symptoms of the inhibited and disinhibited subtype of reactive attachment disorder [RAD]. This study further examines the psychometric properties of the RPQ in children with severe emotional and behavioural problems by testing its measurement invariance across informants and its convergent validity. Parents and teachers of 152 children [mean age [M age] = 7.92] from 20 schools for special education filled out the RPQ and the Strengths and Difficulties Questionnaire [SDQ]. During a home visit in a subsample of 77 children the Disturbances of Attachment Interview [DAI] was administered to the caregiver and the child was observed using an observational schedule for RAD. Exploratory and confirmatory factor analyses revealed the expected two‐factor structure for both parent and teacher RPQ. Configural and metric invariance, but no scalar invariance, were obtained across informants. Both RPQ‐subscales had acceptable to good internal consistencies and correlated as expected with similar DAI‐subscales. Furthermore, the disinhibited RPQ‐scale related with observations of the child's approach to a stranger. Finally, significant associations were found between the RPQ and the SDQ. Overall, the RPQ has good psychometric qualities as a multi‐informant instrument for RAD‐symptoms in children with severe emotional and behavioural problems. Copyright © 2013 John Wiley & Sons, Ltd.

Keywords: reactive attachment disorder, screening instrument, relationship problems questionnaire, validation

Among childhood psychiatric disorders, reactive attachment disorder [RAD] remains one of the most controversial and least understood. Since the criteria first appeared in psychiatric nosologies in the Diagnostic and Statistical Manual of Mental Disorders, Third Edition [DSM‐III] [American Psychiatric Association, 1980], they have been widely criticized and consensus is still lacking [Zeanah & Gleason, 2010]. Although research into RAD expanded, most studies have been conducted among Romanian orphans, and are limited to children younger than six years owing to controversy on the applicability of the criteria in older children [Zilberstein, 2006]. Systematic studies of the diagnosis and treatment of RAD are lagging behind. Relatedly, well‐validated instruments for screening or assessing RAD are missing, especially in school‐age children.

The lack of validated instruments is problematic for both clinical practice and research. A more accurate identification of RAD would enable clinicians to refer and/or treat children with RAD‐symptoms and would enable researchers to study the developmental course, possible outcomes, and the efficacy of interventions. To assist clinicians and researchers in the screening of RAD at school‐age, Minnis et al. [2002] developed the Relationship Problems Questionnaire [RPQ] with promising reliability and validity [Minnis et al., 2009; Minnis et al., 2002; Minnis et al., 2007]. However, a more extensive psychometric evaluation with multiple informants and methods is appropriate to justify its use in a broader clinical sample.

The current multi‐method study seeks to provide psychometric evidence for the RPQ as a multi‐informant tool for RAD‐symptoms in a clinical group of children with severe emotional and behavioural problems. More specifically, we will investigate whether the constructs measured with the RPQ are similar across informants and whether the RPQ converges with other RAD‐measures. Then, the RPQ may enable clinicians and researchers to efficiently and accurately probe RAD‐symptoms and to identify for which children additional interventions addressing relational problems may be needed to improve child outcome.

Establishing indicators of RAD is complicated as criteria have changed over time. Nevertheless, the description of two types of RAD has been retained since their introduction in the DSM‐III [Zeanah & Gleason, 2010]. The inhibited type is characterized by a lack of social approach: these children exhibit emotionally withdrawn and hypervigilant responses, and seem frightened of others. Children with the disinhibited type, on the contrary, are oversociable, fail to show selective attachments and demonstrate clinging and/or indiscriminate friendliness, even in their interactions with unfamiliar persons [Zeanah, 1996; Zilberstein, 2006]. The DSM‐IV [American Psychiatric Association, 2000] brings both types together under a single category, whereas the International Classification of Diseases, 10th revision [ICD‐10] [World Health Organization, 1992] describes two separate disorders. Both nosologies further agree that the disorder is present before the age of five, occurs in most contexts and does not result from the presence of a developmental disorder [e.g. autism spectrum disorders, ASD] or cognitive delay, but that it is linked with pathogenic care [Zeanah, 1996]. Recently, Zeanah and Gleason [2010] suggested a revision of the DSM‐IV criteria for the DSM‐5. Although they proposed to conceptualize the RAD‐types as two distinct disorders instead of two subtypes of a single disorder, they largely retained the behavioural criteria. Pending the publication of the DSM‐5, we will use the term RAD to cover both types as in the DSM‐IV.

Despite the availability of guidelines for the assessment of RAD [e.g. AACAP, 2005; Chaffin et al., 2006], the need for well‐validated standardardized screening instruments to assist the identification of children that might suffer from RAD still remains. Due to the paucity of research this particularly holds true for older children. To address this need, recently, some efforts have been made to develop instruments specifically probing RAD‐symptoms. One of these instruments is the RPQ.

The RPQ was developed as a screening instrument for RAD to be completed by caregivers of children over five. Items were based on the symptoms of institutionalized children from 18 months till 17 years old in a qualitative case study plus the ICD‐10 and DSM‐IV diagnostic criteria [Minnis et al., 2002]. In a study with foster carers of children between 5 and 16 years old, the original version had a sufficient internal consistency [Cronbach's α = 0.70], and the interrater reliability and test–retest reliability were adequate [intraclass correlation coefficient [ICC] values of 0.81 and 0.78, respectively] [Minnis et al., 2002]. In another study with an adapted version [Cronbach's α = 0.85] in twins aged 7–9 years factor analysis revealed 10 items that loaded on two factors referring to behaviours reflecting the inhibited and disinhibited form of RAD, respectively, with two of the inhibited items not being part of diagnostic criteria [Minnis et al., 2007]. A recent study in which these 10 items were adopted in a teacher version of the RPQ suggested that teachers may be reliable informants of RAD‐symptoms as well [Minnis et al., 2009]. As the teacher is an important informant to signal problem behaviour in school‐aged children in addition to parental caregivers, we will test the psychometric qualities of both versions and examine whether both perspectives are comparable.

The main goal of the present study is to examine the psychometric qualities of the RPQ as a multi‐informant instrument for RAD‐symptoms in school‐aged children. Unlike previous studies, evidence was obtained in a broader clinical group of children in elementary schools providing special education for children with emotional and behavioural disorders. Children in these schools show a wide range of behavioural and emotional regulation problems, and family dysfunction. If the RPQ is found to be reliable and valid in this sample, this study would show that the RPQ is a useful instrument to signal inhibited and disinhibited RAD‐symptoms in special education.

To check the psychometric qualities of the RPQ, we first aimed to demonstrate the factorial validity by replicating the two‐dimensional structure of inhibited and disinhibited RAD in a broader clinical sample. Next, we assessed reliabilities of both subscales. Because verifying an instrument's measurement invariance is equally important as verifying its reliability and validity [Vandenberg & Lance, 2000], we subsequently extended previous research by testing three types of measurement invariance across informants, that is configural, metric, and scalar invariance [cf. Chen, 2008]. In the case of configural invariance, the same items are associated with the same underlying construct [i.e. inhibited or disinhibited RAD] in parent and teacher RPQ. Metric invariance additionally implies that the factor loadings are equal across informants which means that the strength of the relations between an item and a construct are the same for parents and teachers. In this case, the underlying constructs have an equivalent meaning across informants and predictive relationships of parent and teacher RPQ scores may be compared. If both versions are scalar invariant, parent and teacher scores have the same origin [i.e. intercept] as well and factor means may be meaningfully compared across informants. Thus, in order to use the RPQ as a multi‐informant tool and to compare correlates of both versions configural and metric invariance should be obtained; scalar invariance is needed only when directly comparing RPQ‐scores for both informants. Then, we checked the convergence between the RPQ‐subscales, a caregiver interview [the Disturbances of Attachment Interview, DAI], and a home observation. We expected both RPQ‐subscales, and particularly the parent RPQ, to converge with similar subscales of the interview. Additionally, we anticipated that disinhibited behaviours, which are the most observable, would show the most significant correlations with observed child interactive behaviours, in particular with approach to strangers, demanding behaviour, and less so with emotional/interactional problems. Finally, we examined relations between the RPQ‐subscales and multi‐informant reports of behavioural and emotional problems. Based on recent research regarding RAD‐subtype correlates [Gleason et al., 2011], we expected moderate positive associations of the inhibited and disinhibited subscales with internalizing and externalizing problems, respectively.

Approval for this study was obtained from the institutional ethical committee of the KU Leuven, Belgium. All 38 schools providing special education for children with emotional and behavioural disorders in Flanders [Belgium] were contacted and asked to participate in the study. The principals and school teams of 21 schools agreed to participate. Next, the primary caregivers of all children born between 2001 and 2004 [n = 425] received a letter with information and were asked for their permission for the child's participation. No specific exclusion criteria were used because we aimed to validate the RPQ for all children with emotional and behavioural problems. Nevertheless, we will control for ASD and general intelligence in the analyses.

Caregivers of 178 children [participation rate of 42%] agreed to participate and filled out the RPQ. Almost 90% of the caregivers was a biological parent of the child, but only 30% of the caregivers daily took care of the child together with the other biological parent. Approximately 13%, 25%, and 42% of the caregivers completed primary, lower secondary, and higher secondary school, respectively, and merely 20% obtained a degree of higher education. The children's teachers were asked to fill out a similar questionnaire, if parental permission for it was obtained [94%]. Additionally, the school psychologists provided information on child psychiatric diagnoses and general intelligence. We received completed parent and teacher questionnaires for 152 children. These children [87% boys] were between 5.67 years and 10.08 years old [M age = 7.92 years]. They had a mean IQ of 85.95 [standard deviation [SD] = 15.42], ranging from 50 to 141. According to their files, the majority of the children [80%] had one or more psychiatric diagnoses of whom about 25% were diagnosed with or suspected to suffer from RAD.

Participants in the home part of the study were selected from the 152 children for whom both parent and teacher questionnaires were available.1 Selection was based on parental permission for a home visit [n = 120] and the RPQ‐scores. The caregivers of the 75% children with the highest scores on parent or teacher RPQ were contacted by telephone to further inform them about the home visits. Seventy‐seven visits [85%] were made. A recruitment flow diagram is available as online supporting information.

At the start of each home visit, caregivers gave informed consent for the second time, in which they also allowed the school psychologists to provide additional background information from the personal files of the children. According to the school psychologists, family risk factors such as tensions in the partner relation, parental psychological problems, problematic parenting history of one or both parents, and large differences in parenting style were [suspected to be] present in about half of the participants. More than three‐quarters of the families frequently dealt with stress or crises. The school psychologists indicated [suspicions of] a history of pathogenic care as evidenced by the presence of physical or emotional maltreatment or neglect or sexual abuse for 48% of the children.

Each home visit was completed by one researcher and lasted about 2.5 hours. In addition to the administration of a caregiver interview, the child's interactive behaviour with the caregiver was observed and reactions toward the [unfamiliar] researcher were used to rate interactive behaviour toward a stranger. Both activities were piloted and evaluated by two researchers in three home visits.

Relationship Problems Questionnaire [RPQ]

Both caregiver and teacher completed the 10 RAD‐items of the RPQ [cf. the bold items in the online appendix to Minnis et al., 2007]. Four items intend to measure disinhibited behaviour [e.g. “Gets too physically close to strangers”], while six items aim to probe inhibited behaviour [e.g. “Sometimes looks frozen with fear, without an obvious reason”]. Each item has four possible responses [“Not at all like”, “A bit like”, “Like”, “Exactly like”] scored 0, 1, 2 and 3. The original items and responses were translated to Dutch using forward‐backward blind translation [100% agreement].

Disturbances of Attachment Interview [DAI]

The DAI is a semi‐structured interview that was originally created for administration to familiar caregivers of one‐ to five‐year‐old children, and later adapted for use with school‐aged children [Zeanah & Smyke, unpublished manuscript]. It gives an indication of the presence of inhibited and disinhibited behaviour, supplemented with information on the occurrence of four secure‐base distortions based on the alternative criteria of Zeanah and Boris [2000]. Prior studies have shown interrater reliability [κ = 0.88] and internal consistencies of the inhibited and disinhibited subscale [α values were 0.80 and 0.83, respectively] to be very good [Smyke et al., 2002]. Furthermore, Gleason et al. [2011] found convergence between the DAI and a psychiatric diagnostic interview, and between the disinhibited subscale and an observational measurement. They also demonstrated its discriminant validity by relating the inhibited and disinhibited type with depression and attention deficit hyperactivity disorder [ADHD], respectively.

Prior to data collection, the first author was trained during a two‐day session to administer and score the Dutch version of the DAI at the VU Amsterdam, in which the second author was one of the trainers. She was herself trained to identify symptoms of attachment disturbances by Neil Boris and Robert Marvin. For this study, we used the school‐aged version which consists of 11 questions on RAD‐symptoms. The four additional items on secure‐base distortions were not included in the analyses. Each question is scored with 0, 1, or 2 depending on the extent to which the probed attribute is present [no, some, and consistent evidence for an attachment disorder symptom]. Principal component analyses [PCAs] with varimax rotation revealed a two‐factor structure with one factor reflecting inhibited [four items] and another representing disinhibited behaviour [seven items]. Cronbach's α values were 0.53 and 0.80, respectively. Approximately 30% of the DAIs [n = 22] were double rated with the second rater completely blind to additional information. Interrater reliability was good for both subscales [Spearman Rho correlations of 0.89 and 0.91].

Observational schedule

In addition to the RPQ, Minnis and colleagues also developed a structured observational schedule for RAD [McLaughlin et al., 2010]. They formulated key areas for observation based on a literature review which were reviewed by experts. The resulting 20‐item tool had respectable internal consistency [α = 0.75], but only half of the items had good interrater reliability and were specific in discriminating children with RAD from the comparison group [McLaughlin et al., 2010]. In this study, these 10 items were used to evaluate the child's observed behaviour. Each item is answered by “yes” [1] or “no” [0], indicating the presence or absence of a behaviour. To reduce the number of variables, three interpretable subscales were created based on PCA with varimax rotation. Four items loaded highly on a factor Approach to Stranger[s] [e.g. “Does the child initiate conversation with the stranger[s] as if previously familiar?”, “Does the child move towards and approach the stranger?”]. The other two subscales, Emotional/Interactional Problem Behaviour [e.g. “Does the child display sudden shifts to the extremes of emotion?”, “Does the child refuse or ignore a request from his/her carer?”] and Demanding Behaviour [e.g. “Does the child display a seemingly insatiable demand for attention?”, “Does the child interrupt conversation between the stranger and his/her carer?”], consisted of three items each [α values were 0.69, 0.73, and 0.42, respectively]. Higher scores on each subscale indicated more negative child behaviour.

Strengths and Difficulties Questionnaire [SDQ]

Parents and teachers also completed the Strengths and Difficulties Questionnaire [SDQ; Goodman, 1997], a 25‐item questionnaire with subscales for emotional problems, conduct problems, hyperactivity, problems with peer relationships, and prosocial behaviour. This screening instrument for behavioural and emotional problems has been well validated [Goodman et al., 2003; Van Leeuwen et al., 2006]. In this study, Cronbach's α values for the Dutch version's subscales were generally sufficient, ranging from 0.63 to 0.75 for the parent and from 0.69 to 0.81 for the teacher questionnaire. Consistent with previous studies [e.g. Van Leeuwen et al., 2006], only the parent subscale for peer problems had lower internal consistency [α = 0.52].

To identify the latent variables that cause the manifest variables to covary, we used exploratory factor analysis. We preferred factor analysis over PCA, because during factor extraction the shared, unique and error variance of a variable are separated. Only the shared variance appears in the solution with the underlying factor structure [Costello & Osborne, 2005]. As the assumption of multivariate normality was not severely violated, maximum likelihood is the recommended extraction method [Fabrigar et al., 1999]. After extraction, the scree plot indicated that two factors should be retained for both the parent and teacher questionnaire. The two‐factor solution explained 36% and 42% of total variance respectively. The items loaded on the Inhibited and Disinhibited subscale as expected. For the Inhibited subscale, factor loadings ranged from 0.31 to 0.61 for the parent version and from 0.37 to 0.81 for the teacher version. For the Disinhibited subscale, factor loadings varied from 0.35 to 0.91 and from 0.42 to 0.93, respectively. Only one item showed a cross‐loading [> 0.30], namely the item “often asks very personal questions” from the parent version of the Disinhibited subscale.

Cronbach's α values for the parent RPQ were 0.63 for the Inhibited subscale, and 0.77 for the Disinhibited subscale. For the teacher version, Cronbach's α values were 0.71 and 0.81 for the Inhibited and Disinhibited subscales, respectively.

Confirmatory factor analysis [CFA] were conducted using Mplus version 6 [Muthén & Muthén, 1998–2010]. We tested the a priori factor structure [i.e. two uncorrelated factors Inhibited and Disinhibited RAD] for both the parent and teacher version of the RPQ using the maximum likelihood method. The estimated models were considered acceptable when Comparative Fit Index [CFI] ≥ 0.90 and good when CFI ≥ 0.95 [Bentler, 1992; Hu & Bentler, 1999]. In addition, for a good fit, standardized root mean square residual [SRMR] and root mean square error of approximation [RMSEA] values should not exceed 0.08 and 0.06, respectively [Hu & Bentler, 1999]. RMSEA values ≤ 0.08 are considered reasonable [Kline, 2005].

Before conducting the CFAs, we created parcels following the assumption that parcels provide a more stable set of manifest variables on which to base structural models [Widaman et al., 2010]. We used the well‐established item‐to‐construct balance parceling method [Little et al., 2002] to create the optimal number of three parcels for each factor of RAD resulting in a total of six parcels. These parcels were used as input for CFAs in which we tested for three types of measurement invariance. Tests for configural, metric, and scalar invariance respectively reveal whether the multi‐dimensional structure, factor loadings, and intercepts of the RPQ‐items are invariant for parents and teachers. After constraining the models according to measurement invariance, we used the criteria for small sample sizes [N ≤ 300] of Chen [2007] to evaluate the loss in model fit. Fit indices for these models are presented in Table 1.

Goodness of fit indicators of models for parent and teacher report of RAD [n = 152]

First, CFAs were performed for parent and teacher version separately to test the hypothesized two‐factor model. The model fit was good for the parent questionnaire and nearly perfect for the teacher questionnaire. To test configural invariance, an unconstrained, multigroup CFA with two groups [parents and teachers] was conducted, which still resulted in good fit indices. Thus, the two‐factor model proved to be a good solution for both parents and teachers as informants.

Second, metric invariance was evaluated by comparing the configural invariance model in which factor loadings were freely estimated for each of the groups, with the fit of a model in which factor loadings were constrained to be equal for both groups. As these constraints did not result in a significant loss in model fit [ΔCFI > 0.005, but ΔRMSEA  0.01, and ΔSRMR > 0.005]. Hence, the assumption of scalar invariance needs to be rejected.

The intercorrelation between parent and teacher questionnaire was significant for the Disinhibited subscales [r = 0.32, p