On Depth: Conceptual and methodological considerations in meta-analysis
Recently, a methodological approach to examining forensic psychological constructs has been appearing on the pages of clinical psychology, sexology, and child abuse prevention journals and at professional conferences. These studies all make use a meta-analytic methodology I have started to call “depth meta-analysis.” By depth meta-analysis, I mean a meta-analysis that tries to drill down as deeply as possible into the existent literature pertaining to a single psychological construct. By restricting itself to a single construct, a meta-analysis can answer a plethora of disparate research questions germane to the construct of interest. In addition, an approach such as this unlocks the potential of moderator analyses, a set of powerful statistical tools in meta-analysis. In the field of forensic psychology focusing on sexual violence, I have come to see some of the main research questions regarding psychological constructs as being:
- Who displays the problematic construct (i.e., comparisons between offender and non-offender groups and between subgroups; discriminative validity using specificity designs)?
- Are problems associated with an increased rate of sexual recidivism (i.e., predictive validity of a construct)?
- Does this problem reduce in severity over the course of intervention (i.e., treatment change)?
- What other psychological constructs are associated with this construct?
- And finally, are reductions in this construct for offenders associated with a reduction in sexual recidivism?
Admittedly, the area of research examining the relationship between treatment change and reductions in sexual recidivism is not well developed in our field currently (for recent notable exemptions, see here and here).
Depth meta-analysis is capable of addressing all of these questions. Indeed, I hold that this type of meta-analytic approach can help us quickly summarize the available research in all of these domains for many of the psychological constructs posited to be involved in sexual violence perpetration. A good example of depth meta-analysis is an article on emotional congruence with children, which I published with a few colleagues recently (open access here). By setting out to examine a single construct (emotional congruence with children) we were able to place the available literature into three of the research domains I mentioned above: comparing groups on their level of emotion congruence; the predictive validity of the construct; and the changeability of the construct over the course of sex offender treatment. One main advantage of depth meta-analysis is that we were able to extract enough effect sizes in all three domains to examine the construct’s functioning in various subgroups and look at some important and interesting moderator variables, such as the offenders’ relationship with their victims or measurement modality effects. Taking this “depth” approach to the construct, we were able to summarize, with surprising specificity, the research available, and in some respects, we were better able to summarize questions that had already been examined in past meta-analytic research. In addition, we were able to see which of the research domains are underdeveloped, leading to logical “next steps” in developing research on the construct.
In contrast to depth meta-analysis, much of the past meta-analytic research has generally taken a “breadth” approach to a single domain of research (for example, reviews identifying all psychological characteristics that predict sexual recidivism). The inherent limitations in this approach include not being able to examine the complexities of the research on a construct due to multiple research domains potentially needing to be summarized. Entire tracks of specific and nuanced questions are left unexamined due to numerous constructs are examined alongside one another and specific consideration generally cannot be given to single constructs. This limitation can be summed up in terms of the limits of asking “either/or” questions: breadth meta-analysis presents findings in such a way that suggests a set of psychological constructs either predict re-offending or they do not. Depth meta-analysis presents findings in such a way that is more capable of answering the questions: “for whom is this risk factor important?” and “under what circumstances does this relationship hold?” These questions are vital for scientific progress.
I am not attempting to under-value the gains made through the use of breadth meta-analytic methods. This methodology is a cornerstone in the field of forensic psychology and has been instrumental in informing the development of risk assessment tools that incorporate dynamic, or changeable, psychological risk factors (Stable-2007 here, Violence Risk Scale: Sex Offender version here, Structured Risk Assessment: Forensic Version here). Instead, there are logical limits to any methodology and we can benefit from identifying limits and promoting methodological expansion in this area.
I argue that using depth methodology in meta-analysis provides us with a rounder and fuller understanding of what we know about the constructs that we use routinely in clinical settings and research. Having a more detailed understanding of the populations for which a psychological risk factor is important and under what circumstances (procedural, measurement modality, etc) the risk factor is important has incredible implications for everyday clinical practice and scientific progress. Further, summarizing as much as possible about a psychological construct and how it operates allows us to formulate new hypotheses and can inform theoretical development in the area.
In way of a summary, here are the characteristics, as I see them, of depth meta-analysis:
- Focuses on a single psychological construct that is important to understanding sexual violence perpetration
- Summarizes literature on the construct from multiple research domains, as much as the available literature will allow
- Uses moderator analysis to examine whether important issues moderate the relationship under study (e.g., does measurement modality affect the strength of a relationship?)
- Identifies the limits of our current knowledge of a psychological construct and points the way for fruitful avenues of future research or theoretical development
McPhail, I. V. (2014, March 30). On depth: Conceptual and methodological considerations in meta-analysis. [Weblog post]. Retrieved from http://wp.me/p2RS15-3y.
Want to submit your own post? Click here to find out how!