Temporal Changes in Effect Sizes of Studies Comparing Individuals With and Without Autism
(Full-text capture 2026-09-21; web artifacts lightly stripped; truncated.)
A Meta-analysis
Eya-Mist Rødgaard, BSc
1Department of Psychology, University of Copenhagen, Copenhagen, Denmark
Kristian Jensen, PhD
2The Novo Nordisk Foundation Center for Biosustainability, Technical University of Denmark, Kgs Lyngby, Denmark
Jean-Noël Vergnes, PhD
3Département de Prévention, Épidémiologie, Économie de la Santé, Odontologie Légale, Université Toulouse III-Paul-Sabatier, Faculté de Chirurgie Dentaire/CHU de Toulouse, Toulouse, France
4Division of Oral Health and Society, Faculty of Dentistry, McGill University, Montréal, Québec, Canada
3,4, Isabelle Soulières
Isabelle Soulières, PhD
5Département de Psychologie, Université du Québec à Montréal, Montréal, Québec, Canada
Laurent Mottron, MD, PhD
6Département de Psychiatrie, Université de Montréal, Montréal, Québec, Canada
7Centre de Recherche du CIUSSS-NIM, Hôpital Rivière-des-Prairies, Montréal, Québec, Canada
6,7,✉
- Author information
- Article notes
- Copyright and License information
1Department of Psychology, University of Copenhagen, Copenhagen, Denmark
2The Novo Nordisk Foundation Center for Biosustainability, Technical University of Denmark, Kgs Lyngby, Denmark
3Département de Prévention, Épidémiologie, Économie de la Santé, Odontologie Légale, Université Toulouse III-Paul-Sabatier, Faculté de Chirurgie Dentaire/CHU de Toulouse, Toulouse, France
4Division of Oral Health and Society, Faculty of Dentistry, McGill University, Montréal, Québec, Canada
5Département de Psychologie, Université du Québec à Montréal, Montréal, Québec, Canada
6Département de Psychiatrie, Université de Montréal, Montréal, Québec, Canada
7Centre de Recherche du CIUSSS-NIM, Hôpital Rivière-des-Prairies, Montréal, Québec, Canada
Accepted for Publication: May 31, 2019.
Published Online: August 21, 2019. doi: 10.1001/jamapsychiatry.2019.1956
Open Access: This is an open access article distributed under the terms of the CC-BY License. © 2019 Rødgaard E-M et al. JAMA Psychiatry.
✉
Corresponding Author: Laurent Mottron, MD, PhD, Centre de Recherche du CIUSSS-NIM, Hôpital Rivière-des-Prairies, 7070, Boulevard Perras, Montréal, QC H1E 1A4, Canada (laurent.mottron@gmail.com).
Author Contributions: Dr Mottron and Ms Rødgaard had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Rødgaard, Jensen, Soulières, Mottron.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Rødgaard, Jensen, Mottron.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Rødgaard, Jensen, Vergnes.
Obtained funding: Mottron.
Administrative, technical, or material support: Mottron.
Supervision: Vergnes, Soulières, Mottron.
Conflict of Interest Disclosures: None reported.
Funding/Support: This study was supported by grant MIRI 15-3736 from Brain Canada (Ms Rødgaard) and Chaire de Recherche Marcel et Rolande Gosselin en Neurosciences cognitives de l’autisme de l’Université de Montréal.
Role of the Funder/Sponsor: The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Additional Contributions: Christiane Belleville, BBA, Noémie Cusson, Janie Degré-Pelletier, BSc, Camille Letendre, BSc, and Vicky Caron, BSc, provided research assistance; William Hempel, PhD, Alex Edelman and Associates, provided help with English editing of the manuscript; and Sylvie Belleville, PhD, participated in discussion about the interpretation of the findings. Sylvie Belleville was not financially compensated for her work. All other contributors were financially compensated for their work.
✉
Corresponding author.
Received 2019 Feb 23; Accepted 2019 May 31; Issue date 2019 Nov.
Copyright 2019 Rødgaard E-M et al. JAMA Psychiatry.
This is an open access article distributed under the terms of the CC-BY License.
PMCID: PMC6704749 PMID: 31433441
See commentary ” The Necessity to Identify Subtypes of Autism Spectrum Disorder.” with doi: 10.1001/jamapsychiatry.2019.1928.
This meta-analysis assesses effect sizes for statistically significant group-level differences between individuals with autism and control individuals for 5 distinct psychological constructs and 2 neurologic markers.
Key Points
Question
Did effect sizes for group-level differences between individuals with autism and control individuals decrease during past decades?
Findings
In this meta-analysis of 11 meta-analyses, effect sizes for 7 distinct differences between groups with autism and control groups decreased over time, with 5 of 7 being statistically significant.
Meaning
The findings suggest that differences between individuals with autism and those without autism have decreased over time, which may be associated with changes in diagnostic practices.
Abstract
Importance
The definition and nature of autism have been highly debated, as exemplified by several revisions of the DSM ( DSM-III, DSM-IIIR, DSM-IV, and DSM- 5) criteria. There has recently been a move from a categorical view toward a spectrum-based view. These changes have been accompanied by a steady increase in the prevalence of the condition. Changes in the definition of autism that may increase heterogeneity could affect the results of autism research; specifically, a broadening of the population with autism could result in decreasing effect sizes of group comparison studies.
Objective
To examine the correlation between publication year and effect size of autism-control group comparisons across several domains of published autism neurocognitive research.
Data Sources
This meta-analysis investigated 11 meta-analyses obtained through a systematic search of PubMed for meta-analyses published from January 1, 1966, through January 27, 2019, using the search string autism AND ( meta-analysis OR meta-analytic). The last search was conducted on January 27, 2019.
Study Selection
Meta-analyses were included if they tested the significance of group differences between individuals with autism and control individuals on a neurocognitive construct. Meta-analyses were only included if the tested group difference was significant and included data with a span of at least 15 years.
Data Extraction and Synthesis
Data were extracted and analyzed according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses ( PRISMA) reporting guideline using fixed-effects models.
Main Outcomes and Measures
Estimated slope of the correlation between publication year and effect size, controlling for differences in methods, sample size, and study quality.
Results
The 11 meta-analyses included data from a total of 27 723 individuals. Demographic data such as sex and age were not available for the entire data set. Seven different psychological and neurologic constructs were analyzed based on data from these meta-analyses. Downward temporal trends for effect size were found for all constructs (slopes: –0.067 to –0.003), with the trend being significant in 5 of 7 cases: emotion recognition (slope: –0.028 [95% CI, –0.048 to –0.007]), theory of mind (–0.045 [95% CI, –0.066 to –0.024]), planning (–0.067 [95% CI, –0.125 to –0.009]), P3b amplitude (–0.048 [95% CI, –0.093 to –0.004]), and brain size (–0.047 [95% CI, –0.077 to –0.016]). In contrast, 3 analogous constructs in schizophrenia, a condition that is also heterogeneous but with no reported increase in prevalence, did not show a similar trend.
Conclusions and Relevance
The findings suggest that differences between individuals with autism and those without the diagnosis have decreased over time and that possible changes in the definition of autism from a narrowly defined and homogenous population toward an inclusive and heterogeneous population may reduce our capacity to build mechanistic models of the condition.
Introduction
Autism was first described in the 1940s,1 and the definition of the condition has been the subject of much debate.2 The diagnostic criteria for autism have been revised several times, and our understanding of autism has evolved from a narrowly defined clinical picture to a spectrum of conditions of uncertain similarity. There has been an increase in the prevalence of autism from less than 0.05% in 19663 to 1.47% among children aged 8 years in the United States4 and to more than 2% in studies5 measuring lifetime prevalence through less stringent case ascertainment. In the absence of a reliable biomarker for the diagnosis of autism, this statistic may reflect multiple factors, such as a true increase in autism in the population, greater public awareness of autism,6 diagnostic substitution,7 a link between diagnosis and support, greater tendency to diagnose individuals with an IQ in the normal range,8 a diminished threshold for clinical diagnosis,9 the use of checklist diagnoses,10 or low specificity of standardized diagnostic instruments in clinical settings.11, 12 Possible changes in diagnostic practices may have resulted in empirical studies assessing an increasingly heterogeneous population, including individuals with less profound deviations from normal that would not have previously been classified as autistic.
We examined how this temporal change in the definition and clinical practices of autism might affect the ability of the scientific community to detect neurocognitive and neurologic differences between autistic and control samples. We predicted that the magnitude of group differences in studies comparing people with and without autism would depend on the period in which it was conducted and, more specifically, become smaller over time. We investigated whether a temporal decrease in effect size could be detected in a variety of cognitive neuroscience constructs commonly studied in autism and associated with group differences. We also studied the temporal evolution of similar variables in schizophrenia, a heterogeneous condition with stable prevalence, to differentiate temporal trends specific to autism from possible confounding factors.
Methods
Selection of Data Material
Meta-analyses of various neurocognitive constructs for which a group difference between those individuals with autism and comparison groups has been identified were used to investigate the correlation between effect size and publication year. The use of meta-analyses facilitated the identification of studies that tested the same or very similar group differences. Meta-analyses also tested the overall statistical significance of the given difference across studies, allowing constructs for which the difference is not significant to be excluded from the analysis because no temporal trend in effect size would be expected.
Potential meta-analyses were found through PubMed using the search string autism AND ( meta-analysis OR meta-analytic). The search spanned the inception of the database (January 1, 1966) through January 27, 2019. The results were reduced to a set of candidate meta-analyses that were written in English, investigated group-level differences between individuals with autism and control groups, and included data on effect size, sample size, and relevant task or method for each primary study. The resulting meta-analyses were organized by the psychological constructs that were investigated (eg, theory of mind) and domain (eg, the social domain). Other inclusion criteria for the meta-analyses were a span of at least 15 years investigating a construct for which at least 1 meta-analysis found a statistically significant difference between a group with autism and a control group. Only domains for which at least 2 constructs could be analyzed were included to determine whether a temporal trend was systematically present or absent within a domain.
Data Extraction
The data selection process is outlined in Figure 1, and analyzed studies are listed in eTables 1-10 in the Supplement. Group difference effect sizes, sample sizes, and task or method for each study were obtained from the meta-analyses. Within each meta-analysis, primary studies were excluded if they used an invalid control group or an improper outcome metric or if other elements of the study design did not allow it to be meaningfully compared with the rest of the studies (eTable 11 in the Supplement). In addition, mean IQ and autistic group diagnosis (autism vs autism spectrum) (eTables 12-19 in the Supplement) were recorded for each primary study.
Figure 1. PRISMA Flowchart.
Assessment of Data Quality
The overall quality of the literature searches of included meta-analyses was assessed according to criteria of the Cochrane Collaboration.13, 14 These criteria make it possible to rate literature searches and the reproducibility of search strategies in meta-analyses. Publication bias was assessed using both original results from meta-analyses and funnel plots aggregating data for each construct. The quality of each primary study was rated using a tailored adaptation of the Newcastle-Ottawa Scale (eTable 19 in the Supplement).15
Statistical Analysis
Quantification of the Temporal Effect Size Trend
A multivariable linear regression model, which is a sensitive method for detecting gradual changes in effect sizes,16 was fitted with effect size as the dependent variable. Publication year was included as an independent variable along with the task or method used (eg, strange stories) because different task variants could be expected to give systematically different effect sizes. Although the expected effect size should in theory be invariant to changes in sample size,17 publication bias might cause small studies to systematically report larger effect sizes than large studies.18 Sample size was also included in the regression analysis to control for such bias. The estimated slope of the correlation between publication year and effect size was used as the outcome to quantify the temporal effect size trend, and F tests were used to quantify the statistical significance of the temporal trend in each individual construct. Furthermore, the association of Newcastle-Ottawa Scale quality score, group comparability score, IQ difference, and autism-group diagnosis with the magnitude of effect sizes was tested by individually adding them to the model and performing F tests. All statistical analyses were conducted in Python 3.5 (Python Software Foundation) using the statsmodels package. Statistical tests were performed as 2-tailed tests with a significance level of .05.
Proteus Phenomenon
Each construct was examined for the presence of the Proteus phenomenon,19 a situation in which the first reported effect size in a given area of study is unrealistically large because of publication bias. This publication bias leads to the earliest effect size being larger than that which can be explained by the regression model. The presence of the Proteus phenomenon was tested by calculating the studentized residuals of the first study for each task. A studentized residual with a t value above the 95th percentile was considered to be evidence of the Proteus effect. This is an adaptation of the method described by Koricheva et al,16 which works in the presence of moderator variables.
Nonautistic Comparison Group
Observed temporal trends could be specific to autism or representative of a general trend across diagnostic categories. A control analysis was performed using data comparing individuals with schizophrenia with the typical population. Schizophrenia was chosen because some neurocognitive deficits, such as theory of mind and executive functioning, have been identified in both groups.20, 21 However, the prevalence of schizophrenia has remained stable for the past 2 decades.22 Meta-analyses for schizophrenia were selected to match those selected for autism as closely as possible.
Results
We found 11 meta-analyses20, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32 comprising 7 constructs within 3 domains: social (emotion recognition and theory of mind), executive (cognitive flexibility, planning, and inhibition), and neurologic (event-related potential P3b and brain size) ( Table). These included a total of 27 723 individuals.
Table. Overview of the Results for the 7 Constructs in Autism and the 3 Constructs in Schizophrenia.
| Construct | Source | R 2 | Year, Slope a | P Value b |
|---|---|---|---|---|
| Year | Participants | |||
| :— | :— | |||
| Autism | ||||
| Social domain c | ||||
| Emotion recognition | Chung et al,20 2014 | 0.28 | −0.028 | .005 |
| Leppanen et al,23 2018 | ||||
| Peñuelas-Calvo et al,24 2019 | ||||
| Uljarevic and Hamilton,25 2013 | ||||
| Theory of mind | Chung et al,20 2014 | 0.54 | −0.045 | <.001 |
| Leppanen et al,23 2018 | ||||
| Executive domain c | ||||
| Planning | Olde Dubbelink and Geurts,26 2017 | 0.54 | −0.067 | .03 |
| Lai et al,27 2017 | ||||
| Cognitive flexibility | Landry and Al-Taie,28 2016 | 0.11 | −0.013 | .18 |
| Lai et al,27 2017 | ||||
| Westwood et al,29 2016 | ||||
| Inhibition | Geurts et al,30 2014 | 0.07 | −0.003 | .82 |
| Lai et al,27 2017 | ||||
| Neurologic domain c | ||||
| P3b amplitude | Cui et al,31 2017 | 0.65 | −0.048 | .02 |
| Brain size | Sacco et al,32 2015 | 0.41 | −0.047 | .003 |
| Schizophrenia | ||||
| Theory of mind | Chung et al,20 2014 | 0.35 | −0.008 | .37 |
| Bora et al,33 2009 | ||||
| Inhibition, Stroop task | Westerhausen et al,34 2011 | 0.23 | 0.011 | .23 |
| Gray matter volume | Haijma et al,35 2013 | 0.02 | 0.008 | .42 |
a
Slope denotes the regression coefficient for the year variable in the linear models with effect size as the outcome variable.
b
P values denote the significance of the association of the year variable and participants with effect sizes and are calculated using F tests on the linear models with effect size as the outcome variable.
c
Some meta-analyses included more than 1 construct.
Quality of the Data
Selection criteria of the primary studies in the 11 meta-analyses are reported in eTables 21-24 in the Supplement, with good comparability among meta-analyses. Inclusion periods largely overlapped (publication years of meta-analyses between 2013 and 2018), indicating a low risk of bias because of the heterogeneity of data sources (eTables 21-23 and 25 in the Supplement). The mean score of the quality of the literature search strategies of the meta-analyses was 5.5 (range, 3.0-8.0) on the 9-item scale, in which higher numbers are considered to indicate better quality (eTable 26 in the Supplement). There was some evidence of publication bias for the 2 constructs of the social domain but not the other constructs (eTable 27 and eFigure in the Supplement).
Autism
The results of the statistical analysis of the 7 neurocognitive constructs are shown in the Table, and the correlations between publication year and effect size are shown in Figure 2 36 (see eResults in the Supplement for a detailed description of the results for each construct). The slope estimates for publication year were negative for all 7 constructs ( Figure 3), indicative of a general tendency for the effect size to decrease over time. The regression models showed that the association of publication year with effect size was statistically significant for 5 of 7 constructs: emotion recognition (slope: –0.028 [95% CI, –0.048 to –0.007]), theory of mind (–0.045 [95% CI, –0.066 to –0.024]), planning (–0.067 [95% CI, –0.125 to –0.009]), P3b amplitude (–0.048 [95% CI, –0.093 to –0.004]), and brain size (–0.047 [95% CI, –0.077 to –0.016]). For the cognitive flexibility construct, effect sizes from 1 primary study37 deviated substantially from those of almost all other studies. This unusual result was also noted by the author, and a reproduction of the study37 did not replicate it, instead reporting findings consistent with the remaining literature. If the abnormal effect sizes were excluded from our analysis, the results changed markedly, with the association of publication year with effect size also becoming significant for this construct (slope, –0.018; P = .02).
Figure 2. Overview of the Development of Effect Sizes Over Time Within Each of the Constructs.
Each point represents an effect size originating from an empirical study. Colors indicate which task or method was used within the study. The black line indicates the fitted linear model. In the analysis of planning and inhibition, method type was defined as the combination of task and outcome metric. Points are colored by task alone for the purpose of visualization. DKEFS indicates Delis-Kaplan Executive Function System; FB-seq, False Belief sequencing task; MASC, Movie for Assessment of Social Cognition; MRI, magnetic resonance imaging; NEPSY, Developmental Neuropsychological Assessment; RMET, Reading the Mind in the Eyes Test; RMVT, Reading the Mind in the Voice Test; and WCST, Wisconsin Card Sorting Task.
Figure 3. Forest Plot of the Estimated Change in Effect Size per Year.
There was evidence of the Proteus phenomenon for only 1 of the included tasks (the strange stories task within the theory of mind construct), but when the analysis was rerun without the outlying effect size,38 the decrease in effect size over time was still significant and prominent ( P < .001) (see eResults in the Supplement). This suggests that the decreasing trends in effect size were not merely explained by the first studies overestimating the effect size. Instead, there appeared to be a steady decrease throughout the examined period.
Schizophrenia
We performed a similar analysis on 4 meta-analyses investigating group-level differences between individuals with schizophrenia and controls. The constructs investigated were theory of mind, cognitive inhibition (Stroop task), and gray matter volume, abnormalities in all of which were found by the meta-analyses to be significantly associated with schizophrenia. The data for the theory of mind analysis were obtained from meta-analyses conducted by Chung et al20 and Bora et al.33 Data to explore the constructs of cognitive inhibition and gray matter volume were extracted from meta-studies conducted by Westerhausen et al34 and Haijma et al,35 respectively. The results of the analysis are shown in the Table, and correlation plots for publication year and group-level effect size for the 3 constructs are shown in Figure 2. The temporal trends were not significant for any of the constructs.
Study Quality, Other Variables, and the Temporal Trend of Effect Size
We tested whether the temporal trend in effect size could be explained by systematic changes in study design over time by testing the association of quality score, group comparability score, group IQ difference, and autism diagnosis type with effect size. There was no significant association with group difference effect size for quality score or comparability score as measured by an adapted Newcastle-Ottawa Scale (eTable 19 in the Supplement), and control for these variables did not alter the significance of the correlation with publication year (eTable 20 in the Supplement). Group IQ differences were significantly associated with effect size only for the 2 constructs for which no temporal trend was identified initially (cognitive flexibility and inhibition). Among the remaining constructs, IQ differences did not have a significant correlation with effect size and the significance of the associations between publication year and effect size was not altered.
Testing how differences in the definition of autism diagnosis were associated with group differences proved to be difficult because different authors used different systems of classification for individuals with autism (eg, autism, high-functioning autism, and Asperger syndrome). Older studies mostly included individuals with an “autism” diagnosis, whereas newer studies more often used mixtures of people with an autism, Asperger, or an “autism spectrum disorder” diagnosis. Whether a study used or did not use a sample with pure autism (or high-functioning autism) was not, however, significantly associated with group difference effect sizes for any of the constructs, and including this variable in the analysis did not change the significance of the association between publication year and effect size.
Discussion
We investigated effect sizes for 5 distinct psychological constructs and 2 neurologic markers for which statistically significant group-level differences between individuals with autism and control individuals have previously been identified. We found that effect sizes decreased over the past 2 decades. The relative decrease in mean effect size from 2000 through 2015 ranged from 45% to more than 80% among the constructs for which the temporal decrease was significant. The trend observed for autism deviated from that observed for schizophrenia, another psychiatric condition with comparable absolute prevalence but for which there was no documented increase in prevalence during the investigated period.
Changes in our understanding and the definition of autism may have occurred in different ways. One factor could be the evolution of diagnostic criteria associated with a gradual expansion in our understanding and the definition of autism. This may have introduced additional extrinsic heterogeneity. As an example, attention-deficit/hyperactivity disorder was considered to be a differential diagnosis in the DSM-IV, whereas it is listed as a possible co-occurring condition in the DSM-5, so that the social effect of severe attention-deficit/hyperactivity disorder may be mistaken for autism.39 Increased attention-deficit/hyperactivity disorder comorbidity could explain why the temporal decrease appeared to be smaller for executive compared with social or neurologic constructs. However, although executive deficits are shared by the 2 conditions, they may encompass distinct executive functions40 and imperfectly overlap with clinical traits.41
Given that the decrease in effect
(Truncated: full text at source URL.)