Eye-Tracking–Based Measurement of Social Visual Engagement Compared With Expert Clinical Diagnosis of Autism
(Full-text capture 2026-09-21; web artifacts lightly stripped; truncated.)
Warren Jones, PhD
1Marcus Autism Center, Children’s Healthcare of Atlanta, Atlanta, Georgia
2Division of Autism & Related Disorders, Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia
3Center for Translational Social Neuroscience, Emory University, Atlanta, Georgia
1,2,3,✉, Cheryl Klaiman
Cheryl Klaiman, PhD
1Marcus Autism Center, Children’s Healthcare of Atlanta, Atlanta, Georgia
2Division of Autism & Related Disorders, Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia
1,2, Shana Richardson
Shana Richardson, PhD
1Marcus Autism Center, Children’s Healthcare of Atlanta, Atlanta, Georgia
1, Christa Aoki
Christa Aoki, PhD
1Marcus Autism Center, Children’s Healthcare of Atlanta, Atlanta, Georgia
Christopher Smith, PhD
4Southwest Autism Research & Resource Center, Phoenix, Arizona
Mendy Minjarez, PhD
5Seattle Children’s Autism Center and Department of Psychiatry, University of Washington, Seattle
Raphael Bernier, PhD
5Seattle Children’s Autism Center and Department of Psychiatry, University of Washington, Seattle
Ernest Pedapati, MD
6Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio
6, Somer Bishop
Somer Bishop, PhD
7University of California San Francisco
7, Whitney Ence
Whitney Ence, PhD
7University of California San Francisco
Allison Wainer, PhD
8Rush University Medical Center, Chicago, Illinois
Jennifer Moriuchi, PhD
8Rush University Medical Center, Chicago, Illinois
8, Sew-Wah Tay
Sew-Wah Tay, PhD
9Libra Medical Inc, Minneapolis, Minnesota
9, Ami Klin
Ami Klin, PhD
1Marcus Autism Center, Children’s Healthcare of Atlanta, Atlanta, Georgia
2Division of Autism & Related Disorders, Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia
3Center for Translational Social Neuroscience, Emory University, Atlanta, Georgia
1,2,3
- Author information
- Article notes
- Copyright and License information
1Marcus Autism Center, Children’s Healthcare of Atlanta, Atlanta, Georgia
2Division of Autism & Related Disorders, Department of Pediatrics, Emory University School of Medicine, Atlanta, Georgia
3Center for Translational Social Neuroscience, Emory University, Atlanta, Georgia
4Southwest Autism Research & Resource Center, Phoenix, Arizona
5Seattle Children’s Autism Center and Department of Psychiatry, University of Washington, Seattle
6Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio
7University of California San Francisco
8Rush University Medical Center, Chicago, Illinois
9Libra Medical Inc, Minneapolis, Minnesota
Accepted for Publication: June 29, 2023.
✉
Corresponding Author: Warren Jones, PhD, Marcus Autism Center, Children’s Healthcare of Atlanta, 1920 Briarcliff Rd NE, Atlanta, GA 30329 (warren.jones@emory.edu).
Author Contributions: Drs Jones and Tay had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Concept and design: Jones, Klaiman, Klin.
Acquisition, analysis, or interpretation of data: All authors.
Drafting of the manuscript: Jones, Klaiman, Pedapati, Klin.
Critical revision of the manuscript for important intellectual content: All authors.
Statistical analysis: Jones, Tay.
Obtained funding: Jones, Klin.
Administrative, technical, or material support: Jones, Klaiman, Richardson, Minjarez, Wainer, Moriuchi, Tay, Klin.
Supervision: Jones, Klaiman, Minjarez, Pedapati, Klin.
Other–data collection: Smith.
Conflict of Interest Disclosures: Dr Jones reported being employed by Emory University and Marcus Autism Center, a subsidiary of Children’s Healthcare of Atlanta (CHOA); and receiving grants from The Marcus Foundation, the Joseph B. Whitehead Foundation, and the Georgia Research Alliance. Marcus Autism Center was the study sponsor during trial data collection. The technology and study reporting responsibilities were transferred to EarliTec Diagnostics Inc as part of technology transfer from CHOA on January 31, 2020. Dr Jones was a scientific co-founder of EarliTec and is an equity holder, and acts as a paid scientific consultant to EarliTec and unpaid board member, although his primary appointment and affiliation remain at Emory. EarliTec consulting fees are paid to Dr Jones. Dr Jones also reported a lecture honorarium from Washington University School of Medicine in St Louis (Third Annual Dr Adolfo and Fanny Rizzo Endowed Lecture). In addition, Dr Jones reported US patent No.s 7,922,670; 8,343,067; 8,551,015; 9,265,416; 9,510,752; 9,861,307; 10,016,156; 10,022,049; 10,617,295; 10,702,150; and 10,987,043, licensed to EarliTec Diagnostics Inc. Dr Klaiman reported being employed by Emory University and Marcus Autism Center, a subsidiary of CHOA; and receiving grants from The Marcus Foundation, the Joseph B. Whitehead Foundation, and the Georgia Research Alliance. Dr Klaiman acts as a paid scientific consultant to EarliTec Diagnostics Inc and as a clinical advisor to Beaming Health, and reported personal fees from EarliTec and Beaming Health. Dr Klaiman is a certified ADOS-2 trainer and received personal fees to conduct ADOS-2 trainings at ABA Centers of America, Dekalb County School District, Cherokee County School District, and Fulton County School District. Dr Klaiman also reported an honorarium from Children’s Health Council. Dr Richardson reported being employed by Emory University and Marcus Autism Center, a subsidiary of CHOA; and receiving funds and/or equipment to the institution from The Marcus Foundation, the Joseph B. Whitehead Foundation, and the Georgia Research Alliance. Dr Aoki reported being employed by Emory University and Marcus Autism Center, a subsidiary of CHOA; and receiving funds and/or equipment to the institution from The Marcus Foundation, the Joseph B. Whitehead Foundation, and the Georgia Research Alliance. Dr Minjarez reported being employed by Seattle Children’s Autism Center & the University of Washington, Seattle; and receiving funds and/or equipment to the institution from Marcus Autism Center as the study sponsor; in addition, Dr Minjarez reported book royalties from Brookes Publishing and travel paid by Seoul National University to Korea to give lectures. Dr Bernier reported being employed by Apple Inc. Dr Bishop reported being employed by University of California, San Francisco; and receiving funds and/or equipment to the institution from Marcus Autism Center as the study sponsor. In addition, Dr Bishop reported personal fees from Western Psychological Services. Dr Moriuchi reported being employed by Rush University Medical Center; and receiving funds and/or equipment to the institution from Marcus Autism Center as the study sponsor. Dr Tay reported being employed by Libra Medical Inc; receiving funds to the institution from EarliTec Diagnostics Inc; and receiving stock from EarliTec Diagnostics. EarliTec Diagnostics Inc assumed study sponsor responsibilities from Marcus Autism Center after technology transfer from CHOA, and contracted Libra Medical to provide clinical trial management and regulatory services. Dr Klin reported being employed by Emory University and Marcus Autism Center, a subsidiary of CHOA; and receiving grants from the Marcus Foundation, the Joseph B. Whitehead Foundation, and the Georgia Research Alliance. Marcus Autism Center was the study sponsor during trial data collection. The technology and study reporting responsibilities were transferred to EarliTec Diagnostics Inc as part of technology transfer from CHOA on January 31, 2020. Dr Klin was a scientific co-founder of EarliTec and is an equity holder. Dr Klin acts as a paid scientific consultant to EarliTec and as an unpaid board member, although his primary appointment and affiliation remain at Emory. EarliTec consulting fees are paid to Dr Klin. Dr Klin also reported lecture honoraria from National Autism Conference, McKnight Endowment Fund for Neuroscience, Sociedade Brasileira de Fonoaudiologia, Alliance for Early Success, and Washington University School of Medicine; in addition, Dr Klin reported US patent No.s 7,922,670; 8,343,067; 8,551,015; 9,265,416; 9,510,752; 9,861,307; 10,016,156; 10,022,049; 10,617,295; 10,702,150; and 10,987,043 licensed to EarliTec Diagnostics Inc. No other disclosures were reported.
Funding/Support: This study was supported by the Marcus Foundation and the Joseph B. Whitehead Foundation.
Role of the Funder/Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Data Sharing Statement: See Supplement 4.
Additional Contributions: We thank the families and children for their participation. We also thank the data collection and study coordination teams at each of the participating study sites. We thank Peter Lewis, MBA, Jose Paredes, MS, Jason Davis, MS, Travis Dennison, MS, Robert Frazier, BS, Mike Glatzer, BS, Theodore Nicholson, PhD, Shyamal Shah, BS, Steven Shifke, BS, Jenny Taylor, BS, and Antonio Terrell, BS, of the Marcus Autism Center for their help in investigational device development and in device deployment to all study sites. We thank Jason Davis, MS, Steven Shifke, BS, and Jenny Taylor, BS, of EarliTec Diagnostics for their help in eye-tracking data processing and analysis. We thank John Shen, PhD, and Weiwei Tao, PhD, of OcTech Consulting Inc, and Yiming Deng, PhD, of Libra Medical Inc for assistance in statistical analysis. None received additional compensation outside of their regular employment salary.
✉
Corresponding author.
Received 2022 Dec 7; Accepted 2023 Jun 29; Issue date 2023 Sep 5.
Copyright 2023 American Medical Association. All Rights Reserved.
PMCID: PMC10481242 PMID: 37668621
See commentary ” Could an Eye-Tracking Test Aid Clinicians in Making an Autism Diagnosis?” on page 815.
See ” Development and Replication of Objective Measurements of Social Visual Engagement to Aid in Early Diagnosis and Assessment of Autism” in JAMA Netw Open, volume 6, e2330145.
Key Points
Question
Can eye-tracking–based measurement of social visual engagement aid in early diagnosis and assessment of autism in young children?
Findings
In a multisite, prospective, double-blind study of 475 children aged 16 to 30 months assessed for autism in 6 specialty clinics, measurement of social visual engagement had 71.0% sensitivity and 80.7% specificity relative to expert clinical diagnosis. In the subgroup of children whose autism diagnosis was certain (n = 335), the test had 78.0% sensitivity and 85.4% specificity.
Meaning
Eye-tracking–based measurement warrants further evaluation for early diagnosis and assessment of autism in young children referred to specialty clinics.
Abstract
Importance
In the US, children with signs of autism often experience more than 1 year of delay before diagnosis and often experience longer delays if they are from racially, ethnically, or economically disadvantaged backgrounds. Most diagnoses are also received without use of standardized diagnostic instruments. To aid in early autism diagnosis, eye-tracking measurement of social visual engagement has shown potential as a performance-based biomarker.
Objective
To evaluate the performance of eye-tracking measurement of social visual engagement (index test) relative to expert clinical diagnosis in young children referred to specialty autism clinics.
Design, Setting, and Participants
In this study of 16- to 30-month-old children enrolled at 6 US specialty centers from April 2018 through May 2019, staff blind to clinical diagnoses used automated devices to measure eye-tracking–based social visual engagement. Expert clinical diagnoses were made using best practice standardized protocols by specialists blind to index test results. This study was completed in a 1-day protocol for each participant.
Main Outcomes and Measures
Primary outcome measures were test sensitivity and specificity relative to expert clinical diagnosis. Secondary outcome measures were test correlations with expert clinical assessments of social disability, verbal ability, and nonverbal cognitive ability.
Results
Eye-tracking measurement of social visual engagement was successful in 475 (95.2%) of the 499 enrolled children (mean [SD] age, 24.1 [4.4] months; 38 [8.0%] were Asian; 37 [7.8%], Black; 352 [74.1%], White; 44 [9.3%], other; and 68 [14.3%], Hispanic). By expert clinical diagnosis, 221 children (46.5%) had autism and 254 (53.5%) did not. In all children, measurement of social visual engagement had sensitivity of 71.0% (95% CI, 64.7% to 76.6%) and specificity of 80.7% (95% CI, 75.4% to 85.1%). In the subgroup of 335 children whose autism diagnosis was certain, sensitivity was 78.0% (95% CI, 70.7% to 83.9%) and specificity was 85.4% (95% CI, 79.5% to 89.8%). Eye-tracking test results correlated with expert clinical assessments of individual levels of social disability ( r = −0.75 [95% CI, −0.79 to −0.71]), verbal ability ( r = 0.65 [95% CI, 0.59 to 0.70]), and nonverbal cognitive ability ( r = 0.65 [95% CI, 0.59 to 0.70]).
Conclusions and Relevance
In 16- to 30-month-old children referred to specialty clinics, eye-tracking–based measurement of social visual engagement was predictive of autism diagnoses by clinical experts. Further evaluation of this test’s role in early diagnosis and assessment of autism in routine specialty clinic practice is warranted.
Trial Registration
ClinicalTrials.gov Identifier: NCT03469986
This study evaluates the performance of eye-tracking measurement of social visual engagement relative to expert clinical diagnosis in young children referred to specialty autism clinics.
Introduction
Autism affects approximately 1 in 36 US children.1 Most parents of children with autism report having had concerns before the second birthday,2, 3 yet the median age of US autism diagnosis is 4 to 5 years.1, 4 For US racial and ethnic minority, low-income, and rural families, the age at diagnosis is later still.5, 6, 7, 8
Delay in diagnosis delays opportunities to support children and families early, when the brain is most malleable and before maladaptive behaviors may become entrenched.9, 10 From time of first concern to time of diagnosis, most families face multiple years of referrals and waiting for evaluation at specialty centers.11, 12
Objective biomarker tests could offer potential assistance for this issue by aiding in accurate, efficient diagnosis. Measurement of social visual engagement—how children look at and learn from the social environment—offers neurobiological and clinical face validity as a possible measure of autism. Individual variation in social visual engagement reflects individual genetic influence (with monozygotic twin concordance of ~0.9)13, 14; developmental change in this factor is evolutionarily conserved15; and autism-related differences13, 16, 17 emerge at young ages and predict later diagnosis.16
Previous research developed eye-tracking–based measurement of social visual engagement to aid in early autism diagnosis and assessment.18 The current study evaluated this test’s performance at sites across the US with multiple expert clinical teams and measurements collected on automated eye-tracking devices operated by clinical site staff.
Methods
This study evaluated diagnostic performance of eye-tracking–based measurement of social visual engagement (index test) in comparison with expert clinical diagnosis (reference standard) in terms of both (1) categorical diagnostic labels (autism or nonautism) and (2) continuous measures of social disability, verbal ability, and nonverbal cognitive ability. The trial protocol and statistical analysis plan are in Supplement 1. The study was preregistered and followed Standards for Reporting of Diagnostic Accuracy (STARD) guidelines19 (see the eMethods in Supplement 2 for details).
Study Design
The study used a multisite, prospective, double-blind, within-subject comparison to evaluate the test’s performance among 16- to 30-month-old children enrolled at 6 specialty centers between April 2018 and May 2019, with participants each completing a 1-day protocol. This age range was selected as the period when overt symptoms of autism are commonly observable,20 with the objective for the test to aid in diagnosis prior to 30 months, leaving 6 months in which children could be referred to early intervention before reaching 36 months. Children were enrolled at specialty clinics in the Emory University School of Medicine, Southwest Autism Research & Resource Center, Seattle Children’s Autism Center, Cincinnati Children’s Hospital, University of California San Francisco, and Rush University Medical Center. The research protocol was approved as a minimal-risk study by the institutional review boards at each participating institution, and parents or legal guardians gave written informed consent for participating children.
To minimize verification bias,21 diagnostic verification was complete (rather than partial or differential). To minimize selection bias,21 participants were enrolled consecutively. Study design was double-blind21: all staff involved with data collection and analysis of eye tracking were blind to reference standard results, and all staff involved with collection and interpretation of reference standard were blind to eye-tracking results. Only reference standard diagnosis and assessments (current clinical best practice) were used in patient care (neither parents nor study center staff were informed of a child’s eye-tracking results).
Two third-party clinical research organizations (STATKING Clinical Services Inc, and Libra Medical Inc) were responsible for study oversight, data management, unblinding, and comparison of index test and reference standard. Although no serious adverse events were anticipated, adverse events were monitored and summarized for safety analyses.
Participants
Children were consecutively enrolled according to the following inclusion criteria: chronological age between 16 and 30 months; normal or corrected-to-normal vision; normal or corrected-to-normal hearing; in generally good health with no acute illness; and parent or guardian able to understand informed consent in English ( Figure 1 and Table; eMethods in Supplement 2). To characterize the study sample, patient demographic data, including race, ethnicity, and maternal education, were collected based on parent report. Race and ethnicity data were collected to enable evaluation of whether test performance varied based on these characteristics.
Figure 1. Participant Enrollment and Outcomes Comparing Eye-Tracking–Based Quantification of Social Visual Engagement With Expert Clinical Diagnosis of Autism.
During a single visit at each clinical testing site, enrolled participants received expert clinical diagnosis using standardized assessments (reference standard diagnosis) as well as eye-tracking–based measurement of social visual engagement (index test). Clinical staff were blind to eye-tracking results and eye-tracking staff were blind to clinical results. For each participant, expert clinicians rated their certainty of diagnosis as in Klaiman et al22 and McDonnell et al.23
Table. Participant Characterization and Demographics.
| Reference standard diagnosis | |
|---|---|
| All participants (N = 475) a | Certain diagnosis (n = 335) |
| :— | :— |
| Autism or possible autism | Nonautism or possible nonautism |
| :— | :— |
| No. | 221 |
| Age, mo | |
| Mean (SD) | 24.9 (4.2) |
| Median (IQR) | 26 (21-29) |
| Sex, No. (%) | |
| Female | 57 (25.8) |
| Male | 164 (74.2) |
| Race, No. (%) b | |
| Asian | 21 (9.5) |
| Black/African American | 22 (9.9) |
| Native Hawaiian or Other Pacific Islander | 2 (0.9) |
| White | 149 (67.4) |
| Other | 26 (11.8) |
| Unknown | 1 (0.5) |
| Ethnicity, No. (%) c | |
| Hispanic | 44 (19.9) |
| Non-Hispanic | 177 (80.1) |
| No response | 0 |
| Maternal education, No. (%) d | |
| Less than eighth grade | 0 |
| Some high school | 4 (1.8) |
| High school or GED | 33 (14.9) |
| Some college, no degree | 39 (17.7) |
| Vocational school | 12 (5.4) |
| Associate’s degree | 13 (5.9) |
| Bachelor’s degree | 71 (32.1) |
| Master’s degree | 41 (18.6) |
| Professional or doctoral degree | 2 (0.9) |
| No response | 6 (2.7) |
| ADOS-2 e, f | |
| Social affect score, mean (SD) | 14.7 (4.3) |
| Median (IQR) | 16 (11-18) |
| RRB score, mean (SD) | 4.9 (2.0) |
| Median (IQR) | 5 (4-6) |
| Total score, mean (SD) | 19.6 (5.1) |
| Median (IQR) | 20 (16-24) |
| Mullen Scales of Early Learning g | |
| Verbal age equivalent score, mean (SD), mo | 12.5 (7.1) |
| Median (IQR) | 11 (8-16) |
| Nonverbal age equivalent score, mean (SD), mo | 18.5 (5.9) |
| Median (IQR) | 18 (15-21) |
| Other nonautism diagnoses h | |
| Developmental disability | 124 (56.1) |
| No other diagnoses (unaffected) | 0 |
Abbreviations: ADOS-2, Autism Diagnostic Observation Schedule, second edition; RRB, restricted and repetitive behavior.
a
For a table summarizing participant characterization and demographics data for all participants eligible and enrolled (N = 499), irrespective of diagnostic outcomes, please see eTable 1 in Supplement 3.
b
Race data were collected as fixed categories by parental selection, with both “other” and “unknown” as options.
c
Ethnicity data were collected as fixed categories by parental selection.
d
Maternal education data were collected as fixed categories by parental selection.
e
ADOS-2 is a standardized diagnostic assessment for autism, administered by a trained clinical specialist using a semistructured play session consisting of a set of social and communication interactions intended to elicit behaviors relevant to autism diagnosis. The social affect domain score includes test items pertaining to communication and reciprocal social interaction; higher scores indicate more autism symptoms: scores shift slightly based on module and age of child, with scores of approximately 0 to 6 indicating minimal symptoms, scores of approximately 7 to 10 indicating moderate symptoms, and approximately 11 and greater indicating more significant symptoms. Score range is from 0 to 20 (toddler module for children with few to no words) or 0 to 22 (module 2 and toddler module for older children with some words). RRB domain score, which includes test items pertaining to restricted and repetitive behaviors; higher scores indicate more autism symptoms: scores of approximately 1 to 2 indicate minimal symptoms, scores of approximately 3 to 4 indicate moderate symptoms, and approximately 5 and greater indicate more significant symptoms. Score range is from 0 to 6 (toddler module for children with few to no words) or 0 to 8 (module 2 and toddler module for older children with some words). Total score is the total of social affect and RRB. Higher scores indicate more autism symptoms, with scores shifting slightly based on module and age of child: scores from approximately 0 to 7/9 indicate minimal concern, from approximately 8 to 11 or 10 to 13 indicate mild to moderate concern, and approximately 12/14 and greater indicate more moderate to severe concern. Ranges vary according to the toddler module algorithm (children with few to no words or older children with some words) or module 2. In this sample, 414 (87.5%) children received the toddler module.
f
There were 252 children with nonautism diagnoses (2 children missing data because they were not yet walking, no ADOS-2 conducted because administration would be considered invalid).
g
The Mullen Scales of Early Learning is a standardized developmental assessment, administered by a trained clinical specialist, to measure a child’s language, motor, perceptual, and cognitive abilities. Mullen verbal ability age equivalent score, in months, calculated as the average of the Mullen expressive language age equivalent score and the Mullen receptive language age equivalent score. Scores range from 1 to 70 months, with higher scores indicating more-advanced receptive and expressive language abilities. Mullen nonverbal ability age equivalent score, in months, calculated as Mullen visual reception age equivalent score. Scores range from 1 to 69 months, with higher scores indicating more-advanced nonverbal problem-solving abilities. Mullen age equivalents indicate the age bracket in the Mullen norming sample that had the same median score as a tested child’s raw score. Age equivalents are helpful in showing differences between performance areas or in emphasizing that a child’s score was similar to or different from abilities expected at a given chronological age.
h
Expert clinician diagnosis of 1 or more co-occurring developmental disabilities, including language, cognitive, or motor delays, vs no clinical diagnosis of any kind. Speech-language and global developmental delays were the most common other diagnoses given.
Reference Standard Diagnosis
Reference standard diagnosis of autism spectrum disorder or nonautism spectrum disorder (labeled herein as autism and nonautism, respectively) was made by expert clinicians using standardized assessments: the Autism Diagnostic Observation Schedule, Second Edition (ADOS-2)24; the Mullen Scales of Early Learning (Mullen)25; medical and developmental histories; and DSM-5 criteria. Expert autism clinician status was confirmed on the basis of clinician training and experience and by establishing ADOS-2 research reliability with a certified ADOS-2 trainer (eMethods in Supplement 2).
These procedures constitute current best practice for diagnosing autism, but are nonetheless based on the expert clinician’s subjective judgment. Given this, together with the study’s focus on diagnosing very young children20, 26 who can present with complex, evolving clinical profiles (eg, language, cognitive, or other challenges, co-occurring with or distinct from autism), expert clinicians were required to prospectively rate their level of certainty in each diagnosis.22, 23 Reference standard certainty is important because performance metrics of comparison tests depend inherently on the reliability and validity of the reference standard: if a participant’s reference standard diagnosis is uncertain, then that participant’s ground truth for comparison will also be uncertain. Certainty was rated prospectively with each expert clinician blinded to index test results.22, 23
Index Test Measurement of Social Visual Engagement
Social visual engagement data were collected using 6 investigational eye-tracking devices, one at each clinical site. Device operators were staff technicians employed by each clinical site, trained for approximately 1 hour when each device was delivered, with no other required specialization or prior training. The device was located in available space within the clinic (ie, examination room, playroom). Data collection procedures were automated, with no instructions required of the child. Software guided the device operator through eye-tracking calibration, validation of calibration, and automated data collection.
The eye-tracking index test measured social visual engagement while children watched video scenes of social interaction, as previously described13, 18 (see also the Experimental Procedures section under eMethods in Supplement 2). As in the study by Jones et al,18 14 video scenes were presented, each with a mean (SD) duration of about 54.0 (21.5) seconds (range, 21.7 seconds to 1 minute 29.7 seconds).
Following collection, index test data were automatically uploaded to a secure cloud-based server for processing and analysis. Processing and analysis were automated, and each participant’s data, once analyzed, yielded categorical determination of either autism or nonautism, as well as 3 continuous measures: a social disability index, verbal ability index, and nonverbal ability index (see the Data Processing and Data Analysis sections under eMethods in Supplement 2 for details).
Planned Analyses
Primary outcome analyses compared results from the eye-tracking index test with results from reference standard diagnosis. The prespecified test positivity threshold was determined in the efficacy study,18 fixed, and applied herein (values ≤0 indicated autism). The comparison was quantified as index test sensitivity and specificity, with 2-sided 95% CIs,27 together with receiver operating characteristic curves and area under the curve metrics. Positive predictive value (PPV), negative predictive value (NPV), and accuracy of the index test were also calculated. Primary end point analyses were tested at a 1-sided significance level of α = .025.
Secondary outcome analyses compared eye-tracking–based indices of social disability, verbal ability, and nonverbal cognitive ability with their respective expert clinician–administered reference standard assessments (ADOS-2 total scores, Mullen verbal age equivalent scores [average of receptive and expressive language], and Mullen nonverbal age equivalent scores). Standard regression diagnostics were performed for all comparisons,28, 29 and Deming regression30, 31 was used to estimate regression coefficients. Pearson correlation coefficients and adjusted R 2 coefficients of determination,32, 33, 34 together with 95% CIs, were calculated. Secondary outcome analyses were tested at a 1-sided significance level of α = .025.
Exploratory Analyses
Exploratory analyses measured effects of race, ethnicity, sex, age, eye-tracking calibration accuracy, false-positive and false-negative clinical outcomes, reference standard certainty, and reference standard measurement error. Reference standard certainty sets an upper limit on performance metrics of comparison tests, but also enabled a post hoc inferential hypothesis test: if an index test is accurate, its performance should be (1) superior when the reference standard is certain and (2) inferior when the reference standard is uncertain (an accurate test should not match an uncertain reference standard). Similarly, reference standard measurement error variance sets an upper limit on the amount of nonerror variance that can be explained by comparison tests32, 33, 34; reference standard instruments in the current study have “good” but not “excellent” test-retest reliability24, 25 (0.87, 0.74, and 0.75, respectively, for ADOS-2 total score and Mullen verbal and nonverbal scales). Finally, we compared the present study results with findings from prior studies18 to assess replicability.
Results
Participants
Of 499 eligible children enrolled, 475 (95.2%) successfully completed the eye-tracking test and comprise the primary analysis set ( Figure 1). Expert clinicians were certain of reference standard diagnoses in 335 (70.5%) of the 475 participants, while there was uncertainty in 140 (29.5%).22
The Table describes participant demographic and clinical characteristics. The mean (SD) age of participants was 24.1 (4.4) months. Additionally, 38 (8.0%) were Asian; 37 (7.8%), Black; 352 (74.1%), White; 44 (9.3%), other; and 68 (14.3%) were of Hispanic ethnicity. Participants with autism had higher ADOS-2 domain and total scores (all t > 23.8, all P < .001). They also had lower Mullen verbal age equivalent scores and lower Mullen nonverbal age equivalent scores ( t = 17.0, P < .001 and t = 11.8, P < .001, respectively). Characteristics of all eligible enrolled participants, including those who did not complete the test, are provided in eTable 1 in Supplement 3, and comparison of certain and uncertain diagnosis subsamples in eTable 2 in Supplement 3. Eye-tracking quality control indicators are provided in eFigure 1 and eTable 3 in Supplement 3.
Among nonautism and possible nonautism participants, 82.7% had 1 or more co-occurring cognitive or developmental delays (210/254, based on Mullen scores), while 17.3% (44/254) did not. Likewise, among autism and possible autism participants, 56.1% had co-occurring cognitive or developmental delays (124/221), while 43.9% did not (97/221).
While chronological age was restricted to 16 to 30 months, children’s demonstrated verbal abilities (in both diagnostic groups) varied from those delayed to levels approximating typical 4-month-old infants (eg, vocalizing with nonspecific syllable sounds35) to those with advanced skills more common to 44-month-old children (3.7 years old) (ie, knowing several hundred or more words; producing competent phrase speech25, 35). For nonverbal abilities, the range varied similarly from skills approximating those of typical 8-month-old infants (eg, working on fine grasping; beginning to actively seek out-of-view objects35) to those more common to typical 50-month-old children (4.2 years old) (independently performing self-help skills; knowing multiple numbers and amounts35).
Primary End Points: Estimates of Diagnostic Accuracy
Figure 2 presents results for the primary effectiveness end point: accuracy of diagnostic classification when compared with reference standard expert clinical diagnosis (see eFigure 2 in Supplement 3 for underlying distribution data). Figure 2 A plots the receiver operating characteristic curve for all participants with both index test and reference standard diagnosis (primary analysis set). As noted above, this includes data for 140 participants (29.5% of the sample) for whom clinicians were uncertain of their reference standard diagnosis; despite that uncertainty, index test performance had area under the curve equal to 0.85 (95% CI, 0.82-0.89), sensitivity of 71.0% (95% CI, 64.7%-76.6%), and specificity of 80.7% (95% CI, 75.4%-85.1%). Crosstabulation data and performance metrics are given in Figure 2 D. These results represent worst-case performance given the amount of reference standard uncertainty in the comparison (29.5%) because reference standard uncertainty necessarily sets an upper limit on comparison test metrics (eFigure 3 in Supplement 3).