Eye tracking in AI-assisted AR glasses enabled by Pancharatnam–Berry optical elements (Benoit et al., 2026)
Source: https://iopscience.iop.org/article/10.1088/2515-7647/ae58e0 · Published: 2026-04-09 (J. Phys. Photonics 8:022001; CC BY 4.0) · Language: English
Note: excerpt: abstract, introduction/architecture review and future-perspectives sections; PBOE physics/design sections and references omitted.
Abstract
The widespread interest and adoption of artificial intelligence (AI)-assisted smart glasses have redefined the wave of next-generation wearable displays. To improve the accuracy of AI applications through visual-spatial contextual awareness, as well as overcoming limitations that restrict true augmented reality (AR) capability in consumer-ready smart glasses, eye tracking has emerged as a solution to overcome the limitations of AI-Assisted AR glasses. However, a socially acceptable form factor that can meet critical eye tracking requirements has yet to be realized for everyday use cases, especially in varying ambient conditions and users. Here, we develop novel illumination architectures around the real-case form factor of smart glasses using Pancharatnam–Berry optical elements (PBOEs) for near infrared light. By taking advantage of properties surrounding PBOEs, such as polarization selectivity and conversion, as well as the principles of geometric phase, our systems demonstrate nearly 100% diffraction efficiency and provide innovative approaches to overcome factor constraints in emerging 2D and 3D eye tracking methods.
Export citation and abstractBibTeX RIS
Original content from this work may be used under the terms of the Creative Commons Attribution 4.0 license. Any further distribution of this work must maintain attribution to the author(s) and the title of the work, journal citation and DOI.
1. Introduction
The convergence of artificial intelligence (AI) and ultracompact and lightweight augmented reality (AR) glasses has created an aligned path to the next generation of consumer electronics: AI-assisted AR smart glasses. Recently, the advancement of multimodal large language models has opened new applications in human-centric computing that improve, advance, and augment human capability not previously possible [ 1– 3]. At the same time, the near-eye display industry has made remarkable progress towards making devices that provide a socially comfortable form factor and project virtual content indistinguishable to objects in the outside world.
To truly augment human capabilities with future smart glasses, significant challenges remain. For AI, most models lack the ability to understand the user’s visual spatial attention, degrading the user’s intent of the input prompt. This can be consequential for the user experience, which degrades adoption. Even in current smart glasses that use smart accessories for the models, prompt accuracy remains a challenge [ 1]. The near-eye display industry also faces challenges with maintaining a large, high uniformity eye box that degrades the efficiency and power consumption for everyday use [ 4]. Techniques that can reduce the eye box size, such as dynamic eye box switching and pupil steering, and techniques to reduce the resolution of the peripheral region for the user to reduce power consumption, called foveated rendering, have gained interest and show promise for everyday use cases. The enablement of AI applications and AR for everyday use has been found through the implementation of an eye-tracking system [ 5– 7].
In this paper, we will begin by introducing the evolution of eye tracking architecture for extended reality (XR), including optical see-through AR glasses and mixed reality (MR), which is a virtual reality (VR) with cameras to enable video pass-through to interact with the real world. Analyzing the advancements of eye tracking helps establish the need for novel architecture that can reduce the form factor and improve the accuracy for AI-assisted AR glasses. Furthermore, by taking advantage of high diffraction efficiency (DE), large angular response, and advancements in Pancharatnam–Berry optical elements (PBOEs) for the display system in AR devices, we introduce PBOE as a promising candidate to enable eye tracking for next generation smart glasses.
1.1. Advancements of eye tracking architectures for AI/XR glasses
1.1.1. Video-oculography, single detector eye tracking
Similar to the display system in near-eye displays, eye tracking hardware has significantly evolved, especially for AR devices. Up until the late 2010’s, eye tracking systems relied on IR light emitting diodes (LEDs) with cameras facing inwards to capture video of the pupil and corneal reflections from the eye, as shown in figure 1(a) [ 8, 9]. This method, called video-oculography, opened the ability to perform gaze estimation and demonstrate a technique called foveated rendering, where the image quality in the peripheral region of the eye is to improve the overall viewing experience. However, implementing eye tracking increased the form factor, significantly increased the cost, and without the proper computing power, suffered significantly with latency and image drift. The devices that incorporated eye tracking, in addition, were solely focused on VR headsets, not AR.
Zoom In Zoom Out Reset image size
Figure 1. The evolution of eye tracking for near-eye displays, from previous, current, to future prospect architectures addressing the need for a socially comfortable form factor. (a) Schematic of early video-oculography techniques for eye tracking in head mounted displays, common for experimental, legacy head-mounted displays (HMDs) before the mid 2010’s. (b) A schematic of integrated eye tracking architectures for current MR/AR devices. Current applications place the illumination source and the detectors along the hinge of the eyepiece, requiring a thick hinge to be placed around the eyepiece to avoid the components being visible to the outside world. (c) Next generation eye tracking architecture proposed by academia and industry that incorporate illumination and imaging combiners on the eyepiece. Like display systems in AR glasses, the motivation is to place the illumination optics and detection system outside of the user’s peripheral. For diagrams, orange components represent the illumination path of near-infrared (NIR) light, and blue components represent the components for sensing and detection.
Download figure:
Standard image High-resolution image
The first commercial AR headset with built-in eye tracking came in 2018, with the introduction of Magic Leap 1 [ 10]. The device, featuring one camera per eye with off-axis IR illuminators, had a 1° gaze error and had a sampling rate of 60 Hz, impressive at the time the device went on the market. Shortly following the release of Magic Leap 1, HoloLens 2 implemented a similar eye tracking architecture, features NIR LEDs and an individual IR camera embedded in the visor, just behind the transparent waveguide optics. HoloLens 2’s waveguides are coated with dichroic layers, meaning that visible light from the display passes through, while NIR from the LEDs is steered/reflected so the IR cameras get a clean signal without interfering with the projected image. Like previous headsets, bulkiness and cost of both the eye tracking system and the headset limited commercial adoption. Further, image drift and latency were still an issue at the time, making the overall viewing experience with gaze-dependent imaging techniques, such as foveated rendering, still inadequate for daily use.
1.1.2. Miniaturized cameras and light sources for advanced XR devices
From the previous architectures discussed, improving the cost, form factor, and accuracy began to be explored in the late 2010’s and early 2020’s. To improve accuracy, eye tracking systems have begun to implement multiple camera sensors, allowing for more data points to estimate the gaze position of the user as shown in figure 1(b). With the miniaturization of CMOS-based sensors and NIR light sources, many companies began to develop eye tracking systems in the eyepiece to capture the NIR reflections off of the eye’s corneal surface. This technique, called glint-based eye tracking, has continued to be the most adoptable solution for low cost, acceptably accurate eye tracking. Two devices, the Magic Leap 2 and Apple Vision Pro, are examples of this evolution in eye tracking architecture.
Magic Leap 2 integrates an eyepiece that consists of 6 LEDs and 2 IR cameras per eye, doubling the number of cameras of Magic Leap 1, while significantly reducing the form factor into a single, integrated eyepiece. Magic Leap 2 also found additional use cases for eye tracking that enhanced the user’s visual experience. One area where eye tracking is used is segmented local dimming [ 11, 12], which is critical to enable occlusion and to improve ambient contrast ratio. Additionally, the company does claim they can provide binocular and color correction improvement through the display engine itself, which is a need discussed before for eye tracking. While Magic Leap does not specifically discuss the functionality of their algorithms publicly, their patents provide some context of where the company aims to go in the field of eye tracking. The Apple Vision Pro featured a similar architecture: two IR cameras per eye with multiple LEDs, and an ultra-low latency of 10 ms, nearly a 67x improvement over the Magic Leap 2. While these devices featured a dramatic improvement in form factor, the overall cost of the Magic Leap 2 and the Apple Vision Pro increased significantly, restricting adoption of these headsets for consumer use. Social comfort and power consumption remain a significant bottleneck as well, because these devices tend to either be wired or have less than 2 h of battery life, making them unsuitable for everyday use.
1.1.3. AI-enabled eye tracking
The advancement of sensing and computational imaging has helped advance camera technologies to show promise in achieving a compact, low latency and low-cost form factor for smart glasses [ 13– 22]. In analyzing promising advancements in computational imaging, recent advancements in both hardware and software have helped bring latency down to sub-microsecond levels and improve gaze error down to below half a degree through the maturity of new computational imaging techniques, such as polarization-based sensing [ 16– 19] and deflectometry [ 20– 22]. However, these technologies have yet to be realized for AI-assisted AR glasses due to a low signal-to-noise ratio (SNR), especially across a high ambient environment, illumination uniformity over a large area of the eye, and bulky optics required for illumination. Therefore, as AI and sensing platforms have significantly evolved over the past decade, the adoption of eye tracking for everyday smart glasses must come from the improvement of the illumination system.
Currently, there are few companies that provide AI glasses with eye tracking, and there are no architectures on the market for AR devices that mimic the form factor of prescription eyewear. Companies, such as Tobii, Pupil Labs and Tobii, offer solutions that integrate their eye tracking platform into a frame, but such devices are currently used for research purposes only. With AR integration, Meta introduced their Orion AI glasses and claimed to feature integrated eye tracking. However, metrics surrounding the eye tracking system have yet to be released, and Orion AI has yet to be demonstrated for commercial use.
That said, many companies have proposed novel illumination systems that aim to integrate eye tracking into prescription eye wear for everyday use, as demonstrated in figure 1(c). Rather than placing emitters and cameras around the lens periphery, companies including Magic Leap [ 23, 24], DigiLens [ 25], Meta [ 26], Apple [ 27], and Samsung [ 28] propose using thin planar waveguides or combiners to guide NIR illumination to the eye and return corneal reflections back to imaging. This approach collapses the illumination and imaging paths into the eyepiece itself, dramatically reducing mechanical volume while enabling in-field illumination. Across these works, the waveguide is no longer a passive display component but an active optical relay for eye tracking; supporting pupil expansion, controlled angular routing, and tight co-integration with display optics to improve scalability and wearability. The use of a waveguide does present inherent challenges to the eye tracking system. Diffractive waveguides struggle with efficiency from the in-coupler to the out-coupler, staying well below 5%. Using two waveguide paths for the illumination and detection system means that the efficiency at the detector can drop to 0.25%, and that is assuming near-perfect reflectance off of the eye surface. Further, illumination uniformity remains a key challenge as well for diffractive waveguides, requiring significant calibration and computational power for the eye tracking system. Geometric/reflective waveguides incorporate partially reflective mirrors into the waveguide, which overcomes eyebox uniformity issues with diffractive waveguides and significantly improves the power consumption due to a drastic improvement of efficiency; up to 10% compared to diffractive waveguides that are below 5% [ 29]. Beyond fabrication costs, stray light and ghost imaging limit their use for eye tracking for accurate detection. To overcome efficiency-accuracy deficiencies with waveguide-based eye tracking, both academia [ 30, 31] and companies like Tobii [ 32] and TruLife Optics [ 33, 34], in collaboration with Bayfol Covestro, and Meta [ 35, 36] have explored using free space, holographic mirrors to attain an image of the eye off-axis. These proposals remain the promising candidate to reduce the form factor, place the optics out of the user’s eyepiece, and provide even more functionality with the implementation of phase to expand illumination and detection coverage. Experimental demonstrations further show that Bayfol’s NIR-sensitive materials can provide up to 85% reflectivity while still transmitting ambient light, and the holographic mirror can be implemented within the optical path through lens casting. However, these works are still in the development stage, and specific parameters for both the illumination and detection system, such as the stray light optimization, angular response, and the overall design structure of the combiner have yet to be formally reported.
Polarization-selective diffractive optics have been of interest in separating eye-tracking illumination from both the display light and ambient light while maintaining high efficiency. Previously, polarization-based imaging of the eye was reported using visible light, unsuitable for eye tracking in display systems. However, in [ 16], Meta demonstrated the use of NIR light to visualize the birefringence of fibrous collagen, which drastically improves the contrast of the cornea and scleral regions for both glint-based and non-glint-based eye tracking methods. An additional benefit, leveraging circular or linear polarization selectivity to route IR illumination and suppress unwanted reflections or display crosstalk is of significant interest to improve the overall SNR, mitigates ghost images and coherent artifacts, and allows designers to place eye-tracking optics directly in the display optical path without visually contaminating the user’s view [ 37– 40]. Collectively, these efforts show polarization selectivity is not a secondary optimization, but a core architectural requirement for viable in-field eye tracking.
Finally, beyond basic routing, these companies introduce accuracy-driven optical ‘tricks’ that are only practical in waveguide- and polarization-enabled systems. Examples include telecentric or quasi-telecentric eye views to reduce gaze error from eyelid occlusion and oblique incidence, multi-view holographic combiners that project several eye perspectives onto a single sensor to maintain tracking robustness across the eye box, and selective imaging of corneal versus scleral regions to improve 3D eye modeling. Form factor is simultaneously addressed by reducing camera count, collapsing relay optics, embedding diffractive optical elements (DOEs) directly into waveguides, and in Apple’s case, exploring camera-less or highly simplified sensing modalities. Taking together, these approaches show that modern eye-tracking architectures are no longer optimized solely for detection, but for co-designed optical efficiency, robustness, and manufacturability [ 41].
From the evolution of the eye tracking system discussed, architectures that provide social comfort, high polarization-selectivity, and use novel optical elements to further improve the capability of the eye tracking system are in demand for next generation smart glasses. In the following sections, we take advantage of the advancements in PBOEs to introduce, analyze, and experimentally characterize a novel eye tracking architecture enabling eye tracking for AI-assisted glasses, with and without AR capability. The principles discussed aimed to open a new wave of capability for both 2D and advanced 3D eye tracking techniques discussed previously, while also providing a large angular response for the collection of eye surface reflections for the detection system.
[…]
4. Future perspectives
4.1. Future perspectives: device improvement
Due to the design freedoms of PBOEs, multiple design improvements can take place. From a materials perspective, increasing the birefringence of the liquid crystal polymer used for PVG can drastically improve uniformity and bring down the thickness of the PVG due to a larger coupling coefficient. By using linear mixing to increase the birefringence, the incidence angle needed to illuminate the corneal-scleral surface discussed before has an improved uniformity. For example, increasing the birefringence to 0.18 improves the uniformity upon reflection from 94% to 98%, as shown in figure 13.
Zoom In Zoom Out Reset image size
Figure 13. (a) The current angular response of the traditional PVG used for illumination. The angle of incidence interval is set to demonstrate the response over the angles for the 3 × 3 beamsplitter for glint-based eye tracking (4.23° × 4.23°). (b) The resulting uniformity improvement when 
is increased to 0.18, achievable through linear mixing of liquid crystal polymer.
Download figure:
Standard image High-resolution image
There have been multiple advancements in increasing the birefringence to improve the angular response and decrease the thickness dramatically. Commercially available RMs, such liquid crystal polymer PLC-037, have a Δ n of 0.32 at 589 nm, nearly double the Δ n of RM257. Intermolecular reported an RM with a high refractive index (
2.0), which could enable a Δ n as high as 0.5, assuming the ordinary index 
remains at 1.51, typical for RM solutions [ 80]. However, such large birefringence RMs have yet to prove good photostability over a long period. Meanwhile, the decreased solubility of higher Δ n LC polymer is another concern. In our experiments, using 90% RM257 and 10% of liquid crystal polymer PLC-037, which has a birefringence of 0.3, can give a birefringence of 0.174, which is close to the birefringence used in figure 13(b), but anything above that is difficult. Therefore, designing high birefringence RM materials with good solubility and excellent photostability is an urgent task.
4.2. Future perspectives: waveguide-based illumination
In our analysis, we reviewed potential waveguide architecture candidates for both the illumination system and detection system. Waveguide-based eye tracking is attractive due to the ability to perform in-field illumination, enabling a bright pupil that improves the contrast of the pupil features for eye tracking. However, there are multiple issues that remain to be addressed.
In exploring both geometric waveguides and diffractive waveguides, there have been significant improvements for both coupling techniques. Geometric waveguides provide a large, expanded eye box, high angular selectivity, and have little to no eye glow due to light being reflected directly to the user, improving both social comfort and power consumption [ 4, 29]. However, due to the backward reflections from mirrors, geometric waveguides are vulnerable to both stray light and ghost images, an issue for eye tracking systems in AI-Assisted AR glasses that need a high SNR and minimized stray light. On the other hand, diffractive waveguides have shown significant progress in improving angular and spectral uniformity, especially through the development of PVGs. The overall optical efficiency, however, remains an issue, especially when considering both the eye tracking and display system.
The second preliminary issue with waveguide-based systems is the coupler position and size required for a large illumination area. For a large illumination area, 2D pupil expansion is necessary to combat eye movement. One solution is to provide a central FoV, as the illumination area, not the FoV, is critical. Figure 14 demonstrates the optimized waveguide system using the beam tracing method explored in [ 81] that provides an illumination area of 19 mm × 11 mm, providing a system that can capture large, rapid eye movements.
Zoom In Zoom Out Reset image size
Figure 14. The waveguide design plot for a 2D pupil expansion waveguide with a central FoV, a candidate for future in-field illumination (bright pupil) architectures for eye tracking.
Download figure:
Standard image High-resolution image
However, when implementing with a display for AI-assisted AR glasses, the design becomes much more difficult, as optimizing the coupler for a large FoV creates overlap with the coupler. Furthermore, since the k-vector for 850 nm is much larger than that for visible light, clipping of both the visible and NIR light can take place. Figure 15 demonstrates the effect of the k-vector at 850 nm.
Zoom In Zoom Out Reset image size
Figure 15. (a) A waveguide design consisting of blue and green channels ( λ = 465 nm and λ = 532 nm respectively). The waveguide can achieve a 25° diagonal FoV, and consists of an input, folding, and output coupler period of 360 nm, 292.4 nm, and 360 nm respectively. The direction of the input, folding, and output coupler is 38°, 90°, and 142° respectively. (b) A waveguide design consisting of red and NIR channels. The waveguide can achieve a 25° diagonal FoV, and consists of an input, folding, and output coupler period of 500 nm, 406.1 nm, and 500 nm respectively. The direction of the input, folding, and output coupler remains 38°, 90°, and 142°, respectively. Adapted with permission from [ 81] © Optica Publishing Group.
Download figure:
Standard image High-resolution image
From preliminary results, incorporating a waveguide design for both the visible and NIR light would require at least two waveguides: Either the green and blue channels in one waveguide (figure 15(a)) and red and NIR in another waveguide (figure 15(b)) or the visible and NIR paths are separated, as figure 14 shows. Further, the FoV can only be expanded to 25°, limiting the display experience for AR applications. Exploring the possibilities with waveguide-based illumination and detection, therefore, has some promise for AI-assisted glasses without a display, but implementing the eye tracking system fully with the display system is a modeling and simulation challenge we aim to explore for future work. A solution of significant interest is expanding beyond liquid crystal-based/anisotropic structure PBOEs and transitioning to metasurface-based PBOEs [ 82– 85]. Such devices would allow for localized control of the anisotropy and phase shift, maximizing efficiency overall for the waveguide system, the phase profile especially advantageous for 3D eye tracking systems, and further expand the bandwidth due to increased anisotropy, enabling potentially full integration of the display and eye tracking system.
5. Conclusion
In this paper, we have analyzed and demonstrated, both through simulation and experiment, a novel illumination architecture that could enable eye tracking for next-generation AI-assisted AR glasses. Our architecture is motivated by the industry’s interest and needs for eye tracking to enable gaze-contingent AI applications and AR capability for next generation smart glasses. Through this work, we have analyzed all paths, the illumination and detection system, to control the polarization state of light to avoid crosstalk, which can optimize the SNR, crosstalk, and stray light accumulated by the eye tracking system, the display system, and the ambient light. With the hardware proven, we welcome collaboration to evaluate this design with eye tracking programs to provide calibration software and analyze the gaze error in comparison to current hardware implementations not suitable for smart glasses.
Beyond future work in implementation, we believe our devices can enable future perspective designs that can further introduce eye tracking technologies in-field to increase the contrast of pupil features and improve the overall performance by implementing new geometric phases.