Eye Tracking in Virtual Reality: a Broad Review of Applications and Challenges (Adhanom, MacNeilage & Folmer, 2023)
Source: https://pmc.ncbi.nlm.nih.gov/articles/PMC10449001/ · Published: 2023-01-18 (Virtual Reality 27:1481–1505; CC BY 4.0) · Language: English
Note: excerpt: abstract, background, technical-limitations section (with the device accuracy/latency table) and conclusion; applications sections and references omitted.
Abstract
Eye tracking is becoming increasingly available in head-mounted virtual reality displays with various headsets with integrated eye trackers already commercially available. The applications of eye tracking in virtual reality are highly diversified and span multiple disciplines. As a result, the number of peer-reviewed publications that study eye tracking applications has surged in recent years. We performed a broad review to comprehensively search academic literature databases with the aim of assessing the extent of published research dealing with applications of eye tracking in virtual reality, and highlighting challenges, limitations and areas for future research.
Keywords: Eye tracking, Virtual reality
1. Introduction
Catering the stimulus to the user’s actions, such as head movement, eye movement, and hand movement, is the core principle of virtual reality (VR). Head-mounted display (HMD)-based VR depends on the ability to track head movements and render visual scene motion contingent on head movement. This advance was made possible by improvements in head tracking technology. Moving forward, similar improvements in HMD-based eye tracking technology will allow for fundamental advances in VR applications based on eye movement.
Consumer VR HMDs have significantly advanced in recent years in terms of tracking, latency, refresh rate, resolution and optics ( Koulieris et al. 2019) with major consumer platforms (HTC Vive, Oculus Rift, Sony VR) already having presented second or third generation HMDs. Eye tracking technology has been commercially available for decades on desktop displays, but in recent years, a number of commercially available solutions have been developed that facilitate eye tracking on consumer VR HMDs. As a result, research and development surrounding eye tracking in HMDs has accelerated and expanded in recent years.
A number of literature reviews have provided an overview of general eye tracking applications ( Duchowski 2002) and gaze-based interaction ( Duchowski 2018) without focusing on VR. Additionally, there are several reviews, that we highlight in this review, that focus on specific applications of eye tracking in VR, including reviews by Rappa et al. (2019), Souchet et al. (2021), Lutz et al. (2017), Harris et al. (2019), and Souchet et al. (2021). However, to the best of our knowledge, no reviews exist that exclusively aim to provide a broad overview of eye tracking applications for VR HMDs. We believe that interest in this area is growing rapidly, given the large number of studies that utilize eye tracking in a VR setting. It is therefore timely to provide a broad overview of the applications, challenges and limitations of eye tracking in VR.
This paper provides a broad review of current and seminal literature of applications of eye tracking in VR and identifies limitations, challenges and areas for future research. The rest of this paper is organized as follows: Sect. 2 discusses background concepts about eye movements and eye movement tracking methods in VR that will help the reader to understand the concepts discussed in later sections. Section 3 presents the applications of eye tracking in VR by organizing them into seven broad application areas. Finally, Sect. 4 discusses the current challenges and limitations faced by eye tracking in VR.
2. Background concepts
2.1. Types of eye movements
There are several distinct types of eye movements that serve different functions and are controlled by distinct neural circuits. Optimization of eye tracking for VR applications often requires identifying and distinguishing these different eye movement types and exploiting knowledge of visual and cognitive processing during eye movements to achieve the desired outcome( Holmqvist et al. 2011).
The need for eye movements arises because human visual acuity is not uniform across the visual field. Visual acuity is highest at the fovea, a central region of the retina( Cowey and Rolls 1974). The foveal region has the highest density of photoreceptors that allows parts of the visual field that fall on it to be seen with the highest detail ( Curcio et al. 1990). Outside the foveal region, visual acuity drops gradually towards the edges of the retina. Below we will describe several eye movement types and their relevance including: saccades that just move the eyes from target to target, smooth pursuit that allow tracking moving objects, reflexive eye movements that stabilize vision, and vergence eye movements that coordinate the two eyes to allow binocular fixation at different depths. In the rest of this section, we describe these eye movements and their functional role in greater detail.
The subjective experience of being able to see the entire visual field in high acuity is achieved by scanning the visual field with successive eye movements known as saccades and integrating the high acuity snapshots of the scene that are obtained, a process known as trans-saccadic integration ( Melcher and Colby 2008). Saccadic velocities may reach as high as 400 to 600 deg/sec, with durations lasting from 30 to 120 msec, and amplitudes in the range 1 to 45 degrees; there is a systematic relationship between amplitude, duration, and velocity known as the saccadic main sequence ( Leigh and Zee 2015). Under natural circumstances, gaze shifts of greater than about 30 degrees are typically achieved by a combination of saccade and head movement. Vision is typically blurred during a saccade which allows for manipulation of the visual scene to go largely unnoticed during the saccadic interval.
Between saccades, the eyes typically remain relatively still for an average duration of 200-300 ms to fixate on objects of interest in the scene. Analysis of fixation locations provides information about the user’s attentional state and the scene content that the user is interested in. However, the eyes are never completely still during fixations, because there are miniature fixational eye movements that occur during this time of relative stillness. Fixational drifts are small and slow movements away from a fixation point ( Otero-Millan et al. 2014). Drifts are often followed by small faster movements known as microsaccades that bring the eyes back to the fixation point; these small movements are typically less than 1 degree ( Otero-Millan et al. 2014).
Pursuit eye movements occur when the eyes track a slowly moving visual target in order to stabilize the image of that target on the retina. Targets with velocities of greater than 30 deg/sec cannot be effectively tracked by pursuit eye movements, but catch-up saccades are triggered to compensate for this lag. Most people are unable to initiate pursuit without a moving visual signal. Nevertheless, pursuit is considered to be a voluntary eye movement ( Robinson 1965).
Another type of eye movement that serves to stabilize the image on the retina is known as the vestibulo-ocular reflex (VOR). These movements act to rotate the eyes to null foveal retinal image motion that would result from head movement relative to the stationary environment. The VOR is driven predominantly by vestibular signals that transduce inertial cues to head motion, both linear and angular ( Ramaioli et al. 2019). Vestibular-driven movements are supplemented by visually driven movements that respond to uncompensated image motion at the fovea known as retinal slip. The resulting stabilizing eye movement, referred to collectively as the visually enhanced VOR, allows us to see the world as clear and stable despite head movement ( Leigh and Zee 2015).
Vergence eye movements are the simultaneous movements of both eyes in opposite directions that allow the two eyes to change binocular fixation distance. The nearer the target, the greater the convergence that is required. Thus, the vergence angle of the eyes along with disparity in the images presented to the left and right eyes determines how users perceive stereoscopic depth. Because vergence is associated with target depth, there is a linkage between neural signals that drive vergence and those that drive accommodation, or focal state, of the lens of the eye, which also needs to vary with target depth to ensure that the retinal image is in focus ( Toates 1974).
Several of these eye movement types can occur simultaneously, such that the total movement of the eye is the result of superposition of contributing movements, making it difficult to identify and quantify distinct types of eye movements. In addition, eye movements measured in head coordinates can be ambiguous and difficult to characterize without information about scene content and head movement. Inaccurate and/or imprecise classification and quantification in turn compromises the use of eye movement data for VR applications. Advanced identification and classification methods that make use of all available information are therefore an active area of research ( Kothari et al. 2020).
2.2. Eye tracking methods in VR HMDs
Three methods have been used to track eye movements in HMDs: (1) electro-oculography (EOG), (2) scleral search coils, and the most common, (3) video oculography (VOG).
EOG measures the orientation of the eye by placing electrodes on the skin around the eye which measure the resting potential of the eye. The electrodes can be easily incorporated into the HMD where it contacts the face ( Bulling et al. 2009; Shimizu and Chernyshov 2016; Xiao et al. 2019; Kumar and Sharma 2016). The method works because the eye is a dipole that is positively charged toward the cornea and negatively charged toward the retina. The difference in voltage across electrodes placed on opposite sides of the eye (e.g., left and right) map well to eye orientation (e.g., horizontal position of the eye) ( Mowrer et al. 1935). One drawback is that EOG provides a fairly imprecise measure of eye position, but EOG is the only method that allows tracking while the eyes are closed.
The scleral search coil method works by tracking the orientation of a wire loop that is embedded in a contact lens worn by the user ( Robinson 1963). The user’s head is positioned between Helmholtz coils which generate a uniform magnetic field. When the eyes move in the uniform and known magnetic field, electric current is induced in the scleral coils indicating the horizontal, vertical and torsional orientation of the eye. This method is highly accurate and precise with reported spatial resolution of less than 0.1° and temporal resolution that is greater than 1KHz ( Whitmire et al. 2016). However, this method is challenging to implement in HMDs not only because of the need for wired contact lenses but also because the Helmholtz coils must be head-mounted and consequently the resulting magnetic field is much smaller in volume. Nevertheless, an HMD with scleral coil system (see Figure 1) has been developed Whitmire et al. (2016) for use in specialized cases where high-precision tracking in an HMD is required, for example to validate alternative tracking systems.
Fig. 1.
An HTC Vive Pro Eye HR HMD with integrated VOG-based eye trackers (left). A VR HMD with scleral search coil-based eye tracking system (right). Image courtesy of Whitmire et al. (2016)
By far, the most common eye tracking method used in HMDs is VOG ( Duchowski 2007). Currently commercially available HMD-based eye trackers including Tobii, Pupil Labs, Varjo, and Fove employ VOG-based eye tracking. Images of the eyes are captured by cameras mounted in the HMD and analysis of the video frames indicates eye orientation. Most often, analysis methods rely on identification of the pupil and perhaps other landmarks to recover eye orientation. There are many analysis methods for recovering eye position from eye images, and methods implemented by commercially available systems are often proprietary ( Holmqvist et al. 2012). This is an active area of research, but in-depth discussion of eye image analysis methods is beyond the scope of the present paper. Regardless of the method, better quality eye images translate into better quality tracking. Camera position is an important factor limiting image quality. Systems that film the eye front-on through the HMD optics using a hot mirror typically perform better than systems that mount the cameras obliquely thereby bypassing the HMD optics ( Richter et al. 2019). Spatial and temporal resolution of the eye cameras also limit tracking performance. The current standard in commercially available systems varies. For example, the Pupil Labs VR/AR plugin system has 192x192 spatial resolution at 200 Hz, while the Varjo system has 1280x800 spatial resolution at 100Hz.
2.3. Technical aspects of eye movement tracking
2.3.1. Eye tracking data quality
High quality eye tracking data are an important prerequisite for any application that uses eye movement data. Eye tracking data quality refers to the performance of an eye tracker as measured by various metrics including spatial precision, spatial accuracy, sampling rate, and latency.
Spatial Precision is related to the ability of an eye tracker to reproduce measurements accurately over time. Spatial precision of an eye tracker is commonly measured by calculating the sample-to-sample root-mean-square angular displacement (RMS) and the standard deviation of samples collected within a given time window. The amount of precision needed for an eye tracking-based application depends on the type of the eye movement that needs to be recorded and the type of eye tracking task. Applications that use small fixational eye movements, like tremors, drifts and microsaccades require high-precision eye trackers. Andersson et al. (2010) point out that for such tasks the eye tracker should have an RMS precision value that is lower than 0.03°.
Spatial Accuracy is the average angular deviation between the true gaze position and the gaze position estimated by an eye tracker ( Holmqvist et al. 2012). Spatial accuracy of an eye tracker is measured as the distance, in degrees, between the positions of a target point and the average position of a set data samples collected from the eye tracker within a given time window. Lack of accuracy may not pose any problems when the areas of interest in an application are large and are distant from each other. However, in applications with closely spaced stimuli, small inaccuracies could be critical, and could lead to the information obtained from the user or the action executed by the user being different from what the user intended.
Latency refers to the delay between eye movement events and the time these events are received by our system. There are various factors that contribute to data latency in the eye movement tracking pipeline including the sampling rate, the time to compute the gaze position from the eye images, the time to filter and post-process the data and the time to transmit the data to our VR display. Latency could cause elements of the stimulus that are supposed to respond to eye movement to lag behind the eye movements. When this lag is big enough that it is perceptible by the user, it could severely affect the virtual experience. This could be particularly problematic in interactive applications or gaze-contingent displays which change some parts of the stimulus in real time in response to eye movements.
Sampling Rate or sampling frequency of an eye tracker refers to how many times per second the eye is recorded by the eye tracker. Sampling rate determines the type of events that can be measured with your eye trackers and the suitability of the eye tracking data for a particular application. For example, the precise measurement of small amplitude eye movements such as fixational eye movements and low amplitude saccades requires eye trackers with high sampling rates. Andersson et al. (2010) argue that based on the Whittaker–Nyquist–Shannon theorem Shannon (1949), the sampling rate should at least be at least twice the speed of the eye movement to be recorded.
2.3.2. Calibration
Calibration is the process by which the eye tracking system finds a mapping function that maps coordinates reported by the eye tracker to the coordinates of a gaze point in the visual environment ( Harezlak et al. 2014). Most HMD-based eye trackers use a standard point-based calibration procedure. The procedure successively shows a set (usually 5 to 16) of small point targets to the user and the user is asked to fixate on each target for a few seconds. The calibration procedure requires willful engagement and cooperation from the user to be successful, because the mapping function requires the user to fixate on each target for the full duration that the target is shown on the environment. Due to differences in eye attributes and the geometry of the eye tracking setup, most eye tracking use-cases require users to calibrate the eye tracking system before they can use it. However, some use-cases, such as smooth pursuit-based interaction, just employ the user’s relative eye motion in reaction to scene objects; therefore, they do not need to calibrate the eye tracking system Zeng et al. (2020).
[…]
4.1. Technological limitations
4.1.1. Eye tracking data quality
Issues with eye tracker data quality are arguably the biggest technological challenges for eye tracking in VR. Table 1 shows the manufacturer reported data quality specifications for the currently most used HMD-based eye trackers.
Table 1.
Current widely used HMD-based eye trackers and their manufacturer specifications. Note that the manufacturer specifications are calculated under ideal conditions and may be hard to reproduce in realistic settings
| Device | Accuracy | Precision | Sampling rate | Latency |
|---|---|---|---|---|
| Fove-0 | 1.15° | Not reported | 120 Hz | Not reported |
| HTC Vive Pro Eye | 0.5° - 1.1° | Not reported | 120 Hz | Not reported |
| Pico Neo 3 Pro Eye | Sub-degree gaze accuracy | Not reported | 60/90 Hz | Not reported |
| Pupil Labs | 1° | 0.08° | 200 Hz | 8.5 ms Camera latency and 3-4 ms processing latency on i5 CPU |
| Varjo VR-1 | 1° | 0.2° | 100 Hz (default) or 200 Hz | 20-30ms |
| Varjo VR-3 | Sub-degree accuracy | Not reported | 200 Hz | Not reported |
Spatial Precision:
Applications that use small fixational eye movements, like tremors, drifts and microsaccades require high-precision eye trackers. Andersson et al. (2010) point out that for such tasks the eye tracker should have an RMS precision value that is lower than 0.03°. Table 1 shows that not all manufacturers of VR-based eye trackers report the precision value of the eye trackers, and those that have reported precision values seem to have precision values that are worse than the recommended range needed to detect small fixational movements.
Spatial Accuracy:
Table 1 shows that the most popular HMD-based eye trackers report accuracy values between 0.5° and 1.1°. However, manufacturer reported specifications could be misleading, as these metrics are often measured under ideal conditions and do not reflect the accuracy under realistic usage scenarios ( Adhanom et al. 2020a).
Additionally, most current HMD-based eye trackers have the highest accuracy and precision in a small central region of the FOV. Outside this region, accuracy and precision drop substantially. For example, the HTC Vive Pro Eye reports accuracy values of 0.5° - 1.1° within the central 20° of the FOV. Outside this region, the accuracy is not guaranteed. This limits researchers or developers from using the whole FOV for eye tracking-based experiences.
Regardless of the application, performance across all categories would benefit from improvements in the precision and accuracy of eye tracking and also from improvements in identification and labeling of eye movement types. Advances in both of these areas can be achieved by leveraging additional sources of information beyond the eye images that underlie VOG. Simultaneous use of eye data and scene information can help constrain eye position estimates based, for example, on the assumption that users are much more likely to look at an object than the empty space next to an object ( Tafaj et al. 2012). Similarly, simultaneous analysis of both head and eye movement data allows reconstructing gaze in world coordinates, which allows identifying eye movement types that would be difficult to identify based on eye movement alone ( Hausamann et al. 2020; Kothari et al. 2020). As with most current information processing challenges, application of machine learning techniques has shown promise in advancing eye tracking precision, accuracy, as well as identification of eye movement types ( Kothari et al. 2020; Yiu et al. 2019).
Latency:
High latency could cause elements of the stimulus that are supposed to respond to eye movement to lag behind the eye movements. When this lag is big enough that it is perceptible by the user, it could severely affect the virtual experience. This could be particularly problematic in interactive applications or GCDs which change some part of the stimulus in real time in response to eye movements. Albert et al. (2017) points out that foveated rendering is tolerant to latencies of about 50-70 ms. For latencies beyond this range, foveated rendering could be perceptible to the user and lose its effectiveness. Although latencies have greatly decreased in current HMD-based eye trackers, a recent study analyzed the latency for most of the currently available HMD-based eye trackers and reported that the latencies for the mostly used HMD-based eye trackers ranged from 45 ms - 81 ms ( Stein et al. 2021). Although these latencies do not cause significant issues for non-real-time applications, they could hamper real-time use of eye tracking data in VR.
Sampling Rate:
As discussed above, the precise measurement of small amplitude eye movements such as fixational eye movements and low amplitude saccades requires eye trackers with high sampling rates. In Sect. 2.3.1, we have discussed that the sampling rate should be at least twice the speed of the eye movement to be recorded. Table 1 shows that current HMD-based eye trackers have sampling rates that fall in the range 100-200 Hz. These relatively low sampling rates show that most of the commonly used HMD-based eye trackers are not well suited to accurately record low amplitude eye movements.
4.1.2. Calibration
The time consuming and repetitive nature of the calibration procedure could be an obstacle for the wide adoption of eye tracking and could make eye tracking unattractive for applications that require instant use.
Moreover, some users, such as children and users with attentional deficits, have difficulty completing the calibration procedure as they lose interest in the procedure after a few targets have been shown resulting in an unsuccessful calibration procedure ( Blignaut 2017). Alternative eye tracking procedures have been explored to address the issues with the calibration procedure, the most common of which use smooth pursuit eye movements to dynamically calibrate the eye tracker without explicitly asking the user to look at point targets ( Blignaut 2017; Drewes et al. 2019). Although these methods require less time and can be performed without the user being aware of the calibration procedure, they generally produce lower quality eye tracking data. As a result, these calibration methods are not in use in any HMD-based eye tracker we are aware of ( Drewes et al. 2019).
After the calibration procedure, changes in lighting, eye geometry and the relative position of the eye tracking camera with respect to the user’s eyes could cause calibration errors. The latter is the main cause of calibration error in HMD-based eye trackers, as small movements of the VR headset due to the user’s movements could cause calibration errors. All these factors together cause the calibration to decay—the calibration error and spatial accuracy of the eye tracker worsens over time. We call this decay drift. Drift is a common cause of low quality eye tracking with some eye trackers showing calibration drift of about 30% in the first 4 minutes and 30 seconds after calibration ( Ehinger et al. 2019). Calibration errors are hard to deal with due to their dynamic nature. However, the severity of calibration errors could be reduced by making sure the HMD does not move relative to the head after calibration and repeating the calibration procedure multiple times during long sessions. Further research is still needed to develop calibration procedures that are easy, comfortable, and robust to drift.
[…]
5. Conclusion
It is clear based on the many applications mentioned above that eye tracking will soon become an integral part of many, perhaps most, HMD systems. We therefore expect research and development surrounding eye tracking in HMDs to accelerate and expand in the coming years.
Eye tracking in VR has shown great potential to improve rendering efficiency in VR systems and to help enable more comfortable and immersive VR experiences. The most prominent example in this area is gaze-contingent (foveated) rendering. However, more research is needed in this area to understand the perceptual effects of gaze-contingent rendering. More work also needs to be done to reduce the end-to-end latency of eye tracking systems in VR to make gaze-contingent changes in the environments imperceptible to end users.
Moreover, as the user base in VR grows and the need for developing accessible interaction techniques increases, eye tracking has been explored as powerful modality to develop accessible and hands-free interaction techniques for VR systems. Eye tracking could be explored to enable individuals with severe motor impairments to interact with VR systems. There are still challenges with using eye tracking for precise interaction, and more work could be done to enable multimodal or semi-autonomous interaction techniques that solve the challenges with eye tracking-based interaction. Moreover, with more and more sensors being embedded in VR HMDs, sensor fusion (i.e., combining eye tracking with other sensors) is another area for future research that holds promise to further increase the accuracy of eye tracking-based interaction.
VR is used for developing learning and skills training applications as it provides trainees an immersive environment that is safe to fail and allows repetition and correction with minimal cost. Eye tracking in VR has been used for the assessment of learning outcomes and improving the learning process. More research is needed in this area to develop better eye tracking data interpretation methods to measure learning outcomes; to ensure that VR-based training tasks are effective; and to ensure that the skills learned in simulated environments and can be transferred to the real task.
Eye tracking could supplement VR to provide powerful tools that can help advance marketing and CX research in various areas including new product development, advertising, and assessment of consumer behavior and engagement. Previous research has mostly focused at the pre-purchase and purchase stages of the customer journey. However, there is potential for interesting future research on the post-purchase stage that could help companies to analyze consumer behavior and interactions with the purchased product or service.
Eye tracking has been used in the clinical context for diagnostic, therapeutic and interactive uses. However, there is still need for further research to develop uniform and more usable clinical applications that are easy to comprehend and use for the patient as well as the clinician practitioner. This has the potential to allow patients to self-diagnose or self-treat neurological disease and provide clinicians with easy to use tools. There is also need for the development of VR and eye tracking hardware.
Additionally, eye tracking data can be used to easily identify and authenticate users in an implicit way without interrupting the user’s main task. On the other hand, this makes eye tracking data vulnerable and sensitive to data privacy and security challenges. Therefore, more work needs to be done to develop tools and standards that preserve users’ privacy and strengthen security of eye tracking systems.
In summary, eye tracking, i.e., the ability to precisely and accurately measure the user’s eye gaze, has potential to become a standard feature on consumer VR headsets. The availability of high-precision, low-latency and low-cost eye trackers in VR HMDs has led to the emergence of a host of new applications spanning multiple disciplines. This paper provides a broad review of these applications and highlights some areas for future research.