Direct retinal projector

Sep 13, 2021

- Apple

A direct retinal projector may include a gaze tracking system that tracks position of a subject’s pupil and automatically adjusts projection of a scanned light field so that the light field enters the pupil. A control loop adjusts a scanning mirror to substantially center an IR beam on a position sensing detector (PSD). In so doing, the scanning mirror is correctly positioned so that the scanned light field from the projector enters the subject’s pupil. In addition, a direct retinal projector may include an adjustable focusing element that adjusts focus of a combined light beam generated by a projector as the light beam is scanned to an ellipsoid mirror that reflects the light beam to the subject’s pupil. The focusing of the scanned beam may be adjusted as the beam is scanned across the azimuth angle of the curved ellipsoid mirror.

Latest Apple Patents:

• BILLBOARD FOR CONTEXT INFORMATION SHARING
• TECHNOLOGIES FOR DYNAMIC CONTROL OF PROTOCOL DATA UNIT SET DISCARDING
• TECHNOLOGIES FOR QUALITY OF COMPUTING SERVICES FOR SUBNETWORKS
• TRANSMISSION CONFIGURATION INDICATOR SWITCHING FOR NON-TERRESTRIAL NETWORK
• TIMING ADVANCE MANAGEMENT FOR LAYER 1 OR LAYER 2-TRIGGERED MOBILITY

Skip to:Description · Claims · References Cited · Patent History · Patent History

Description

PRIORITY INFORMATION

This application is a continuation of U.S. patent application Ser. No. 15/413,310, filed Jan. 23, 2017, which claims benefit of priority of U.S. Provisional Application Ser. No. 62/299,137, filed Feb. 24, 2016, the content of which are incorporated by reference herein in their entirety.

BACKGROUND

Virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene. When the user views images through a virtual reality system, the user may thus feel as if they are moving within the scenes from a first-person point of view. Similarly, augmented reality (AR) combines computer generated information with real world images to augment, or add content to, a user’s view of the world. The simulated environments of virtual reality and/or the enhanced content of augmented reality may thus be utilized to provide an interactive user experience for multiple applications, such as interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the internet, or the like.

However, conventional virtual reality and augmented reality systems may suffer from accommodation-convergence mismatch problems that cause eyestrain, headaches, and/or nausea. Accommodation-convergence mismatch arises when a VR or AR system effectively confuses the brain of a user by generating scene content that does not match the depth expected by the brain based on the stereo convergence of the two eyes of the user. For example, in a stereoscopic system the images displayed to the user may trick the eye(s) into focusing at a far distance while an image is physically being displayed at a closer distance. In other words, the eyes may be attempting to focus on a different image plane or focal depth compared to the focal depth of the projected image, thereby leading to eyestrain and/or increasing mental stress. Accommodation-convergence mismatch problems are undesirable and may distract users or otherwise detract from their enjoyment and endurance levels (i.e. tolerance) of virtual reality or augmented reality environments.

SUMMARY

Various embodiments of methods and apparatus for providing virtual reality (VR) or augmented reality (AR) systems are described. Embodiments of a direct retinal projector are described that may, for example, resolve the convergence-accommodation conflict in head-mounted AR and VR systems. A VR or AR headset system is described that may include or implement different techniques and components of the direct retinal projector.

Embodiments of a gaze tracking component or system are described that may be used in a direct retinal projector system to track position of a subject’s pupil and automatically adjust projection of a scanned light field generated by a projector component of the system so that the scanned light field from the projector enters the subject’s pupil. In some embodiments, a control loop adjusts a two-dimensional (2D) scanning mirror to substantially center an IR beam on a position sensing detector (PSD). In so doing, the 2D scanning mirror is correctly positioned so that the scanned light field from the projector enters the subject’s pupil.

In addition, embodiments of a beam focusing system for a direct retinal projector system are described. In some embodiments, the beam focusing system may include an adjustable focusing element to provide adaptive optical functionality for the projector. In some embodiments, the adjustable focusing element may be located on the path of a combined light beam generated by light sources of the projector component of the direct retinal projector system and a scanning mirror of the projector component. In some embodiments, the direct retinal projector system adjusts focus of a combined light beam generated by the light sources via the adjustable focusing lens as the light beam is scanned to a curved ellipsoid mirror of the direct retinal projector system that reflects the scanned light beam to the subject’s pupil. Since the curved ellipsoid mirror has optical power, the focusing of the scanned beam may be adjusted as the beam is scanned across the azimuth angle of the curved ellipsoid mirror.

In various embodiments of a direct retinal projector system, either the gaze tracking component or the adjustable focusing element may be used, or both may be used in combination.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an example of different types of eye focus.

FIG. 2 illustrates one embodiment of a conventional near-eye virtual reality system.

FIG. 3 illustrates an example of parallel light beams entering an eye.

FIG. 4 A illustrates a curved, substantially ellipsoid mirror, according to some embodiments.

FIG. 4 B illustrates light from a curved ellipsoid mirror of a direct retinal projector striking the pupil at different positions, according to some embodiments.

FIG. 4 C illustrates elevation and azimuth scans to a curved ellipsoid mirror, according to some embodiments.

FIGS. 5 A and 5 B illustrate principle of operation of a gaze tracking system, according to some embodiments.

FIG. 6 illustrates a direct retinal projector that includes a gaze tracking system, according to some embodiments.

FIGS. 7 A and 7 B provide a flowchart of a method of operation for a gaze tracking system in a direct retinal projector as shown in FIG. 6, according to some embodiments.

FIGS. 8 A through 8 C illustrate focusing at different points relative to an azimuth, according to some embodiments.

FIG. 9 illustrates a direct retinal projector that includes an adjustable focusing lens, according to some embodiments.

FIG. 10 is a flowchart of a method of operation for an adjustable focusing lens in a direct retinal projector, according to some embodiments.

FIGS. 11 A and 11 B illustrate an adjustable focusing lens, according to some embodiments.

FIG. 12 is logical block diagram of a virtual reality (VR) and/or augmented reality (AR) device, according to some embodiments.

FIG. 13 is a logical block diagram of a raster scan generated using an array of MEMS mirrors, according to some embodiments.

FIG. 14 is a logical block diagram of multiple fields of view, according to some embodiments.

FIG. 15 is a logical block diagram of a configuration of a light emitting device, according to some embodiments.

FIG. 16 is a logical block diagram of light source focusing and/or collimating lenses, according to some embodiments.

FIG. 17 is a logical block diagram of a frame for a VR/AR device, according to some embodiments.

FIG. 18 is a logical block diagram of a device that provides augmented reality (AR) to a subject, according to some embodiments.

FIG. 19 is a high-level flowchart illustrating a method of operation for a virtual reality device, according to some embodiments.

FIGS. 20 A and 20 B illustrate a dynamically adjustable MEMS mirror that may be used in a VR/AR device, according to some embodiments.

This specification includes references to “one embodiment” or “an embodiment.” The appearances of the phrases “in one embodiment” or “in an embodiment” do not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

“Comprising.” This term is open-ended. As used in the claims, this term does not foreclose additional structure or steps. Consider a claim that recites: “An apparatus comprising one or more processor units … . ” Such a claim does not foreclose the apparatus from including additional components (e.g., a network interface unit, graphics circuitry, etc.).

“Configured To.” Various units, circuits, or other components may be described or claimed as “configured to” perform a task or tasks. In such contexts, “configured to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs those task or tasks during operation. As such, the unit/circuit/component can be said to be configured to perform the task even when the specified unit/circuit/component is not currently operational (e.g., is not on). The units/circuits/components used with the “configured to” language include hardware—for example, circuits, memory storing program instructions executable to implement the operation, etc. Reciting that a unit/circuit/component is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112, paragraph (f), for that unit/circuit/component. Additionally, “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue. “Configure to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.

“First,” “Second,” etc. As used herein, these terms are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations for “first” and “second” values. The terms “first” and “second” do not necessarily imply that the first value must be written before the second value.

“Based On” or “Dependent On.” As used herein, these terms are used to describe one or more factors that affect a determination. These terms do not foreclose additional factors that may affect a determination. That is, a determination may be solely based on those factors or based, at least in part, on those factors. Consider the phrase “determine A based on B.” While in this case, B is a factor that affects the determination of A, such a phrase does not foreclose the determination of A from also being based on C. In other instances, A may be determined based solely on B.

“Or.” When used in the claims, the term “or” is used as an inclusive or and not as an exclusive or. For example, the phrase “at least one of x, y, or z” means any one of x, y, and z, as well as any combination thereof.

DETAILED DESCRIPTION

Various embodiments of methods and apparatus for providing virtual reality (VR) or augmented reality (AR) systems are described. Embodiments of a direct retinal projector are described that may, for example, resolve the convergence-accommodation conflict in head-mounted AR and VR systems. A VR or AR headset system is described that may include or implement different techniques and components of the direct retinal projector.

Embodiments of a gaze tracking component or system are described that may be used in a direct retinal projector system as described herein to track position of a subject’s pupil and automatically adjust projection of a scanned light field generated by a projector component of the system so that the scanned light field from the projector enters the subject’s pupil. In some embodiments of a gaze tracking component, a light source (e.g., an infrared (IR) LED) projects a beam of IR light. One or more beam splitters help ensure that the IR beam is roughly q to the center of the scanned light field generated by the projector component of the direct retinal projector system. Both the projected light and the IR beam are reflected off a 2D scanning mirror and the curved ellipsoid mirror before reaching the subject’s eye. A portion of the IR beam entering the pupil of the eye reflects off the retina and emerges from the pupil again (forming a “bright pupil”). The returning IR beam reflects off the curved ellipsoid mirror, the 2D scanning mirror, and the beam splitters to reach a position sensing detector (PSD), for example a quadrant cell technology PSD (also referred to as a quad cell PSD). A control loop adjusts the 2D scanning mirror to substantially center the returning IR beam on the PSD. In so doing, the 2D scanning mirror is correctly positioned so that the scanned light field from the projector enters the subject’s pupil. FIGS. 5 A through 7 B further illustrate components and methods of operation of a gaze tracking component in a direct retinal projector system, according to at least some embodiments.

In addition, embodiments of an adjustable focusing element (e.g., an adjustable lens), also referred to as an optical actuator component, and focusing method for the projector component of the direct retinal projector system are described. In some embodiments, the adjustable focusing element may be an optical microelectromechanical system (MEMS) configured to dynamically change the shape of a flexible optical element to provide adaptive optical functionality for the projector. In some embodiments, the adjustable focusing element may be located on the path of a combined light beam generated by multiple light sources (e.g., lasers) of the projector component of the direct retinal projector system and a scanning mirror component (e.g., a MEMS mirror module) of the projector component. In some embodiments, a controller component of the direct retinal projector system adjusts focus via the adjustable focusing lens as the light beam generated by the projector component is scanned across the azimuth angle of a curved ellipsoid mirror of the direct retinal projector system that reflects the scanned light beam to the subject’s pupil. Since the curved ellipsoid mirror has optical power, the focusing of the scanned beam is adjusted as the beam is scanned across the azimuth angle of the curved ellipsoid mirror. In some embodiments, beam focusing is not changed with the elevation angle during the scan as the direct retinal projector system is rotationally symmetric. In some embodiments, the beam focus may follow a scanning cycle at the projector frame rate (e.g., 60 Hz, 90 Hz, etc.). In some embodiments, the beam generated by the projector and reflected off the curved ellipsoid mirror is not completely collimated when entering the subject’s eye, but is focused to the system hyperfocal distance. In some embodiments, depending on system parameters such as beam diameter and resolution which effect the depth of field, the adjustable focusing lens may provide control over the beam focus to enable focusing at object distances closer than half the hyperfocal distance. FIGS. 8 A through 11 B further illustrate components and methods of operation of an adjustable focusing element in a direct retinal projector system, according to at least some embodiments.

In various embodiments of a direct retinal projector system, either the gaze tracking component or the adjustable focusing element may be used, or both may be used in combination.

FIGS. 12 through 20 B and the section titled Example virtual reality device describe embodiments of a virtual reality headset that provide direct retinal projection and that may implement or incorporate embodiments of the scan tracking system, adjustable focus element, and various other methods and apparatus for direct retinal projector systems as described herein. However, note that embodiments of the scan tracking system and adjustable focus element for a projector in a scanning system may be implemented in various other direct retinal projector systems, in other AR or VR technology systems, or in other types of scanning projection systems.

Accommodation and Convergence in AR/VR Systems

The human brain typically uses two cues to gauge distance: accommodation (i.e., eye focus) and eye convergence (i.e., the stereoscopic perspective difference between the two eyes). Conventional near-eye VR systems typically use separate miniature screens for each respective eye to project the images intended for the left eye and the right eye, as well as optics to allow a user to comfortably focus the eyes at a far distance during viewing of the left eye and right eye images. Conventional near-eye VR systems thus produce conflicting visual cues since the resulting three-dimensional (3D) image produced by the brain effectively appears at a convergence distance that is closer than the accommodation distance that each eye focuses on separately, thereby leading to the possibility of headache and/or nausea over time. Heavy users of conventional VR systems may potentially train themselves to compensate for accommodation-convergence mismatch, but a majority of users might not.

AR systems typically add information and graphics to an existing scene being viewed by a user. In some embodiments, AR may be a powerful experience, since the user can see both the projected images and/or sprites (i.e., the augmented world) as well as the surrounding scene (i.e., the real world) directly through the AR system rather than using camera systems to project a version of the surrounding scene less accurately onto screen displays for each eye.

FIG. 1 depicts an example of different types of eye focus. In system 100 of FIG. 1, an eye 110 A may be selectively configured to focus at a far distance, as shown by the incident light originating from a distant location and focusing onto the retina (i.e., the back internal surface) of eye 110 A by the internal lens of eye 110 A. In another embodiment, eye 110 A may instead be selectively configured for a close focus scenario, as shown by light from a nearby location being incident upon the eye and focusing onto the retina.

FIG. 2 illustrates one embodiment of a conventional near-eye VR system 200. As depicted, right eye 210 and left eye 220 are focused on a focal plane 230 where an image for right eye 240 and an image for left eye 250, respectively, are displayed. As right eye 210 and left eye 220 focus on their respective images on focal plane 230, the brain of the user combines the images into a resulting 3D image 260. In one embodiment, the accommodation distance may be the distance between focal plane 230 and an eye of the user (e.g., right eye 210 and/or left eye 220), and the convergence distance may be the distance between resulting 3D image 260 and an eye of the user. Since, as depicted in FIG. 2, the accommodation distance differs from the convergence distance, conventional near-eye VR system 200 therefore results in an accommodation-convergence mismatch and may cause discomfort for the user as described above.

FIG. 3 illustrates an example of parallel light beams entering an eye 300. As shown, various sets of parallel light beams that enter eye 300 are focused by eye 300 such that the parallel beams within a respective set land at the same place on the retina of eye 300.

Direct Retinal Projector System

In embodiments of a direct retinal projector system for AR and/or VR applications as described herein, a light beam is generated by a scanning projector, reflected off a curved mirror (e.g., a curved ellipsoid mirror) in front of the subject’s eye and through the subject’s pupil, and forms an image on the subject’s retina—there is no intermediate image on a screen or surface that the subject views. In some embodiments, with relatively small diameter laser beams, the effective depth of focus of the eye can be greatly increased. The direct retinal projector system may at least partially eliminate eye lens accommodation from the retinal projection focus to help eliminate the accommodation convergence mismatch. In some embodiments, the direct retinal projector system may help compensate for user eye lens problems, such as short- or long-sightedness.

FIG. 4 A illustrates a side view and an inner surface view of a curved ellipsoid mirror 2008, according to some embodiments. The curved ellipsoid mirror 2008 reflects and focuses the light field from the scanning projector into the subject’s eye pupil, thus simplifying the optics and reducing the scanning degrees of freedom required when compared to conventional systems. In some embodiments, the curved ellipsoid mirror 2008 may be “see through”, i.e. allowing at least some light from the subject’s environment to pass through, thus enabling a much more natural augmented reality (AR) experience. For example, AR content projected by the direct retinal projector system may be “overlaid” on or viewed in the environmental scene that the subject is viewing.

FIG. 6 illustrates a direct retinal projector system 2000, according to some embodiments. The direct retinal projector system 2000 of FIG. 6 may, for example, be used in a virtual reality headset as shown in FIGS. 12 through 20 B. Note that the size, shape, and arrangement of the components within the direct retinal projector system 2000 are provided by way of example and are not intended to be limiting. Also note that there may be more or fewer components in direct retinal projector system 2000 than shown. Further note that the direct retinal projector system 2000 is shown for only one eye; generally but not necessarily, there will be a second direct retinal projector system 2000 for the second eye

The projector 2004 scans an image (e.g., an RGB image) (also referred to as a light field 10) into the pupil 2092 of the subject’s eye 2090. Light from the projector 2004 is reflected off a 2D scanning mirror 2006 and then the curved ellipsoid mirror 2008 before entering the pupil 2092. In at least some embodiments, the 2D scanning mirror 2006 is not used directly to generate the light field. In some embodiments, the 2D scanning mirror 2006 is part of the gaze tracking system that also includes the IR light source 2010 and the position sensing detector (PSD) 2014. In some embodiments, the angle of the 2D scanning mirror 2006 may be adjusted based on the position of the pupil 2092 of the subject’s eye 2090 so that the light field 10 enters the pupil 2092.

FIG. 4 B illustrates light (field rays) from a curved ellipsoid mirror 2008 of a direct retinal projector system 2000 striking the pupil 2092 at different positions, according to some embodiments. In some embodiments, the curved ellipsoid mirror only focuses the light field to a point at one pupil position. At other positions, it focuses to a region. As long as the light enters the pupil, it does not matter where it enters. In some embodiments, the curved ellipsoid mirror 2008 may be modified from the mathematical ellipsoid shape so as to even up the focus region sizes for different pupil positions.

FIG. 4 C illustrates elevation and azimuth scans to a curved ellipsoid mirror, according to some embodiments. In some embodiments, the scanning projector of the direct retinal projector system may be configured to scan pixels from a source VR or AR image or frame to the curved ellipsoid mirror 2008 in a pattern in which the pixels are scanned on the elevation (fast) axis (each elevation scan corresponding to a column of the source image), with the elevation scans proceeding across the curved ellipsoid mirror 2008 across the azimuth (referred to as the azimuth, or slow, scan). Note that the direction of the arrows in FIG. 4 C are given by way of example, and are not intended to be limiting. VR or AR images or frames may be scanned at a frame rate, e.g. 60 or 90 Hz.

The following describes components and operations of a direct retinal projector system 2000 that includes a gaze tracking system, according to some embodiments. Reference is made to FIGS. 6 and 8. In some embodiments, a direct retinal projector system 2000 may include a projector 2004 that generates a combined light beam (e.g., an RGB laser beam) that is scanned using a first 2D scanning mirror (e.g., mirror 3042 of MEMS mirror module 3040 in FIG. 9). In some embodiments, an adjustable focusing element ( 3020 in FIG. 9) may be located on the path of the combined light beam generated by the light sources 3010 (e.g., lasers) of the projector 2004 and MEMS mirror module 3040. In some embodiments, the direct retinal projector system adjusts focus of the light beam via the adjustable focusing lens 3020 as the light beam is scanned across the azimuth angle of the curved ellipsoid mirror 2008 by the MEMS mirror module 3040.

The scanned combined beam (10 in FIG. 6) is reflected off a second 2D scanning mirror 2006 that is used as part of the gaze tracking system to adjust the location that the scanned light field 10 is projected to (which should be in the location of the eye pupil 2092). In some embodiments, the gaze tracking system uses a sensor (position sensing detector (PSD)) that senses the position of an IR beam 20- 30 generated by an IR light source 2010, such as a quad cell or other PSD technology sensor, shown as PSD 2014 in FIG. 6. The IR beam is generated by an IR light source or emitter 2010, such as an IR LED. In some embodiments, beam splitters 2012 A and 2012 B to align the IR beam 20 with the center of the scanned light field 10 from the projector 2004 before it reflects of the second 2D scanning mirror 2006.

Once reflected off the second 2D scanning mirror 2006, useful light (visible light in the case of the projected light field 10 and IR light in the case of the gaze tracking system IR beam 20) is reflected off a curved, substantially ellipsoid mirror 2008 located in front of the subject’s eye 2090 (with the concave side of the mirror 2008 facing the eye 2090), for example mounted on a frame to which the projector 2004 and gaze tracking apparatus are also attached. An example frame for a virtual reality headset is shown in FIG. 17. In some embodiments, the mirror 2008 may not be exactly ellipsoid so as to optimize the light field focusing over a range of pupil positions, but may be generally close to ellipsoid. The light 10 and 20 is reflected off the mirror 2008 towards the pupil 2092 of the subject’s eye 2090.

In the gaze tracking system, at least some of the IR light 20 reflected off the mirror 2008 enters the pupil 2090 and is reflected off the retina back through the pupil 2090 (shown as returning IR beam 30), to create a bright pupil image. The returning IR light beam 30 is reflected back off the curved ellipsoid mirror 2008 and second 2D scanning mirror 2006, and then the returning IR light beam 30 is directed onto the PSD 2014 by one or more beam splitters 2012 B. In some embodiments, a control loop (e.g., executed by a controller 2002) is used to alter the angle of the second 2D scanning mirror 2006 according to PSD 2014 sensor data so that the reflected IR beam 30 is substantially centered on the PSD 2014. The angle of the second 2D scanning mirror 2006 may thus be corrected to a position so that the scanned light field 10 from the projector enters the subject’s pupil 2092.

TABLE 1 provides parameters for at least some features of example embodiments of a direct retina projection system as described herein. Note that these parameters are provided as examples, and are not intended to be limiting.

TABLE 1FirstSecondexampleexampleFEATURESembodimentembodimentCommentsLaser beam diameter 1 mm3 mmBetter resolutionat projectorwith bigger beamResolution pixels/deg.20>30 Depends on beamquality, diffract

(Excerpt: record head (abstract, description, patent history); full page at source URL.)