Full documents Title Abstract Claims All Any Exact NotAdd AND condition
These CPCs and their children
These exact CPCs Add AND condition
Exact Exact Batch Similar Substructure Substructure (SMARTS) Full documents Claims onlyAdd AND conditionAdd AND condition
Application Numbers Publication Numbers EitherAdd AND condition
Display device
Abstract translated from
An augmented reality headset may include a reflective holographic combiner to direct light from a light engine into a user’s eye while also transmitting light from the environment. The combiner and engine may be arranged to project light fields with different fields of view and resolution to match the visual acuity of the eye. The combiner may be recorded with a series of point to point holograms; one projection point interacts with multiple holograms to project light onto multiple eye box points. The engine may include a laser diode array, a distribution waveguide, scanning mirrors, and layered waveguides that perform pupil expansion and that emit wide beams of light through foveal projection points and narrower beams of light through peripheral projection points. The light engine may include focusing elements to focus the beams such that, once reflected by the holographic combiner, the light is substantially collimated.
Images (31)
Classifications machine-classified cpc-machine-classified fterm-machine-classified fterm-family-classified
The classifications are assigned by a computer and are not a legal conclusion.
Google has not performed a legal analysis and makes no representation as to the accuracy of the classifications listed.
The CPC classifications are assigned by a computer and are not a legal conclusion.
Google has not performed a legal analysis and makes no representation as to the accuracy of the classifications listed.
The F-term classifications are assigned based on a patent family member containing these classification codes.
The F-term classifications are assigned by a computer and are not a legal conclusion.
Google has not performed a legal analysis and makes no representation as to the accuracy of the classifications listed.
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/01Head-up displays
G02B27/017Head mounted
G02B27/0172Head mounted characterised by optical features
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/0093Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/01Head-up displays
G02B27/017Head mounted
G02B27/0176Head mounted characterised by mechanical features
GPHYSICS
G06COMPUTING OR CALCULATING; COUNTING
G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
G06T19/00Manipulating three-dimensional [3D] models or images for computer graphics
G06T19/006Mixed reality
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/01Head-up displays
G02B27/0101Head-up displays characterised by optical features
G02B27/0103Head-up displays characterised by optical features comprising holographic elements
G02B2027/0105Holograms with particular structures
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/01Head-up displays
G02B27/0101Head-up displays characterised by optical features
G02B27/0103Head-up displays characterised by optical features comprising holographic elements
G02B2027/0105Holograms with particular structures
G02B2027/0107Holograms with particular structures with optical power
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/01Head-up displays
G02B27/0101Head-up displays characterised by optical features
G02B2027/0123Head-up displays characterised by optical features comprising devices increasing the field of view
G02B2027/0125Field-of-view increase by wavefront division
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/01Head-up displays
G02B27/017Head mounted
G02B27/0172Head mounted characterised by optical features
G02B2027/0174Head mounted characterised by optical features holographic
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/01Head-up displays
G02B27/017Head mounted
G02B2027/0178Eyeglass type
GPHYSICS
G02OPTICS
G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
G02B27/00Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
G02B27/01Head-up displays
G02B27/0179Display position adjusting means not related to the information to be displayed
G02B2027/0187Display position adjusting means not related to the information to be displayed slaved to motion of at least a part of the body of the user, e.g. head, eye
View 9 more classifications
Hide more classifications
Landscapes
Physics & MathematicsGeneral Physics & Mathematics
Show more
Show less
InventorRichard J. ToplissPaul J. Gelsinger-AustinThomas M. GregoryRichard H. TsaiAlexander ShpuntCurrent Assignee
The listed assignees may be inaccurate.
Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.
Apple Inc
2017WO
Application number: PCT/US2017/052573
Filing date: 2017-09-20
Legal status: Ceased
2019US
Application number: US16/359,924
Filing date: 2019-03-20
Legal status: Active
2021US
Application number: US17/566,477
Filing date: 2021-12-30
Legal status: Active
2019-03-20
Application filed by Apple Inc
2019-03-20
2019-09-19
Publication of US20190285897A1
2021-12-30
2022-01-04
Application granted
2022-01-04
Status
Active
2037-09-19
Anticipated expiration
InfoPatent citations (42)Cited by (70)Legal eventsSimilar documentsPriority and Related ApplicationsExternal linksUSPTOUSPTO PatentCenterUSPTO AssignmentEspacenetGlobal DossierDiscuss
Description translated from
- [0001]
This application is a continuation of International Application No. PCT/US2017/052573, filed Sep. 20, 2017, which claims benefit of priority to U.S. application Ser. No. 15/709,398, filed Sep. 19, 2017, which is abandon, which claims benefit of priority to U.S. Provisional Application No. 62/397,312, filed Sep. 20, 2016. The above applications are incorporated herein by reference. To the extent that any material in the incorporated application conflicts with material expressly set forth herein, the material expressly set forth herein controls.
BACKGROUND- [0002]
Virtual reality (VR) allows users to experience and/or interact with an immersive artificial environment, such that the user feels as if they were physically in that environment. For example, virtual reality systems may display stereoscopic scenes to users in order to create an illusion of depth, and a computer may adjust the scene content in real-time to provide the illusion of the user moving within the scene. When the user views images through a virtual reality system, the user may thus feel as if they are moving within the scenes from a first-person point of view. Similarly, augmented reality (AR) and mixed reality (MR) combine computer generated information with views of the real world to augment, or add content to, a user’s view of their environment. The simulated environments of VR and/or the enhanced content of AR/MR may thus be utilized to provide an interactive user experience for multiple applications, such as interacting with virtual training environments, gaming, remotely controlling drones or other mechanical systems, viewing digital media content, interacting with the internet, or the like.
- [0003]
However, conventional VR, AR, and MR systems may suffer from accommodation-convergence mismatch problems that cause eyestrain, headaches, and/or nausea.
- [0004]
Accommodation-convergence mismatch arises when a VR or AR system effectively confuses the brain of a user by generating scene content that does not match the depth expected by the brain based on the stereo convergence of the two eyes of the user. For example, in a stereoscopic system the images displayed to the user may trick the eye(s) into focusing at a far distance while an image is physically being displayed at a closer distance. In other words, the eyes may be attempting to focus on a different image plane or focal depth compared to the focal depth of the projected image, thereby leading to eyestrain and/or increasing mental stress. Accommodation-convergence mismatch problems are undesirable and may distract users or otherwise detract from their enjoyment and endurance levels (i.e. tolerance) of virtual reality or augmented reality environments.
SUMMARY- [0005]
Various embodiments of an augmented reality (AR), and/or mixed reality (MR) direct retinal projector system that may include an AR headset (e.g., a helmet, goggles, or glasses) that uses a reflective holographic combiner to direct light from a light engine into the user’s eye, while also transmitting light from the user’s environment to thus provide an augmented view of reality. The holographic combiner may be recorded with a series of point to point holograms; one projection point interacts with multiple holograms to project light onto multiple eye box points. The holograms may be arranged so that neighboring eye box points are illuminated from different projection points. The holographic combiner and light engine may be arranged to separately project light fields with different fields of view and resolution that optimize performance, system complexity and efficiency, so as to match the visual acuity of the eye. The light engine may implement foveal projectors that generally project wider diameter beams over a smaller central field of view, and peripheral projectors that generally project smaller diameter beams over a wider field of view.
- [0006]
The light engine may include multiple independent light sources (e.g., laser diodes, LEDs, etc.) that can independently project from the different projection points, with a proportion being foveal projectors and a proportion being peripheral projectors. In some embodiments, the light engine may include two or more two-axis scanning mirrors to scan the light sources; the light sources are appropriately modulated to generate the desired image. The light engine may include a series of optical waveguides with holographic or diffractive gratings that move the light from the light sources to generate beams at the appropriate angles and positions to illuminate the scanning mirrors; the light is then directed into additional optical waveguides with holographic film layers recorded with diffraction gratings to expand the projector aperture and to maneuver the light to the projection positions required by the holographic combiner.
- [0007]
In some embodiments, the light engine may include at least one focusing element (e.g., optical lens, holographic lens, etc.) for each projector to focus emitted light beams such that, once reflected off the holographic combiner, the light is substantially collimated when it enters the subject’s eye. The required focal surface may be complicated by the astigmatism of the holographic combiner, but is a curved surface in front of the combiner. The ideal focal surface is different for different eye box positions, and errors may lead to less collimated output. However, in some embodiments, this can be compensated by reducing the beam diameter for different angles where the errors between the ideal focal surface and the actual best fit focal surface are greatest, which alleviates the problem by increasing the F-number and hence the depth of focus of the beam.
- [0008]
In some embodiments, active beam focusing elements may be provided for each projection point. This may reduce or eliminate the need to change beam diameter with angle. This may also enable beams that diverge into the eye to, rather than being collimated, match the beam divergence of the supposed depth of the virtual object(s) being projected by the light engine.
- [0009]
The AR system may not require extra moving parts or mechanically active elements to compensate for the eye changing position in the eye box or for the changing optical power from the holographic combiner during the scan, which simplifies the system architecture when compared to other direct retinal projector systems. Further, the holographic combiner may be implemented by a relatively flat lens when compared to curved reflective mirrors used in other direct retinal projector systems.
BRIEF DESCRIPTION OF THE DRAWINGS[0010]
FIG. 1 is an example of different types of eye focus.
[0011]
FIG. 2 illustrates one embodiment of a conventional near-eye virtual reality system.
[0012]
FIG. 3 illustrates an example of parallel light beams entering an eye.
[0013]
FIG. 4 illustrates a direct retinal projector system that uses a curved ellipsoid mirror to direct light from a projector into a subject’s eye, while also transmitting light from the environment to the subject’s eye.
[0014]
FIG. 5 illustrates an augmented reality (AR) system that uses a reflective holographic combiner to direct light from a light engine into a subject’s eye, while also transmitting light from the environment to the subject’s eye, according to some embodiments.
[0015]
FIG. 6 illustrates an AR headset that includes a reflective holographic combiner to direct light from a light engine into a subject’s eye, while also transmitting light from the environment to the subject’s eye, according to some embodiments.
[0016]
FIG. 7 illustrates high-level components of an AR system, according to some embodiments.
[0017]
FIG. 8 illustrates foveal and peripheral projectors of a light engine in an AR headset, according to some embodiments.
[0018]
FIG. 9 illustrates light beams from foveal projectors in an AR system, according to some embodiments.
[0019]
FIG. 10 illustrates light beams from peripheral projectors in an AR system, according to some embodiments.
[0020]
FIG. 11 illustrates foveal and peripheral eye boxes for an AR system, according to some embodiments.
[0021]
FIGS. 12A through 12C illustrate a laser array for an AR system, according to some embodiments.
[0022]
FIGS. 13A through 13C illustrate collimating lenses for a laser array in an AR system, according to some embodiments.
[0023]
FIG. 14 illustrates a laser array projector, according to some embodiments.
[0024]
FIG. 15 illustrates a laser array projector and waveguide with holograms, according to some embodiments.
[0025]
FIG. 16 illustrates a 2D scanning microelectromechanical systems (MEMS) mirror, according to some embodiments.
[0026]
FIG. 17 illustrates foveal waveguides, according to some embodiments.
[0027]
FIG. 18 illustrates beam angles in cosine space for foveal waveguides, according to some embodiments.
[0028]
FIG. 19 illustrates peripheral waveguides, according to some embodiments.
[0029]
FIG. 20 illustrates beam angles in cosine space for peripheral waveguides, according to some embodiments.
[0030]
FIG. 21 further illustrates peripheral waveguides, according to some embodiments.
[0031]
FIGS. 22A through 22C are graphs illustrating angular selectivity for a holographic combiner, according to some embodiments.
[0032]
FIG. 23 illustrates foveal projections for a holographic combiner, according to some embodiments.
[0033]
FIG. 24 illustrates peripheral projections for a holographic combiner, according to some embodiments.
[0034]
FIG. 25 illustrates a best fit focus curve and a focusing element for peripheral projections in an AR system, according to some embodiments.
[0035]
FIG. 26 is a graph of peripheral projector resolution vs. pupil angle in an AR system, according to some embodiments.
[0036]
FIG. 27 illustrates a best fit focus curve and a focusing element for foveal projections in an AR system, according to some embodiments.
[0037]
FIG. 28 illustrates projector scan angle for foveal projections, according to some embodiments.
[0038]
FIG. 29A is a graph of foveal projector resolution vs. pupil angle in an AR system, according to some embodiments.
[0039]
FIG. 29B is a graph of beam diameter for foveal projections in an AR system, according to some embodiments.
[0040]
FIG. 30 is a high-level flowchart of a method of operation for an AR system as illustrated in FIGS. 5 through 29B, according to some embodiments.
- [0041]
This specification includes references to “one embodiment” or “an embodiment.” The appearances of the phrases “in one embodiment” or “in an embodiment” do not necessarily refer to the same embodiment. Particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.
- [0042]
“Comprising.” This term is open-ended. As used in the claims, this term does not foreclose additional structure or steps. Consider a claim that recites: “An apparatus comprising one or more processor units … .” Such a claim does not foreclose the apparatus from including additional components (e.g., a network interface unit, graphics circuitry, etc.).
- [0043]
“Configured To.” Various units, circuits, or other components may be described or claimed as “configured to” perform a task or tasks. In such contexts, “configured to” is used to connote structure by indicating that the units/circuits/components include structure (e.g., circuitry) that performs those task or tasks during operation. As such, the unit/circuit/component can be said to be configured to perform the task even when the specified unit/circuit/component is not currently operational (e.g., is not on). The units/circuits/components used with the “configured to” language include hardware—for example, circuits, memory storing program instructions executable to implement the operation, etc. Reciting that a unit/circuit/component is “configured to” perform one or more tasks is expressly intended not to invoke 35 U.S.C. § 112, paragraph (f), for that unit/circuit/component. Additionally, “configured to” can include generic structure (e.g., generic circuitry) that is manipulated by software or firmware (e.g., an FPGA or a general-purpose processor executing software) to operate in manner that is capable of performing the task(s) at issue. “Configure to” may also include adapting a manufacturing process (e.g., a semiconductor fabrication facility) to fabricate devices (e.g., integrated circuits) that are adapted to implement or perform one or more tasks.
- [0044]
“First,” “Second,” etc. As used herein, these terms are used as labels for nouns that they precede, and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations for “first” and “second” values. The terms “first” and “second” do not necessarily imply that the first value must be written before the second value.
- [0045]
“Based On” or “Dependent On.” As used herein, these terms are used to describe one or more factors that affect a determination. These terms do not foreclose additional factors that may affect a determination. That is, a determination may be solely based on those factors or based, at least in part, on those factors. Consider the phrase “determine A based on B.” While in this case, B is a factor that affects the determination of A, such a phrase does not foreclose the determination of A from also being based on C. In other instances, A may be determined based solely on B.
- [0046]
“Or.” When used in the claims, the term “or” is used as an inclusive or and not as an exclusive or. For example, the phrase “at least one of x, y, or z” means any one of x, y, and z, as well as any combination thereof.
DETAILED DESCRIPTION- [0047]
Various embodiments of an augmented reality (AR), and/or mixed reality (MR) direct retinal projector system are described that may, for example, resolve the convergence-accommodation conflict in head-mounted AR, MR, and VR systems. While at least some embodiments may provide mixed reality, for simplicity the system may generally be referred to herein as an AR system. Embodiments of an AR headset (e.g., a helmet, goggles, or glasses) are described that may include or implement different techniques and components of the AR system. In some embodiments, an AR headset may include a reflective holographic combiner to direct light from a projector light engine into the user’s eye, while also transmitting light from the user’s environment to thus provide an augmented view of reality. In some embodiments, the holographic combiner may be recorded with a series of point to point holograms; one projection point interacts with multiple holograms to project light onto multiple eye box points. In some embodiments, the holograms are arranged so that neighboring eye box points are illuminated from different projection points.
- [0048]
In some embodiments, the holographic combiner and light engine may be arranged to separately project light fields with different fields of view and resolution that optimize performance, system complexity and efficiency, so as to match the visual acuity of the eye. In some embodiments, the light engine may include foveal projectors that generally project wider diameter beams over a smaller central field of view, and peripheral projectors that generally project smaller diameter beams over a wider field of view.
- [0049]
In some embodiments, the light engine may include multiple independent light sources (e.g., laser diodes, LEDs, etc.) that can independently project from the different projection points, with a proportion being foveal projectors and a proportion being peripheral projectors. In some embodiments, the light engine includes two or more two-axis scanning mirrors to scan the light sources; the light sources are appropriately modulated to generate the desired image. In some embodiments, the light engine includes a series of optical waveguides with holographic or diffractive gratings that move the light from the light sources to generate beams at the appropriate angles and positions to illuminate the scanning mirrors; the light is then directed into additional optical waveguides with holographic film layers recorded with diffraction gratings to expand the projector aperture and to maneuver the light to the projection positions required by the holographic combiner.
- [0050]
In some embodiments, the light engine includes a lens for each projector to focus emitted light beams such that, once reflected off the holographic combiner, the light is substantially collimated again when it enters the subject’s eye. The required focal surface may be complicated by the astigmatism of the holographic combiner, but is a curved surface in front of the combiner. The ideal focal surface is different for different eye box positions, and errors may lead to less collimated output. However, in some embodiments, this can be compensated by reducing the beam diameter for different angles where the errors between the ideal focal surface and the actual best fit focal surface are greatest, which alleviates the problem by increasing the F-number and hence the depth of focus of the beam. In some embodiments, these features may be incorporated into a holographic lens.
- [0051]
In some embodiments, active beam focusing elements may be provided for each projection point. This may reduce or eliminate the need to change beam diameter with angle. This may also enable beams that diverge into the eye to, rather than being collimated, match the beam divergence of the supposed depth of the virtual object(s) being projected by the light engine.
- [0052]
With the methods and apparatus presented above, the AR system may not require extra moving parts or mechanically active elements to compensate for the eye changing position in the eye box or for the changing optical power from the holographic combiner during the scan, which simplifies the system architecture when compared to other direct retinal projector systems.
Accommodation and Convergence in AR/VR Systems- [0053]
The human brain typically uses two cues to gauge distance: accommodation (i.e., eye focus) and eye convergence (i.e., the stereoscopic perspective difference between the two eyes). Conventional near-eye systems typically use separate miniature screens for each respective eye to project the images intended for the left eye and the right eye, as well as optics to allow a user to comfortably focus the eyes at a far distance during viewing of the left eye and right eye images. Conventional near-eye systems thus produce conflicting visual cues sin
(Excerpt: record head incl. abstract, classifications, claims; full page at source URL.)