What we know so far about Apple’s mixed reality headset

Something big is coming our way in June, if you believe the countless reports. Because in the course of WWDC 2023, Apple is to unveil its first new product category in eight years. The last new category was the Apple Watch, which was announced in 2014 and is now one of the most successful products in the Apple universe after the iPhone.

With the now expected mixed reality headset, the manufacturer is entering a difficult market segment that has been served by other players such as Oculus and Meta, HTC and others for years. The products are less successful in the mass market and are used more by gamers or in industry. In the past few months, sales of these headsets have also tended to decline.

Some industry representatives assume that Apple’s MR headset will shake up the entire market for “Metaverse products”. According to some people who have tried the headset, Apple’s debut should far outperform the competition in terms of capabilities and features, offering a higher level of performance and immersion, the latest reported Wall Street Journal.

For several years, information has been circulating about the headset, which Apple has been working on for seven years and which is said to be the first product to be created entirely under the aegis of Tim Cook. Steve Jobs even had a hand in the Apple Watch and the Apple silicon chips.

With the expected $3,000 headset, which could be called either Reality One or Reality Pro based on trademark filings, the company aims to deliver a new approach to virtual meetings and immersive content, disrupting the currently Meta-dominated VR industry. According to Bloomberg reporter Mark Gurman, Apple’s goal is to “bring something new to the table.”

The headset, which presumably looks like ski goggles, not only supports content in the virtual world (VR), but also that of augmented reality (AR). AR fuses digital content with the real world using outward-facing cameras. The Apple boss is particularly taken with AR: he has been talking about its strength for years and considers it to be just as important a core technology as an iPhone.

As for the hardware, Apple is expected to rely on two Sony 4K displays, the battery pack will not be attached to the headset, which can reduce the weight, which is rumored to be around 350 grams. The battery pack, which is to be attached to the hip, is about the size of two iPhone 14 Pro Max stacked on top of each other (test) and should offer around two hours of runtime, which is roughly on par with the competition.

In contrast to the VR headsets from the competition, Apple should not rely on plastic, but on higher-quality materials such as aluminum and glass. It is said to be reminiscent of the Airpods Max premium headphones. The product will feature a curved screen on the front that can expose the wearer’s eyes, the reports said. It also has speakers on the sides and a headband to adjust the device to your head. Airpods can be connected for more immersive sound.

As the computing unit of the Reality Pro headset, Apple will use a variant of the M2 chip, which is also installed in the company’s current Macs. In addition, a special processor for graphics and mixed reality experiences should also support the M2 chip. This second chip is to bear the name Reality Processor, as can be seen from the trademark applications submitted.

Apple’s headset should not only be better positioned in terms of hardware, but also the control will differ: Instead of classic controllers that are held in the hands, it is possible to navigate by eye and hand tracking.

The control is implemented using up to 15 cameras and sensors: Four 3D sensor sets are directed outwards, similar to Apple’s Face ID in the iPhone, and should be able to analyze the user’s hand movements. The number of sensor arrays means that higher accuracy is achieved than with individual TrueDepth camera arrays in iPhones.

Furthermore, inward-facing sensors in the device’s housing are designed to detect the eyes. With these, the wearer can control the device by looking at an item such as a button, an app icon, or a list item on the screen to select the item.

The user can squeeze their thumb and forefinger together to trigger the action without having to hold anything. This approach differs from the headsets from other manufacturers like Meta or Sony, which usually require a hand controller.

According to Gurmann, it is also possible to switch between VR and AR mode using a digital crown, similar to the Apple Watch. The Information magazine reported on this feature in early January. With the AR function, digital content is superimposed on the real views in pass-through mode.

Not unimportant for people who wear glasses: According to Bloomberg, Apple should offer users with corrective glasses customized lenses that sit in the housing itself.

In contrast to the Apple Watch, an iPhone is not required to operate the MR headset, according to the rumours. Apple is developing technology that will allow users to type with their hands in mid-air, according to Mark Gurman. It is unlikely that this feature will be available at launch.

Alternatively, text can also be entered by voice command via Siri or via passthrough mode on an iPhone, iPad or Mac keyboard.

Hardware is nothing without software or the operating system: xrOS, the Extended Reality Operating System, is apparently used here, which was specially developed for the new product category. The user interface and the range of functions are similar to those of iPhones and iPads, but the sub-interface is displayed in a 3D environment.

Among other things, the Safari web browser, photos, mail, messages and the calendar app are on board. There will also be apps for other Apple services such as the App Store for installing third-party software, Apple TV Plus, music and podcasts. The company is also working on health monitoring features.

Due to the proximity to iOS and iPad OS, the user experience of the MR headset should appear familiar. “When they put on the headset, the main interface is almost identical to that of the iPhone and iPad, with a home screen with a grid of icons that can be reorganized,” says Mark Gurman. As with the iPhone, users could also pin widgets such as weather, calendar events, emails and stock prices between their app icons.

To expand the ecosystem, Apple will provide a software development kit (SDK). With the SDK, developers can optimize or develop new apps for the headset.

In terms of functionality, the headset should be able to “show immersive video content, serve as an external screen for a connected Mac and replicate many functions of iPhones and iPads”.

According to a Bloomberg report, during video calls via Facetime with the headset, the face and the entire body of the user should be able to be realistically displayed in virtual reality. These avatars allowed two people wearing Apple headsets to communicate with each other “and feel like they were in the same room.” This approach differs from the meta-headsets’ virtual meeting rooms, the Horizon Workrooms, which create a more cartoonish avatar of the user.

A lot of computing power is required for these high-resolution full-body avatars, so they can only be used for video chats with two people. Facetime sessions with several people are also possible, additional users would be displayed as a symbol or memoji, it says.

In addition to the video calls with full-body avatars, another feature of the headset is the immersive viewing of videos. Apple has negotiated the development of VR content for the platform with about half a dozen media partners, including Disney and Dolby Laboratories. Apple is also working on adapting its own Apple TV Plus material for the headset. As part of these plans, Apple acquired streaming company Next VR in 2020 with the goal of creating sports content in VR. Gaming will also play a certain role, they say.

Even though Apple has put a lot of work into the hardware and software, according to Gurman, the killer app is still missing. Ultimately, Apple packs in everything that is possible, and customers can pick out the features that appeal to them.

The company did a similar thing with the Apple Watch, which was packed with numerous functions at the start. In the meantime, it is primarily used for health monitoring, for notifications, and the dials with many complications have also turned out to be popular.

Recently, reports circulated in the US media, according to which Apple internally still have all sorts of doubts about the sense and purpose of the new product category. “Some internal skeptics have questioned whether the new device is a solution to finding a problem,” according to the report.

Skeptics took the wind out of Apple CEO Tim Cook’s sails in an interview, saying that there were “tons of skeptics” about almost everything Apple has ever done. “Anyone who dares to innovate will always encounter skepticism,” added Cook.

Apple probably doesn’t expect high sales figures for the first generation anyway. Because for the vast majority, the first headset is not planned, as it will be quite expensive with a presumed price of 3,000 US dollars.

Although the headset will probably be announced during the WWDC keynote on June 5th, the latest reports say that the market launch will take place much later, which is not unusual for completely new Apple products. Apple also took a lot of time between the announcement and the start of sales for the modular Mac Pro presented in 2019, the first Homepod and other products. If you believe the analyst Ming-Chi Kuo, the late market launch of the headset in 2023 could be smaller than previously assumed: he expects 200,000 to 300,000 units, which is little by Apple standards. He had previously assumed that around half a million devices would be delivered this year.