How Apple Built The iPhone X

The iPhone X is the smartphone Apple always wanted to make.

Sure, the company released three phones this fall, and like a parent who insists they love all of their children equally, Apple claims that the iPhone 8, 8 Plus, and iPhone X are each unique and special in their own way. But it’s obvious that the company is bending the truth — because the iPhone X is special in its own right.

“[It’s] something just entirely new,” said Apple SVP of worldwide marketing Phil Schiller. “I think customers were ready to have one more thing in the iPhone line that was new and very different.”

Mashable recently sat down with Schiller and other senior members of Apple’s executive team including SVP of software engineering Craig Federighi, SVP hardware engineering Dan Riccio, and VP of user interface design Alan Dye for a wide-ranging discussion about how they built what is perhaps the most eagerly-anticipated smartphone since Apple’s founder Steve Jobs unveiled the original iPhone more than a decade ago.

“It is one of those projects at Apple where you set up to do something that, at outset, you think, there’s no way we’re going to pull this off.”

Like virtually all other Apple products, the iPhone X was born out of a collaborative, cross-departmental process. It relied on teams and disciplines the company has been testing for years. Some, like 3D-depth mapping and OLED technology have never or rarely been used in an Apple product before. The Apple executives also revealed for the first time just how difficult it was for them to remove the home button, and why they don’t think “the notch,” where the iPhone X’s TrueDepth imaging module resides, is that big of a deal.

“It is one of those projects at Apple where you set up to do something that, at outset, you think, there’s no way we’re going to pull this off,” said Federighi who was recovering from the flu and joked that he had “iPhone X fever.”

“We all knew where we were going,” he said, “but there were a dozen things we knew we had to solve, and any one of them could have failed to come together.”

It was a sunny and unusually warm fall day when I arrived at Apple’s original campus in Cupertino (they’re still moving into the new “spaceship campus” down the road), a place where the iPhone X is almost as commonplace as the iPhone 7 or 8. I hoped to leave with a device that has, some might argue, gobbled up much of the attention from the also new iPhone 8 and 8 Plus. Generally well-received, the iPhone 8 has the unfortunate role of being the warm-up act for the real star of the smartphone show.

In the beginning

Depending on who you ask, work on the iPhone X (pronounced “10”) either started almost three years ago or way back in 2007.

It was around 2014 that Apple’s silicon team decided to include a neural engine inside what would eventually become the A11 Bionic CPU. “At the time, we didn’t know exactly what we’d use it for, but because silicon takes a lot of time in the oven, we knew we had to include it back then,” said Riccio.

That silicon enabled things on the device like augmented reality and powerful Face ID facial-recognition features. The groundwork, though, for an all-screen iPhone was laid a full decade ago. “It’s been a dream we’ve had since iPhone 1,” said Schiller. “We’ve had a dream since Day One to make it all screen, edge to edge.”

Perhaps it’s no secret that Apple wanted to do away with buttons and even the “head” and “chin” — mostly dead-space on the iPhone — but the Apple executives offered some fascinating insight into why this is happening now and how Apple identified the moment when all the necessary technologies percolating for years were ready to make an all-screen iPhone possible.

And Apple was going to deliver it… next year.

“As far as last-minute design changes? Actually, we didn’t have time for it.”

As Riccio described it, Apple had “the line of sight” for a 2018 iPhone X launch, “but with a lot of hard work, talent, grit, and determination we were able to deliver them this year,” he told Mashable.

That choice, one that in hindsight seems a little odd considering how the very existence of a better iPhone may have impacted iPhone 8’s fortunes, apparently put additional pressure on the teams.

In a rare, albeit brief, moment of stunning transparency, Riccio revealed that the compressed timeline left little room to consider functional alternatives. Such changes are part of Apple’s iPhone lore. Steve Jobs famously demanded the team swap out the original iPhone 1’s scratchable plastic screen for glass just weeks before its release.

When Apple made the choice to drop the home button and Touch ID fingerprint scanning in favor of Face ID, Riccio said they went “all in” with that functional decision. “We spent no time looking at [putting] fingerprints on the back or through the glass or on the side,” he said. Apple did it because they believed in the quality of Face ID security and screen unlocking, with executives describing it as good as second-generation Touch ID, but also because there simply wasn’t time.

“As far as last-minute design changes? Actually, we didn’t have time for it,” said Riccio, who seemed energized by the memory of that intense development period. “Quite frankly, this program was on such a fast track to be offered [and] enabled this year. We had to lock [the design] very, very early. We actually locked the design, to let you know, in November,” said Riccio before he was cut off by Apple PR. Riccio appeared to realize he’d said maybe too much, and then reaffirmed with a smile, “We had to lock it early.”

It’s not all that surprising that Apple didn’t entertain any big, last minute alterations. They were, after all, making some of the most aggressive design and functionality changes in the history of the iPhone.

Gone home

Few things upset Apple product fans like a deletion. Apple’s been pilloried in recent years for “courageously” removing the headphone jacks on iPhones and virtually all the third-party ports on MacBooks and MacBook Pros. I wondered if Apple had any trepidation over what may be its most aggressive deletion of all.

Apple’s continuing pride in the home button was clear to me as Schiller described how hard they worked on it and how, over each iPhone generation, it has changed, becoming more powerful, and, when they added Touch ID, critical to the iPhone operation and security.

It’s hard to overstate the importance of the home button, as it’s still the first way everyone interacts with their iPhone. (Pick up your phone, where does your thumb go?)

Schiller sounded, for a moment, like he was making a case for keeping the home button.

“So here we are at the pinnacle of the best single button interface ever on a device — truly, I don’t think that’s hyperbole — and with incredible strategic advantage compared to anyone else — no one has a touch sensor that works as well as Touch ID and just as we’re at a pinnacle, we replace it,” said Schiller. Then he laughed and added “That is so, so, so, so like us, like other things we’ve done.”

Replacing a feature people like, maybe love, and that Apple believes is as one of their greatest technological innovations is risky and only intensifies the pressure on every replacement technology in the iPhone X.

“We didn’t replace it because it didn’t work well,” said Schiller. “We replaced it because we wanted to do something else and therefore the bar was super high that it had to be great.”

“So here we are at the pinnacle of the best single button interface ever on a device…and just as we’re at a pinnacle, we replace it.”

Each of those replacement technologies presented their own challenges, “I think the toughest challenge we had was to replace Touch ID. Very, very hard,” said Riccio.

Apple contends that Face ID, their new biometric security, is a better, more natural authentication strategy. No one, Riccio reminded us, examines someone’s fingerprint when they’re trying to identify them on the street.

“You actually use your eyes and recognize them that way,” he says. Riccio makes an obvious point. Less evident was how Apple would implement facial recognition on the iPhone.

Fortunately for Apple, it already had the core technology in house, even if it never envisioned using it for this purpose.

In 2013, Apple acquired PrimeSense, the company best-known for building the first 3D motion-tracking technology in the original Microsoft Kinect. “We didn’t know what home it would have,” said Riccio. But Apple chose to invest and develop it anyway, figuring, “we might have a home for it someday,” recalled Riccio.

Even as Apple approached the idea of integrating the 3D depth-sensing technology into the iPhone, they never considered using it for authentication. Instead, Apple focused on enabling lightweight experiences like the new animojis, which use a 3D depth map of your face to sync your expressions with an on-screen iMessage cartoon character.

At the same time, Riccio’s team decided to see what would happen if they married the depth sensing capabilities with an IR camera for facial recognition. The results were better than they expected.

“We were not that far off from the spec that we had on Touch ID, both in what we call false acceptance rate, which is somebody else trying to use your phone, and false reject rate, which is more annoying, where it’s you, but it says it’s not you,” said Riccio.

With the True Depth hardware in place, the team focused on the A11 Bionics’ Neural Engine and other software and algorithms to reduce the false acceptance rate from one in 50,000 to one in a million.

The notch

Apple’s solution for a button-free home screen, though, had a major consequence. The TrueDepth module resides on the front of the iPhone X, digging a nearly quarter-inch-deep notch into Apple’s perfect OLED screen. While choosing a similar placement for IRIS-scanning authentication technology on the Galaxy S8, Samsung, notably, chose to just give it the full top edge of the device and not cut into their screen.

Schiller, however, offered a spirited and unusual defense of the iPhone X notch.

“To me, it’s something that, as you use it, you quickly grow comfortable with. You look at it as just the way the screen is. You don’t see at it as anything unusual or different, anymore than the bite of the Apple [logo] looks wrong to be bitten out of the apple. It’s a bite out of the apple and that’s just the way it is. It’s just the way the screen is,” said Schiller.

He’s also proud of the sheer amount of technology squeezed into the notch space. There are eight components in the space, including the infrared camera, flood illuminator, ambient light sensor, a speaker, microphone, front camera, and dot projector. “It’s one of the densest technology sections we’ve ever created,” he said.

See me

And that notch will be busier than the iPhone home button ever was. Apple is using the TrueDepth camera and its face-tracking capabilities to track attention, as well.

Federighi said Apple is taking advantage of attention detection and managing how often the iPhone X looks back at you.

“It’s one of the densest technology sections we’ve ever created.”

In addition to identifying your face among millions of others for secure unlocking, the iPhone X does a less personalized attention detection scan. “Just a, ‘Is someone looking at the phone right now’,” said Federighi.

While that may sound creepy, it’s really a battery-efficiency move. Current iPhones keep their screen on for a while after you put them down on the off chance you’re still looking at them. If you’re reading on the iPhone, you may have to tap it every once in a while, to keep it from dimming or turning off.

“Now we’re able to use your eyes as that virtual tap,” said Federighi. “If you’re reading something, the phone can periodically check, ‘Oh, someone’s still looking at me,’ so don’t go dim the phone away, but if no one is looking at the phone, the phone can be can be more aggressive about dimming down and saving power.”

Federighi told me the iPhone X checks for this kind of attention roughly every 30 seconds.

Wake up

Obviously, the iPhone X isn’t so hyper alert that it never goes to sleep. Put it on the desk and depending on your settings, it will auto lock and the screen will go dark. Which leads to another core and radical change.

Waking up the iPhone X is another reminder of what’s missing. Without a home button, how do you open the phone?

“It’s the biggest adjustment in the experience, but I think it comes pretty quickly,” said Federighi who has spent so much time with the iPhone X that, when he picks up his wife’s phone, he starts swiping up on it and then can’t figure out why nothing is happening.

Even as Apple was committed to removing the home button hardware, it considered other ways of representing the same idea.

Teaching iPhone users new habits meant that, whatever they did, it had to be obvious and clear.

“There’s no question, we explored whether it’s the form of the visual affordance or whether its wholly new or, perhaps…digital versions of the home button. Of course, we went very, very broad, as we do, kind of, with everything,” said Dye who claimed the teams worked closely with Apple’s chief designer Jony Ive, as well as all the hardware and software teams.

Teaching iPhone users new habits meant that, whatever they did, it had to be obvious and clear.

The result is a fairly non-descript bar that appears at the bottom of the screen and, if you don’t use it right away, it reminds you to “Swipe up to open.” Dye said the look belies all the work that when into it and the incredible attention paid to placement, size and the padding around it.

Federighi added that smaller versions of what he called a “grabber bar” have appeared before on the iPhone, most notable in iOS 7’s control center. Coincidentally, the iPhone X gesture bar forced the movement if the Control Center from a swipe up from the base to a less intuitive swipe down from the right side of the notch.

When pressed on how that vertical shift will impact users with smaller hands, Dye said people are already reaching up there, by which he may have meant the swipe down to view notifications. Federighi reminded us that the “reachability” option under accessibility still exists, and it does move Control Center access halfway down the screen.

Unlike the home button, this gesture bar serves one purpose: swiping up to open the iPhone X. However, even after people learn the new gesture, you can’t switch off the bar, confirmed Federighi.

iPhone X users will only grow accustomed to that new gesture if it works precisely and instantly every single time.

“We worked to get every millisecond of latency out of this gesture. If the moment your finger touches the bottom of the display, if the UI didn’t respond, we wouldn’t be happy about it” said Federighi who recalled early development, and walking around with “makeshift prototypes, crazy TrueDepth bolt-ons,” and even the new interface running on iPads.

Apple’s first iPhone OLED screen and its improved refresh rates helped some with this effort. Still, even adding OLED came with its own set of challenges.

More than a pretty face

Apple had been working on OLED for years, but knew that they couldn’t use off-the-shelf OLED technology for the iPhone X. “In order to enable the edge-to-edge design we wanted, it could not be a typical OLED, had to be a flexible OLED to be able to go top to bottom and side-to-side,” said Riccio.

They also had to work through some of the challenges of existing OLED technology including color accuracy, off-axis view, and burn in.

“We’re very particular about system-wide colors.”

Apple confirmed that the iPhone X does use a Samsung OLED display, but it’s not an off-the-shelf component. They worked with Samsung to create bespoke technology and then, Federighi told us, did a lot of low-level software work to overcome OLED’s inherent drawbacks.

Even perfect OLED technology will handle color representation differently than LCD.

“Making sure the colors were consistent to our expectations was a bit of a challenge,” said Dye whose team spent time tuning the displays and working on how the OLED would display system colors. “We’re very particular about system-wide colors.”

Apps at ease

For as many changes as the iPhone X promises, there is much that is blessedly unchanged or that will at least simply work despite the changes.

Third-party apps that used Touch ID to authenticate will automatically work with Face ID, said Federighi.

There isn’t even a custom version of iOS 11 for the iPhone X. Instead, Apple engineered the underlying the mechanics of how UI operates and the engine behind it to support both iPhone X and the traditional iPhone experience.

When I asked Federighi about how app developers were dealing with the new edge-to-edge screen, he noted that most are prepared.

“Generally, most developers now have taken advantage of size classes and auto-layouts and other technology to adapting to a wide variety of screen sizes…There’s very little dependency where developers have essentially hard-coded their screen ratios,” he said.

Still, this recognizable, though different iPhone X design might confound or frustrate some. Schiller acknowledged the challenge while spinning it to another Apple win. “Yes, there’s a moment when you need to learn. [It’s a] whole new experience. Actually, love that. To me, technology is exciting when you get to not get stuck in your ways for a decade, but learn something new.”

You’ve been asking Apple to innovate for years. Now, you have to ask yourself, are you ready to accept what innovation really means?

Original Source: Mashable.com