Before taking a closer look at each of the elements that go into Beauty Tech, it’s helpful to get an overview of how it all works together. According to EOS Intelligence, “The use of AI, augmented reality (AR), virtual reality (VR) as well as complex beauty devices has revolutionized the way consumers perceive, apply, and select beauty products. Moreover, in the age of online retail, it enables companies to maintain a similar personalized level of service that would otherwise require a physical interaction with a beauty consultant.” Once a futuristic gimmick, AR & AI beauty tech has become an integral part of the beauty shopping experience for makeup, skincare, nail polish, men’s grooming, and hair brands of all sizes.
Technology Basics
Beauty Tech represents a merging of hardware and software. Some companies are creating specific-use hardware devices — such as for analyzing current skin conditions to provide more precise product usage or using handheld inkjet printers to hide blemishes or create masks.
However, the most common hardware element is the ubiquitous smartphone. These days everyone has one. Model upgrade by model upgrade, their cameras become more precise and their operating systems, and portfolio of apps, become more powerful. Add to that the increasing speed of mobile internet connectivity, and you have a powerful environment in which AR and AI can flourish.
Ideally, Beauty Tech is implemented as an efficient platform in which the software maximizes the potential of the hardware.
The basic framework of Beauty Tech includes:
Capture an Image
The process begins by capturing an image of the face—either with a dedicated Beauty Tech hardware device, or most commonly, a smartphone.
Process the Image
The real power kicks in with the software used to process the image. This includes, among other factors, deciphering the elements of the face—establishing precisely where eyebrows begin and end, defining the eyes and surrounding skin, identifying the exact boundaries of the lips, and recognizing each strand of hair. Perfect Corp.’s AR beauty technology is based on the innovative real-time high-definition facial 3D live mesh with 3,900 meshes, ensuring the accuracy of the virtual try-on experience.
Apply AI
The best Beauty Tech solutions incorporate AI to bring greater precision into the facial mapping and tracking required to accomplish all of the above—and to enable these boundaries to be fluidly identified even as the user turns to see how a virtual try-on looks from different angles—providing a magic mirror experience.
Launch AR
The AR experience begins as mathematical algorithms are executed behind the scenes to change values—applying different colors to the lips, overlaying foundation, changing brow shapes, trying new hair colors, testing new eye shadow, or exploring various nail polish shades. Precision AR technology is required to achieve the real-world look and feel of an immersive AR experience.
BIG Data
Big data—and more AI—can be used to further the personalization of Beauty Tech. Big Data and AI can be harnessed, for example, to power a foundation finder application that can better match a person’s skin than the user could. Of course, the user can try whatever foundation they like, but solid AI and Big Data can tap into thousands of skin types to generate optimal suggestions. Data can also be harvested from previous interactions, as an application learns the preferences of individual users—as well as the experimentation that similar users enjoy.
Make Up for Ever Foundation Shade Matcher Facial Tracking
A fully immersive AR experience depends upon faultless facial tracking. Users don’t sit still when working with a virtual try-on. They want to move their head about, just as they would in front of a mirror. The lack of faultless facial tracking can mean that lipstick lags behind when moving from side to side, or that a new hair color gets stuck on the wall when the user moves to see how a new hair color will look from a different angle. An enormous amount of technology goes into real-time facial tracking. Without this technology, makeup can take on a clownish appearance, and the session, while perhaps a bit amusing, won’t satisfy nor bring users back. Perfect Corp.’s patented AgileFace® tracking technology accurately detects all facial features and overlays virtual makeup on them with the utmost precision, even aligning virtual makeup with facial movement. This new technology properly identifies all variations of facial features, resulting in the most inclusive virtual try-on experience yet.
Precision virtual try-on for all categories
The value of a virtual try-on experience—for the customer as well as the seller—is to provide an authentic presentation of how a product will actually look on the customer. This can be especially challenging with hair color, as the color applied will differ according to the customer’s natural hair color. “Lashes are extremely hard to capture in AR,” says American International Industries’ Marwan Zreik. “They can look pasted on. And the same is true for hair, which is also a major technical challenge to precisely render. We were all blown away by the accuracy of Perfect Corp’s AI and AR. The color went onto the hair in a completely natural way. It actually works with the tone of your natural hair color, which means that each of our Punky colors looks different on every person depending on their base hair color.”
Exacting Lip and Eyeliner Colors
As with hair color, the value of a virtual try-on experience is completely dependent upon the ability to render in AR exactly how a product will look on the user. For lipstick, this means not only rendering colors precisely but also capturing the texture. If lipstick is glossy, or metallic, shimmers, or has other unique features, precision AR is required to enable the user to see exactly how it will look on their own lips. Failure at this results in unhappy customers and can have devastating effects on social media. Stila Cosmetics partnered with Perfect Corp. to offer precisely this kind of hyper-realistic virtual try-on experience on its website. Customers can instantly try on dozens of popular makeup products, including lipstick, eye shadow, mascara, highlighter, and more. The digital effects of a virtual makeup try-on so closely mimic those of a physical try-on that consumers can be confident they are making the right purchase decision.
High-Definition Brows
Hair is generally considered one of the greatest challenges for AR because it needs to be rendered strand by strand. The challenge is even greater when working with brows. Without precise definition, the virtual try-on will be unsatisfactory. Again, facial tracking is important, too. If a user arches a brow, that brow must arch.
Foundation is often puzzling for users
Just which foundation is right? Finding the perfect foundation shade can be a challenging process for many consumers, but AI beauty tech can simplify the process. For example, Estée Lauder’s latest in-store beauty app combines precision foundation shade detection with a virtual try-on-color match experience. With iMatch™ Virtual Shade Expert, Perfect Corp.’s AI deep learning for real-time skin-tone detection uses its knowledge base of 89,969 skin tones to help customers find their preferred foundation shade in Estée Lauder’s number one foundation, Double Wear Stay-in-Place Makeup, and see it virtually applied in real-time. “We found a customer’s perfect foundation shade match is not always her preferred shade,” says Gen Obata, Vice President Global Retail Experience at Estée Lauder. “So iMatch™ Virtual Shade Expert answers this need. It finds her perfect shade and then provides options that are lighter, darker, cooler, or warmer so she can choose the shade she thinks suits her best.” This kind of personalization creates a strong customer bond.
Unique Virtual Manicure
Finding that perfect nail polish shade can be a daunting task, considering how challenging it can be to try nail products at the store and especially online. Perfect Corp.’s AI-powered AR virtual try-on for nails greatly simplifies this undertaking by delivering a real-time ultra-realistic 3D live preview of the nail products directly on consumers’ hands, without messy clean ups and time-consuming application.
Innovative Approach to Men’s Grooming
Beauty Tech can be beneficial for men as well. Latest AI & AR developments have made it easy for men to virtually experiment with different beard styles and colors, with no need to shave or commit to a new look. Perfect Corp.’s AI Beard technology allows users to virtually experience beard dye products and try different beard styles using any device with a camera before committing to a certain look. The advanced AI technology can also simulate beard removal, allowing men’s grooming customers to visualize how they would look clean-shaven.
Powerful AI
AI has been mentioned throughout as it provides foundational support to great AR. In addition to its role in facial detection and tracking, and its powerful mining of Big Data, AI can power personalization that creates tight customer bonds. Jon Roman, Senior Vice President, Consumer Marketing and Online at Estée Lauder puts it this way: “The combination of precision shade matching with personalization not only increases conversions, but you are also delighting the customer because she’s really getting something that’s personalized for her. This kind of personalization really builds customer loyalty.” AI can also be used for ‘look transfer’ in which a user can capture the essence of a favorite celebrity or influencer, and map that look to their own face.
Delta E Precision
We have spoken repeatedly about the need to precisely match colors displayed in a virtual try-on experience with those found in real life. And we’ve spoken of the variability of colors when applied to someone’s unique hair or their own unique skin texture and color. While this may sound subjective, color variations can be measured down to the point of imperceptibility through the use of Delta E metrics. Delta E is a metric for calibrating how the human eye perceives color differences. The term delta comes from mathematics, meaning change in a variable or function. The suffix E references the German word Empfindung, which broadly means sensation. Generally, a Delta E value of 3 or less means that color differentiation can only be detected by careful study. A value of 2 or less is considered to be imperceptible to the human eye. With our Perfect Corp. technology, the Delta E value is often below 1.5—making for a seamless experience when moving from AR virtual try-on colors to real-world products, and helping every consumer find their perfect product match.
Multi-Product Capability
Finally, a Beauty Tech platform should provide multi-product and multi-vendor capability. Don’t expect that consumers will download a separate app for each product they like. We all suffer from app overload, and it disrupts the AR experience to have to move from one company’s lip app to another company’s eye app to a hair color app. That’s where Perfect Corp.’s AR and AI technology comes in to provide a unified and fun shopping experience. With this technology, users can create complete looks and then make their purchases accordingly. Ekta Chopra, VP of Digital, at e.l.f., speaks of the joy of product discovery. “Looking at the full consumer journey, discovery is a huge element,” Chopra says. “You want to create a digital environment where they can discover new products, try them on, and make the purchase. Perfect Corp gives us the augmented reality technology to provide our consumers with an amazing, personalized, discovery experience.”