face off Facebook research aimed at providing really realistic avatars


2 min read

Facebook's Virtual Reality Laboratories (FRL), headed by Chief Scientist Michael Abrash, is experimenting with a human avatar, Michael believes the future of us all is in virtual reality (me too).


AR and VR will serve as the primary way for people to work, play, and connect in the future. Heather regularly speaks of the time he expects these fundamental milestones in technology to occur, but in a new blog series this year he wants to explore exactly how that will happen.


Abrash revealed on the blog today more about the company's research on live avatars, something done by the Pittsburgh branch of the company's skunkworks, Facebook's Virtual Reality Labs (FRL).


The software we developed on Facebook for transcribing human into virtual reality is called "Codec Avatars," and we use it in a groundbreaking way they call "3D Capture in AI Upgrade" to create realistic virtual avatars that can provide the creator's foundation in a quick, easy and personal way


Our Facebook vision says FRL research director, "to close physical distances and create social connections in virtual reality as" natural and common as in the real world. "

"It's not just the cutting graphics or advanced motion tracking," Sheikh says. "It is as natural and effortless to communicate with people in virtual reality as you can communicate with someone right in front of you. The challenge lies in creating authentic interactions in artificial environments."


Our goal is 'social presence', to provide realistic avatars that people embrace; The team calls the process "passing the ego test and the mother test".


"You have to love your avatar and your mom has to love your avatar before you both feel comfortable interacting like you do in real life. It's a very high bar," Sheikh says.


The video shows users using real avatar which certainly gives an interesting perspective on what the future of avatars in VR could be.

The company says that at this stage of real-time avatar, photorealistic avatars require expensive equipment.


The ultimate goal, however, is to achieve all this with light virtual reality goggles, though FRL Pittsburgh is currently using its own motion capture system (HMCs) equipped with cameras, accelerometers, gyroscopes, magnetometers, infrared lighting, microphones to capture the full range of human expression .

"Codec Avatars should capture your 3D profile, including all the subtleties of how you move and the unique features that make you instantly recognizable by friends and family," the company says. "And if billions of people want to use codec avatars every day, they need to develop an easier and cheaper system."


Using a small group of participants, the lab captures 1GB of data per second in an effort to create a database of physical features. In the future, the hope is that consumers will be able to create their own avatars without a capture studio and without much data.

Currently in terms of volumetric the system takes a face scan around 15 minutes, and requires multiple cameras to create the photorealistic avatars. That's why Facebook is developing a smart AI system that will turn the use of equipment and software into something that consumers can easily work with, humans have many different shapes and sizes, so the challenge ahead isn't simply Heather said.


"It taught me to appreciate how unique each one is. We caught people with exaggerated hairstyles and someone wearing an electroencephalographic hat. We scanned people with earrings, lobe rings, nose rings, and so much more," says Hugh. "We need to capture all these sophisticated clues to get it all to work properly. It's both challenging and empowering because we work so that you can be with us in virtual reality.


There are still many challenges to consider on the way there, Sheikh says. One big problem that appears on the horizon is "depths," or the act of recreating a person's appearance or voice to mislead others.


"Depths are an existential threat to our telepresence project because trust is intrinsically linked to communication," Sheikh says. "If you hear your mother's voice in the conversation, you have no doubt that what she said is what you heard. You have this confidence even though her voice feels a loud, cramped microphone, transmitted over many miles, reproduced on the far side and worn by an imperfect speaker .


Sheikh argues that we are still far from seeing Avatar's level of photorealism, though the lab is currently exploring the idea of securing future avatars through an authentic account, as well as several security and detection options for future devices.