What Are Deepfakes: Synthetic Media Explained

Jun 15, 2021 6:43:48 AM

Deepfakes are one of the most unique phenomena of the last five years in the world of synthetic media. Many are afraid of this technology, others have figured out how to put it to productive use. It's time to figure out what deepfakes are exactly, and what makes them so significant in a world of modern media technologies.

What is synthetic media and what makes up its market landscape?

Synthetic media is an AI-generated or AI-modified content. With traditional media, people relied on broadcasting networks (radio and TV) and social networks to create and distribute their content.

This method of receiving information came with certain restrictions, which, of course, were significantly weakened by the emergence of social networks.

With synthetic media, content creators can create content with a high level of quality that was previously only available to major studios with massive budgets.

AI content is cheaper and easier to scale. However, this new trend raises several ethical concerns. They all boil down to one essential problem: how to distinguish AI-synthesized content from authentic content?

While the tech community and policymakers formulate proper regulation, movie studios, video bloggers, and the education sector are reaping the benefits of this technology.

The current landscape for the synthetic media market has been covered in detail by a recent Samsung Next study. Here are the key media sectors disrupted by this emerging technology: 

  1. Speech and voice synthesis
  2. Music & sound synthesis
  3. Image synthesis
  4. Video synthesis
  5. Game content synthesis
  6. Digital avatar synthesis
  7. Mixed reality synthesis
  8. Natural-language generation

Keep in mind that the majority of synthetic media use cases are run on deepfake technology.

Deepfakes in a nutshell 

In short, deepfakes are artificial intelligence-based images and sound synthesis techniques. They are used to join and overlay existing images, videos, and soundtracks onto original content.

In most cases, deepfakes use generative adversarial neural networks (GANs) to create this type of content. One part of the algorithm learns from a real media object. It creates an image that literally "competes" with the second part of the algorithm until it starts confusing the generated copy with the original.

Here's how deepfakes work in three key steps (using video production as an example):

  1. It begins with inserting the original video or voice of the target character into a neural network. Autoencoder and GAN algorithms go to work by analyzing the subject's facial expressions and main features. 
  2. Combining an autoencoder with GAN allows the algorithm to generate fake images until it can no longer distinguish them from the original.
  3. The video with the stunt double is then inserted into the neural network. After having analyzed the facial characteristics of the target subject, the network can easily generate a deepfake. The target subject's face is then overlaid onto the video of the stunt double. 

In voice cloning, where other algorithms are used, the process itself practically does not change.

Deepfake use cases 

The most common examples of deepfakes are videos in which the authors swap people's faces with other actors. You can find many deepfake cosplays of Hollywood actors like Tom Cruise or Arnold Schwarzenegger on the web. Less often, there are genuinely unique projects where the technology is used at the level of an art form.

One such project is the resurrection of Vince Lombardi for the Super Bowl. Respeecher created Vincent's speech for this project, and you can appreciate how brilliant the final product turned out.

Here's what Abigail Savage, Sound Designer and Actress that starred in Orange Is the New Black, had to say about Respeecher's AI-synthesized voice cloning:

Respeecher is a remarkable tool for Sound Editors. It delivers very high-fidelity recreations of a target voice, with transparent performance-matching of its source. It blows text-to-speech out of the water! The effect is uncanny and incredibly effective and I can imagine a whole slew of uses going forward.

Abigail-Savage-actress-sound-department-composer-Orange-Is-the-New-Black-testimonial-Respeecher-voice-cloning-software

Abigail Savage, Sound Designer and Actress

The range of deepfake use cases is not limited to video production. It spans multiple industries, from marketing to museum and education. Here you can find more examples using deepfake in marketing projects.

Another industry widely utilizing deepfake technology is education. British advertising holding WPP trains employees using a neural network.

Fifty thousand people learn the basics of marketing through video courses with a virtual mentor who lectures in three languages: English, Spanish and Chinese. At the same time, the mentor addresses each employee personally by name.

The most prominent architecture project to date is NVIDIA's GauGAN. Its neural network turns sketches into actual images. The program helps architects collect building designs from drawings and game designers to create levels in games faster.

Investigative journalists use deepfakes to change the appearance of sources who want to remain anonymous in reports. This technique, for example, was used by HBO when creating the documentary film, "Welcome to Chechnya."

And one of the most famous examples is Wireless Lab's Faceapp application that changes people in photos: their gender, age, appearance, and ethnicity.

Benefits of using deepfake in audio and video production

Using deepfake can significantly reduce production timelines and costs while helping to scale it to the next level. In short, here are a few of the most critical selling points for production studios:

  • Freeing up the actor's time. Artificial intelligence systems can generate entire scenes with actors without having to bring them in for live filming sessions.
  • Post-production savings. Re-recording and additional dialogue replacement (ADR) can be completed without involving actors.
  • Digital avatars instead of living people. Once created, virtual characters can run customer service, a reception desk, or act like a chatbot, increasing customer engagement. 
  • Content localization. Voice synthesis allows you to quickly duplicate audio tracks in many languages while preserving an actor's original voice. The final result sounds as if the actor speaking in the foreign language is doing so with the native tongue.

If you're looking to learn more about deepfake technology, we encourage you to take this LinkedIn course: Understanding the Impact of Deepfake Videos.

Conclusion

It's easy to get overwhelmed by the onset of this era of synthetic media, feeling like it cannot be stopped. But if you can manage to ride the trend and use these new opportunities to develop your projects and business, you'll be well ahead of the curve for years to come.

If you need to synthesize an original voice, rejuvenate a voice, or transmute a voice into another language - contact us today and we will help you identify the best options for your project.

Related Articles

Feb 25, 2021 7:52:37 AM

Deepfake videos are a subject of increasing interest and scrutiny worldwide. To this...

Mar 12, 2021 7:46:40 PM

Technology is a disruptor for many industries, especially when it comes to...

Apr 29, 2021 6:17:03 AM

It has always been part of our mission at Respeecher to help content creators scale...

Ready to Replicate Voices?

Want to see our technology in action? Get a demo today.

REQUEST A DEMO