Respeecher-synthesized-a-younger-Luke-Skywalkers-voice-for-Disney+s-The-Mandalorian-Respeecher-voice-cloning-software-case-study

Respeecher synthesized a younger Luke Skywalker's voice for Disney+'s The Mandalorian

 

The Mandalorian is a Star Wars television series created by Jon Favreau for the streaming service Disney+.

The Mandalorian became the most successful Disney+ original series to date, topping the charts for television series in 2019 and 2020. Critics widely acclaimed the first two seasons, which received ratings of 93 and 94 on Rotten Tomatoes. 

The Mandalorian’s success is not only due to the outstanding work of the cast, the showrunner, and the screenwriters. The technology used to create the environments of the Mandalorian plays a vital role in its production quality.

The Mandalorian was the first production to be filmed using real-time rendering for realistic, parallax environments. It's also one of the first TV shows that introduced Respeecher's voice cloning to first-tier movie production.

The Challenge

Luke Skywalker was set to make an appearance in the final episode of the second season of The Mandalorian.

With Mark Hamill being 68 years young, and Skywalker barely twenty years old in the time period of The Mandalorian, Respeecher had its work cut out.

In the Making of Season 2 Finale chapter that aired in August 2021, production teams revealed some details about applying voice cloning in the series.

"When we realized that we wanted to show a younger Luke Skywalker, we looked at all the techniques available," said Mandalorian's showrunner, Jon Favreau.

The producers settled on using deepfake technologies for visuals and voice to make a younger-looking version of Luke Skywalker.

Jon Favreau's team partnered with Lola VFX for video production and Respeecher for synthesizing the young Skywalker's speech.

The Project in Numbers

The best-kept secret of “The Mandalorian” Season 2 was Mark Hamill’s surprise cameo in the finale as a young Luke Skywalker, who offers to complete baby Grogu’s training in The Force. Yet how Hamill was de-aged has been kept under wraps until the launch of the episode of “Disney Gallery: Star Wars: The Mandalorian” on Disney+.

24

Emmy Awards nominations

2.5D

de-aging process

30 years

age difference

Respeecher-synthesized-a-younger-Luke-Skywalkers-voice-for-Disney+s-The-Mandalorian-Respeecher-Mark-Hamill-tweet

 

How Does Voice Cloning Technology Work

In order to clone a voice, Respeecher’s AI-powered speech synthesis software needs examples of that voice (which we call the target voice). Usually, 1-2 hours of quality recordings are enough. The software then analyzes the voice until it can produce a clone. Further testing is then conducted to ensure that the synthesized speech cannot be distinguished from the original voice.

Discover more about how we ensure (and even improve) the quality of the recording in this whitepaper about Respeecher’s audio super-resolution algorithm.

Hamill's target voice was captured using many early years' worth of radio broadcasts, interviews, ADRs, and dubs.

After analyzing the younger Luke Skywalker's voice, Respeecher could recreate it in a way that it is almost identical to the real version.

"Something people didn't realize is that his voice isn't real. His voice, the young Luke Skywalker’s voice, is completely synthesized using an application called Respeecher," says Jon Favreau.

Interestingly, this went unnoticed for the whole 9 months between the release dates of the Season 2 and the documentary.

Key Highlights

Aside from using Respeecher’s unique technology, there are several factors that the Mandalorian team makes note of.

One of the crucial goals during The Mandalorian’s production was keeping Luke Skywalker's cameo a secret.

"In the first season, Jon was able to successfully keep the name Grogu a secret before it was revealed. For the second season, we had to make it through the entire season to the airing of the last episode to not have this spoiled. In hindsight, it's a bit of a miracle that it remained a secret, and I'm thrilled because for people to discover that in real-time was magical," says Peyton Reed, Mandalorian's Director.

None of the hundreds of people working on the show leaked the details. This delivered the surprise to all the fans of the Star Wars universe on the day the final episode was released.

“The fact that we were able to keep my involvement a secret for over a year with no leaks is nothing less than a miracle. A real triumph for spoiler-haters everywhere!” - Mark Hamill tweeted.

The second factor that the creators of The Mandalorian paid great attention to was the ethics of using Deepfake technologies.

Considering that we have reached a point where the original video and voice are almost indistinguishable from the fake, the ethical use of this technology is more relevant now than ever.

Jon Favreau shared his thoughts on what anti-fake protection might look like:

"We were alluding to the fact that there's a type of blockchain identification that everybody has. I wonder if certain images or videos that are released in an official capacity could have some kind of a stamp on them. Because it is becoming harder to tell the real from the fake."

While the industry has not yet developed uniform standards for combating the malicious use of deepfake technology, Respeecher has taken some steps in this direction. 

We spend a lot of time and energy to educate the public on the threats of synthetic media technologies and promote ethical standards among deepfake creators and content creators, by showcasing amazing uses of voice cloning technologies in real projects and participating in a number of initiatives in order to guard against the misuse of such technologies.

Conclusion

Firstly, the entire Respeecher team was honored to work on such a huge project.

Secondly, the result of this cooperation is sure to attract even more high-profile projects to voice cloning technology. This is important for both the entertainment industry and audiences worldwide. The former gets unprecedented opportunities to create unique content with iconic stars, while the latter gets to see their long-loved heroes on the big screen again.

Thirdly, voice cloning brings unprecedented opportunities for new talent and content creators, allowing them to better balance their workload between high-demand actors and those who are not facing that intense demand. Respeecher transformed the way productions handle ADR and dubbing, saving time for actors and money for production teams.

Lastly, the widespread attention and adoption of deepfake technologies will accelerate the need to regulate the use of the technology. Respeecher always prioritizes eliminating the possibility of others being harmed by deepfake technologies, which cannot be done without uniting various media industry players and stakeholders around the goal of adopting synthetic media technologies in the most ethical way.

We deeply appreciate Jon Favreau and his team bringing attention to how synthetic media can reshape media creation in the future, how it could be harmful in the wrong hands, and how we can control it. We also greatly appreciate his showcasing of voice cloning technology in the best possible light on one of the biggest TV shows of 2020.

Reviews

review_avatar

Halsey Burgund

Co-Director In Event of Moon Disaster

review_avatar

Ryan Steelberg

President and co-founder of Veritone

review_avatar

David Romberg

Media, Communication & Film Studies, Muhlenberg College

review_avatar

Abigail Savage

Sound Designer and Supervisor, Actress

review_avatar

Michael York

Actor, Producer

review_avatar

Tim Jones

Emmy Award Winning Animation Producer

review_avatar

Joseph López Osorio

Marketing Technology Director, DDB Latina

review_avatar

Edgardo Rivera

President & CEO, DDB Puerto Rico

review_avatar

Joshua Madoff

Composer, Actor

review_avatar

James Tumminia

Producer & President at Italus Films

We worked with Respeecher on a film called ‘In Event of Moon Disaster’ first shown at the Amsterdam Documentary Film Festival 2019. They helped us create a synthetic voice of Richard Nixon to bring to life a never-read contingency speech in case the Apollo 11 mission went badly. We created a highly realistic film, in a large part due to their work. ‘In Event of Moon Disaster’ shows the creative possibility of voice replacement technology as well as highlighting just how realistic deepfake technologies can be, acting as a civic engagement project for the public along with contextual and educational resources.
For Veritone, the most important thing is ensuring our clients get access to technology without compromising on the ability to protect, the delivery speed and quality, and in this case, of the generated content. Respeecher's technology meets both requirements and has a proven track record of working with tier one projects.
When I approached the team at Respeecher with a very sensitive and personal documentary project, I didn't know what to expect. I’m happy to say that it has been an amazing experience. They are extremely professional, ethically minded and are genuinely passionate about bringing their innovative technology to the world. Their approach to the voice cloning process is highly personal and transparent. As a documentarian, it was important to me that the approach was ethical and human focused. The Team at Respeecher delivered on their promises and I could not be happier with the results.
Respeecher is a remarkable tool for Sound Editors. It delivers very high-fidelity recreations of a target voice, with transparent performance-matching of its source. It blows text-to-speech out of the water! The effect is uncanny and incredibly effective and I can imagine a whole slew of uses going forward. I am very excited to have discovered Respeecher, and it will be my go-to for voice recreation in the future, without question.
I am very grateful to Respeecher for giving my words and work a whole new lease on life. The technique involved is as ingenious as it is amazing, and promises a great future for all those involved in voice work. With Respeecher, the quality of the re-worked passages is indistinguishable from the original. History comes almost miraculously alive again. I heartily endorse this product.
Respeecher's source/target conversion of Mr. York's speaking voice demonstrably opens doors for the entertainment, education, and healthcare fields. Suffice it to say, it was an absolute honor to collaborate with them. Their brilliant accomplishment and timely delivery have allowed our humble project to be a success, exceeding our expectations for what's even possible.
Discovering Respeecher was the highlight of our project. They agreed to work with the task even when our training data was not of the best quality. The result was fantastic, and the level of detail was just outstanding.
Working with Reespecher was a unique experience. Their attentiveness to detail and dedication to this project allowed us to reconnect with one of our lost narration idols in Puerto Rico "Manuel Rivera Morales" on a new level. It gave his family and PR a glimpse of what could have been if he was still alive and narrating all sorts of Olympic sports games. It was truly magical to see this project come alive.
Working with Respeecher has been incredible. Their team was meticulous and patient, walking us through the process of developing an accurate model... and the final result was nothing short of miraculous. It really sounds like we’re hearing our (deceased) subject speaking again. Something I never would have thought was possible.
From the introduction to Respeecher’s innovative and state-of-the-art products to the thorough and quick technical support, I immediately felt that I found the perfect partner and technology for my projects’ needs. During our discussions, the team offered other creative options that assured me we were on the same page and had my project’s success as a top priority. The Respeecher team members are consummate professionals.
review_avatar
review_avatar
review_avatar
review_avatar
review_avatar
review_avatar
review_avatar
review_avatar
review_avatar
review_avatar

Ready to Replicate Voices?

Want to see our technology in action? Get a demo today.

REQUEST A DEMO