De-aging, Resurrection and other AI Tricks

Machine Learning and AI are being put to innovative and creative use in filmmaking, proving that Data Science is not all about boring scientists pouring over sheets of data!

How about making a film casting now 90-plus Clint Eastwood as a 30-year-old? Possible, you say – with high-end prosthetic make-up.

Then what about shooting a brand-new blockbuster pairing Marlon Brando and Merilyn Monroe in the lead roles? Impossible, you say – because both these Hollywood stars are long dead and gone!

And therein lies the catch! Both these films are absolutely do-able using technology based on Artificial Intelligence (AI) and Machine Learning (ML) algorithms. And no, Eastwood will not require elaborate make-up with painstakingly glued-on prosthetic pieces. Because even if you do, he will never be able to match the lithe steps of the 30-year-old Western hero that he was.

Journey in Time

We all agree that movies require characters to go back-and-forth in time. And by now we are familiar with the age-old techniques they use to get around this poser. To show a character’s childhood, cast a child with matching features. To show a mature character as a young adult, cast a young adult with matching features. To show a mature character as an adult not so back in time, use make-up. To show a youthful character as an old adult, use heavy/prosthetic make-up. Not too many options, but it worked reasonably, helped by the viewers’ share of “willing suspension of disbelief”.

However, as technology progressed cracks began to show up. Digital cameras and HD images ensured that the viewer gets to see every wart in high-resolution. Although prosthetic make-up techniques got sophisticated too, there was always going to be a limit to it. New-age cinematography required state-of-the-art solutions. And that was when AI-powered computer-generated imagery (CGI) took over. It could take the same actor back or front in time, and also achieve the impossible task of bringing a dead performer back to life – if only on screen!

De-aging the AI Way

In a recent Hollywood movie, The Irishman, two legendary senior actors, Robert De Niro and Al Pacino were cast as 40-year-olds. They played the roles themselves, although De Niro was 76 and Al Pacino79. The technique used is known as“de-aging”. In short, it is an Artificial Intelligence-driven 3D effect technology used to edit specially shot digital images or to apply CGI touch-ups,that make the actor look younger on screen.

No make-up is involved here. Tiny markers remain attached to the faces of the actors during the shoot. These capture the real-life coordinates of the actor’s head as it moved during the shot. Based on these marker inputs, AI programming recreates a 3D version of the head. This virtual head is then synchronized with the movements of the actor’s head on screen. Colour filters or patches are overlaid to recreate the original character’s skin tone. Finally a digital “facelift” works on the obvious facial points that are ravaged by aging – the chin, the cheekbones, the nose and the ears.

One problem might still be there – the movements of an aged actor might not be matching his artificially acquired youthful looks. For this, often a younger actor with similar physical features is used as a body double. The double enacts the same movements the original actor had performed, and these youthful steps are accepted, along with the recreated young head, as the final shot.

The Technology

ML has become a key area of 3D R&D at most of the major VFX facilities. There have been efforts to incorporate available ML algorithms into a traditional 2D compositing framework. But a lot of studios are also using customised in-house programming for their productions. Mostly, ML Generative Adversarial Networks (GANs) are used to train the algorithm on images before and after digital clean-ups.

Basically, a GAN lets two different AI engines fight it out to come up with the best result. This cross-checking mechanism allows continuous self-improvement, making GANs the ideal choice for such visual-enhancement programs.

Experts mostly agree that the best way forward might be to combine the science of machine learning with the art and existing skills of VFX professionals, to recreate magic on screen.

Resurrecting the Dead

Harnessing AI/ML for CGI can have much more extensive impact on movie-making than simple de-aging. The most scintillating of these possibilities would be recreating images of deceased actors and make them appear in brand-new productions. No body double would be involved for this; in fact, no physical actor would be involved for these roles. The algorithm will analyse the existing video footage of the deceased actor, identify the key movement styles, and then – based on that “learning” – simply recreate new movements for that person depending on the requirements of the new role.

What we have here is not a repetition of actions in previous footages, but an entirely new recreation of that actor for an entirely new role – all based on the machine-generated inference of how that person would have walked, talked, laughed, reacted as conjectured from his style captured in existing footage.

Recreating the voice, too, would not be an issue at all. We already have efficient audio generation algorithms that can reproduce any style of speech in authentic audio after being “trained” on existing audio recordings of a person.

Imagine the possibilities this could open up – with star actors remaining eternally in demand! The implications are yet to be fully understood, but there would definitely be controversies.

The Ethical Question

That brings us to the ethical aspect of this technology. AI-generated CGI has already been called into ethical controversy because this is the technology that is the source of deep fakes. We all know the harmful potential deepfakes carry – especially since it has been used to create illegal video clippings, superimposed pornographic material, fake images, doctored news footage, and similar counterfeits that have already caused enough harm. Coupled with audio generation algorithms, they could easily produce motivated hate speeches or fake bites from top leaders that could wreck political havoc. Of course, a technology itself cannot be blamed for its misuse.

Resurrecting dead actors would certainly provoke intellectual property rights disputes, royalty litigations, and moral questions – like would the deceased actor have approved of a role he is being made to do in a recreated avatar? And what about the young and emerging actors who might not get meaty roles simply because legendary superstars would be digitally available for eternity!

The Cost Factor

As of now, it appears that such technology would remain exorbitantly costly for quite some time to come. So, they are not going to be mainstream right away. However, the industry will have to cross that bridge once technology becomes more affordable – as is usually the case.

Until then, filmmaking is going to see some exciting times as de-aging and other similar technology will allow the scriptwriters to explore more innovative plots, with the actors’ age being no longer a bar!