I've been wondering about the possibility of using this sort of tech (or the API offerings from Azure or GCP) to provide voice overs in video games.
By that I mean for smaller budget Indie development, it would be certainly interesting to either be able to generate voice audio from transcripts in order to add voices to background NPCs and so on (or even the possibility of doing it at run time to produce much more dynamic worlds).
I guess the biggest blocker is the difficulty in conveying emotion with what is currently available as well as the difficulty in getting pronunciation correct (especially with nouns).
There are half a dozen startups in this space that provide the tech. They use embedded style tokens or sliders to change the emotion, pitch, timbre, etc. I don't have links off hand, but they're not too difficult to find.
These companies tend to focus on off-the-shelf turnkey solutions, so they'll have a suite of a few voice actors to choose from for different character archetypes.
Yep, that's what we're doing at https://replicastudios.com - though we have strict ethical guidelines to not clone a person's voice without their permission.
Can this service be used with preexisting recordings? I have family that are losing the ability to speak and I’d love to be able to give them that back somehow.
I'm not a lawyer, but I think we're entering into a legal gray area. There are the existing frameworks of copyright, parody, free speech, slander, libel, etc. that are all somewhat tangential to this.
I believe (I'm not certain) that celebrity voice impersonation is legal as long as it is not used to sell or endorse a product.
Most models are trained on the original speaker's voice, but maybe only a little bit. Models might incorporate learning from many speakers. We might even be able to boil down a speaker representation to a small vector encoding in the future. It'll be interesting if we can capture the representation of a person with just a few numbers.
I don't think the legislature should be overly protective against machine learning. It seems obvious to me that neural networks will play a huge role in creating entirely virtual musicians and influencers. We're already seeing this start to happen. r9y9 on github has published some models that rival Vocaloid in lyrical ability.
At the same time, we don't want these techniques used to commit fraud, slander, or have them be used to falsely accuse someone of committing some act. These are things we might need new legal protections for.
But I don't know what I'm talking about. I'm not a lawyer.
I asked not because I expected an answer, but because I figured you'd have an insightful opinion.
It's essentially the performance of a composition vs the composition question again: at what point am I mimicking someone to the extent they have a valid claim on a portion of my work?
I expect it'll enter the courts a few milliseconds after someone clones a dead actor (without their estate's permission) for a new performance.
There's always been an inherent tension in the US distinction between a law of nature and a creative work though. It seems a bit silly for me to claim patent / trademark on a vector that encodes my likeness.
I suspect there's some plausible deniability built-in that might allow for such matters to be legal.
For example, lots of people sound like Arnold Schwarzenegger. So if you trained a model with tall, deep-voiced Austrian man, you could probably get something that people will immediately associate with Arnold without actually being his voice, or someone emulating him. Because much of what Americans associate with his voice is really a regional accent which is relatively uncommon in the US.
There may be a little bit more difficulty getting away with with someone like Gilbert Gottfried, whose voice is much more unique. But I do think you could get away with creating a voice that people think sounds just like him, but doesn't hold up in a side-by-side comparison.
What I think will happen is celebrities like Morgan Freeman will use their voice to train models like this, then gift these to their estates for use in the future.
> So if you trained a model with tall, deep-voiced Austrian man, you could probably get something that people will immediately associate with Arnold without actually being his voice, or someone emulating him. //
I think "passing off", an unregistered element of trademark laws, may be pertinent here. If the public think that there's an association and you're knowingly trading on that, even if the public are wrong, then you can be 'passing off' your output as someone else's goods/services/[vocal renditions].
It's likely you'd have to be very careful about use of copyright material for training the voice (eg extracting metrics that describe the voice). Fair Use might apply in USA though (even commercially).
>"Most models are trained on the original speaker's voice, but maybe only a little bit."
Really cool that you got this to work. I used to work on TTS (a few years ago, now), and we trained on celebrity voices, but used full audiobooks. https://github.com/Kyubyong/tacotron
It's not settled caselaw so anyone basing a business off this should expect to spend a lot of money defending it in courts
I saw once a company that offered to be the sole purveyor of a celebrity's synthesized voice. I haven't been able to find them again, but that seems like a much safer way to monetize this.
I'm not sure that making it easier to profit off of the likeness of others is a positive side. If it's legal for indie studios to do, it's legal for 20th Century Fox, Universal, and so forth.
It reduces cost to produce a specific good. I would think it would be measured similar to other technology advancements that do the same. Good for society, bad for the craftsman that were made obsolete, overall net positive.
This is purposefully not counting in the effect of being able to fake people and the damage that does to society, but I think that was implied by the previous poster specifying looking for the more positive side of the technology.
It’s not so obvious that it’s a net positive. Music for instance is way easier to get ahold of now, but we also don’t get to enjoy sophisticated long-form music as much, owing to how there’s no money in making it.
Check out Modulate.ai! We make real-time, emotive voice skins aimed at gaming voice chat. Audio watermarking is also built-in to prevent fraud. Currently in a closed alpha stage but if you're part of a game studio and have interest please reach out!
I've been wondering about the possibility of using this sort of tech (or the API offerings from Azure or GCP) to provide voice overs in video games.
By that I mean for smaller budget Indie development, it would be certainly interesting to either be able to generate voice audio from transcripts in order to add voices to background NPCs and so on (or even the possibility of doing it at run time to produce much more dynamic worlds).
I guess the biggest blocker is the difficulty in conveying emotion with what is currently available as well as the difficulty in getting pronunciation correct (especially with nouns).