AI

Runway’s new video-generating AI, Gen-3, offers improved controls

Comment

Runway Gen-3
Image Credits: Runway

The race to high-quality, AI-generated videos is heating up.

On Monday, Runway, a company building generative AI tools geared toward film and image content creators, unveiled Gen-3 Alpha. The company’s latest AI model generates video clips from text descriptions and still images. Runway says the model delivers a “major” improvement in generation speed and fidelity over Runway’s previous flagship video model, Gen-2, as well as fine-grained controls over the structure, style and motion of the videos that it creates.

Gen-3 will be available in the coming days for Runway subscribers, including enterprise customers and creators in Runway’s creative partners program.

“Gen-3 Alpha excels at generating expressive human characters with a wide range of actions, gestures and emotions,” Runway wrote in a post on its blog. “It was designed to interpret a wide range of styles and cinematic terminology [and enable] imaginative transitions and precise key-framing of elements in the scene.”

Gen-3 Alpha has its limitations, including the fact that its footage maxes out at 10 seconds. However, Runway co-founder Anastasis Germanidis promises that Gen-3 is only the first — and smallest — of several video-generating models to come in a next-gen model family trained on upgraded infrastructure.

“The model can struggle with complex character and object interactions, and generations don’t always follow the laws of physics precisely,” Germanidis told TechCrunch this morning in an interview. “This initial rollout will support 5- and 10-second high-resolution generations, with noticeably faster generation times than Gen-2. A 5-second clip takes 45 seconds to generate, and a 10-second clip takes 90 seconds to generate.”

Gen-3 Alpha, like all video-generating models, was trained on a vast number of examples of videos — and images — so it could “learn” the patterns in these examples to generate new clips. Where did the training data come from? Runway wouldn’t say. Few generative AI vendors volunteer such information these days, partly because they see training data as a competitive advantage and thus keep it and info relating to it close to the chest.

“We have an in-house research team that oversees all of our training and we use curated, internal datasets to train our models,” Germanidis said. He left it at that.

Runway Gen-3
A sample from Runway’s Gen-3 model. Note the blurriness and low resolution is from a video-to-GIF conversion tool TechCrunch used, not Gen-3.
Image Credits: Runway

Training data details are also a potential source of IP-related lawsuits if the vendor trained on public data, including copyrighted data from the web — and so another disincentive to reveal much. Several cases making their way through the courts reject vendors’ fair use training data defenses, arguing that generative AI tools replicate artists’ styles without the artists’ permission and let users generate new works resembling artists’ originals for which artists receive no payment.

Runway addressed the copyright issue somewhat, saying that it consulted with artists in developing the model. (Which artists? Not clear.) That mirrors what Germanidis told me during a fireside at TechCrunch’s Disrupt conference in 2023:

“We’re working closely with artists to figure out what the best approaches are to address this,” he said. “We’re exploring various data partnerships to be able to further grow … and build the next generation of models.”

Runway also says that it plans to release Gen-3 with a new set of safeguards, including a moderation system to block attempts to generate videos from copyrighted images and content that doesn’t agree with Runway’s terms of service. Also in the works is a provenance system — compatible with the C2PA standard, which is backed by Microsoft, Adobe, OpenAI and others — to identify that videos came from Gen-3.

“Our new and improved in-house visual and text moderation system employs automatic oversight to filter out inappropriate or harmful content,” Germanidis said. “C2PA authentication verifies the provenance and authenticity of the media created with all Gen-3 models. As model capabilities and the ability to generate high-fidelity content increases, we will continue to invest significantly on our alignment and safety efforts.”

Runway Gen-3
Image Credits: Runway

Runway has also revealed that it’s partnered and collaborated with “leading entertainment and media organizations” to create custom versions of Gen-3 that allow for more “stylistically controlled” and consistent characters, targeting “specific artistic and narrative requirements.” The company adds: “This means that the characters, backgrounds, and elements generated can maintain a coherent appearance and behavior across various scenes.”

A major unsolved problem with video-generating models is control — that is, getting a model to generate consistent video aligned with a creator’s artistic intentions. As my colleague Devin Coldewey recently wrote, simple matters in traditional filmmaking, like choosing a color in a character’s clothing, require workarounds with generative models because each shot is created independently of the others. Sometimes not even workarounds do the trick — leaving extensive manual work for editors.

Runway has raised over $236.5 million from investors, including Google (with whom it has cloud compute credits) and Nvidia, as well as VCs such as Amplify Partners, Felicis and Coatue. The company has aligned itself closely with the creative industry as its investments in generative AI tech grow. Runway operates Runway Studios, an entertainment division that serves as a production partner for enterprise clientele, and hosts the AI Film Festival, one of the first events dedicated to showcasing films produced wholly — or in part — by AI.

But the competition is getting fiercer.

Runway Gen-3
Image Credits: Runway

Generative AI startup Luma last week announced Dream Machine, a video generator that’s gone viral for its aptitude at animating memes. And just a couple of months ago, Adobe revealed that it’s developing its own video-generating model trained on content in its Adobe Stock media library.

Elsewhere, there’s incumbents like OpenAI’s Sora, which remains tightly gated but which OpenAI has been seeding with marketing agencies and indie and Hollywood film directors. (OpenAI CTO Mira Murati was in attendance at the 2024 Cannes Film Festival.) This year’s Tribeca Festival — which also has a partnership with Runway to curate movies made using AI tools — featured short films produced with Sora by directors who were given early access.

Google has also put its image-generating model, Veo, in the hands of select creators, including Donald Glover (aka Childish Gambino) and his creative agency Gilga, as it works to bring Veo into products like YouTube Shorts.

However the various collaborations shake out, one thing’s becoming clear: Generative AI video tools threaten to upend the film and TV industry as we know it.

Runway Gen-3
Image Credits: Runway

Filmmaker Tyler Perry recently said that he suspended a planned $800 million expansion of his production studio after seeing what Sora could do. Joe Russo, the director of tentpole Marvel films like “Avengers: Endgame,” predicts that within a year, AI will be able to create a full-fledged movie.

A 2024 study commissioned by the Animation Guild, a union representing Hollywood animators and cartoonists, found that 75% of film production companies that have adopted AI have reduced, consolidated or eliminated jobs after incorporating the tech. The study also estimates that by 2026, more than 100,000 of U.S. entertainment jobs will be disrupted by generative AI.

It’ll take some seriously strong labor protections to ensure that video-generating tools don’t follow in the footsteps of other generative AI tech and lead to steep declines in the demand for creative work.

More TechCrunch

Fisker is just a few days into its Chapter 11 bankruptcy, and the fight over its assets is already charged, with one lawyer claiming the startup has been liquidating assets…

The fight over Fisker’s assets is already heating up

A hacker is advertising customer data allegedly stolen from the Australia-based live events and ticketing company TEG on a well-known hacking forum. On Thursday, a hacker put up for sale…

Hacker claims to have 30 million customer records from Australian ticket seller giant TEG

Welcome to Startups Weekly — Haje‘s weekly recap of everything you can’t miss from the world of startups. Sign up here to get it in your inbox every Friday. Elon…

Tesla makes Musk best-paid CEO of all time and Fisker bites the dust

Dot is a new AI companion and chatbot that thrives on getting to know your innermost thoughts and feelings.

Dot’s AI really, really wants to get to know you

The e-fuels startup is working on producing fuel for aviation and maritime shipping using carbon dioxide and other waste carbon streams.

E-fuels startup Aether Fuels is raising $34.3 million, per filing

Fisker was facing “potential financial distress” as early as last August, according to a new filing in its Chapter 11 bankruptcy proceeding, which the EV startup initiated earlier this week.…

Fisker faced financial distress as early as last August

Cruise, the self-driving subsidiary of General Motors, has agreed to pay a $112,500 fine for failing to provide full information about an accident involving one of its robotaxis last year.…

Cruise clears key hurdle to getting robotaxis back on roads in California

Feel Therapeutics has a pretty original deck, with some twists we rarely see; the company did a great job telling the overall story.

Pitch Deck Teardown: Feel Therapeutics’ $3.5M seed deck

The Rockset buy fits into OpenAI’s broader recent strategy of investing heavily in its enterprise sales and tech orgs.

OpenAI buys Rockset to bolster its enterprise AI

The U.S. government announced sanctions against 12 executives and senior leaders of the Russia-based cybersecurity giant Kaspersky. In a press release, the Department of the Treasury’s Office of Foreign Assets…

US government sanctions Kaspersky executives

Style DNA, an AI-powered fashion stylist app, creates a personalized style profile from a single selfie. The app is particularly useful for people interested in seasonal color analysis, a process…

Style DNA gets a generative AI chatbot that suggests outfit ideas based on your color type

Rates of depression, anxiety and suicidal thoughts are surging among U.S. teens. A recent report from the Center of Disease Control found that nearly one in three girls have seriously…

Khosla-backed Marble, built by former Headway founders, offers affordable group therapy for teens

Cover says what sets it apart is the underlying technology it employs, which has been exclusively licensed from NASA’s Jet Propulsion Laboratory.

A new startup from Figure’s founder is licensing NASA tech in a bid to curb school shootings

Spotify is introducing a new “Basic” streaming plan in the United States, the company announced on Friday. The new plan costs $10.99 per month and includes all of the benefits…

Spotify launches a new Basic streaming plan in the US

Photographers say the social media giant is applying a ‘Made with AI’ label to photos they took, causing confusion for users.

Meta is tagging real photos as ‘Made with AI,’ say photographers

Website building platform Squarespace is selling Tock, its restaurant reservation service, to American Express in a deal worth $400 million — the exact figure that Squarespace paid for the service…

Squarespace sells restaurant reservation system Tock to American Express for $400M

Featured Article

Change Healthcare confirms ransomware hackers stole medical records on a ‘substantial proportion’ of Americans

The February ransomware attack on UHG-owned Change Healthcare stands as one of the largest-ever known digital thefts of U.S. medical records.

19 hours ago
Change Healthcare confirms ransomware hackers stole medical records on a ‘substantial proportion’ of Americans

Google said today that it globally paused its experiment that aimed to allow new kinds of real-money games on the Play Store, citing the challenges that come with the lack…

Google pauses its experiment to expand real-money games on the Play Store

Venture firms raised $9.3 billion in Q1 according to PitchBook data, which means this year likely won’t match or surpass 2023’s $81.8 billion total. While emerging managers are feeling the…

Kevin Hartz’s A* raises its second oversubscribed fund in three years

Google is making reviews of all your movies, TV shows, books, albums and games visible under one profile page starting June 24, according to an email sent to users last…

Google is making your movie and TV reviews visible under a new profile page

Zepto, an Indian quick commerce startup, has more than doubled its valuation to $3.6 billion in a new funding round of $665 million.

Zepto, a 10-minute delivery app, raises $665M at $3.6B valuation

Speak, the AI-powered language learning app, has raised new money from investors at double its previous valuation.

Language learning app Speak nets $20M, doubles valuation

SpaceX unveiled Starlink Mini, a more portable version of its satellite internet product that is small enough to fit inside a backpack.  Early Starlink customers were invited to purchase the…

SpaceX debuts portable Starlink Mini for $599

Ali Rathod-Papier has stepped down from her role as global head of compliance at corporate card expense management startup Brex to join venture firm Andreessen Horowitz (a16z) as a partner…

Brex’s compliance head has left the fintech startup to join Andreessen Horowitz as a partner

U.S. officials imposed the “first of its kind” ban arguing that Kaspersky threatens U.S. national security because of its links to Russia.

US bans sale of Kaspersky software citing security risk from Russia 

Apple has released Final Cut Pro for iPad 2 and Final Cut Camera, the company announced on Thursday. Both apps were previously announced during the company’s iPad event in May.…

Apple releases Final Cut Pro for iPad 2 and Final Cut Camera

Paris has quickly established itself as a major European center for AI startups, and now another big deal is in the works.

Poolside is raising $400M+ at a $2B valuation to build a supercharged coding co-pilot

The space industry is all abuzz about how SpaceX’s Starship, Blue Origin’s New Glenn, and other heavy-lift rockets will change just about everything. One likely consequence is that spacecraft will…

Gravitics prepares a testing gauntlet for a new generation of giant spacecraft

LTK (formerly LiketoKnow.it and RewardStyle), the influencer shopping app with 40 million monthly users, announced on Thursday the launch of a free direct message tool for creators to instantly share…

Influencer shopping app LTK gets an automatic direct message tool

YouTube appears to be taking a firm stance against Premium subscribers who attempt to use a VPN (virtual private network) to access cheaper subscription prices in other countries. This week,…

YouTube confirms crackdown on VPN users accessing cheaper Premium plans