Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More
As enterprises increasingly harness the power of generative AI, organizations are racing to develop more advanced offerings. An example of this is Lumiere, a space-time diffusion model created by researchers from Google, the Weizmann Institute of Science, and Tel Aviv University for realistic video generation.
The research paper detailing Lumiere has just been published, although the models are not yet available for testing. If made accessible, Lumiere could become a strong contender in the AI video space, currently dominated by companies like Runway, Pika, and Stability AI. The researchers highlight that their model uses a unique approach to synthesize videos with realistic, diverse, and coherent motion, addressing a significant challenge in video synthesis.
Lumiere, which means light, is a video diffusion model that allows users to generate realistic and stylized videos and edit them as needed. Users can input text descriptions to generate corresponding videos or upload still images and add prompts to transform them into dynamic videos. The model also supports features like inpainting, which inserts specific objects into videos using text prompts; Cinemagraph, which adds motion to specific parts of a scene; and stylized generation, which uses the style of one image to create videos.
While features like these are not new and are available from players like Runway and Pika, Lumiere addresses a critical gap. Most existing models handle the additional temporal data dimensions in video generation using a cascaded approach, which often compromises temporal consistency and limits video duration, visual quality, and realistic motion.
Lumiere uses a Space-Time U-Net architecture to generate the entire temporal duration of a video in one pass, resulting in more realistic and coherent motion. The model employs both spatial and temporal down- and up-sampling and leverages a pre-trained text-to-image diffusion model to generate full-frame-rate, low-resolution videos by processing them in multiple space-time scales.
Trained on a dataset of 30 million videos with their text captions, Lumiere can generate 80 frames at 16 fps. The source of this data is unclear at this stage.
In comparisons with models from Pika, Runway, and Stability AI, the researchers noted that while these models produced high-quality frames, their four-second outputs had limited motion, sometimes resulting in near-static clips. ImagenVideo, another model in this space, produced reasonable motion but lagged in quality.
In contrast, Lumiere produces five-second videos with higher motion magnitude while maintaining temporal consistency and overall quality. Surveyed users preferred Lumiere over the competition for text and image-to-video generation.
Although Lumiere shows promise in the rapidly evolving AI video market, it is important to note that it is not yet available for testing. Additionally, the model has certain limitations, such as an inability to generate videos with multiple shots or transitions between scenes, which remains an area for future research.
Stay in the know! Get the latest news in your inbox daily. Subscribe
By subscribing, you agree to VentureBeat’s Terms of Service.