Cooking Up Narrative Consistency for Long Video Generation

The latest public launch of the Hunyuan Video generative AI mannequin has intensified ongoing discussions concerning the potential of huge multimodal vision-language fashions to someday create whole motion pictures.

Nonetheless, as we’ve got noticed, this can be a very distant prospect in the intervening time, for quite a few causes. One is the very brief consideration window of most AI video turbines, which wrestle to keep up consistency even in a brief single shot, not to mention a collection of photographs.

One other is that constant references to video content material (akin to explorable environments, which shouldn’t change randomly for those who retrace your steps via them) can solely be achieved in diffusion fashions by customization methods akin to low-rank adaptation (LoRA), which limits the out-of-the-box capabilities of basis fashions.

Subsequently the evolution of generative video appears set to stall except new approaches to narrative continuity are developed.

Recipe for Continuity

With this in thoughts, a brand new collaboration between the US and China has proposed the usage of tutorial cooking movies as a potential template for future narrative continuity programs.

Click on to play. The VideoAuteur challenge systematizes the evaluation of components of a cooking course of, to supply a finely-captioned new dataset and an orchestration methodology for the era of cooking movies. Consult with supply website for higher decision.  Supply: https://videoauteur.github.io/

Titled VideoAuteur, the work proposes a two-stage pipeline to generate tutorial cooking movies utilizing cohered states combining keyframes and captions, reaching state-of-the-art leads to – admittedly – an under-subscribed house.

VideoAuteur’s challenge web page additionally contains quite a few quite extra attention-grabbing movies that use the identical method, akin to a proposed trailer for a (non-existent) Marvel/DC crossover:

Click on to play. Two superheroes from alternate universes come nose to nose in a faux trailer from VideoAuteur. Consult with supply website for higher decision.

The web page additionally options similarly-styled promo movies for an equally non-existent Netflix animal collection and a Tesla automobile advert.

In growing VideoAuteur, the authors experimented with numerous loss features, and different novel approaches. To develop a recipe how-to era workflow, additionally they curated CookGen, the most important dataset centered on the cooking area, that includes 200, 000 video clips with a mean period of 9.5 seconds.

At a mean of 768.3 phrases per video, CookGen is comfortably probably the most extensively-annotated dataset of its variety. Various imaginative and prescient/language fashions have been used, amongst different approaches, to make sure that descriptions have been as detailed, related and correct as potential.

Cooking movies have been chosen as a result of cooking instruction walk-throughs have a structured and unambiguous narrative, making annotation and analysis a better job. Apart from pornographic movies (more likely to enter this specific house sooner quite than later), it’s tough to consider another style fairly as visually and narratively ‘formulaic’.

The authors state:

‘Our proposed two-stage auto-regressive pipeline, which features a lengthy narrative director and visual-conditioned video era, demonstrates promising enhancements in semantic consistency and visible constancy in generated lengthy narrative movies.

By experiments on our dataset, we observe enhancements in spatial and temporal coherence throughout video sequences.

‘We hope our work can facilitate additional analysis in lengthy narrative video era.’

The brand new work is titled VideoAuteur: In direction of Lengthy Narrative Video Era, and comes from eight authors throughout Johns Hopkins College, ByteDance, and ByteDance Seed.

Dataset Curation

To develop CookGen, which powers a two-stage generative system for producing AI cooking movies, the authors used materials from the YouCook and HowTo100M collections. The authors examine the size of CookGen to earlier datasets centered on narrative improvement in generative video, such because the Flintstones dataset, the Pororo cartoon dataset, StoryGen, Tencent’s StoryStream, and VIST.

CookGen focuses on real-world narratives, notably procedural actions like cooking, providing clearer and easier-to-annotate tales in comparison with image-based comedian datasets. It exceeds the most important present dataset, StoryStream, with 150x extra frames and 5x denser textual descriptions.

The researchers fine-tuned a captioning mannequin utilizing the methodology of LLaVA-NeXT as a base. The automated speech recognition (ASR) pseudo-labels obtained for HowTo100M have been used as ‘actions’ for every video, after which refined additional by giant language fashions (LLMs).

As an example, ChatGPT-4o was used to supply a caption dataset, and was requested to give attention to subject-object interactions (akin to fingers dealing with utensils and meals), object attributes, and temporal dynamics.

Since ASR scripts are more likely to comprise inaccuracies and to be usually ‘noisy’, Intersection-over-Union (IoU) was used as a metric to measure how intently the captions conformed to the part of the video they have been addressing. The authors word that this was essential for the creation of narrative consistency.

The curated clips have been evaluated utilizing Fréchet Video Distance (FVD), which measures the disparity between floor reality (actual world) examples and generated examples, each with and with out floor reality keyframes, arriving at a performative end result:

Moreover, the clips have been rated each by GPT-4o, and 6 human annotators, following LLaVA-Hound‘s definition of ‘hallucination’ (i.e., the capability of a mannequin to invent spurious content material).

The researchers in contrast the standard of the captions to the Qwen2-VL-72B assortment, acquiring a barely improved rating.

Methodology

VideoAuteur’s generative section is split between the Lengthy Narrative Director (LND) and the visual-conditioned video era mannequin (VCVGM).

LND generates a sequence of visible embeddings or keyframes that characterize the narrative movement, just like ‘important highlights’. The VCVGM generates video clips primarily based on these decisions.

The authors extensively talk about the differing deserves of an interleaved image-text director and a language-centric keyframe director, and conclude that the previous is the simpler method.

The interleaved image-text director generates a sequence by interleaving textual content tokens and visible embeddings, utilizing an auto-regressive mannequin to foretell the subsequent token, primarily based on the mixed context of each textual content and pictures. This ensures a good alignment between visuals and textual content.

In contrast, the language-centric keyframe director synthesizes keyframes utilizing a text-conditioned diffusion mannequin primarily based solely on captions, with out incorporating visible embeddings into the era course of.

The researchers discovered that whereas the language-centric methodology generates visually interesting keyframes, it lacks consistency throughout frames, arguing that the interleaved methodology achieves increased scores in realism and visible consistency. Additionally they discovered that this methodology was higher in a position to study a practical visible fashion via coaching, although generally with some repetitive or noisy components.

Unusually, in a analysis strand dominated by the co-opting of Steady Diffusion and Flux into workflows, the authors used Tencent’s SEED-X 7B-parameter multi-modal LLM basis mannequin for his or her generative pipeline (although this mannequin does leverage Stability.ai’s SDXL launch of Steady Diffusion for a restricted a part of its structure).

The authors state:

‘Not like the traditional Picture-to-Video (I2V) pipeline that makes use of a picture because the beginning body, our method leverages [regressed visual latents] as steady situations all through the [sequence].

‘Moreover, we enhance the robustness and high quality of the generated movies by adapting the mannequin to deal with noisy visible embeddings, for the reason that regressed visible latents might not be good as a result of regression errors.’

Although typical visual-conditioned generative pipelines of this sort typically use preliminary keyframes as a place to begin for mannequin steering, VideoAuteur expands on this paradigm by producing multi-part visible states in a semantically coherent latent house, avoiding the potential bias of basing additional era solely on ‘beginning frames’.

Exams

In keeping with the strategies of SeedStory, the researchers use SEED-X to use LoRA fine-tuning on their narrative dataset, enigmatically describing the end result as a ‘Sora-like mannequin’, pre-trained on large-scale video/textual content couplings, and able to accepting each visible and textual content prompts and situations.

32,000 narrative movies have been used for mannequin improvement, with 1,000 held apart as validation samples. The movies have been cropped to 448 pixels on the brief facet after which center-cropped to 448x448px.

For coaching, the narrative era was evaluated totally on the YouCook2 validation set. The Howto100M set was used for knowledge high quality analysis and in addition for image-to-video era.

For visible conditioning loss, the authors used diffusion loss from DiT and a 2024 work primarily based round Steady Diffusion.

To show their competition that interleaving is a superior method, the authors pitted VideoAuteur in opposition to a number of strategies that rely solely on text-based enter: EMU-2, SEED-X, SDXL and FLUX.1-schnell (FLUX.1-s).

The authors state:

‘The language-centric method utilizing text-to-image fashions produces visually interesting keyframes however suffers from a scarcity of consistency throughout frames as a result of restricted mutual info. In distinction, the interleaved era methodology leverages language-aligned visible latents, reaching a practical visible fashion via coaching.

‘Nonetheless, it sometimes generates photos with repetitive or noisy components, because the auto-regressive mannequin struggles to create correct embeddings in a single move.’

Human analysis additional confirms the authors’ competition concerning the improved efficiency of the interleaved method, with interleaved strategies reaching the best scores in a survey.

Nonetheless we word that language-centric approaches obtain the very best aesthetic scores. The authors contend, nevertheless, that this isn’t the central concern within the era of lengthy narrative movies.

Click on to play. Segments generated for a pizza-building video, by VideoAuteur.

Conclusion

The most well-liked strand of analysis in regard to this problem, i.e., narrative consistency in long-form video era, is anxious with single photos. Initiatives of this sort embrace DreamStory, StoryDiffusion, TheaterGen and NVIDIA’s ConsiStory.

In a way, VideoAuteur additionally falls into this ‘static’ class, because it makes use of seed photos from which clip-sections are generated. Nonetheless, the interleaving of video and semantic content material brings the method a step nearer to a sensible pipeline.

First printed Thursday, January 16, 2025