KD-CVG: A Knowledge-Driven Approach for Creative Video Generation


Anonymous Submission
Code (coming soon) Data (coming soon)

Abstract

Creative Generation (CG) leverages generative models to automatically produce advertising content that highlights product features, and it has been a significant focus of recent research. However, while CG has advanced considerably, most efforts have concentrated on generating advertising text and images, leaving Creative Video Generation (CVG) relatively underexplored. This gap is largely due to two major challenges faced by Text-to-Video (T2V) models: (a) ambiguous semantic alignment, where models struggle to accurately correlate product selling points with creative video content, and (b) inadequate motion adaptability, resulting in unrealistic movements and distortions. To address these challenges, we develop a comprehensive Advertising Creative Knowledge Base (ACKB) as a foundational resource and propose a knowledge-driven approach (KD-CVG) to overcome the knowledge limitations of existing models. KD-CVG consists of two primary modules: Semantic-Aware Retrieval (SAR) and Multimodal Knowledge Reference (MKR). SAR utilizes the semantic awareness of graph attention networks and reinforcement learning feedback to enhance the model's comprehension of the connections between selling points and creative videos. Building on this, MKR incorporates semantic and motion priors into the T2V model to address existing knowledge gaps. Extensive experiments have demonstrated KD-CVG's superior performance in achieving semantic alignment and motion adaptability, validating its effectiveness over other state-of-the-art methods. The code and dataset will be open source at https://kdcvg.github.io/KDCVG/.

Advertising Creative Knowledge Base         

Pipeline

Overview of the two core modules in the KD-CVG framework: The SAR module uses reinforcement learning to train a SC-GAT that perceives semantic relationships between selling point texts. It extracts content from the ACKB that is highly relevant to the current selling points, serving as a knowledge reference for the LLM. This helps in understanding the deep meaning of the selling points and the transformation rules between selling points and general scripts. The MKR module utilizes information retrieved by SAR to extract multimodal priors. Semantic priors enhance the generation of creative scripts, while motion priors are distilled into the T2V model, ultimately generating realistic and diverse ACVs that highlight the selling points provided by the user.

Comparison with other methods         

KD-CVG significantly improved movement regularity.

Ablation Studies         

Reference content and script generation for SAR.

Input Selling Points:
Diverse Fruit Aroma, Babies Love It at First Use.
Reference Script:
A stream of water is poured into the glass, causing the tea leaves to swirl around in the liquid.
Generation Script:
A refreshing blend of fruit juices being poured into a glass, filling the air with a sweet aroma that instantly delights babies.

Input Selling Points:
Cool and Refreshing.
Reference Script:
Two women are holdingglasses of wine and drinking.
Generation Script:
Two women are holding glasses of chilled lemonade, sipping and enjoying the cool, refreshing taste on a sunny day.

Input Selling Points:
Herbal Fragrance, Fresh Breath
Reference Script:
A person's hand is disturbing the water, causing ripples and splashes.
Generation Script:
A person's hand is gently stirring a cup of herbal tea, causing ripplesthat release a calming fragrance, leaving a refreshing sensation with every splash.

Importance of MR-LoRA in MKR.

More Qualitative Results of KD-CVG