STYLING VIDEO WITH AI
What level of control is possible with Runway Aleph and Seeddance?
Test 1: Convert Trello to an Oil Pastel Style
Products used:
ChatGPT Plus, Seeddance 2.5, After Effects
Cost:
$300 and 50 hours
Details:
Rendering in an oil pastel style invited a lot of variation in color and font, even when I expressly forbade it. The majority of my time was spent rendering stills one at a time, then adjusting colors using Lumetri and After Effects plugins. It also took many tries to find the precise prompt wording needed to minimize errors.
In the end, creating 8-second snippets with 3 reference stills provided the best results.
Test 2: Convert Target to a Sumi-E Style
Products used:
ChatGPT Plus, Seeddance 2.5, After Effects
Cost:
$200 and 30 hours
Details:
Despite my best efforts I couldn’t get the fast-moving 3D landscape lines or circles to inherit the sumi e style whilst moving accurately. I also was unable to get the fish at 3 seconds to behave as it does in the original video - I had to comp it in after.
The renderer sometimes produced understandable but strange results: paint dots that moved uniformly, speed lines, and objects that bled ink. Achieving uniformity was a more due to persistence than than skill.
Note: I kept the Target logo unstylized intentionally.
Test 3: Convert Kitchen Animals to a Claymation Style
Products used:
ChatGPT Plus, Runway Aleph 2.0, After Effects
Cost: $100 and 20 hours
Details:
Runway Aleph is cheap and fast, but drifts easily and only takes one reference image at a time. I would “ride” the drift by making a 5 second video segment, then using a frame from that segment as the reference image for the next one. I moved between After Effects and Aleph a lot. Runway Aleph is very weak when it comes to big, sudden movements, and would not have worked for the other two videos.
Claymation is probably the easiest style for an AI renderer to get right. I would recommend it for any organization or person trying to take their first steps with the workflow.
Workflow Instructions
Collect stills from the original video. Collecting additional stills and video without background, text or logos is often useful.
Stylize stills uniformly using ChatGPT Plus (medium thinking). Achieving 100% accuracy at this stage is everything and will save you time further down the line. Use previously stylized frames as reference, and utilize After Effects / Photoshop / Lumetri to fix what ChatGPT won’t. Key phrasing for ChatGPT: “treat these like they are separate frames of the same animation.”
Use the original video and edited stills to make the new video, section by section. Make sure to relate the names of the stills to their place in the video timecode. Render in draft mode / 480p until you get the results you want. Sometimes it helps to remove transitional stills.
For areas that the AI consistently misses, render zoomed-in or isolated sections of the original video and enact the workflow for just those sections.
Use After Effects to stitch snippets together, minimize timing and other inconsistencies, and reintroduce logos/text. Asking AI for specific Lumetri Color settings can help match videos together.
Workflow Challenges
The amount of time and money needed to get the desired result is difficult to predict.
Both engines are poor at responding to notes. You’re better off starting a new chat with a more detailed, possibly AI-assisted prompt and updated reference material.
Content that moves subtly or quickly may get skipped over by the AI.
Logos / typography get garbled. Rendering blanks and compositing afterwards can help.
AI content generation sometimes conflicts with a policy. That includes content involving public figures, stock footage, legal advice, children, violence, or sensitive information.