Name: Using Text to Video and Image to Video
Uploaded: 2024-12-10T16:04:10.991Z
Description: Learn how to generate videos using text prompts and images in Runway.

Use this quick tutorial to learn how to use Runway's mobile app for iOS.

- Download the Runway app from the App Store app
- Sign into Runway using the same account you use on the web
- Circular purple button will initiate a new generation
- 1 second of video generation = 14 credits
    - To top up credits, click profile icon in the top right corner of the Home page and tap “Credits”
- To create a video, simply hold down the record button and let go when you’re finished
- Three main style settings:
    - Presets by Runway
    - Reference image: click “Add” to choose an image from your library; adjust style strength and seed number here
    - Text prompt
- Hit “Generate” — video will be queued and show a percentage status circle
- Generate new options in the meantime; videos will continue generating in the background
- Generations will be stored in library tab


Using the Mobile App

Runway Mobile

Transform a low resolution image into a high resolution image with Upscale Image.

- Use Upscale Image to add more detail to increase image resolution
- Upload an image asset
- Select desired size under “Scale” and click “Process”
- That’s all there is to it!
- Download the image or add to your assets by hovering over it and clicking the icons on the top right


Upscale Images

Learn how to use AI Tool Erase and Replace to easily modify images.

- Head to AI Tools to find Erase and Replace
- Choose an image asset from your computer or asset library
- Adjust brush size using the slider on the right sidebar
- Brush over the portion of the image you’d like to replace
- Enter a text prompt and click “Replace”
- “Add to assets” to add to your Runway asset library
- “Download” to download locally to your computer
- Use “Clear” to generate something new with a new prompt


Erase and Replace

Learn to create a bespoke image sequence using Image to Image.

- Remove the background of your video with Green Screen and save as a PNG sequence for the most flexibility down the road
- Open Image to Image and use first frame in the sequence as a test frame
    - Describe the entire scene, touch on style that you want
    - Make sure the look is right — you’ll be committing to it in this workflow
    - Lock seed number and don’t change the prompt for images you like
    - Save to assets to make sure you can find them later
- Select every few frames if you don’t want to select them all
- String the frames together in the video editor


Image to Image

Use this tutorial to learn how to use the Text to Image AI Tool in Runway.

- Head to AI Tools to find Text to Image
- Enter a simple text prompt
- Open up “Advanced” under “Settings” sidebar to explore refinement options
    - Size, style, medium, mood, prompt weight
- Prompt weight will determine how much the text prompt is taken into consideration; higher number = most consideration, lower number = least
- Experiment with more complex prompts
- “Add to assets” to save to your Runway asset library
- “Download” to download locally to your computer


Text to Image

Bring still images to life and learn how to create a living portrait in Runway.

Step 1: Brush out and replace the parts of your image you'd like to change using either Erase & Replace or Infinite Image.
Step 2: Head over to Frame Interpolation to animate your various images into a video.
Step 3: Behold the beauty of your new living image.


Create a Living Portrait

Learn how to seamlessly transition images into one another by using our Frame Interpolation AI Tool.

- Frame Interpolation works well to create videos with subtle movement from still images
- The more frames you have to work with, the better
- Keep the differences between photos subtle if possible for realistic effect
- Try using Erase and Replace to create new images to interpolate between!


Frame Interpolation

Learn to use the Backdrop Remix AI Tool, and remix your images into anything you want them to be.

- Backdrop Remix changes the background of your photos while keeping the subject in tact
- Upload your photo, and choose a preset or a custom prompt
- Click “Generate,” and Runway will get to work replacing the background automatically — that’s it!


Backdrop Remix

Use this lesson to learn tips and tricks for using Text to Color Grade.

- The main thing to keep in mind: you can never go wrong with natural language.
- Typing in a movie title alone might be counterproductive sometimes; add in a description to make the movie title interpretation less literal.
- Same goes for specific film stocks — add additional descriptors.
- Try different seed numbers for different options.


Text to Color Grade

- Find Infinite Image in the AI Tools section. Start with a text prompt to generate an original image, or upload your own image.
- Move the generation frame to desired area. Make sure it’s overlapping the original image!
- Be descriptive with your text prompt and click “Generate.” You’ll then be presented with 4 initial generated options to choose from.
- Repeat this process as many times as you’d like to continue extending your image.
- Use adjustment button to adjust prompt weight; lower for more surprising and creative results, higher to stay more true to the prompt.
- Try opening your final image in Erase and Replace to clean up any elements in your image you’d like to change.


Infinite Image

Learn how to create generative images using various AI Tools.

- Learn how to create a generative image using AI Tools
- AI Tools used:
    - Text to Image: creates an original generated image from a text-based prompt
    - Erase and Replace: eliminates a specific part of an image and replaces it seamlessly with a new image generated from a new text based prompt
    - Image to Image: uses an image as the input, and generates a brand new image based on both the original and a descriptive text based prompt
- When writing prompts, be descriptive. Include object colors, orientations, and artistic styles in your prompt.
- Refine as you go — the initial output does not have to be perfect. You can slowly work and refine your image with tools like Erase and Replace and Image to Image to gradually get your content to where you want it.


Generative Image Workflow

AI Tools

Learn how to combine Gen-1 with Green Screen and additional techniques to create a dynamic edit.

- Main idea: take multiple sources of footage and combine them into one shot
- Start by using Gen-1 to style a source video
- Once you’ve landed on a generation you like, bring it into Remove Background to isolate the subject
- Export the mask into a transparent ProRes file
- Start adjusting layers independently using scale and effects
- Experiment with speed ramps for exciting motion
- Export


Gen-1 Workflow w/ Green Screen

Learn how to create new worlds to jump through with Gen-1.

- This tutorial is inspired by Martin Haerlin’s viral Twitter post
- Record a video of yourself walking through a space, snapping every time you want to transform yourself
- Cut up your video so that each snap becomes its own file
- Use Text to Image or Image to Image to generate some scenes and characters
    - Add images to you assets
- Open Gen-1 and click on one of your saved images
- Use Advanced Settings to fine-tune your generations
- Preview & Generate
- Put your clips together and add sound design to create an ambient effect


Gen-1 World Jumping

Turn simple comps into compelling videos with Gen-1.

- Gen-1 source video doesn’t have to be a “normal” straight out of camera video; try creating a rough composite instead!
- Compositing = combining multiple visual elements for a cohesive final output
- Take rough mockups into Gen-1 with an image prompt for surprising results


Using Composites in Gen-1

Learn how to take your Gen-1 workflow to the next level by combining multiple AI Tools.

- Head to Text to Image in AI Tools to generate images to use as a reference image in Gen-1
- Edit the generated image in Erase and Replace or iterate in Image to Image
- Use Inpainting to remove subject from the scene, then use Green Screen to extract the subject by itself, for complete foreground and background separation
    - Affect the background with one set of prompts, and affect the foreground with another
    - Combine them back together for unique results!


Gen-1 Workflow w/ AI Tools

Looking for Gen-1 inspiration? Use this tutorial and experiment with unexpected results.

- Upload the video you want to use as your driving source
- Incorporate a prompt that is wildly different or opposite from the source video
- Open “Advanced” in Gen-1
    - Turn “Structural consistency” to 6
    - Turn “Weight” to 4 or 5
    - Turn “Frame consistency” to 1.2
- Click “Generate” and expect the unexpected


Unexpected Results in Gen-1

Use Gen-1 to turn your videos into animated sketchbooks.

- Upload you video and open Gen-1
- Find or generate a reference image that has a hand-drawn, sketch-type feel to it
    - It doesn’t need to resemble the source video; the art style is most important
- Open “Advanced”
    - Turn “Structural consistency” to 0
    - Turn “Weight” to 5 or 6
    - Turn “Frame consistency” to .6
- Click “Generate”


Sketchbook Look with Gen-1

Keep your Gen-1 results sharp and polished with this tutorial.

- Use this tutorial to get your Gen-1 results to be clean and as close to the original video as possible
- Upload the video to your Runway account, and then open it in Gen-1
- If you’ll be using a reference image, it’s helpful to use a tool like Image to Image to generate the look you’re going for, so the image and videos line up 1:1
- Open “Advanced” in Gen-1
    - Turn “Structural consistency” to 0
    - Turn “Weight” to a lower number (between 3-7 works well)
    - Turn “Frame consistency” to 1.2
- Click “Generate”


Clean Results with Gen-1

Learn the basics of using Gen-1 in Runway.

- Head into AI Tools in Runway to find Gen-1
- Select an asset
- Three style reference options:
    - Image (choose your own image)
    - Presets by Runway
    - Prompt (enter a text prompt)
- Keep settings simple or dive into advanced settings: https://help.runwayml.com/hc/en-us/articles/15161225169171-Gen-1-Gen-2-Advanced-Settings
- Find recommendations for settings by hovering over tool tips


Gen-1 Basics

Gen-1

Learn about workspaces and how they function within Runway.

- When you first sign up for a Runway account, you’ll be assigned a personal workspace, which can be viewed by clicking on the top left icon on the Dashboard
- Invite other members to your workspace to turn it into a team workspace
    - Note: you will be charged on a per person basis for each additional member
- If you’ve been invited to and accepted to a team, you can toggle between teams/workspaces using the top left icon on the Dashboard
- Make sure you are using the correct team workspace since each workspace’s plans and credits may differ


Navigating Workspaces

Learn about credits and how they’re used in Runway.

- Runway uses a credit system for generative tools and custom model training
- Your plan tier will determine how many credits are in your account
- To see plan tier and credit balance, head to “Workspace Settings” > “Plans & Billing”
- Only account owners and admins can see these settings
- 1 credit = $.01
    - Image generation: 5 credits per image
    - Gen-1: 14 credits per second
    - Gen-2: 5 credits per second
    - Custom model: 1000 credits per model
- For individual plans, credit balance will refresh every month
    - These allotments do not carry over into the following month(s)
- Additional credits may be purchased at any time
    - Purchased credits do carry over into the following month(s)
- For Enterprise plans, credit balance will refresh every month, but everyone on the team is taking credits from the same pool
    - Each Enterprise team has their own unique credit allotment
    - More credits may also be purchased on Enterprise plans if needed


Generation Credits

Learn how to add, organize and share assets in Runway.

- Navigate to “Assets” and click “Upload”
- Media is labelled by type for easy sorting later
- Folders will automatically be created after using certain AI Tools, like Gen-1 or Gen-2
- View all recent exports, or toggle between “Private” and “Shared” at the top of your Assets screen
- If you’d like to share assets with other members of your team, select the asset(s), right click, and “Move to Team”
- You can always move assets back to “Private”
- To organize assets, you can sort by many different types
- Hold down Shift + click for batch actions, like organizing or deleting


Asset Management

Learn how to navigate the Runway Dashboard.

- From the main dashboard homepage, you have access to featured AI Tools, popular AI Tools, tutorials, and community content
- If you click on “view all tools” button, you’ll see the full list of the AI Tools
- Or, you can use the categorized buttons on the left side
- “Projects” is where you’ll find all of your saved Runway projects, including editing timelines, Green Screen clips, and Inpainting clips
- “Shared with me” contains projects that you’ve been specifically invited to
- “Assets” is where all of your uploaded and creative assets will be stored
    - Sort and organize assets with folders and categories
    - Sort assets by type for easier navigation
    - Shift + click to batch organize or delete assets
    - By default, all assets that you create or upload will appear in your “Private” assets
- Dashboard may change slightly over time


Dashboard Overview

- Click on the “Invite Collaborators” button from the primary account holder’s dashboard
- Enter the emails of the team members you’d like to invite
- Choose from view access, editing access, or admin access
    - can view: can view and download assets, but can’t utilize AI Tools
    - can edit: can access all AI Tools, use credits, and manage projects and assets
    - admin: can adjust workspace roles, adjust billing information, and edit/view assets
- Typical team: 1-2 admins, plus editors
- To change name/profile image, head to “My Workspace Settings”


How to Create a Team

Learn how to join a team/workspace in Runway.

- Once you’ve been invited to a team workspace, you will get an invite to your email inbox
- Log in using Google, Apple, or sign up with Runway
- Team workspace is indicated by top left name and icon on your dashboard


How to Join a Team

Getting Started

Learn how to train a Style Generator in Runway for a cohesive look across multiple generations.

- Head to “AI Training” in the left-hand panel of your dashboard and select “Train a Style Generator”
- Collect 15-30 images
    - in this example, we’re using generated images that have been color corrected for a unified feel
- Upload the images into the Style Generator
- Once the training is finished, the model will be available for use in Text to Image and Image to Image
- When prompting with your styles, nothing elaborate is required; try out simple prompts for great results
- To recap:
    - Curate an aesthetically cohesive set of 15-30 images
    - Upload images to the Style Generator
    - Apply style with simple prompts in other Runway tools


Training Custom Style Generators

Learn how to continue using your model in Text to Image for hyper-customized images.

- Continue using your model in other mediums, like Text to Image
- Important tip to keep in mind: include the name of your model in the text prompt in addition to selecting the model from the dropdown
- Add in-depth description to fine-tune results


Using Your Model

AI Training

Learn how to use Generative Audio to add speech to your images and videos.

- Find Generative Audio on the Runway dashboard
- At the top, you’ll find an area to write anything you want and turn it into a spoken audio file
- After you type out your text, you can preview and choose voices from the default voice menu
- Once you choose a voice, click on the “Generate” button
- Generate times depend on script length, but are generally pretty quick
- Audio generations are automatically saved to the “Generative Audio” folder inside of the main assets folder, but can be saved elsewhere
- Custom Voice training only requires a few minutes of clean audio with no background noise — import or record in Runway
- Finally, to create a Lip Sync video, you’ll need a video or image of a person whose face is fully in the frame
    - upload your own or use one of the characters in the list
    - Lip Sync can be used with generated audio from Text to Speech, recorded audio or uploaded audio
    - choose a voice and click “Generate”
- Tip: turn your image into a video with Gen-2, then upload that video into the Generative Audio tool to add Lip Sync to your video
- If your audio is longer than your video, once your video reaches the end of its duration, it will reverse and go back to the beginning for the duration of the audio
    - avoid using general camera motion controls and instead use Motion Brush, which makes the reverse effect less noticeable


Generative Audio Overview

Audio

Learn how to generate videos using text prompts and images in Runway.

- Select the Text/Image to Video tool from your dashboard
- Start by adding a text prompt for a video you want to create
- Next, click on the “Style” icon to add a style to your prompt, or use a custom style with the “Model” icon
- Choose an aspect ratio for your video
- Click “Generate” to generate a video based on your settings and prompt
- Use “Free previews” to see the what the first frame of your generations will look like, without spending credits, and then use the image as input to continue the creation process
- General Motion slider ranges from 1-10
    - 1 for little motion, 10 for extreme motion
- For more control over the camera and subjects in your video, you can use some of the other settings
    - Camera Motion: choose direction and speed of camera motion in your video
    - Motion Brush: select up to 5 different parts of your frame, and assign specific motion values to each of them
- Once you’re happy with your result, you can favorite it for easy access later, download it locally to your computer, or extend the video to make it longer
- Extending a video will automatically generate and stitch an additional 4s onto the end of your video, continuing the motion from the first half automatically, but motion can also be adjusted
- Check out custom presets, prompt enhancer and Lip Sync within Text/Image to Video to help speed up your generative workflows even more


Using Text to Video and Image to Video

Learn how to use Multi Motion Brush to create life-like movement across varying depth layers of a video.

- First, determine what would be realistic movement
- Envision your video in “slices,” each representing a different depth/distance
    - objects closer are in the front slices, and objects farther away are in the back slices
    - closer objects will appear to move faster than those further away — this concept is called Motion Parallax
- Head into Gen-2 and begin with the closest layer
    - intensify movement given “close” proximity
- Move onto Brush 2 for your next layer — apply reduced motion since the layer is farther back in proximity
- Keep continuing through layers, reducing motion as you reach the farther “slices”/layers
- As a recap
    - visualize image in layers based on distance
    - assign brushes to different layers
    - apply more motion to closer layers, with intensity decreasing for farther layers


How to Create Realistic Motion with Multi Motion Brush

Learn to generate expressions using Multi Motion Brush in Gen-2.

- Create a character using Gen-2
- Use Multi Motion Brush to draw various masks on the character’s face
    - eyebrows, eyes, and mouth positions work best
    - experiment with different directions for each brush, taking into account the expression you are going for in the output


How to Generate Expressions

Learn how to use Multi Motion Brush for precise movement control.

- Open Gen-2 and start with an image prompt
- Click the “Motion Brush” button
    - You now have access to five different brushes on the Motion Canvas that can be independently controlled
- All brush strokes will be visible on the canvas, and beside each brush number, you’ll see icons representative of how the brush is affecting movement
- Different brush strokes cannot be combined — brushing over another stroke will replace it on the canvas
- Using the eraser tools will erase _all_ brush strokes in the erased area
- To adjust values of any of the brushes, click on the brush number button


How to Use Multi Motion Brush

Learn to generate film assets with Gen-2.

- Open Gen-2 and generate a desired scene
- Use Gen-2 to then generate a texture or element like a light flare, dust or light leak
- Overlay the texture element over your scene in a timeline editor, and adjust the blend mode to match your original scene
    - modes like "lighten" and "soft light" work well
- Adjust the opacity of the top texture layer to control the intensity of the added element


Generating Film Assets

Learn how to master prompting with these tips in Gen-2.

- Open up Gen-2 and enter an extremely simple prompt (in this example, a dog)
- Use the preview functionality to get a feel for what your prompt will generally generate
- From there, get more specific with details
    - For this example: color, breed, location, time of day
- Taking it a step further, include details about the type of shot you’re going for
    - For this example: adding “an extreme close-up shallow depth of field portrait” to the prompt
- Two things to focus on: what it looks like + what it is
    - Example: a painting of (+) a bowl of fruit
    - Example with detail added: a textured historical oil painting of (+) a bowl of fruit on a wooden table in a farmhouse kitchen
- Find Style Presets at the bottom of the prompting box in Gen-2 if you’re looking for a specific style
- Use prompt modifiers at the end of a prompt for further customization
- Examples include:
        - cinematic
        - shallow depth of field
        - warm color palette
        - moody
        - smooth texture
    - imperfect
- In summary, focus on:
    - What it looks like
    - What it is
    - Relevant modifiers
    - Keep it focused


Prompting for Text to Video

Learn how to create text animations with images and words in Gen-2.

- Start by creating an image of text on a green background
- Bring the image into Gen-2 using Image + Description mode
- Add a prompt to describe what you want the letters to do and turn the General Motion slider up
- When you find an animation that works well, take note of the seed value
- Take the seed value and reuse it for new generations while changing up the prompt
- Seeds that worked for us:
    - 3479942342
    - 1887763640
    - 442533868
- The last thing to do is key out the green background, and overlay the text animation over some generated or live action footage


Gen-2 Text Animation

Learn how to smoothly composite multiple Gen-2 videos into one scene.

- Begin by generating videos in Gen-2 — for example, one video featuring a foreground subject and one featuring your desired background for the final comp
- Use Motion Brush and Camera Motion to customize the movement in your generations
- Use Remove Background (found under “Edit Videos” on left-hand dashboard panel) to isolate your foreground subject and export
- Layer the isolated subject video on top of your generated background video in a timeline editor
- Style further for a more seamless and cohesive composite by blurring the background and color grading


Gen-2 Compositing Workflow

Learn how to create cohesive scenes with Image to Image and Gen-2.

- Collect a series of images
- Use Image to Image to transform each image using **Style** and **Prompt** options. Maintain the same options for each image to create a cohesive style throughout
- Open up Gen-2 and use Image to Video mode
- Adjust Camera Motion to your liking, and generate videos from your images
- Stitch them together for a connected scene


Composing Scenes with Photos

Learn how to generate match cuts using Text to Image, Image to Image and Gen-2.

- Generate an image using Text to Image
- Open up Image to Image, and change the subject of your prompt. For example, change your prompt from "a woman staring intensely at the camera" in Text to Image, to "a man standing in the street."
- Continue iterating until you've generated you desired amount of images with similar composition
- Open Gen-2, and head to image to video mode. Input one of your images, and use Motion Brush and Camera Motion to add a specific movements along with your generation. Repeat this step for the rest of your images, ensuring to apply the same movements to all
- Stitch them together for seamless match cuts


How to Generate Match Cuts

Learn how to convey emotion through your generations by viewing these examples.

- Generate an image using Text to Image
- Open up Gen-2 and use Director Mode to adjust Camera Control and Motion Brush values
- Use Image + Description mode to provide additional context for your generations with text, like prompting for a specific emotion or character action


How to Generate Emotion

Learn how to add camera movements to your Motion Brush outputs.

- Motion Brush and Camera Control work independently from one another, allowing for more control in your generations
- If you want the camera to move to the right, pop into Motion Brush, add mask on the desired area, adjust values, and save
- Then, open Camera Motion to add horizontal movement and save
- Experiment moving the camera in different directions; the Motion Brush movement will remain the same
- Try experimenting with abstract images for mesmerizing and intricate generations, or multiple masks to control movement over different areas


Motion Brush + Camera Control

Learn brush and movement techniques using Motion Brush.

- For this first example, we’ll make our subject’s head move up
- You’ll want to brush a mask over the subject’s entire head, but you may see some background motion in your output if you happen to brush over the background while making your mask
- Try a smaller mask on the face, since Motion Brush understands what you’re trying to move
- If you’re trying to make a character’s clothes blow in the wind, create a small mask that doesn’t touch the background or other parts of the character
- Try this same mechanic with hair, eye-line, facial expressions and timelapse elements like moving clouds
- If you want even more specific control, try using Motion Brush in combination with Image + Description mode
    - Provide simple prompts for this method, don’t over-complicate


Motion Brush Techniques

Learn how to use Motion Brush to have more control over movement in your video.

- Motion Brush works with image prompts inside of Gen-2
- Once your image is loaded into Gen-2, click on the Motion Brush button to get started
- Brush over the area of the image you want to control
- If you want to make your brush size bigger, use the slider in the upper left
- If you accidentally paint too big of an area, use the eraser tool or undo/redo arrows
- Once you have a basic mask, head down to the Horizontal, Vertical and Proximity sliders
- You can think of these as an X, Y and Z grid
        - X = horizontal movement, left or right
        - Y = vertical movement, up or down
    - Z = proximity, closer or farther
- Each slider is controlled with a decimal point value with a range from -10 to +10
- You can manually input numerical value, drag the text field left or right or use the sliders
- If you need to reset everything, click the “Clear” button to reset everything back to 0
- Click “Save” once you’re done adjusting your mask


Gen-2 Motion Brush Interface

- Open Gen-2 and add an image prompt
    - If you want to use Text to Video, make sure you generate previews, then use one of the preview images as your image input
- Click on Motion Brush
- Paint over the area you want to control, and assign a motion direction and speed to your selection
- Control speed for Horizontal, Vertical, and Proximity independently
- Motion Brush works independently from Camera Motion, so feel free to experiment with both in tandem


Gen-2 Motion Brush

Learn how to create dynamic scenes by replicating a three camera setup in Runway.

- Today we’ll recreate a three camera setup, which typically comprises of a wide shot, a medium shot, and a close up
- Open Text to Image, and generate a wide shot
- Take note of seed numbers — it may be helpful to jot it down for consistency between shots
- Use a similar prompt for a medium shot, but experiment with how you want the shot to look
- Generate a close-up shot
- Remember to use other tools like Infinite Image or Upscale Image to perfect your shot
- Load your images into Gen-2 to create your final scene


Generating Scenes

Learn how to use Director Mode to customize camera controls in Gen-2.

- Add an image to Gen-2, click on the “Motion” button, and click “Custom camera control”
- Choose one or multiple directions to move your camera
- Then, adjust “Speed” for your motion
- Clicking the “Extend” button will reuse the existing camera direction for your extensions to continue the motion seamlessly, up to 16s
- For even more control, try using the Image + Description feature, which will allow you to dial in a more specific prompt


Gen-2 Director Mode

Learn how to make a looping video that shows off dynamic movement with Gen-2.

- Add an image to Gen-2
- Navigate to “Advanced camera controls” section to use Director Mode
- Turn on Upscale
- Choose a camera path and generate a video for each direction that you want present in your video
- Pro tip: the next step is easier if you rename outputs with keywords or directions
- When editing video files, play the clip forwards; when it reaches the end, duplicate the video and play it backwards, so that it resets back to the first frame of the video
- Play with keyframes to make things feel more dynamic and snappy
- For even more loop-ability, try exporting your video as a GIF


Gen-2 Directional Looping Video

Learn how to control motion in Gen-2 generations.

- Open Gen-2 and input a prompt
- Increase or decrease motion value by typing or dragging the slider
- The default is 5; the range is 0-10 (0 = less motion, 10 = more motion)
- Click “Generate”


Gen-2 Motion Slider

Learn how to extend generations up to 18s in Gen-2.

- Open Gen-2 and generate a video
- Once your video is done generating, use the “Extend” button to extend your generations up to 18s
- Each time you click “Extend,” the video output will look different; so, try a few extensions to get different results each time


Extend Generations in Gen-2

Explore the three different modes you can use in Gen-2.

- Modes in Gen-2:
    - Text prompt
    - Text prompt + Image
    - Image
- Try starting out using a text prompt only; check advanced options and generate
- If you have an image in mind you’d like to use to influence your generations, there are two ways to do that:
    - Include an image reference along with a text prompt (for a _blend_ of text prompt and image reference)
    - Click into “Image” tab to use the image only
- To recap:
    - Text to Video: explore creative ideas, compositions, and creating new worlds
    - Image + Text to Video: narrow in on a specific concept, with flexibility of prompting
    - Image to Video: bring a very specific vision to life
- Rate your generations to help us keep improving Gen-2


Gen-2 Modes

Learn how to use Text to Image for use with Gen-2 in Runway.

- Open Text to Image and explore the newly updated styles in the “Advanced” menu
- Customize output with mediums and moods
- A notable mention is the “Cinematic” style — adept at crafting photorealistic, cinematic visuals
- Once you have a generation you like, navigate over to Gen-2, and use the image as your prompt
- Before generating, open the advanced camera controls to set the direction and intensity of motion in your video


Image to Video Workflow

Combine Gen-2 with other AI Tools for a smooth and polished workflow.

- Start with Text to Image
    - generate an image you want to bring to life with Gen-2, then add it to your assets
- Open Gen-2, and use that image as your reference image alongside a text prompt
- Try bringing your video into Super-Slow Motion to extend the clip time
- Or, use your video into Text to Color Grade (LUT) to give it a color treatment
- Bring your video into Inpainting to clean up any artifacts or parts of the frame
- Toss it into Green Screen to extract your creation from the scene or create alpha mattes


Gen-2 Workflow w/ AI Tools

Gen-2

Learn how to use Character Reference Videos in Act-One.

- Head to Generative Sessions tool and open up Act-One in Gen-3 Alpha or Gen-3 Alpha Turbo
- You can now use a video as your character input, in addition to images
- Upload or record your driving performance, choose a video clip for your character reference video, adjust motion intensity if you’d like, and hit “Generate”
- Keep in mind: if your driving performance video is longer than your character reference video, the output will automatically reverse at the end of the clip to fill the duration
- Character reference videos in Act-One can be live action videos, giving you the freedom and flexibility of changing performances in post
- You can also bring in your Gen-3 Alpha videos as character videos, giving your generative workflows a huge boost in expressiveness
- You can even run your video clips through Video to Video before bringing them into Act-One to completely control every aspect of your performance


Character Videos in Act-One

Learn how to use Expand Video in Gen-3 Alpha Turbo.

- Expand Video allows you to seamlessly generate beyond the edges of a frame to turn horizontal videos into vertical videos, and vice versa - without compromising details or framing
- Open Gen-3 Alpha Turbo to use Expand Video
- If you upload a horizontal video, the model will automatically default into turning it into a vertical video
- You’ll have a few options:
    - Click the Generate button without providing a text prompt
    - Provide a detailed text prompt telling the model what you’re hoping to see in the parts of the frame that aren’t visible
    - Provide the model with an outpainted image of the first frame of your video
- Best practices:
    - Subject is close to center of the frame
    - No graphics or text present in the video
    - Use a guiding text prompt
- If you run multiple expansions in a row, you can create wide shots of the scene — then by overlaying the original footage onto the expanded canvas, you can create super zooms


Gen-3 Alpha Expand Video

Learn how to use Camera Control in Gen-3 Alpha Turbo.

- Head to Generative Video, then Gen-3 Alpha Turbo and upload an image
- Camera Control allows you to plan out specific camera movements for your scene: left/right, up/down, in/out, pan left/right, tilt up/down, rotate clockwise/counter-clockwise
- The camera speed is determined by how close your subject is to the camera
    - Use smaller values when your subject is closer and larger values when your subject is far away
- Camera Control will generally work better if you provide a text prompt to help guide your generation (example: where you want the camera to move, what you want your subject to be doing, add more to the scene)
- Try using presets for inspiration, found underneath the prompt box


Gen-3 Alpha Camera Control

Learn how to use Act-One to generate expressive performances using a single driving video and character image.

- Head into Generative Video with Gen-3 Alpha selected
- Click “Act-One” icon on the left sidebar
- Add your source footage
    - Act-One can work with characters positioned at varying angles as long as it’s not too extreme
    - Maintain lively facial expressions with the entire face in view, with minimal body movement and clear audio
- Add your reference image
    - Upload your own image or choose from one of our presets
- Once ready, hit “Generate”
    - Your result will be a seamless animation preserving all of the subtle expressions and movements from your performance, but transposed onto the character from your reference image


Getting Started with Act-One

Learn how to use Video to Video in Gen-3 Alpha.

- Open up Generative Video and select Gen-3 Alpha
- Upload a source video
    - input videos 5s or less cost 50 credits, input videos longer than 5s (up to 10s) cost 100 credits
- Add a text prompt describing how you’d like to stylize your video
- Adjust structure transformation slider in settings
    - lower values will adhere to the source video structure more
    - higher values will allow you to slightly deviate from structure of input video
- Click “Generate”
- Use custom style presets to experiment with new styles
    - edit the purple text in the text prompt with a quick description of your scene


Gen-3 Alpha Video to Video

Learn how to extend Gen-3 Alpha generations.

- Choose Gen-3 Alpha from the model dropdown in the Text/Image to Video tool (extensions aren’t available in Gen-3 Alpha Turbo yet)
- Generate a video from a text prompt or an image
- Click “Extend” — choose to extend it by 5s or 10s for a maximum of three extensions total
- If you click the “Extend” button with no additional prompt, Gen-3 Alpha will automatically continue the motion from the previous video
- If you’d like, you can add a text prompt to your extension for more control, or to add something new
- From your assets folder, you can always navigate back to a previous Gen-3 Alpha generation and extend it from there
- Extensions are a great way to get more out of Gen-3 Alpha — showcasing the environment, transitioning the scene or just continuing the action


Gen-3 Alpha Extensions

Learn how to blend live action video with Generative Visual Effects.

- Find or shoot a source video
    - Suggested camera settings: auto focus off, auto white balance off, camera on tripod
- Next, scrub through your footage and export a still image from where you want to add visual effects
- Bring the still image into Gen-3 Alpha Image to Video
- Depending on your idea, you can choose to input the image as the first frame or last frame
- In this example, we’ll select the first frame and add a text prompt describing the effect
- Once you have a shot you like, composite (combine two or more images together to make one final image) the source video with the generated video
- For the final output, feel free to add color correction or masking if needed
- Gen-3 Alpha Image to Video allows you to achieve various levels of VFX — from singular assets to fully blended live action and generated content


Generative Visual Effects

Learn how to use Image to Video with Runway’s newest video model, Gen-3 Alpha.

- From your Runway dashboard, click on “Text/Image to Video” and upload an image
- If you’d like, you can click the “Generate” button with no additional prompt guidance, and the model will interpret the image to give you the best results
- You can also provide a text prompt alongside your image which will help guide the result better or, introduce new elements or movements throughout the video
- Image to Video is a great way to start with a specific style, character or composition for added control and intention when generating


Using Image to Video in Gen-3 Alpha

Learn how to use Text to Video with Runway’s newest video model, Gen-3 Alpha.

- Head to Text/Image to Video tool and select Gen-3 Alpha from the model dropdown
- Add a text prompt
- Select 5s or 10s video to generate
- 720p generations take about 60s for a 5s video, or 90s for a 10s video
- The more detailed you can make the text prompt, the more Gen-3 Alpha will shine
    - Include things like subject, scene, lighting, camera, movements and more
- In addition to visual details, Gen-3 Alpha can be prompted for subject action, camera action, speed, transition and more
- Gen-3 Alpha works with a wide variety of prompt structures, from super simple to more complex
- If you’re looking for inspiration, separate prompt into a visual description and a camera description
    - Example: Visual: A pillow fort in a cozy living room. The pillow fort is made from an assortment of quilts, fabrics and pillows Camera motion: Hand held camera smoothly zooms into the entrance of the pillow fort, revealing an ancient castle in the interior
- For more helpful resources, check out our [Help Center](https://help.runwayml.com/hc/en-us) or [Discord](http://discord.gg/invite/runwayml)
