Name: How to Composite
Uploaded: 2023-09-28T17:55:20.999Z
Description: Walk through a full compositing example in Runway using Green Screen, Inpainting, and Motion Tracking.

Learn how to continue using your model in Text to Image for hyper-customized images.

- You aren’t limited to the first 100 images — continue using your model in other mediums, like Text to Image
- Important tip to keep in mind: include the name of your model in the text prompt in addition to selecting the model from the dropdown
- Add in-depth description to fine-tune results


Using Your Model

Use this tutorial to learn the basics of AI Training in Runway.

- Gather your images… here are some tips:
    - Use at least 15 images
    - Subject in focus
    - Different lighting/context
    - Only contain subject you’re aiming to train on
- Give model unique and simple keyword
- Train your model — it should take approximately 30 minutes depending on number of images, image sizes, etc.
- Try using your model in Text to Image for further generations!


Introduction to AI Training

AI Training

Use this quick tutorial to learn how to use Runway's mobile app for iOS.

- Download the Runway app from the App Store app
- Sign into Runway using the same account you use on the web
- Circular purple button will initiate a new generation
- 1 second of video generation = 14 credits
    - To top up credits, click profile icon in the top right corner of the Home page and tap “Credits”
- To create a video, simply hold down the record button and let go when you’re finished
- Three main style settings:
    - Presets by Runway
    - Reference image: click “Add” to choose an image from your library; adjust style strength and seed number here
    - Text prompt
- Hit “Generate” — video will be queued and show a percentage status circle
- Generate new options in the meantime; videos will continue generating in the background
- Generations will be stored in library tab


Using the Mobile App

Runway Mobile

Transform a low resolution image into a high resolution image with Upscale Image.

- Use Upscale Image to add more detail to increase image resolution
- Upload an image asset
- Select desired size under “Scale” and click “Process”
- That’s all there is to it!
- Download the image or add to your assets by hovering over it and clicking the icons on the top right


Upscale Images

Learn how to use AI Magic Tool Erase and Replace to easily modify images.

- Head to AI Magic Tools to find Erase and Replace
- Choose an image asset from your computer or asset library
- Adjust brush size using the slider on the right sidebar
- Brush over the portion of the image you’d like to replace
- Enter a text prompt and click “Replace”
- “Add to assets” to add to your Runway asset library
- “Download” to download locally to your computer
- Use “Clear” to generate something new with a new prompt


Erase and Replace

Learn to create a bespoke image sequence using Image to Image.

- Remove the background of your video with Green Screen and save as a PNG sequence for the most flexibility down the road
- Open Image to Image and use first frame in the sequence as a test frame
    - Describe the entire scene, touch on style that you want
    - Make sure the look is right — you’ll be committing to it in this workflow
    - Lock seed number and don’t change the prompt for images you like
    - Save to assets to make sure you can find them later
- Select every few frames if you don’t want to select them all
- String the frames together in the video editor


Image to Image

Use this tutorial to learn how to use the Text to Image AI Magic Tool in Runway.

- Head to AI Magic Tools to find Text to Image
- Enter a simple text prompt
- Open up “Advanced” under “Settings” sidebar to explore refinement options
    - Size, style, medium, mood, prompt weight
- Prompt weight will determine how much the text prompt is taken into consideration; higher number = most consideration, lower number = least
- Experiment with more complex prompts
- “Add to assets” to save to your Runway asset library
- “Download” to download locally to your computer


Text to Image

Bring still images to life and learn how to create a living portrait in Runway.

Step 1: Brush out and replace the parts of your image you'd like to change using either Erase & Replace or Infinite Image.
Step 2: Head over to Frame Interpolation to animate your various images into a video.
Step 3: Behold the beauty of your new living image.


Create a Living Portrait

Learn how to seamlessly transition images into one another by using our Frame Interpolation AI Magic Tool.

- Frame Interpolation works well to create videos with subtle movement from still images
- The more frames you have to work with, the better
- Keep the differences between photos subtle if possible for realistic effect
- Try using Erase and Replace to create new images to interpolate between!


Frame Interpolation

Learn to use the Backdrop Remix AI Magic Tool, and remix your images into anything you want them to be.

- Backdrop Remix changes the background of your photos while keeping the subject in tact
- Upload your photo, and choose a preset or a custom prompt
- Click “Generate,” and Runway will get to work replacing the background automatically — that’s it!


Backdrop Remix

Use this lesson to learn tips and tricks for using Text to Color Grade.

- The main thing to keep in mind: you can never go wrong with natural language.
- Typing in a movie title alone might be counterproductive sometimes; add in a description to make the movie title interpretation less literal.
- Same goes for specific film stocks — add additional descriptors.
- Try different seed numbers for different options.


Text to Color Grade

Learn how to use AI Magic Tool Infinite Image.

- Find Infinite Image in the AI Magic Tools section. Start with a text prompt to generate an original image, or upload your own image.
- Move the generation frame to desired area. Make sure it’s overlapping the original image!
- Be descriptive with your text prompt and click “Generate.” You’ll then be presented with 4 initial generated options to choose from.
- Repeat this process as many times as you’d like to continue extending your image.
- Use adjustment button to adjust prompt weight; lower for more surprising and creative results, higher to stay more true to the prompt.
- Try opening your final image in Erase and Replace to clean up any elements in your image you’d like to change.


Infinite Image

Learn how to create generative images using various AI Magic Tools.

- Learn how to create a generative image using AI Magic Tools
- AI Magic Tools used:
    - Text to Image: creates an original generated image from a text-based prompt
    - Erase and Replace: eliminates a specific part of an image and replaces it seamlessly with a new image generated from a new text based prompt
    - Image to Image: uses an image as the input, and generates a brand new image based on both the original and a descriptive text based prompt
- When writing prompts, be descriptive. Include object colors, orientations, and artistic styles in your prompt.
- Refine as you go — the initial output does not have to be perfect. You can slowly work and refine your image with tools like Erase and Replace and Image to Image to gradually get your content to where you want it.


Generative Image Workflow

AI Magic Tools

Learn how to combine Gen-1 with Green Screen and additional techniques to create a dynamic edit.

- Main idea: take multiple sources of footage and combine them into one shot
- Start by using Gen-1 to style a source video
- Once you’ve landed on a generation you like, bring it into Remove Background to isolate the subject
- Export the mask into a transparent ProRes file
- Start adjusting layers independently using scale and effects
- Experiment with speed ramps for exciting motion
- Export


Gen-1 Workflow w/ Green Screen

Learn how to create new worlds to jump through with Gen-1.

- This tutorial is inspired by Martin Haerlin’s viral Twitter post
- Record a video of yourself walking through a space, snapping every time you want to transform yourself
- Cut up your video so that each snap becomes its own file
- Use Text to Image or Image to Image to generate some scenes and characters
    - Add images to you assets
- Open Gen-1 and click on one of your saved images
- Use Advanced Settings to fine-tune your generations
- Preview & Generate
- Put your clips together and add sound design to create an ambient effect


Gen-1 World Jumping

Turn simple comps into compelling videos with Gen-1.

- Gen-1 source video doesn’t have to be a “normal” straight out of camera video; try creating a rough composite instead!
- Compositing = combining multiple visual elements for a cohesive final output
- Take rough mockups into Gen-1 with an image prompt for surprising results


Using Composites in Gen-1

Learn how to take your Gen-1 workflow to the next level by combining multiple AI Magic Tools.

- Head to Text to Image in AI Magic Tools to generate images to use as a reference image in Gen-1
- Edit the generated image in Erase and Replace or iterate in Image to Image
- Use Inpainting to remove subject from the scene, then use Green Screen to extract the subject by itself, for complete foreground and background separation
    - Affect the background with one set of prompts, and affect the foreground with another
    - Combine them back together for unique results!


Gen-1 Workflow w/ AI Magic Tools

Looking for Gen-1 inspiration? Use this tutorial and experiment with unexpected results.

- Upload the video you want to use as your driving source
- Incorporate a prompt that is wildly different or opposite from the source video
- Open “Advanced” in Gen-1
    - Turn “Structural consistency” to 6
    - Turn “Weight” to 4 or 5
    - Turn “Frame consistency” to 1.2
- Click “Generate” and expect the unexpected


Unexpected Results in Gen-1

Use Gen-1 to turn your videos into animated sketchbooks.

- Upload you video and open Gen-1
- Find or generate a reference image that has a hand-drawn, sketch-type feel to it
    - It doesn’t need to resemble the source video; the art style is most important
- Open “Advanced”
    - Turn “Structural consistency” to 0
    - Turn “Weight” to 5 or 6
    - Turn “Frame consistency” to .6
- Click “Generate”


Sketchbook Look with Gen-1

Keep your Gen-1 results sharp and polished with this tutorial.

- Use this tutorial to get your Gen-1 results to be clean and as close to the original video as possible
- Upload the video to your Runway account, and then open it in Gen-1
- If you’ll be using a reference image, it’s helpful to use a tool like Image to Image to generate the look you’re going for, so the image and videos line up 1:1
- Open “Advanced” in Gen-1
    - Turn “Structural consistency” to 0
    - Turn “Weight” to a lower number (between 3-7 works well)
    - Turn “Frame consistency” to 1.2
- Click “Generate”


Clean Results with Gen-1

Learn the basics of using Gen-1 in Runway.

- Head into AI Magic Tools in Runway to find Gen-1
- Select an asset
- Three style reference options:
    - Image (choose your own image)
    - Presets by Runway
    - Prompt (enter a text prompt)
- Keep settings simple or dive into advanced settings: https://help.runwayml.com/hc/en-us/articles/15161225169171-Gen-1-Gen-2-Advanced-Settings
- Find recommendations for settings by hovering over tool tips


Gen-1 Basics

Gen-1

Learn how to make a looping video that shows off dynamic movement with Gen-2.

- Add an image to Gen-2
- Navigate to “Advanced camera controls” section to use Director Mode
- Turn on Upscale
- Choose a camera path and generate a video for each direction that you want present in your video
- Pro tip: the next step is easier if you rename outputs with keywords or directions
- When editing video files, play the clip forwards; when it reaches the end, duplicate the video and play it backwards, so that it resets back to the first frame of the video
- Play with keyframes to make things feel more dynamic and snappy
- For even more loop-ability, try exporting your video as a GIF


Gen-2 Directional Looping Video

Learn how to use Director Mode to customize camera controls in Gen-2.

- Add an image to Gen-2, click on the “Motion” button, and click “Custom camera control”
- Choose one or multiple directions to move your camera
- Then, adjust “Speed” for your motion
- Clicking the “Extend” button will reuse the existing camera direction for your extensions to continue the motion seamlessly, up to 16s
- For even more control, try using the Image + Description feature, which will allow you to dial in a more specific prompt


Gen-2 Director Mode

Learn how to control motion in Gen-2 generations.

- Open Gen-2 and input a prompt
- Increase or decrease motion value by typing or dragging the slider
- The default is 5; the range is 0-10 (0 = less motion, 10 = more motion)
- Click “Generate”


Gen-2 Motion Slider

Learn how to extend generations up to 18s in Gen-2.

- Open Gen-2 and generate a video
- Once your video is done generating, use the “Extend” button to extend your generations up to 18s
- Each time you click “Extend,” the video output will look different; so, try a few extensions to get different results each time


Extend Generations in Gen-2

Explore the three different modes you can use in Gen-2.

- Modes in Gen-2:
    - Text prompt
    - Text prompt + Image
    - Image
- Try starting out using a text prompt only; check advanced options and generate
- If you have an image in mind you’d like to use to influence your generations, there are two ways to do that:
    - Include an image reference along with a text prompt (for a _blend_ of text prompt and image reference)
    - Click into “Image” tab to use the image only
- To recap:
    - Text to Video: explore creative ideas, compositions, and creating new worlds
    - Image + Text to Video: narrow in on a specific concept, with flexibility of prompting
    - Image to Video: bring a very specific vision to life
- Rate your generations to help us keep improving Gen-2


Gen-2 Modes

Combine Gen-2 with other AI Magic Tools for a smooth and polished workflow.

- Start with Text to Image
    - generate an image you want to bring to life with Gen-2, then add it to your assets
- Open Gen-2, and use that image as your reference image alongside a text prompt
- Try bringing your video into Super-Slow Motion to extend the clip time
- Or, use your video into Text to Color Grade (LUT) to give it a color treatment
- Bring your video into Inpainting to clean up any artifacts or parts of the frame
- Toss it into Green Screen to extract your creation from the scene or create alpha mattes
- The possibilities are endless with 30 AI Magic Tools!


Gen-2 Workflow w/ AI Magic Tools

Use this tutorial to learn the basics of using Gen-2 in Runway.

- Use Gen-2 via web or mobile (iOS)
- Navigate to Gen-2, and simply type in what you want to generate
- Adjust settings like seed value, interpolation, upscaling, and watermark
- Click “Generate”
- Rate your generation to help the Runway team improve Gen-2
- Generation will automatically save to your Runway assets folder
- If you’re feeling stuck or need inspiration, view Gen-2 inspiration feed below the prompt bar to remix existing prompts
- Gen-2 also works with an image reference
- Helpful to structure prompts like this:
    - base prompt
    - “in the style of”
    - description


Gen-2 Basics

Gen-2

Walk through a full compositing example in Runway using Green Screen, Inpainting, and Motion Tracking.

- We’re going to use a combination of tooling in Runway to composite this example: Green Screen, Inpainting, and Motion Tracking.
- Drag desired background layer into the timeline below your subject layer.
- To remove the background from the top subject layer, select Green Screen to mask the subject out. Refine as needed.
- Track the background layer using Motion Tracking to emulate the movement in the foreground layer. Tip: select an area of high contrast for best results.
- Rescale or adjust position of tracked background layer to fill the frame.
- Use Inpainting to paint out any undesired objects or people in your background layer.
- Correct color of background to match foreground layer by using effects and LUTs. In this example, we use a LUT, the Hue/Saturation/Lightness effect, and a small amount of blur.
