Meta works on Emu Video and Emu Edit: Generative AI tricks for GIFs, photos and 4-second videos،
Meta announces via a blog post that they are busy working on new research on “controlled image editing based solely on text instructions and a method of generating text in video based on diffusion models.”
Which, in simpler terms, means they want to integrate generative AI tools from Facebook and Instagram. The projects that Meta is developing are called Emu Video and Emu Edit.
What is Emu Video?
This tool, as its name suggests, is used to generate videos. Meta describes it as “a simple method of generating text to video based on delivery models.” Emu Video must respond to a variety of inputs: text only, image only, and both text and image. The process is divided into two steps, Meta says: first, generating images conditioned by a text prompt, then generating a video conditioned by both the text and the generated image.
Our state-of-the-art approach is simple to implement and uses just two streaming models to generate four-second 512 x 512 videos at 16 frames per second.
What is Emu Edit?
This should allow “precise image editing” via recognition and generation tasks. As Meta says, using generative AI is often a process and not a single task.
“Emu Edit is capable of free-form editing via instructions, encompassing tasks such as local and global editing, removing and adding a background, color and geometry transformations, detection and segmentation, and much more. Current methods often tend to over-edit or under-perform various editing tasks. We argue that the primary goal should not only be to produce a “credible” image. Instead, the model should focus on precisely editing only the pixels relevant to the edit request. Unlike many current generative AI models, Emu Edit follows instructions precisely, ensuring that pixels in the input image unrelated to the instructions remain intact. For example, when adding the text “Aloha!” “Compared to a baseball cap, the cap itself should remain unchanged,” says the Meta team.
Potential use cases
The path forward is definitely AI-driven for Meta.
“While this work is purely fundamental research at the moment, the potential use cases are clearly evident. Imagine generating your own animated stickers or smart GIFs on the fly to send into the group chat rather than having to search for the perfect media for your response. Or edit your own photos and images, no technical skills required. Or add some extra oomph to your Instagram posts by animating static photos. Or generate something entirely new,” the blog post concludes.
