AI and Storytelling

Mission

Stories are the fundamental way people share experiences and make sense of the world. They surprise us, delight us, inform us, open our eyes, and connect us to others. They make us think and feel. Generative AI has startling abilities to generate content, but that doesn't necessarily make its creations interesting or meaningful to people.

How can we use AI to express and share experiences in new ways?

This is a studio-style class focused on making and sharing our creations. Typically, we create and share on social media. The staff have experience using AI to create and share stories in text, image, and video formats. Our recent focus has been on video, but all mediums of storytelling are equally interesting to us.

As we create we will also reflect and share our thoughts on questions about the nature of AI and storytelling:

In what ways does AI impact the storytelling process?
What new possibilities does AI create for storytelling?
How do we channel our experiences, intentions, thoughts, and emotions into AI?
How do audiences perceive the stories we tell through an AI medium?
What is the role of human creativity in the age of AI?

We will publish blog posts, position papers, video essays, and other works to share our thoughts with the world.

We aim to do a majority of the creating during studio time, which is Wednesdays 3-6pm in CSB 453.

Examples

Here are example of the types of work our staff have done before in explore the potential for AI and storytelling.

This video captures the phrasing and emotions of a song clip through abstract moving imagery. We used AI to analyze the music and identify beats and salient moments in the clip. While AI served as a starting point, the annotations were made by people based on the AI’s analysis. Using the segments of the song as visual "scenes," we created imagery that reinforces the beats visually, primarily through motion speed and zoom levels. Additionally, color was used to convey aspects of the emotional trajectory and energy of the song. AI assisted in analyzing the audio and generating visual elements, but the interpretation of the music—and its transformation into visual forms—was entirely human-driven.

This video explores various works by J.M.W. Turner. It connects his turbulent marine paintings, focusing on his famous textures and colors. AI add motion, but our artistic choices are what allows the video to highlight the unique and interesting elements of Turner's work.

@nccnewsonline
Vera House warns of the dangers for people in domestic violence situations that an emergency alert from FEMA could cause.
♬ original sound - NCC News

This is a "news reel" - a way of telling journalistic stories on social media. This was made for the Syracus University news station - NCC News -as a part of a similar studio class for making news on social media with AI. AI is capable to reframing traditional print media into social media "role plays", and then generating script and storyboards for the video, which journalists film. AI makes suggestions and allow people to explore multple options, but people are responsible for preserving the message, ensuring the information is salient, engaging, and conveyed with the appropriate tone. This video had over 273,000 views in December 2024.

Instruction and Partners

As a studio class, there will be no lectures - our time will be spent creating, critiquing, and reflecting. AI and storytelling is a topic where there is no definitive expert yet - we will discover what this topic is together as a class. To aid our critique and discussions, we are fortunate to have a number of "partners" to join us for studio time. Partners are creative professionals interesting in helping us explore AI and storytelling. Each of them is a master of their trades - both technical and creatively.

Prof. Lydia Chilton
Lydia Chilton is a professor in the Columbia University Computer Science Department. She teaches User Interface Design and Designing for Generative AI. Her research is on computational design - how people and AI can work together to analyse, discover, design and communicate novel insights for creativity and innovation. She is the instructor of record for this class.

Prof. Lance Weiler
Lance Weiler is an American filmmaker, writer, and Director of the Digital Storytelling Lab at Columbia University School of the Arts . He is a storyteller and emerging media artist working in film, theatre, games and code.

Anh Truong
Anh Truong is a Research Engineer at Adobe. Her research focuses on building interactive tools and techniques that make it easier for people to create and consume stories using video. She received her M.S in Computer science from Stanford University, where she was advised by Maneesh Agrawala. Anh is part of Project Blink, an AI-powered video editor on the web.

Sean Kim
Sean Kim is a recent Columbia Alum (MS 2024, CC 2023). He is head of AL/ML at Rizz, the #1 dating assistant on the Apple App Store. He has done work on AI generated humor and AI generated video narratives.

Prof. Tiffany Tseng
Tiffany Tseng is a professor in the Computer Science Department at Barnard. She develops design tools that empower beginners to realize their creative potential. Prior to joining Barnard, she was a research scientist at 🍎 Apple and a Project Assistant Professor at the 🗼 University of Tokyo. I also spent several years as a lead UX designer and design engineer on Tinkercad at 🧰 Autodesk. She received my PhD from the Lifelong Kindergarten Group at the ✨ MIT Media Lab. .

Vansh Murad Kalia
Vansh Murad Kalia is a film maker, data scientist and machine learning researcher. He graduated from Columbia with an MS in Quantiative Methods in Social Science. .

Samia Menon
Samia Menon is an software engineer and researcher. She has published groundbreaking papers on AI and Storytelling systems that are used by industry professionals (ReelFramer, MoodSmith). She graduated from Columbia CS in 2023. .

Duy Nguyen
Duy Nguyen is a senior machine-learning engineer at the New York Times AI Initiative Team. He previously work at The Times creating interactive data visualizations, particularly for scrollytelling articles. He was a senior machine-learning engineer at the Brown Institute’s Local News Lab at Columbia Journalism School, helping newsrooms use A.I. to solve complex business problems. He has authored an academic publication on using AI for storytelling in the newsroom..

Max Kreminski
Max Kreminski is a research scientist at Midjourney, where he leads the Storytelling Lab. He recently released Patchwork is a collaborative AI-supported worldbuilding canvas. Users work together to craft a coherent fictional setting and tell stories in that setting with multimodal AI. Before Midjourney, he was an professor of computer science and engineering at Santa Clara University.

Katherine Chou
Katherine Chou is the Head of Product and UX for Google Research with a specific focus on nurturing scientific and technical breakthroughs with global impact for health, climate, education, science and the advancement of platform technologies for our developers and researchers.

Interested in joining?

If you're interested in the class, please fill out this form

Also, get on the waitlist for COMS 3998 sec 60. Or COMS 4901 sec 60 - it's the same class, just with different numbers.

The class will meet Wednesdays from 3-6pm in CSB 453.

Prof. Lydia Chilton is the instructor of record for this class. She administers the course, and gives academic credit to students. However, she is not "the instructor" of this course, she is one of many partners who will engage in creation, critique, and discussion alongside the students..

Send questions or bureaucratic matters to: Prof. Lydia Chilton - chilton@cs.columbia.edu

AI and Storytelling

Columbia University | Spring 2025

Mission

Examples

Instruction and Partners

Interested in joining?