AI Course Pushes Computer Vision into Real-World Problem Solving
Inspired by Assistant Professor Tahiya Chowdhury’s new sequence of classes, students explore machine learning's potential to improve lives
To illustrate how artificial intelligence could assist in clinical areas, David Watts, director of Colby’s Davis Institute for Artificial Intelligence, cited a Colby student’s project investigating plant disease during a recent meeting with representatives of a large Maine health organization.
“If the full spectrum of AI skills, algorithmic rigor, and responsible application can be developed for diagnosing diseases in agricultural plants, could they also be applied to diagnose diseases in human beings?” asked Watts.
Watts highlighted the ongoing work of Dave Boku ’26, a computer science major and math minor, as an example of how the Davis Institute empowers faculty and students to use machine learning in a variety of fields to improve lives across society.
“By integrating an understanding of the fundamentals, the possibilities, and the potential pitfalls of machine learning into the liberal arts curriculum, the Davis Institute empowers faculty and students to maximize the potential and minimize the negative impacts of AI,” Watts said.
Boku began developing an app to help diagnose and treat plant diseases as part of a new fall-semester computer vision course, introduced and developed by Clare Boothe Luce Assistant Professor of Computer Science Tahiya Chowdhury. The course, CS366, included high-level theory and practical application in areas of personal interest to the 23 students who signed up. Throughout the semester, they learned about the potential and limitations of what computers can and cannot see.
‘Few U.S. universities offer a dedicated course on the interdisciplinary topic of multimodal interaction and machine learning. This is an emerging topic that will transform the future of robotics, social interaction, assistive technology, and many other application areas.’
Tahiya Chowdhury, Clare Boothe Luce Assistant Professor of Computer Science
This spring, in the second part of the course, students are expanding on the ideas and concepts presented in the fall, incorporating multimodal data, such as images, text, and audio. Students in the spring portion of the course, CS466 Multimodal Interaction and Learning, will present their final projects, with topics ranging from drones and robots to protecting sensitive information from AI agents, as part of the day-long Colby Liberal Arts Symposium (CLAS26) on April 30.
The seminar course addresses the challenges and opportunities of the important and emerging field of multimodal interaction. While artificial intelligence and machine learning are ubiquitous in U.S. higher education, the specific interdisciplinary niche of multimodal interaction combined with machine learning remains a specialized field, said Chowdhury, who was the Davis Institute’s first postdoctoral research fellow during the 2023-24 academic year. Her work focuses on human-centered tools and technology for social good.
“Few U.S. universities offer a dedicated course on the interdisciplinary topic of multimodal interaction and machine learning,” she said. “This is an emerging topic that will transform the future of robotics, social interaction, assistive technology, and many other application areas.”
A first-of-its-kind course
In Chowdhury’s first-of-its-kind course at Colby, students learn not just how to incorporate artificial intelligence into a range of coursework across disciplines, but how to build better AI with real-world implications, Watts said. “These pioneering computer vision courses are the epitome of Colby’s vision, to help students move beyond traditional boundaries, turning AI frameworks into human-centered solutions for some of the world’s pressing challenges, from agricultural sustainability to global health,” Watts said.
Computer vision is a field of artificial intelligence that teaches computers to “see” and understand the world through digital images and videos. If a camera acts as an eye, computer vision acts as a brain, processing and analyzing images, people, and actions.
‘Professor Chowdhury told us, “Pick whatever excites you.” She wanted us to think about areas and topics we might have an interest in, and then we met with her one-on-one to figure out the technical details we needed to implement our ideas.’
Dave Boku ’26
The class meets in the Davis Science Center, and coursework involves reading and discussing papers, hearing from outside experts, and hands-on work on building projects of personal interest, nearly all of which take place outside the classroom.
Inspired by his interest in agriculture and a study-abroad experience in Copenhagen focused on medical imaging, Boku decided to focus on plant diseases.
“Professor Chowdhury told us, ‘Pick whatever excites you.’ She wanted us to think about areas and topics we might have an interest in, and then we met with her one-on-one to figure out the technical details we needed to implement our ideas,” including identifying relevant data sets and how to access them, said Boku, a Davis Institute research assistant, who describes himself as an engineer and designer interested in solving “real human problems.”
As he embarked on his project in the fall, Boku met with Colby biology professors to understand how diseases manifest in leaves and how to detect them. He researched what farmers struggle with regarding plant diseases, what solutions already exist, and what remains to be done.
Then he built and began testing his model. It involves an app that captures an image of a diseased leaf and analyzes it to provide a diagnosis. His project is still in progress, and he has work to do to improve its effectiveness across a wider range of diseases.
After graduation this spring, Boku hopes to use his AI skills and “multidisciplinary lens of thinking” at a small-scale startup in a major tech city, while advocating for ethical and best practices that are “fair to people and beneficial to everybody.”
Student presentations at CLAS26
Several students in Chowdhury’s course will present their work from 10:30 a.m. to noon April 30 in room 117 of the Davis Science Center, as part of CLAS26.
Aidan Kwok ’26 and Saad Khan ’26 will lead a discussion about their ongoing work to build a drone from scratch for high-stakes and emergency-response scenarios. They are receiving assistance from Max Vieira ’28 and Ben Manasse-Pols ’26.
Their drone is intended to be autonomous and deployable in situations involving someone with a firearm. The drone would be equipped with technology to detect firearms and neutralize a possible suspect through kinetic force.
‘We’re really exploring how this might work, and what pieces of our project are applicable in the real world.’
Aidan Kwok ’26
Kwok is a computer science major and math minor, who is interested in robotics. He got the idea for the project while working at a robotics startup during Jan Plan.
Khan is a computer science major and a physics minor. Vieria and Masasse-Pols are not enrolled in Chowdhury’s seminar course, but they are working with their friends because of their personal interest in the topic and to gain experience.
Boku and Philip Booth ’26 will present a project titled Poetry as a Multimodal Dream, which explores how generative models interpret and transform poetic expression across modalities. What that means: They are building a machine to generate imagery based on poetry.
They got the idea after taking a poetry course together and became curious about creating a pipeline in the computer science course to generate images from the poem that would serve as an art installation as the poem is read. “Poetry is a visceral experience, so we are seeing if we can add a visual element to it,” Boku said.
The poetry project relates to a topic the Davis Institute explored this spring in its Distinguished AI Speaker Series with Ross Goodwin, a self-described “data poet,” who spoke to students and faculty about his experiences with language, writing, and large language models.
As with the plant-disease project, Poetry as a Multimodal Dream is a work in progress. Exploration and experimentation are part of the learning process, Chowdhury said, adding that she appreciates that students are experimenting with the limits and possibilities of AI across disciplines.
“In both my teaching and research, I aim to highlight not only the capabilities of AI systems, but also the hidden dimensions that are critical to make AI work but are often overlooked,” she said.
