Interactive Exploration of Journalistic Video Footage through Multimodal Semantic Matching

Sarah Ibrahimi
Shuo Chen
Devanshu Arya
Arthur Camara
Yunlu Chen
Tanja Crijns
Maurits van der Goes
Emiel van Miltenburg
Daan Odijk
William Thong
Jiaojiao Zhao
Pascal Mettes
ACM Multimedia(2019)
Google Scholar


This demo presents a system for journalists to explore video footage for broadcasts. Daily news broadcasts contain multiple news items that consist of many video shots and searching for relevant footage is a labor intensive task. Without the need for annotated video shots, our system extracts semantics from footage and automatically matches these semantics to query terms from the journalist. The journalist can then indicate which aspects of the query term need to be emphasized, e.g. the title or its thematic meaning. The goal of this system is to support the journalist in their search process by encouraging interaction with the system.