2. Artificial Intelligence at the service of the RTS Audiovisual Archives
Pietro Rezzonico | Radio Télévision Suisse | Switzerland
One of the objectives of RTS is to be able to explore and exploit its audiovisual archives and to open them to the public. Faced with the lack of metadata on content, we have set up a development team over the past two and a half years that uses artificial intelligence (AI) technologies to perform automatic metadata extraction to achieve this objective. The first development was the synchronization of videos between different versions of a program, such as live feeds and clean feeds. Based on the exact visual recognition, it was a quick win that helped documentalists by saving manual work time and generates a basic segmentation for all programs.
Secondly, we have developed quality control algorithms to identify specific defects on the files resulting from the digitalization. The manual quality control of 400,000 digital files would mean 20 years of work. Instead of that, an interface allows humans to examine the results of the algorithms in a quickest mode. Based on these successes, we have developed a unique solution that we label Visual Feature Extraction Pipeline. This tool implements recognition capabilities using Machine Learning techniques to identify patterns within facial, object and landmark features. All algorithms are based on features that are extracted once at the beginning of the pipeline. Three applications were developed and are in production:
1) The facial recognition. To perform the facial recognition, a public figures database was created by documentalists. It contains now 5,500 people. The results of the facial recognition are integrated in the digital assets management system. This automatic metadata extraction helps the documentalists during the indexation process and makes data searchable by the end users in the search interface.
2) The visual search, which makes it possible to search for similar faces, landmarks or objects from a given shot or image. It allows a powerful exploration and exploitation of the audiovisual content by finding similar undescribed images or finding images with rights issues. At the end 2019, 100% of the RTS video archives will be searchable through this revolutionary way.
3) Classification allows content to be automatically categorized using custom classifiers, which can be trained with objects or landmarks features. The first implementation is carried out by our documentalists who are creating a sport classifier, in order to automatically categorize and index old sport collections for which no metadata are available.
Several new uses are planned, such as the traceability of the use of archives and the speaker identification, which are currently under development. What we have learned: the AI technologies can really help us in our goal to open up the archives by extracting metadata automatically, enrich our legacy archives, and giving discoverability to our archival content. Internal developments allow us to be independent of commercial solutions, to protect our sensitive data and to meet our specific needs (local public figures, classifiers). The use of open source tools opens collaboration possibilities with other public broadcaster. Putting into production the AI based applications early in the development process allows early feedback and results in a user focused development. The costs of ownerships are more than half as high as with a commercial solution.