Structuring and Embedding Image Captions: the V.I.F. Multi-modal System

Abstract

Within the context of historical photographic annotated collections, we observe the frequent occurrence of some subsets of important characters, usually described in captions. For many years, image captions were annotated using natural language texts intended to be read by humans. Today, the information retrieval of structured information is appealing and the migration of natural language captions to structured information is desirable in a variety of photographic collections. In this paper, we describe the Very Important Faces (V.I.F.) system, which is designed to graphically document the occurrence of distinguished characters within photographic collections and store this information in a structured format useful for retrieval purposes. The V.I.F. system implements face detection in the image data and detects proper names in previously inserted captions if any are present. The user matches names to faces throughout the software interface in order to produce a photo annotation that is stored considering structured information principles. Once the matching is done, an efficient verification tool is proposed, which helps the expert to review the annotation, taking advantage of such multi-modal databases. The concept of annotation maturity level is also introduced.