An integrated approach to semantic evaluation and content-based retrieval of multimedia documents

We present an overview of a large combined querying and retrieval system that performs content-based on-line searches in a large database of multimedia documents (currently text, tables and colour images). Queries are submitted as sentences in natural language and are transformed into the language of the target database. The documents are analyzed semantically for their information content; in a data fusion step the individual pieces of information extracted from these documents are aggregated into cognitively adequate result documents. There is no pre-indexing necessary when new documents are stored into the system. This retains a high degree of flexibility with respect to the questions that may be asked. It implies, however, that both huge amounts of data must be evaluated rapidly and that intelligent caching strategies must be employed. It is therefore mandatory that the system be equipped with dedicated high-speed hardware processors. The complete system is currently available as a prototype; the paper outlines its architecture and gives examples of some real sample queries in the knowledge domain of weather data documents.