Comparative Evaluation of Multilingual Information Access Systems

The fourth campaign of the Cross-language Evaluation Forum (CLEF) for European languages was held from January to August 2003. Participation in this campaign showed a slight rise in the number of participants from the previous year, with 42 groups submitting results for one or more of the different tracks (compared with 37 in 2002), but a steep rise in the number of experiments attempted. A distinctive feature of CLEF 2003 was the number of new tracks and tasks that were offered as pilot experiments. The aim was to try out new ideas and to encourage the development of new evaluation methodologies, suited to the emerging requirements of both system developers and users with respect to today's digital collections and to encourage work on many European languages rather than just those most widely used. CLEF is thus gradually pushing its participants towards the ultimate goal: the development of truly multilingual systems capable of processing collections in diverse media. The campaign culminated in a two-day workshop held in Trondheim, Norway, 2122 August, immediately following the 7th European Conference on Digital Libraries (ECDL 2003), and attended by more than 70 researchers and system developers. The objective of the workshop was to bring together the groups that had participated in the CLEF 2003 campaign so that they could report on the results of their experiments.

Inhalt
CLEF 2003 Methodology and Metrics.- Analysis of the Reliability of the Multilingual Topic Set for the Cross Language Evaluation Forum.- Evaluation of Information Access Technologies at the NTCIR Workshop.- I. Ad-hoc Text Retrieval Tracks.- CLEF 2003 Overview of Results.- Mainly Cross-Language Experiments.- Report on CLEF-2003 Multilingual Tracks.- The Impact of Word Normalization Methods and Merging Strategies on Multilingual IR.- JHU/APL Experiments in Tokenization and Non-word Translation.- Report on CLEF-2003 Experiments: Two Ways of Extracting Multilingual Resources from Corpora.- Combining Query Translation and Document Translation in Cross-Language Retrieval.- Cross-Language Experiments with the IR-n System.- Multilingual Information Retrieval Using Open, Transparent Resources in CLEF 2003.- ITC-irst at CLEF 2003: Monolingual, Bilingual, and Multilingual Information Retrieval.- Language-Dependent and Language-Independent Approaches to Cross-Lingual Text Retrieval.- Multilingual Retrieval Experiments with MIMOR at the University of Hildesheim.- Concept-Based Searching and Merging for Multilingual Information Retrieval: First Experiments at CLEF 2003.- CLEF 2003 Experiments at UB: Automatically Generated Phrases and Relevance Feedback for Improving CLIR.- SINAI at CLEF 2003: Decompounding and Merging.- Merging Results by Predicted Retrieval Effectiveness.- MIRACLE Approaches to Multilingual Information Retrieval: A Baseline for Future Research.- Experiments to Evaluate Probabilistic Models for Automatic Stemmer Generation and Query Word Translation.- Clairvoyance CLEF-2003 Experiments.- Simple Translations of Monolingual Queries Expanded Through an Association Thesaurus.- Two-Stage Refinement of Query Translation in a Pivot Language Approach to Cross-LingualInformation Retrieval: An Experiment at CLEF 2003.- Regular Sound Changes for Cross-Language Information Retrieval.- Exeter at CLEF 2003: Experiments with Machine Translation for Monolingual, Bilingual and Multilingual Retrieval.- Monolingual Experiments.- Lexical and Algorithmic Stemming Compared for 9 European Languages with Hummingbird SearchServerTM at CLEF 2003.- Océ at CLEF 2003.- Comparing Weighting Models for Monolingual Information Retrieval.- Pruning Texts with NLP and Expanding Queries with an Ontology: TagSearch.- Report on CLEF-2003 Monolingual Tracks: Fusion of Probabilistic Models for Effective Monolingual Retrieval.- Selective Compound Splitting of Swedish Queries for Boolean Combinations of Truncated Terms.- COLE Experiments at CLEF 2003 in the Spanish Monolingual Track.- Experiments with Self Organizing Maps in CLEF 2003.- Ricoh at CLEF 2003.- MediaLab at CLEF-2003: Using Keyword Disambiguation.- II. Domain-Specific Document Retrieval.- The GIRT Data in the Evaluation of CLIR Systems from 1997 Until 2003.- A Data-Compression Approach to the Monolingual GIRT Task: An Agnostic Point of View.- UC Berkeley at CLEF-2003 Russian Language Experiments and Domain-Specific Retrieval.- University of Hagen at CLEF 2003: Natural Language Access to the GIRT4 Data.- III. Interactive Cross-Language Retrieval.- The CLEF 2003 Interactive Track.- iCLEF 2003 at Maryland: Translation Selection and Document Selection.- UNED at iCLEF 2003: Searching Cross-Language Summaries.- Comparing Syntactic-Semantic Patterns and Passages in Interactive Cross Language Information Access (iCLEF at University of Alicante).- Continued Experiments on Cross-Language Relevance Assessment.- IV. Cross-Language Question Answering.- The Multiple Language Question Answering Track at CLEF2003.- Creating the DISEQuA Corpus: A Test Set for Multilingual Question Answering.- Bridging Languages for Question Answering: DIOGENE at CLEF 2003.- Cross-Language Question Answering at the USC Information Sciences Institute.- How Frogs Built the Berlin Wall.- Cross Lingual QA: A Modular Baseline in CLEF 2003.- Question Answering in Spanish.- Quantum, a French/English Cross-Language Question Answering System.- A CrossLanguage Question/AnsweringSystem for German and English.- Cross-Language French-English Question Answering Using the DLT System at CLEF 2003.- V. Cross-Language Image Retrieval.- The CLEF 2003 Cross Language Image Retrieval Track.- Assessing Translation Quality for Cross Language Image Retrieval.- Foreign Name Backward Transliteration in Chinese-English Cross-Language Image Retrieval.- Image Retrieval: The MIRACLE Approach.- Scene of Crime Information System: Playing at St. Andrews.- VI. Cross-Language Spoken Document Retrieval.- The CLEF 2003 Cross-Language Spoken Document Retrieval Track.- Exeter at CLEF 2003: Cross-Language Spoken Document Retrieval Experiments.- N-Grams for Translation and Retrieval in CL-SDR.- Spoken Document Retrieval Experiments with IR-n System.- ITC-irst at CLEF 2003: Cross-Language Spoken Document Retrieval.