Objective

The importance of reusable, large-scale standard test collections in Information Access research has been widely recognized. The success of TREC, CLEF, and NTCIR has clearly established the importance of an evaluation workshop that facilitates research by providing the data and a common forum for comparing models and techniques. The Forum for Information Retrieval Evaluation (FIRE) follows in the footsteps of TREC, CLEF and NTCIR with the following aims:

  • to encourage research in South Asian language Information Access technologies by providing reusable large-scale test collections for ILIR experiments
  • to explore new Information Retrieval / Access tasks that arise as our information needs evolve, and new needs emerge
  • to provide a common evaluation infrastructure for comparing the performance of different IR systems
  • to investigate evaluation methods for Information Access techniques and methods for constructing a reusable large-scale data set for ILIR experiments.

The Tasks

  1. Ad-hoc monolingual document retrieval in Bengali, Hindi Marathi and English.

  2. Ad-hoc cross-lingual document retrieval
    - documents in Bengali, Hindi, Marathi, and English,
    - queries in Bengali, Hindi, Marathi, Tamil, Telugu , Gujarati and English.
    - Bengali and Hindi topics will also be transliterated and made available in Roman script. Adhoc monolingual task participants are encouraged to submit runs using these queries as well.

  3. Retrieval and classification from mailing lists and forums.
    This is a pilot task being offered by IBM India Research Lab.

  4. Ad-hoc Wikipedia-entity retrieval from news documents
    - Entities mined from English Wikipedia
    - Query documents from English news website
    This is a pilot task being offered by Yahoo! Labs, Bangalore.

Task Co-ordinators

Ad-hoc retrieval:
Pushpak Bhattacharyya (pb@cse.iitb.ac.in)
Dipasree Pal (dipasree_t@isical.ac.in)
Retrieval from mailing lists and discussion forums:
Debapriyo Majumdar (debapriyo at in dot ibm dot com)
Ayan Bandyopadhyay (ayan_t@isical.ac.in)
Ad-hoc Wikipedia-entity retrieval from news documents (WikEND):
Ashwin Tengli (ashwint@yahoo-inc.com)
Pabitra Mitra (pabitra@cse.iitkgp.ernet.in)

Overall co-ordinators

Prasenjit Majumder (p_majumder@daiict.ac.in)
DAIICT, Gandhinagar
Mandar Mitra (mandar@isical.ac.in)
Indian Statistical Institute, Kolkata