SMS-Based FAQ Retrieval

FIRE 2011 Task


Home
Dataset
Important Dates
People
Submission
Attendance
Resources
Contact
Results
Joint Task Coordinators
COER
and
IBM Research


Run Submission Guidelines



1. You can submit upto 3 runs per subtask.

2. Each run will have its own file with the following file name format : emailaddress$subtaskName$RunNo.txt
where - emailaddress is the Email address used for registration
subTaskNames could be one of the following:
a) "eng-mono" - For the English Monolingual Retrieval Task
b) "hin-mono" - For the Hindi Monolingual Retrieval Task
c) "mal-mono" - For the Malayalam Monolingual Retrieval Task
d) "eng-multi" - For the English Multi-lingual Retrieval Task
e) "hin-multi" - For the Hindi Multi-lingual Retrieval Task
f) "mal-multi" - For the Malayalam Multi-lingual Retrieval Task
g) "cross" - For the cross lingual retrieval task
runNo : Integer between 1 and 3.


For example, the first run submission by user : john@somedomain.com for the Malayalam multilingual run would be:
john@somedomain.com$mal-multi$1.txt

3. In the retrieval tasks, you may submit upto 5 matches per SMS query along with the scores between 0 and 1, where 1 being the highest.

(a) The format of the run submission file is as follows (FAQs to be sorted in decreasing order of scores - left to right).

SMS_ID_x,FAQ_MATCH_ID_1,Score_1,FAQ_MATCH_ID_2,Score_2,FAQ_MATCH_ID_3,Score_3,FAQ_MATCH_ID_4,Score_4,FAQ_MATCH_ID_5,Score_5
SMS_ID_y,FAQ_MATCH_ID_1,Score_1,FAQ_MATCH_ID_2,Score_2,FAQ_MATCH_ID_3,Score_3,FAQ_MATCH_ID_4,Score_4,FAQ_MATCH_ID_5,Score_5
...
...


(b) In case there are less than 5 matches :

The entry would stop at the appropriate count, For example, if there are only 3 matches - the row ends at the third match.
SMS_ID_z,FAQ_MATCH_ID_1,Score_1,FAQ_MATCH_ID_2,Score_2,FAQ_MATCH_ID_3,Score_3
...
....
....
(c) In case a query returns no matches, the entry should be as follows:
SMS_ID_z,NULL.
(Comma separated and entry as NULL)

Please use the FIRE 2011 SMS Task Submission System to upload your runs.

Please prepare a flat zip/tgz archive containing only your run submissions and upload it it in the submission system. Only one tgz/zip archive needs to be submitted.

The upload system refers to the submission as a "paper" but we would like to clarify you do not need to upload any pdf/write up at this stage and you should use the upload mechanism to submit only your zip/tgz archive containing your task runs.

All participating teams also need to submit a working note that outlines the approach followed by them. The Working note is due on November 28, 2011.

The post-proceedings will be published in Lecture Notes in Computer Science, Springer. Authors will be invited to submit a revised and expanded version of their working note to be published in the post-proceedings.