Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
Submitting Runs to INTENT-2@NTCIR-10 (updated: June 19, 2012)

Run Types and Run Names

For the Chinese/Japanese Subtopic Mining and Document Rankings subtasks, we accept three types of runs:

R-run: A "Revived" run from NTCIR-9 INTENT-1. This applies only to those who also participated in the NTCIR-9 INTENT task. If the system that you used to generate runs for INTENT-1 is still available, we encourage you to use the system again to process all 200 topics (100 INTENT-1 and 100 INTENT-2 topics) to generate an R-run. This will help us monitor technological advances. (The system does not have to be completely identical to your INTENT-1 system. For example, you might have replaced a tokenizer after the INTENT-1 run submission. Also, please use the OFFICIAL query suggestions we provided if your INTENT-1 system makes use of query suggestions.)

B-run: Any new run that uses the Baseline nondiversified search results we provided in one way or another. B-runs should also process all 200 topics.

A-run: Any other new run. A-runs should also process all 200 topics.

For the English Subtopic Mining subtasks, there will only be A-runs, as we did not privide any baseline search results, and as the subtask was not run at NTCIR-9 INTENT-1. As there are only 50 topics (kindly provided by the TREC 2012 diversity task), Any English A-run should process 50 topics.

Run Names

Run files should be named as follows. Make your your <teamID> is exactly what you registered.

Document Ranking R-runs:

<teamID>-D-[CJ]-R[priority].txt

e.g.

MSRA-D-J-R1.txt (J means Japanese - use C for Chinese; 1 means this run has the highest priority to be included in the pool)

Document Ranking A-runs and  B-runs:

<teamID>-D-[CJ]-[priority][AB].txt

e.g.

MSRA-D-J-1A.txt

MSRA-D-J-2A.txt

MSRA-D-J-3B.txt

Subtopic Mining R-Runs:

<teamID>-S-[CJ]-R[priority].txt

e.g.

MSRA-S-J-R1.txt (J means Japanese - use C for Chinese; 1 means this run has the highest priority to be included in the pool)

Subtopic Mining A-runs and B-runs:

<teamID>-S-[CEJ]-[priority][AB].txt

e.g.

MSRA-S-J-1A.txt

MSRA-S-J-2A.txt

MSRA-S-J-3B.txt

Number of Runs

For each subtask-language pair (e.g. Chinese Subtopic Mining), a participating team can submit up to five NEW runs (i.e. A- and B-runs), plus up to two R-runs. Thus:

subtask-language 

max. number of runs per team 
Chinese Document Ranking  5 new runs + 2 R-runs = 7 runs 
Japanese Document Ranking  5 new runs + 2 R-runs = 7 runs 
Chinese Subtopic Mining  5 new runs + 2 R-runs = 7 runs  
Japanese Subtopic Mining  5 new runs + 2 R-runs = 7 runs  
English Subtopic Mining  5 A-runs 

 

NOTE: Please don't do this: you have only one algorithm, but you submit two runs by trying two different parameter settings: e.g. param=0.8 for Run 1, param=0.2 for Run 2. THIS IS LIKE TUNING WITH THE TEST DATA! (Exceptions: if, for example, param=0.0 means switching off a certain component of your system, then yes you might want to try both param=0.0 and param=0.8 etc.)

Document Ranking Run Submission Format

For those of you who are familiar with the TREC run format, our Document Ranking run format is basically the same. The only difference is that we require a one-sentence description of the run in line 1 of each runfile.

For A-runs and B-runs for Document Ranking, Line 1 of the runfile must be of the form:

<SYSDESC>[insert a short description in English here]</SYSDESC>

For R-runs, Line 1 must be of the form:

<SYSDESC>[INTENT1NAME]<space>[insert a short description in English]</SYSDESC>

where [INTENT1NAME] is the corresponding runname from NTCIR-9 INTENT-1. This is because we want to know which of your R-runs are equivalent to your runs submitted to INTENT-1.

 

The rest of the file should contain lines of the form:

[TopicID] 0 [DocumentID] [Rank] [Score] [RunName]\n

Use a white space as the field separator. For example, a Japanese Document ranking run should look like this:

0301 0 clueweb09-ja0006-97-23810 1 27.73 MSRA-D-J-1A

0301 0 clueweb09-ja0009-08-98321 2 25.15 MSRA-D-J-1A

0301 0 clueweb09-ja0003-71-19833 3 21.89 MSRA-D-J-1A

0301 0 clueweb09-ja0002-66-03897 4 13.57 MSRA-D-J-1A

Return no more than 1,000 documents per topic. The runfile should be saved as an UTF-8 encoded file (for consistency with Subtopic Mining).

In fact, during evaluation using NTCIREVAL, we will in completely ignore all of the above fields except for [TopicID] and [DocumentID]. The ranked document lists are taken "as is."

Your Chinese Document Ranking run should contain ranked lists for 200 topics - 0001-0100 (i.e. INTENT-1 C topics) and 0201-0300 (i.e. INTENT-2 C topics).

Your Japanese Document Ranking run should contain ranked lists for 200 topics - 0101-0200 (i.e. INTENT-1 J topics) and 0301-0400 (i.e. INTENT-2 J topics).

It's okay if your ranked lists are empty for some topics.

Subtopic Mining Run Submission Format

The Subtopic Mining run format is basically the same as the Document Ranking run format. The only difference is the separator: instead of a white space, a semicolon ";" is used, as some subtopics may contain white spaces.

For A-runs and B-runs for Subtopic Mining, Line 1 of the runfile must be of the form:

<SYSDESC>[insert a short description in English here]</SYSDESC>

For R-runs, Line 1 must be of the form:

<SYSDESC>[INTENT1NAME]<space>[insert a short description in English]</SYSDESC>

where [INTENT1NAME] is the corresponding runname from NTCIR-9 INTENT-1. This is because we want to know which of your R-runs are equivalent to your runs submitted to INTENT-1.

 

The rest of the file should contain lines of the form:

 [TopicID];0;[Subtopic];[Rank];[Score];[RunName]\n”

 For example, an English Subtopic Mining run should look like this:

0401;0;Windows Phone 7;1;0.98;MSRA-S-E-1A

0401;0;Windows 7;2;0.97;MSRA-S-E-1A

0401;0;Windows Update;3;0.9;MSRA-S-E-1A

0401;0;House Windows;4;0.85;MSRA-S-E-1A

Return no more than 100 subtopics per topic. The runfile should be saved as an UTF-8 encoded file. Make sure your subtopics do not contain any semicolon ";" or backslash "\".

In fact, during evaluation using NTCIREVAL, we will in completely ignore all of the above fields except for [TopicID] and [Subtopic]. The ranked subtopic lists are taken "as is."

Your Chinese Subtopic Mining run should contain ranked lists for 200 topics - 0001-0100 (i.e. INTENT-1 C topics) and 0201-0300 (i.e. INTENT-2 C topics).

Your Chinese Subtopic Mining run should contain ranked lists for 200 topics - 0101-0200 (i.e. INTENT-1 J topics) and 0301-0400 (i.e. INTENT-2 J topics).

Your English Subtopic Mining run should contain ranked lists for 50 topics - 0401-0450 (TREC 2012 web topics 151-200).

Checking Your Runs

All Subtopic Mining and Document Ranking run files should be checked for formatting errors using the CheckIntent script. We will not evaluate badly formatted run files.

Some common problems found in Subtopic Mining run files:

  • (1) Bad character codes: from U+E000 to U+F8FF, U+FFFD (replacement char), and U+200B (zero-width space).
  • (2) Unnecessary white space before and after subtopic string.
  • (3) More than one white space continuously occurs in subtopic string.
  • (4) Use of "\" within subtopic string.
  • (5) Text visually garbled for other reasons.

 

CheckIntent.pl may detect problems (1)-(4). First, try

% CheckIntent.pl <runfile>

and if a problem is detected, execute

% CheckIntent.pl <runfile> <fixedrunfile>

to fix the problems. (Please ignore messages such as "Wide character in print at [path]\CheckIntent.pl line 133, <RES> line xxx.") Then remember to check and rename your runfiles, e.g.

% diff <fixedrunfile> <runfile>

% mv <fixedrunfile> <runfile>

Although CheckIntent.pl does NOT automatically solve problem (5), this problem often occurs along with bad character codes. So it is worthwhile to manually check the lines for which warning messages were obtained.

Uploading Your Runs

  1. Create a single file for your team called "<teamID>-INTENT2.tar.gz". This file should contain ALL of your Subtopic Mining and Document Ranking run files, PLUS a plain text file called "<teamID>-INTENT2.txt" which contains a complete list of your runs, one run file name per line. Thus if you are submitting five runs, the text file should contain exactly five lines. The summary file should contain no more than 33 lines: see the Number of Runs section.
  2. Upload the gzipped tar file to our run submission site. The URL will be announced to registered participants only.