MSR-Bing Image Retrieval Challenge (IRC) - FAQ 2014

Q. Do I have to register to participate?
A. Yes. To participate in the Challenge, participants need to register your team in the CMT system (to be announced). The CMT system will be also the venue for you to submit your prediciton results (and papers, if there is no paper system set up by the main conference).

Q. Can I compete even if I am not affiliated with an academic or industrial lab?
A. Yes.

Q. How do you evaluate the system?
A. The measurements are main page of this website. This year, we will do offline evaluation.

Q. Is this competition open to entrants who live outside the United States?
A. Individuals who live in Cuba, Iran, North Korea, Sudan, or Syria are not eligible to participate in the competition. Individuals living in other countries or regions may participate.

Q. Who is not eligible to compete in the challenge?
A.The challenge is not open to:

  • Entrants who are younger than 18 years of age;
  • Residents of any of the following countries: Cuba, Iran, North Korea, Sudan, and Syria;
  • Employees of Microsoft Corporation or an employee of a Microsoft subsidiary;
  • Persons involved in any part of the administration and execution of the competition; or
  • Immediate family (parent, sibling, spouse, child) or household members of a Microsoft employee, an employee of a Microsoft subsidiary, or a person involved in any part of the administration and execution of this Contest.

Q. Why does the challenge include separate training, development and the test datasets?
A. The test dataset is used to determine the winners of the challenge. There, we use expert judgments to determine the retrieval quality. In contrast, the training dataset is sampled from Bing's usage logs that contain only query, image and click information. Since clicked images are not necessarily the most relevant to the queries, and not-clicked images are not necessarily irrelevant, the challenge fpr the participants is to distill the large amount of click information to arrive at good relevant judgments. To help training such a system, a development set created in the same manner as the test dataset is provided so that participants can utilize the development set to refine their systems. Please see the "Datasets" section of the website for more details.

Q. Why does the challenge restrict the number of submissions on the final day of the competition?
A. The challenge will only accept three final test entries from each team. One of these entries must be listed as a primary, which is the only entry that is eligible to win a prize. The other two entries will be run against the final dataset to provide teams with feedback about the quality, but they will not be eligible for prizes (even if they outperform primary entries). We have restricted the number of submissions to avoid giving an unfair advantage to teams with the ability to develop a large number of submissions.

Q. Why do I need to choose one of my submissions as my "primary entry?" Can’t you just do that for me?
A. You are required to determine which system is your primary entry to be considered for the cash prizes. This limitation is designed to prevent participants with fewer resources from being disadvantaged, since participants with more resources could just provide three entries and get three chances at winning a prize. This also requires teams to judge for themselves which system they think is best – and then see if their intuition was correct.

We permit secondary submissions in order to enable teams to get feedback on more than one system, but the secondary submissions are not eligible for any prizes (regardless of whether they outperform other primary ones).

Q. Will you release the ground truth of the final Test Data set?
A. No. We will not. We will build a service to automatically evaluate prediction results submitted after the challenge.

Q. Who can I contact for other questions on the Challenge?
A. You can contact us at xshua@microsoft.com or participate in the community discussion (see below).

Q. Is there a place where I can interact with the participants of the Challenge?
A. Yes. We encourage participants to use the discussion forum of the Facebook and Twitter.