Share on Facebook Tweet on Twitter Share on LinkedIn Share by email
MSR-Bing Image Retrieval Challenge @ ICME 2014

Detailed Message About the Challenge (@ICME 2014)

Update History

  • April 23: Evaluation result is available here. Congratulations! Paper notification will be sent on 4/26(New!).
  • April 7: Password-protected test set is avaliable for downloading. Please check your email (sent from CMT).
  • April 1: Trial set is available for download  (see details below)
  • Mar 27: CMT System is up
  • Mar 24: Announce evaluation rules

Evaluation Process

To enable more team to participant the grand challenge, we will change the evaluation method from online to offline:

You will be asked to download one compressed file (evaluation set) which contains two files in text formats. One is a list of query-key pairs, and the other is a list of key-image pairs. You will be running your algorithm to give a relevance score for each pairs in the first file, and the image content, which will be Base64 encoded JPEG files, can be found in the second file through the key.

The evaluation set, which is encrypted, will be available for downloading 3 days before the challenge starts. A password will be delivered to all participants to decrypt the evaluation set when the challenge starts.

One full day (24 hours) will be given to the participants to do predictions on all the query-image pairs in the evaluation set.Before the end of the challenge, participants need to submit the evaluation results (which is a text file containing a list of triad: query, key, score) to a CMT system (will be available soon). The order of the triads in the text file is not important. Running prediction in parallel is encouraged.

The evaluation set will be different from what we used last year. The number of query-image pairs will be increased significantly this time. A trial set will be available around one week before the start of the challenge.

One team can submit up to 3 runs in 3 text files, and each file corresponds to the results of one run.

Trial Dataset Download (New!)

  1. The trial dataset can be downloaded here (13.6M).
  2. It is a .tar file. If you can untar it successully, you should get the file correctly.
  3. After untar, you will get a zip file that is protected by a password.
  4. The password will be sent to registered users via emails.
  5. We will follow the sampe process to deliver final test set.
  6. You can try uploading your prediction results and then email to confirm the submission format.

Important Dates (Updated)

  • April 1: Trial set available for download.
  • April 2: Deadline for the participants to register on the CMT system for submitting evaluation results. April 7: Encrypted evaluation set is available for downloading.
  • April 10: Challenge start 0:00AM PDT, password will be sent to emails that are registered in the challenge CMT system.
  • April 11: Challenge ends 0:00AM PDT. Submission system closes.
  • April 16: Challenge paper submission deadline (4-6 pages following the same guideline of the main conference).
  • April 23: Notification.
  • April 30: Camera ready.

Prediction Result Submission Entrance:

Participants are required to register their teams by April 2. Please note the prediction submission site share the same main entrance with ICME 2014 workshop papers. Please choose "MSR-Bing Image Retrevial Challenge" to submit you results (by the end of April 10) and papers (by the end of April 16.

Useful Links


Please email to