Home   >  Competitions   > 

 

Deliverables


1、Work-in-process validation results. Each team can submit validation set forecast results anytime during the course of the contest in a csv file. The csv file should be named “temp.csv” and be in the following format (separated by a comma):

                                                 qid,uid,label

                                                 111,222,0.5

                                                 ......

    The qid is the question ID, the uid is the expert ID, and the label is the probability of that expert answering the question(float type,range from 0-1)

2、Final calculation result. Each team should submit the test set forecast results by the submission deadline. The csv file should be named “final.csv” and be in the same format with “temp.csv”.

Important: no extra spaces; add an empty line to the end of the file; file encode is utf-8 without bom


Evaluation

 

    In terms of evaluation, given a certain question, we will rank the experts based on the forecasted probability, and evaluate the NDCG@5 and NDCG@10 of ranking results. The final evaluation formula is:  NDCG@5 * 0.5 + NDCG@10 * 0.5.