Agriculture-Vision Challenge 2022
We use mean Intersection-over-Union (mIoU) as our main quantitative evaluation metric, which is one of the most commonly used measures in semantic segmentation datasets. The mIoU is computed as:
Where c is the number of annotation types (c = 9 in our dataset, with 8 patterns + background), Pc and Tc are the predicted mask and ground truth mask of class c respectively.
Since our annotations may overlap, we modify the canonical mIoU metric to accommodate this property. For pixels with multiple labels, a prediction of either label will be counted as a correct pixel classification for that label, and a prediction that does not contain any ground truth labels will be counted as an incorrect classification for all ground truth labels.
Concretely, we construct the confusion matrix Mc×c with the following rules:
For each prediction x and label set Y:
If x ⊆ Y, then My,y = My,y + 1 for each y in Y
Otherwise, M x,y = M x,y + 1 for each y in Y
The mIoU is finally computed by (true_positive) / (prediction + target - true_positive), averaged across all classes.
We are now hosting our challenge on Codalab. The competition page can be found here (Agriculture-Vision Challenge). Each participating team is required to register for the challenge. To register your team, fill out the registration form here (registration form) and register on the competition page.
*Make sure your Codalab account email matches one of the member emails in the registration form. Each team can only register once per challenge track.
All registered teams can evaluate their results on Codalab and publish their results on the leaderboard. The submission file should be a compressed .zip file that contains all prediction images. All prediction images should be in png format and the file names and image sizes should match the input images exactly. The prediction images will be converted to a 2D numpy array with the following code:
In the loaded numpy array, only 0-8 integer labels are allowed, and they represent the annotations in the following way:
0 - background
1 - double_plant
2 - drydown
3 - endrow
4 - nutrient_deficiency
5 - planter_skip
6 - water
7 - waterway
8 - weed_cluster
IMPORTANT: following our paper, the "storm_damage" category will not be evaluated.
This label order will be strictly followed during evaluation.
All teams can have 2 submissions per day and 20 submissions in total.
Final submission and prize reward
The Codalab leaderboard will be closed after the deadline. Top-tier teams in each challenge track will be invited through email to provide their final submission for the prize reward. The Final submission should also include a detailed report of the method and the necessary code to reproduce the results. If the final submission result can not be reproduced by the code provided, the participants of the respective submission will not be considered for the prize reward. The final submission will be a compressed .zip file that contains the following materials:
(field id #1)_(x1)-(y1)-(x2)-(y2).png (label predictions that matche the best mIoU on the leaderboard)
(field id #2)_(x1)-(y1)-(x2)-(y2).png
code/ (the training and inference code for the method)
models/ (pretrained model (if applicable) and the final model)
challenge_report.pdf (detailed description of the method)
To be considered as a valid submission for the prize reward, all submissions must satisfy the following requirements:
Model size will be limited below 150M parameters in total.
The mIoU derived from the "results/" folder in the final submission should match the mIoU on the leaderboard.
Predictions in "results/" in the final submission can be reproduced with the resources in "code/" and "models/".
The training process of the method can be reproduced and the retrained model should have a similar performance.
The test set is off-limits in training.
For fairness, teams need to specify what public datasets are used for training/pre-training their models in their challenge_ report.pdf. Results that are generated from models using private datasets, and results without such details will be excluded from prize evaluation.
The prize award will be granted to the top 3 teams for each challenge track on the leaderboard that provide a valid final submission.
NOTE: since our challenge deadline will now be after the paper submission deadline, challenge papers will no longer be accepted or included in the workshop proceedings.