Frequently Asked Questions (FAQ)
--- Challenge ---
1. What is CholecTriplet2022?
An Endoscopic vision challenge on the recognition and localization of tool-tissue interactions in surgical videos in the form of triplets. It is an upgrade of the CholecTriplet2021 challenge with localization task.
2. What is a triplet?
A combination of {instrument,verb,anatomical target} that describes a surgical action.
3. Who can participate?
Anyone who signs the challenge agreement except the members of the organizing lab.
4. Will the challenge submission still be open after the submission deadline?
Likely yes, but only submissions made before the deadline will be eligible for awards.
--- Registration ---
1. Why is my registration not yet approved?
For your registration to be approved, you must send a signed challenge contract. Check the Getting Started page for more details.
2. Must every member of my team submit a signed contract?
One contract per team is sufficient. However, every member of the team must abide by the terms and conditions in the signed contract. The dataset obtained after signature of this contract remains confidential and cannot be transferred to someone outside the team.
3. Is it mandatory to register as a team?
Yes.
4. What if I am working alone, must I still register as a team?
Yes. A team can comprise of only one person.
5. Must every member of my team register on the challenge website?
Yes.
6. When is the deadline for team registration?
All registrations end on July 1, 2022.
--- CholecT50 dataset ---
1. What labels in the dataset should be used to train the model?
Triplet labels. The standalone instrument labels, verb labels, and target labels are provided as additional labels should in case they can help your modeling. Their usage is optional and entirely depends on your propose methods. The localization can be learnt by weak supervision preferably using the instrument presence labels or the triplet labels.
2. I found some action triplet marked null for the verb & target components in the ground truth, is this an error?
No, some clinically valid triplets are not in the considered 100 triplet classes, due to their occurrence frequency and clinical relevance to the considered procedure.
3. How many types of null triplets are possible in the dataset?
The possible null triplet classes can be grouped into two as follows:
a.) Instrument inclusive null: this is seen in a situation where there is no instrument in the frame, or where the instrument involved in the action is not from valid classes in the dataset. In this case, the label is {null-instrument,null-verb,null-target}. Since triplet is a multi-label classification problem, this class is true only when all other classes are negative in one frame. It is NOT included in the 100 triplet classes.
{grasper,null-verb,null-target}, {bipolar,null-verb,null-target}, {hook,null-verb,null-target}, {scissors,null-verb,null-target}, {clipper,null-verb,null-target}, and {irrigator,null-verb,null-target}.4. Is the train/val/test split different from the one used in the published papers?
Yes. For the challenge, we restrict the test set to only videos that are not in the public domain. We recommend 40/5 train/val split on the provided data but participants are entirely free to define their own splits. The training set is the entire CholecT45 dataset [7].
5. Is the challenge test dataset publicly available?
No, while the trained dataset are 45 videos of publicly available Cholec80 [1], the test set is a private dataset (not Cholec80) of the same type of surgery.
6. I have observed that there are some black images in the dataset, are these images corrupted?
No, as a privacy protection measure, we zeroed out all images that display the faces of the clinicians or the patients. For temporal consistency reason, we do not remove the zeroed frames from the dataset.
--- My Challenge Methods ---
1. What is the expected output of the model?
A model produces two type of outputs per frame:
- A vector of N=100 probability scores for the triplet recognition in the format
[score1, score2, ..., scoreN]. - A list of box-triplet pairing for each positive triplet instance in the format: [
[tripletID, instrumentID, confidence, x, y, w, h], [tripletID, instrumentID, confidence, x, y, w, h], . . .]
2. Do I need to predict the instrument, verb and target separately?
No, while you may want to leverage the extra annotations provided for instrument, verb and target to improve your model, you are only required to predict the final triplet IDs as a vector[100] of probability scores.
3. Will the inference pipeline preserve the temporal information?
Yes. Participants are free to train their model either on sequential or shuffled frames. During testing, we will use an input setup that preserves the temporal frame order per video.
4. What is the frame-rate for the test set?
1 FPS. Same as the train data.
5. Is the testing going to be an online prediction?
Yes. We will maintain a real-time scenario during testing. This means that our test input setup will collect your model's outputs at time t before feeding the input frame at time t+1. Your method can accumulate and utilize previous frames information but not a future one.
5. My model performance is quite low, do I still need to submit?
Triplet recognition task is generally challenging. The average performance of a random model is 0.01%. So if you beat this performance, you have a good method to submit for the competition.
--- Baseline Methods ---
1. Where can I find a published paper/article on surgical triplet recognition?
* Tripnet [2]: first deep learning model for the recognition of action triplets in surgical videos:
[Nwoye C.I. et.al, Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets, MICCAI 2020]
Please, note that the models in this paper are trained and evaluated on CholecT40 (a subset of CholecT50)
* Rendezvous [3]: the journal extension of Tripnet baseline, trained on CholecT50:
[Nwoye C.I. et.al, Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos]
* Summary of methods and results in the previous action triplet challenge [4]:
[Nwoye C.I. et.al, Cholectriplet2021: A benchmark challenge for surgical action triplet recognition]
2. Where can I find a trained model (code) on triplet recognition?
Some code are available on CAMMA public git repo: https://github.com/CAMMA-public
We also provide a sample code in the colab code blog to help you get started.
3. Must I follow the same strategy as in the published papers?
No, you are free to develop any method that works for you: deep learning, machine learning, rule-based inference, etc.
4. Can I submit exactly the same model as in the published papers?
Submitting original and novel method is highly recommended, however, you are not constrained on what to submit.
--- Training ---
1. Is pretraining on a surgical dataset allowed?
Yes, you are free to pretrain your model on any third-party public dataset. Additional use of any private dataset is not allowed for this challenge.
--- Submission ---
1. How do I submit my method?
Methods are to be submitted as docker file. We will provide a docker template and submission guideline by June 2022.
The submission channel will open on July 1, 2022.
--- Evaluation ---
1. What is the metrics for the evaluation?
Our evaluation will be based on mean average precision (mAP) provided by ivtmetrics library [6]. The ivtmetrics is a special library for the evaluation of tool-tissue interaction detection. It can be installed using either pip or conda python library installer. More details about the metrics and its usage can be found on the method and evaluation page.
2. What will my model be evaluated on?
We will evaluate each model on 3 criteria:
- Triplet recognition performance (mean average precision mAP).
- Instrument localization performance (mAP at box IoU of 0.5).
- Triplet detection performance (correct triplet-box matching).
--- Publication ---
1. Will my challenge submission be published?
We plan a joint publication of the surgical action triplet detection which will include the submitted challenge models and results. More information will be provided on this as time goes on.
2. Who will be co-authors?
Top N performing team can submit at most 2 qualifying authors. The sub-challenge organizers determine the order of the authors in a joint challenge paper. Information on the N number of teams will be provided before the challenge presentation and would depend on the number of participation.
3. When can a participant publish an independent research on this dataset?
Participants are allowed to publish their own results separately on triplet recognition using only the publicly released CholecT45 dataset [7]. However, triplet detection and localization results which is the prime focus of this challenge cannot be published until after a publication of a joint challenge paper. Publication cannot be made on CholecT50 [3] before the joint publication.
4. When will the joint results be published?
This should be expected before the end of 2023.