CVPR 2024 Suggested Practices for Authors
Reproducibility: Refer to this Reproducibility Checklist as a guide for making sure your paper is reproducible. Reviewers should follow this guide when evaluating papers as well. We highly encourage authors to voluntarily submit their code as part of supplementary material, especially if they plan to release it upon acceptance. Reviewers may optionally check this code to ensure the paper’s results are reproducible and trustworthy, but are not required to. We expect (but do not require) that the accompanying code will be submitted with accepted papers.
Release of code and data: In the spirit of reproducibility, we strongly encourage researchers to release the code and data associated with their papers. Both code and data (or representative samples or details thereof) can be submitted as part of the supplementary material to be optionally considered by reviewers.
If a paper submission is claiming a dataset release as one of its contributions, it is expected that the dataset will be made publicly available no later than the camera-ready deadline. Note that this does NOT imply that all datasets used in CVPR submissions must be public. The use of private or otherwise restricted datasets for training or experimentation is acceptable, but such datasets cannot be claimed as contributions of the paper as they do not become available to the scientific community.
Attribution of existing assets: Just like papers are expected to cite previous work that inspired a submission or on which a submission is built, we expect CVPR papers to cite assets, such as code or datasets, that have been used in the creation of the submitted manuscript. If there are multiple versions of an asset, specify the version you have been using. This attribution of assets can be made either in the main paper or in the supplemental material. We furthermore encourage authors to discuss the license and/or copyright terms of the assets used. The inclusion of a URL is encouraged as well, where appropriate.
Personal data / human subjects: If a paper makes use of personal data and/or data from human subjects, including personally identifiable information or offensive content, we expect that the collection and use of such data has been conducted carefully in accordance with the ethics guidelines. In many countries and institutions, the collection and use of personally identifiable data or data from human subjects is subject to approval from an Institutional Review Board (IRB, or equivalent). If the use of such data was approved by an IRB, stating this is sufficient. If the use of such data has not (yet) been approved by an IRB, authors should provide information on any pending approval process, how the data was obtained, as well as discuss if and how consent was obtained (or why it, perhaps, could not be obtained). This discussion can be included either in the main paper or in the supplemental material.
IRB reviews for the US or the appropriate local ethics approvals are typically required for new datasets in most countries. It is the dataset creators' responsibility to obtain them. If the authors use an existing, published dataset, we encourage, but do not require them to check how data was collected and whether consent was obtained. Our goal is to raise awareness of possible issues that might be ingrained in our community. Thus we would like to encourage dataset creators to provide this information to the public.
Discussion of limitations: Considering the limitations of an approach is an important part of good academic scholarship. Such discussion should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only held locally). Authors need to reflect on how these assumptions might be violated in practice and what the implications would be.
Authors should also reflect on the scope of their claims, e.g., if they only tested their approach on a specific type of imagery or did a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. The discussion should reflect on the factors that influence the performance of the approach. For example, a recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting.
We understand that authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection. It is worth keeping in mind that a worse outcome might be if reviewers discover limitations that are not acknowledged in the paper. In general, we advise authors to use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community.
FAQs
About Datasets
Q. I use a private dataset for my experiments that I cannot distribute publicly. Can I still submit a paper?
A. YES, you can absolutely submit a paper. You simply cannot claim the dataset as one of the paper’s contributions. The paper must stand on its other scientific merits (e.g. its technical contribution).
Q. I wish to claim a dataset contribution in my paper and release the dataset, but the data is not ready for release at submission time. Is this a problem?
A. NO, you can make minor modifications to the dataset until the camera-ready deadline. However, the scientific conclusions based on the dataset at submission time must be sound and must continue to hold. You must provide a link to the dataset only when submitting the camera ready.
Q. I wish to claim a dataset contribution in my paper and release the dataset, but the data may/will not be ready for release at the camera-ready deadline. Is this an issue?
A. YES, it is the expectation that your dataset will be ready and available at the time when submitting the camera ready, since at this point the paper that describes the dataset is considered final as well. If you cannot ensure that you can meet this deadline, then the release of the dataset should not be one of the major scientific contributions of your paper.
Q. I plan to release a dataset with my paper, but my dataset website requires users to create a log-in or otherwise request permission, before they can access the dataset. Is this permitted?
A. YES, this is permitted as long as you do not unreasonably withhold or delay access to members of the research community.
Q. Releasing the dataset is not fully under my control, e.g. because it needs to still be approved by an institutional review board. What should I do?
A. We recommend waiting with your submission until you have obtained the appropriate approval.
Q. Are there any specific license requirements for dataset contributions?
A. NO, the expectation is that the data is available for research use by members of the research community.
About the Attribution of Assets (Datasets & Code)
Q. Do I need to attribute assets that I used in my paper?
A. YES, you should treat assets such as datasets or code just like scientific papers.
Q. Do I really need to cite ALL assets that I have been using? For example, do I need to include a citation to Python, since I used Python to write my code?
A. NO, you should cite research assets. Use good judgement, just like you would for citations of previous scientific work. As an example, most papers will happily make use of the Gaussian distribution without making reference to Carl Friedrich Gauss. Similarly, it is fine to use general purpose programming languages or widely used programming tools without explicit citation. Research datasets or specific algorithms, even if widely used such as ImageNet or Adam, call for a citation, on the other hand.
Q. Do I need to include URLs of assets?
A. You are encouraged, but not required to do so. Including a URL is particularly sensible if the asset is not widely used or if the URL is not easily associated with the paper that introduced the asset.
Q. Do I need to discuss the licensing terms / copyright terms of the assets?
A. You are not required but encouraged to do so. It should go without saying that assets should be used in accordance with their license / copyright terms. Including information on the licensing terms / copyright terms shows that authors carefully considered this.
About Personally Identifiable Data / Human Subjects
Q. I use personally identifiable data or data from human subjects and I obtained IRB (or similar) approval. Do I need to include proof of approval?
A. NO. It is sufficient at submission time to state that you obtained IRB approval (either in the paper or in the supplemental material). Do not include any material that would identify your institution or de-anonymize your submission in another way.
Q. I use personally identifiable data or data from human subjects, but my IRB (or similar) approval is not completed. What should I do?
A. We recommend holding off on a submission. Note also that changes in the experimental protocol mandated by the IRB can no longer be accommodated once the paper has been submitted.
Q. I use personally identifiable data or data from human subjects, yet I do not / cannot get IRB approval, e.g. as my institution does not have an IRB. What should I do?
A. Carefully describe in your paper or supplemental material how you ensure that the collection and use of data follows the ethical principles set up in the ethics guidelines. One possible path would be to obtain clearance from an independent IRB service.
Q. I have a specific application, which makes it difficult to obtain consent for data collection. What should I do?
A. Explain your situation either in your paper or in the supplemental material and ensure to include sufficient information to support that you followed the ethical principles set up in the ethics guidelines.
Q. Can my paper get rejected for an inappropriate use of personal data or data from human subjects?
A. We want to increase awareness in the CVPR community of this important issue, hence asking authors to give information on the use of such data. Reviewers will be asked to flag any significant ethical concerns. These will be referred to an ethics committee, which will assess the situation and advise the program chairs. The program chairs reserve the right to reject papers with grave ethical issues, but expect this to occur only in exceptional circumstances.
Q. My research uses datasets that have been withdrawn by their creators, such as DukeMTMC-ReID or MS-Celeb-1M. What should I do?
A. Generally, papers should not use datasets that have been withdrawn by their creators, as doing so may involve ethical violations or even legal complications. Under some circumstances, authors may feel they need to use such datasets — for example, if fair comparison is impossible in any other way. However, authors who use such datasets should always explain the need to do so carefully and in some detail as such claims will be carefully scrutinized. Note that in many cases alternative datasets exist. The recommended course should be to not use the dataset, and (if necessary) explain that this may affect certain comparisons with prior art. It is a violation of policy for a referee or area chair to require comparison on a dataset that has been withdrawn without a detailed consultation with PCs or DEI chairs or ombuds.
Q. My research relies on broadly used public datasets of others, which have not been withdrawn, but for which it is unclear if they have been approved by an IRB. Is this allowed?
A. In the case of broadly used datasets that are still offered by their creators, for which IRB approval status is unclear, authors are encouraged to discuss the situation, e.g. why no better alternatives are available
About Potential Negative Societal Impact
Q. Does a paper automatically get rejected if it has a potential negative impact?
A. NO, technologies often have two sides. We encourage authors to discuss this, because it is beneficial for the community to be aware of this issue.
About Code Submission
Q. Is a code submission required?
A. NO, it is completely optional. But we encourage code submissions to aid reproducibility.
Q. I cannot submit my code, e.g. because it is proprietary. What should I do?
A. We understand that code may be proprietary, preventing you from submitting it. In this case, simply state so in the submission form. If there are other reasons for not being able to submit code, you are encouraged to explain this in your submission, such as in the supplemental material. Examples include code that is not sufficiently stand-alone to include (though see below for suggestions for how to handle such cases).
Q. Does submitted code need to be anonymized?
A. CVPR is a double blind conference, so authors should make a reasonable effort to anonymize the submitted code and data. This means that author names, institution names (also in copyright / license statements) should be removed. If the paper gets accepted, we expect the authors to replace the submitted code by a non-anonymized version or link to a public GitHub repository.
Q. Are anonymous GitHub links allowed?
A. Yes. However, they have to be on a branch that will not be modified after the submission deadline. Please enter the GitHub link in a standalone text file in a submitted ZIP file.
Q. How will the submitted code be used for decision-making?
A. The submitted code will be used as additional evidence provided by the authors to add more credibility to their results. We anticipate that high-quality papers whose results are judged by our reviewers to be credible will be accepted to CVPR, even if code is not submitted. However, if something is unclear in the paper, then code, if submitted, will provide an extra chance for reviewers to verify it.
Q. If code is submitted, do you expect it to be published with the rest of the supplementary? Or, could it be withdrawn later?
A. YES, we expect submitted code to be published with the rest of the supplementary. However, if the paper gets accepted, then the authors will get a chance to update the code before it is published by adding author names, licenses, etc.
Q. Do you expect the code to be standalone? For example, what if it is part of a much bigger codebase?
A. We expect your code to be readable and helpful to reviewers in verifying the credibility of your results. It is possible to do this through code that is not standalone -- for example, with proper documentation.
Q. My code requires a dataset that is too large to include as part of the supplemental material. Can I provide an anonymous link to the dataset?
A. Similar to links to anonymous code repositories on GitHub, it is permitted to provide anonymized links to datasets that are required to execute the code. Such links must not allow the discovery of the authors’ identity in any way. Moreover, such anonymous data storage must not allow authors to discover who accessed the data in any way
Q. What about pseudocode instead of code? Does that count as code submission?
A. Yes, we will count detailed pseudocode as code submission as it is helpful to reviewers in validating the credibility of your results.
Q. Do you expect authors to submit data?
A. We understand that many of our authors work with highly sensitive datasets, and are not asking for private data submission. If the dataset used is publicly available, there is no need to provide it. If the dataset is private, then the authors can submit a toy or simulated dataset to illustrate how the code works.
Q. Who has access to my code? For how long?
A. Only the program chairs, the technical chairs, and the reviewers and area chairs assigned to your paper will have access to your code. We will instruct reviewers and area chairs to keep the code submissions confidential (just like the paper submissions), and delete all code submissions from their machine at the end of the review cycle. Please note that code submission is also completely voluntary.
Q. I would like to revise my code/add code during author feedback. Is this permitted?
A. Unfortunately, no. But please remember that code submission is entirely optional.