Review History


To increase transparency, PeerJ operates a system of 'optional signed reviews and history'. This takes two forms: (1) peer reviewers are encouraged, but not required, to provide their names (if they do so, then their profile page records the articles they have reviewed), and (2) authors are given the option of reproducing their entire peer review history alongside their published article (in which case the complete peer review process is provided, including revisions, rebuttal letters and editor decision letters).

New to public reviews? Learn more about optional signed reviews and how to write a better rebuttal letter.

Summary

  • The initial submission of this article was received on November 28th, 2013 and was peer-reviewed by the Academic Editor.
  • The Academic Editor made their initial decision on December 3rd, 2013.
  • The first revision was submitted on December 20th, 2013 and was reviewed by 4 reviewers and the Academic Editor.
  • A further revision was submitted on February 24th, 2014 and was reviewed by the Academic Editor.
  • The article was Accepted by the Academic Editor on February 27th, 2014.

Version 0.3 (accepted)

· · Academic Editor

Accept

Paul,

I happy to accept your paper. It is a very interesting and important study. I am glad you were able to come up with a way to publish that data.

Version 0.2

· · Academic Editor

Minor Revisions

Dear Paul,

To recap I originally decided after feedback from two external reviewers that it would not be appropriate to publish your paper without the approval of your institutional review board. After you had clarified that your employer did not want any involvement with this project and that you had maintained the web site and conducted this study as a private individual we agreed to reconsider the manuscript.

Four additional consultants with a range of backgrounds have reviewed your manuscript. I am pleased to be able to accept the paper for publication in PeerJ if you would be willing to make some modifications.

1. The first review who is an expert in medical ethics feels your study would be exempt as long as the individuals who supplied the tips remain anonymous. Please confirm this is the case in the manuscript and explain how they were able to email the allegations to your site without revealing their identities.

2. The allegations of data integrity problems that make up the data for this study are not well characterized. It would be helpful to provide more information about the types of allegations that were submitted to your website.

It appears most of the complaints consist of alleged image manipulation. I believe in most cases these are images from research in the life sciences such as western blots. For those readers not familiar with the issues involved in potentially manipulating these images, it would be helpful to provide a few paragraphs framing the controversies going on in this area.

3. The fourth point of the second reviewer concerning unfounded allegations seems quite compelling. I believe it would enhance the paper to address this issue in the discussion.

4. Both the third and fourth reviewers raise a very difficult issue. PeerJ's policy is to require research data to be made available for review. Given the extremely sensitive nature of the allegations that formed the data for this study, information about the specific allegations that would allow the articles and hence authors involved to be identified could not be made available to readers or the reviewers. After a great deal of thought I feel the benefits of publishing your paper outweigh the very legitimate concern of these reviewers. That said, the paper should make it clear the database of allegations is not being made available and the reasons why.

There is additional feedback and suggestions from the four reviewers in their comments below. Please consider their suggestions implementing what you feel to be appropriate.

As long as you can address the four issues above in a revision, I will be happy to accept your paper.

·

Basic reporting

This is an interesting paper that tackles an important question, and is clearly written

Experimental design

This is a problem. Although the major independent variable the author wants to study is whether the allegations were public or private, there are any number of uncontrolled factors that could have influenced retraction rates, etc., and no way to determine whether these factors vary significantly between the public and private groups.

See my comments on the manuscript. To his credit, the author acknowledges this, on line 225.

Validity of the findings

The chief problems here are described above.

Reviewer 2 ·

Basic reporting

This paper reports an analysis of 497 ‘problem’ papers and concludes that the set for which the problems were widely known was much more likely to be retracted or corrected. Although the results are not surprising considering human nature the demonstration of this effect is a very important finding. Overall this is an important study because it suggests that sunshine can help the scientific community clean up the literature more rapidly. The paper is well written and the author is appropriated cautious on its conclusions, especially given the low numbers for some of the retracted paper categories.
Other comments.
1. The first sentence states that the scientific process is inherently ‘self-correcting’. This is a widely held assumption for which there is no hard evidence.
2. In paragraph 2 references 4 and 5 are contrasted as contradicting. The PNAS study found a much larger prevalence of misconduct because it used a different methodology which showed that one cannot rely on retraction notices for accurate information. Basically they correlated retraction notices with office of research integrity and found that many of the retraction notices for papers subsequently found to include misconduct were misleading.
3. In line 223 the author notes as a caveat the fact that this paper was done outside of IRB review. It is not clear to this reviewer why this study would be subject to human research oversight and it is not clear that IRB oversight would have improved this study.
4. Although this reviewer has no problem with the conclusions and is a supporter of more sunshine in science, I have a nagging concern that unfounded allegations by themselves can tarnish papers and lead to great efforts by investigators to defend their work – effort that would not go into new investigations and cause additional waste. Anonymous allegations can be used to derail the work of competitors and for payback in conflicts between scientists. When there is no misconduct an allegation of misconduct can lead investigators into the difficult situation of having to prove a negative. Hence, increased publicity can be a double edge sword and we need to find ways to maximize its benefits and minimize its debits. Some discussion of this conundrum could be a welcomed addition to the paper.

Experimental design

see above

Validity of the findings

see above

Comments for the author

see above

·

Basic reporting

Please see general comments.

Experimental design

Please see general comments.

Validity of the findings

Please see general comments.

Comments for the author

While I appreciate the sensitive issues around this manuscript, and welcome all attempts to correct the scientific literature, I am reluctant to offer a review without being able to see the data upon which the findings are based. The decision to not make those data available is based on sound reasoning, but it still means that this paper is not being held to the same standard as others. If we demand deposition of data, it should be for all papers. This doesn't mean I think the author should necessarily reverse his decision, just that I would be uncomfortable making a decision without access to the data.

I have chosen "major revision" because I think the paper is worthy of review and I would consider making the data available a required major revision.

Reviewer 4 ·

Basic reporting

no comments

Experimental design

As pointed out by the author in Notes for Reviewers: "The key problem here is that the data set itself comprises a list of problems in the published literature. It's easy to see how this could be construed as a series of allegations of misconduct, and as such this could be a libelous document. It is therefore impossible to make the primary data set available for readers."

The manuscript fails to meet PeerJ's fourth criterion under Experimental Design: "Methods should be described with sufficient information to be reproducible by another investigator."

Therefore, the Experimental Design is deemed to be faulty.

Validity of the findings

The manuscript fails to meet PeerJ's second criterion under Validity of Findings: "The data on which the conclusions are based must be provided or made available in an acceptable discipline-specific repository."
As pointed out by the author in Notes for Reviewers: "The key problem here is that the data set itself comprises a list of problems in the published literature. It's easy to see how this could be construed as a series of allegations of misconduct, and as such this could be a libelous document. It is therefore impossible to make the primary data set available for readers."

Version 0.1 (original submission)

· · Academic Editor

Major Revisions

Dear Dr. Brookes,

This is an interesting and important study however the research protocol needs to be reviewed by an institutional review board. Since it is a somewhat unusual situation, after consulting with Dr. Binfield we asked two experts in human subject protections to read the manuscript and both agreed this study needs be reviewed by an IRB. Even if the IRB decides the study is exempt, that determination should be made by an IRB not by the researcher.

I am sorry for the inconvenience this causes you and urge you to resubmit after receiving IRB approval. Again, this is a valuable study that I hope you will be able to publish. I feel however it is in the best interest of PeerJ as well as for your own protection that you receive approval from an IRB before it goes out for further review and can be published.

Best wishes,

All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.