All reviews of published articles are made public. This includes manuscript files, peer review comments, author rebuttals and revised materials. Note: This was optional for articles submitted before 13 February 2023.
Peer reviewers are encouraged (but not required) to provide their names to the authors when submitting their peer review. If they agree to provide their name, then their personal profile page will reflect a public acknowledgment that they performed a review (even if the article is rejected). If the article is accepted, then reviewers who provided their name will be associated with the article itself.
Thanks for considering our requested changes and submitting your revised manuscript. I verified that the new version of the Colab notebook works as advertised - this is sufficient notwithstanding future changes to Colab, in which case only minor changes to the notebook would be necessary to reproduce the trained network.
Whilst reviewing the manuscript I noticed a couple of very minor issues that should be addressed in the final proof:
1. Line 25 - abstract "these results are GOOD ..." - capitalisation is undesirable and not advised here, though I fully understand your sentiment ! Perhaps a more traditional 'surprisingly good' could be used in instead ?
2. Line 109-110: "Applications such as iNaturalist and Epicollect are generally used to report enlistments to detect certain species present in the images". - 'enlistments' seems to not be the correct choice of word for this sentence, which its context suggest should state that iNaturalist and Epicollect are designed to measure species presence/absence rather than abundance estimation or surveillance.
Finally, may I simply echo the sentiment of Reviewer 1 in the previous round, and wish your project the very best of success in the future.
[PeerJ Staff Note - this decision was reviewed and approved by Paula Soares, a PeerJ Section Editor covering this Section ]
Thank you for your resubmission. The resubmitted manuscript reads very well overall, but there still remain certain outstanding issues that need to be addressed.
R1 notes a small number of typographic and more critical issues - citing lines in the 'Differences' pdf rather than the revised manuscript. The critical issues are:
1. more clearly articulate why existing crowdsourcing tools were not appropriate. Here R1 highlights two places where more specific explanation is needed. Firstly, why do existing platforms not allow bulk download of images relevant to the study, and secondly, why do existing platforms not offer the 'diversity' necessary - here the reviewer's suggestions suggest they too have not fully understood what you mean, since I recognise the critical need here is 'observations' (ie periodic recordings over time) rather than 'sightings' (an image captured to indicate a species is present).
2. Reproducibility. Conda's requirements.txt needs to be made truly portable. The reviewer did not see the comment in the python script about downloading the Sargassum image dataset into the correct location - however that could be addressed by simply printing these instructions if the script cannot locate the file. The reviewer suggests providing all files as supplementary information - I suggest instead you do the following
i. adapt the script to tell the user to download the files - I would here recommend using the figshare doi https://doi.org/10.6084/m9.figshare.13256174.v5 rather than the hard link, since that includes a version number for the file. Please also reference the figshare data as a citation rather than a URL in the main manuscript.
ii. Consider creating a truly portable and reusable version of the notebook and/or python script by committing the scripts, requirements.txt and notebook files to github and verifying the notebook can be run via mybinder.org. To archive this repository in compliance with PeerJ's supporting data & code rules, you need to first link the github repository to zenodo and then create a version tag in the repository.. that will mint a DOI via Zenodo which can be cited in the paper.
3. R2 is also concerned that no detail is provided about the provenance of the training data. Ideally, details of the recruitment and training process should be documented somewhere - I do not see this as a critical concern for assessing training data bias, but it is good practice to record and publish these details as supporting information.
In addition, my own revisions are noted below:
4. Figure 3 legend. Use the Figshare citation rather than the raw URL. Please also give image filenames for the respective images so they can be located in the training data. I also assumed these photos are also all positive rather than negative examples? The legend should clearly state the class of these images.
5. Line 417-418: "It is also clear that because the AlexNet loss (Figure 12B) for the evaluation dataset remained below the test dataset, the accuracy achieved by the network is of 94%."
This statement is problematic. Accuracy is usually understood as a function of performance on the test data - so even if the training accuracy was higher, it is not appropriate to quote it without explicitly stating so, ie 'accuracy during training', or 'accuracy for recall' (as you refer to this number later). However, I note from Fig 13 that (225+215)/480 ~ 91.6 - so reported accuracy should be 92% - and this is quoted in table 3 in the next section. To avoid confusion, please explicitly state 'the accuracy achieved by the network during training is 94%' here.
6. Figure 13. axis Labels on the confusion matrices are wrong (0 and 1 switched around for one of them!).
7. Line 426: "92%. This is considered a significant result considering that the AlexNet Neural Network training was performed with a relatively small set of unbalanced images."
Significance implies statistical assessment, which hasn't really been done here. I suggest you reword to: 'This can be considered a good result considering..' to avoid implying that is the case. Ideally you would compare this performance with accuracies for trained AlexNet instances for other problems to give a better idea of whether 94% is a 'significant' result for this class of problem, but I don't know of such a reference.
You should also change 'unbalanced' to 'balanced' - this looks like a typo, since you have equal numbers of positive/negative images in your original dataset, and presumably these were evenly sampled in your random subsets.
This sentence also appears in line 482-483, and should be similarly revised, or even better, omitted, since it could mislead the reader about the statistical validity of the AlexNet performance stats.
Another approach would be to comment in your discussion or conclusions that 92% accuracy makes this network sufficiently accurate to be used to conduct surveys on sargassum distribution. Ideally of course you could then use this 92% value as a prior to assign confidence scores to the inferred sargassum distribution in your spatial distribution surveys: I don't think that is necessary for this proof of principle, but should be considered as essential for further published work or in guidance for remedial activities that may be proposed using your system.
Rewordings and typographic revisions
T1. A scattering of typos: 'iNaturalis' rather than 'iNaturalist', 'Epicollet' rather than 'Epicollect', s with an accent used instead of apostrophe 's' for 'AlexNet's' (e.g. line 402)
T2. Line 117 - space needed after '.'
S1. Line 401 recommend revising sentence to "This has a definite effect on the model’s ability to generalize."
S2. Line 458 recommend revising ".. or Playa del Carmen, which is why the presence of Sargassum in Progreso is not as critical." to "..or Playa del Carmen, so the presence of Sargassum in Progreso is not as critical."
Line 50: "Rodríguez-Martínez et al., 2020" does not need parentheses when directly referenced in the sentence.
Lines 66-67: "Outlook of 2020" should be cited
Lines 119-121: "The main advantages of this platform are the data is openly accessible on the Web and that much of it is licensed for re-use or free of intellectual property restrictions." This is missing the word "that" between "are" and "the."
Lines 134-138: While the revision emphasizes the subjective nature of the authors' choice not to use existing crowdsourcing tools, it does not adequately justify that choice, in my opinion. What about these existing platforms makes it "difficult to automate the acquisition, storage, and processing of images"? Do they lack application programming interfaces that would enable data extraction? Are they not translated into the languages that your expected data contributors speak? Do they make data available in a format that was unfamiliar to the authors or whoever built the custom tools for this project?
Lines 186-191: Again, terms like "diversity" and "attributes" are not defined in context to the point that a scientist in another field would understand what they mean. I assume that the authors mean "diversity" as in the diversity of visual attributes like angle, background elements, lighting, lens distortion, etc. that would make an ML model robust enough to classify input images with a high degree of variance in these attributes, but that needs to be explained.
The requirements.txt that the authors included (cs-55549-requirements.txt in the supplementary materials) is not usable to install the dependencies required to run their code because it still contains numerous references to files on the authors' computers as opposed to simply naming packages and versions. For example, the first line is `anyio @ file:///home/conda/feedstock_root/build_artifacts/anyio_1610315727443/work/dist`. In addition to referencing a directory hierarchy that is specific to the authors' computers, it doesn't reference a version. These should be re-generated so a line like this would appear as `anyio==2.1.0`.
Since the authors mentioned they had intended their notebook to be run in Google Colab, I tried uploading the .ipynb file there to see if it would run, but it errored out on this line:
zip_ref = zipfile.ZipFile("./data/sargassum_dataset.zip", 'r')
The file "./data/sargassum_dataset.zip" was not included with the supplementary materials so this code also will not run. I see the URL to this zip on figshare was included in the caption to Figure 3, and using that file in Google Colab I can get the notebook to run, but that is not where I would have expected to find the data required to run the code. In my opinion, it should all be included with the supplementary materials.
I would highly recommend that the authors try to run their own code on a fresh machine using only the materials provided with the paper to ensure all required software and data are accessible and referenced correctly.
no comment
I am still not seeing an explanation of how participants were acquired or trained and how that acquisition and other aspects of participant behavior may have biased the training data. You did a good job of explaining the limitations of a crowdsourcing approach due to these kinds of bias in your rebuttal, but I wish you had included that in the paper as well.
N/A
N/A
N/A
I think the authors have done a good job in strengthening their manuscript and addressing comments from both Reviewers. It sounds like a cool project and a good example of interdisplinary research to help a fundamental problem in many coastal parts of the world. Good luck in growing the project and keeping it afloat.
All the best.
Both reviewers have provided quite detailed comments on your manuscript - highlighting and suggesting ways to address issues of clarity, typography and reproducibility. They also note that the paper's focus on identification of a model for quantification of Sargassum distribution in images does not help the reader assess the overall design, implementation, and current state of the 'CollectiveView' project as a whole.
In addition to their observations, please consider figshare or Zenodo for depositing data and any associated scripts used to generate figures. Both these services would allow individuals responsible for figure creation to gain attribution for their work, which is superior to a simple acknowledgement, and also improve reproducibility for Figure 14 and 15. See also https://peerj.com/about/author-instructions/#figure-referencing and https://peerj.com/about/author-instructions/#data-and-materials
"Diadema antillarum" on line 48 should be italicized because it is a scientific name.
The reference to "Rodriguez-Martinez et al." on line 52 should be cited to a specific source; there are a number of citations matching that description. Also, consistent use of í should be used.
"open data" should not be hyphenated when used as a noun as it is on line 62
Line 68, "However, the large presence of clouds in the region is an issue that causes false positives" should probably be rewritten as, "However, the frequent presence of clouds in the region can cause false positives." The largeness of the presence (assuming that makes sense) probably isn't the problem so much as the temporal frequency of said clouds. If size of clouds is an issue, perhaps, "However, the frequency and spatial extent of clouds in the region can cause false positives"
The authors have not consistently capitalized several terms, including "citizen science," "crowdsourcing," and "crowd-mapping" A single approach should be chosen and followed.
Line 91: the "content areas" do not need to be capitalized as they are not proper nouns or specialized terms.
Line 98: since the authors mentioned the managing institution for Epicollect, they should do the same for iNaturalist, which is managed by the California Academy of Sciences and the National Geographic Society.
Line 101: the fact that iNaturalist is "open-sourced," i.e. that the source code for its software is publicly accessible and licensed for re-use at no cost, is not much of an advantage to scientists. Perhaps it would be more accurate to say that iNaturalist data is openly accessible on the Web and that much of it is licensed for re-use or free of intellectual property restrictions.
Line 106: "Using this approach can become complicated unless a way to process the images automatically is found." This is vague and the paragraph it leads does not resolve that vagueness. What is it about crowdsourcing image collection or classification that is complicated, and/or how do the unique properties of this process complicate the use of its outputs? What kind of processing are the authors referring to?
Lines 152-156: references to specific projects on Epicollect, iNaturalist, and Naturalista should be cited or at least linked to in footnotes
Lines 158-159: what is image "diversity" and why is it beneficial to Sargassum monitoring? Similarly, why is a "constant flow of photographs" important for monitoring, how constant does the flow need to be to be useful for monitoring, and how does the flow of images on crowdsourcing platforms compare to constancy of flow that would be useful?
Line 183: if this paper purports to describe the entire Collective View platform as a methodology for generating real-time Sargassum maps, the Data Collection section needs to go into more detail about the design of the mobile app, the training users received in how and what to collect, information about the abilities and motivations of the people who were involved in outreach, etc. Even if the photographers didn't provide any classification data (e.g. marking photos as containing Sargassum or not), they might still have been biased by being asked to look for Sargassum, as opposed to being asked to photograph any marine algae, or indeed any beach under any conditions to provide meaningful absence data. If outreach was focused on tourist hotels and their clients, then the training data will have a spatial bias toward those places and any resulting models might fail to detect Sargassum in less popular areas or beaches predominantly used by locals. Without this kind of information it seems impossible to assess the quality of the training data.
Line 225: Google Images (https://images.google.com/) is not a primary source of photos. While it's useful to know it was used to find images, it says nothing about the provenance of those images or the reliability of any metadata data contained in those images. The authors should probably cite the actual sources of the images, perhaps by URL in the supplementary materials, or better yet, include the images themselves in the supplementary materials.
Line 231: how was the presence or absence of Sargassum determined in these images? Manually or automatically? By the authors, graduate students, Mechanical Turk workers? If photographers weren't contributing to image classification, why did the authors build a separate mobile application when there are any number of existing applications they could have used to collect geotagged images?
Line 240: these images should be included in the supplementary materials and referenced here.
Since this is more of a methods paper there isn't really an experiment being conducted. I lack the expertise to comment on the machine learning methodology, but any description of the data generation process seems completely absent aside from the the fact that the authors built an Android app and asked some people to use it. If this paper was intended to address only the ML aspects of the project, I think that's appropriate, but the scope of the paper should be narrowed to reflect that.
The tools used to build the platform also go without mention, even though it seems like the authors used ESRI products for data storage and analysis, given the inclusion of https://arcg.is/1STq0C in a footnote. Replication of the full platform would be impossible without these kinds of details.
The supplemental materials do not contain enough data to replicate the results included in the manuscript. One reason is that there is no description of the required software dependencies and their versions. The authors could address this by including a requirements.txt file exported from pip and including the versions (see https://pip.pypa.io/en/stable/user_guide/#repeatability). The Python notebook in the supplemental materials also references a file ("/content/drive/MyDrive/datasets/Sargazo/sargassum_dataset.zip") that is not included with the materials, making replication impossible. Furthermore, there are references to other resources that are not included in the archive, including "/content/drive/My Drive/GIS LATAM 2020/vgg16_cm.eps" and "https://www.elpuntosobrelai.com/wp-content/uploads/2018/10/sargazo-majahual.jpeg". Resources on the authors' computers are not accessible to readers, and even web resources are not guaranteed to exist in perpetuity.
I have no reason to believe the model performance attributes reported are inaccurate. However, regarding the ultimate utility of the proposed solution to the problem of real-time Sargassum monitoring, I think the authors will have to go to greater lengths to describe their process of data generation, its biases, and how data quality was ensured and assessed. This information has serious bearing on the utility and even the performance of any machine learning models derived from such data.
I think you've described an interesting application of machine learning technology in natural resource management, but given that this paper focuses almost exclusively on the ML part of that process I think you should narrow the scope of the work to only that aspect and defer most of the project description and background to a more comprehensive future methodological description.
Aside from the issues of training data quality and bias I've already mentioned, it seems to me that the success of such a platform depends on keeping volunteers motivated over the time span required, and/or how to maintain recruitment of new volunteers to compensate for the loss of existing ones. These are sociological questions more than computer science ones, but they are just as important to the success of this project as the technical feasibility of ML-based maps.
No comment
I think the experimental design was sufficient.
I believe the authors convinced me that AlexNet was indeed the best neural network to train the sargassum images.
It was a pleasure to read the manuscript “Collective View: Mapping Sargassum distribution along beaches”. This was an interesting manuscript aimed at environmental monitoring of sargassum on beaches. The authors used three different neural networks to classify images with and without sargassum and found that AlexNet was the most productive in terms of accuracy and loss. The results are convincing. And I believe that crowdsourcing/citizen science will undoubtedly form an important role in future environmental monitoring, which the authors help to demonstrate here. I believe this Collective View program is in its infancy. Is it still running? Are the number of submissions still increasing? Or was this a one off? Some more information about the crowdsourcing part of this manuscript will help. For example, I interpreted Figure 1 and parts of the introduction that this Collective View program was continuously updated as this is where the maximal benefit in environmental monitoring will be. But I’m not 100% convinced this is the case, or that it is possible. If the authors can make this clear and fully highlight that Collective View will be a dynamic resource for local policy makers/tourists etc. then I think this can be an important paper! This is highlighted below regarding a major limitation of the current study which is that the authors train all images from August to December – but sargassum presence is dynamic in space and time. Currently, however, the paper is largely about the use of neural networks to train sargassum images. I hope the authors can use these suggestions and improve their manuscript. I provide some further comments below.
Lines 30-59: I actually would recommend switching the order of the first two paragraphs to start broadly about sargassum as an ‘issue’ and the economic damage it can cause etc. Then switch to a more specific case about sargassum relevant to the Mexican coast etc. I guess I just recommend starting a bit broader and then narrowing in.
Lines 81-82: What is the resolution of these sensors? I’d list it here and then add a sentence demonstrating why we need more fine-scale resolution – which is what you are attempting here.
Lines 106-115: I’d consider cutting this paragraph. Or at least trim it and integrate it with the paragraph below on ‘maps’. You repeat a fair bit of this below in the paragraph starting on line 122.
Line 163: So here it becomes clear that your major advancement is the ability to make maps and monitor Sargassum at the ‘beach scale’. I think this is a good goal! But, I think a couple sentences here highlighting why at the beach scale is important would help to highlight why your work is indeed an advancement.
Line 164: I’d suggest a paragraph break at “This study proposes…”
Lines 181-182: When the authors say that each of the components were used in different stages it implies to me that they are not continuously being updated. But my interpretation is that users can continue to upload photographs and then this leads to map generation in a dynamic and updated fashion in near real-time. Unless I am misinterpreting something? Assuming I am right, maybe just a bit different wording here to highlight this. Also, could somehow switch Figure 1 to a circle or have a ‘feedback’ arrow from map generation to data collection to highlight the updated fashion of your project!
Line 205: Used by citizens and tourists for what purpose? This is relevant to a previous comment above and this should be just a little bit more explicit in the introduction.
Line 206: It would be interesting to see a graph showing the accumulation of images over time. Is it still increasing? Was there a big spike in the beginning and then quickly levelled off? Maybe as a supplementary figure or something?
Line 222: I’m a little confused as to why there were so many images not geotagged? If they were taken with a smartphone shouldn’t they have been automatically geotagged?
Line 231: This is the result of someone manually looking at them all?
Line 246: This whole section could be improved if some references are added throughout here and there.
Line 402-403: Is there a reference for this?
Line 422: This is probably my most substantial comment. How does the presence of sargassum change through time? Presumably the quantity of sargassum can wax and wane through time, especially over a long period as that from August to December? This is pointed out by the authors here. Can your collective view program account for this? How so? I think more details are needed here. This is made clear in Figure 15 where you have present and absent photos pretty much ‘on top’ of one another. Presumably these photos were taken at different times? Can the authors attempt to use smaller subsets of their data and fit through AlexNet, say for each month? Would this show the presence/absence of sargrassum through time?
Line 438: I’m curious how many photos there are now!
I really liked all the figures and thought they did a great job of conveying the relevant information.
All text and materials provided via this peer-review history page are made available under a Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.