Using Bumble Bee Watch to investigate the accuracy and perception of bumble bee (Bombus spp.) identification by community scientists

Introduction

Community science, which is also known as citizen science, is a popular tool in conservation biology that involves public participation in scientific data collection (Silvertown, 2009; Crall et al., 2011; Kremen, Ullmann & Thorp, 2011; Lye et al., 2011; Dickinson et al., 2012; Lebuhn et al., 2012; Roy et al., 2012; Birkin & Goulson, 2015; Kobori et al., 2016; Kosmala et al., 2016; McKinley et al., 2017). The main strengths of community science include increasing the spatial and temporal scale and magnitude of sampling efforts; reducing the cost of sampling; and creating educational and recreational benefits for participants (Bonney et al., 2009; Kremen, Ullmann & Thorp, 2011; Dickinson et al., 2012; Lebuhn et al., 2012; Birkin & Goulson, 2015; McKinley et al., 2017; Falk et al., 2019).

Accurate species level identification is an important first step in conservation and is an essential pre-requisite for effective conservation management decisions (Le Féon et al., 2011; Kremen, Ullmann & Thorp, 2011). Previous research has found that community scientist data, when unreviewed or unverified by experts, can contain errors that significantly influence the interpretation of results by experts (Gardiner et al., 2012; Comont & Ashbrook, 2017). Problematic errors include the overestimation of rare or at-risk species, as well as underestimation of common species, inflated species richness and significant increases in species diversity (Dickinson, Zuckerberg & Bonter, 2010; Gardiner et al., 2012; Silvertown et al., 2013; Comont & Ashbrook, 2017; Falk et al., 2019).

Verification is part of expert assisted community science programs where experts in the field review a submission to determine if the identification is correct or incorrect, which can increase data quality (Le Féon et al., 2011; Gardiner et al., 2012; Comont & Ashbrook, 2017; Falk et al., 2019). As such, expert assisted community science projects have a higher scientific value, particularly for conservation management decisions (Dickinson et al., 2012; Crall et al., 2011; Gardiner et al., 2012; Falk et al., 2019).

Bumble bees (Hymenoptera: Apidae: Bombus) are native throughout the Northern Hemisphere and in South America with their diversity centered in temperate and mountainous regions (Williams et al., 2014). They are important pollinators in natural and agro-ecosystems (Corbet, Williams & Osborne, 1991; Buchmann & Nabhan, 1996). A portion of North America’s bumble bee species have been found to be in decline in recent years (Colla & Packer, 2008; Grixti et al., 2009; Cameron et al., 2011; IUCN Red List of Threatened Species (IUCN), 2019) (see Table 1). There are several factors that have been identified as drivers of bumble bee decline, including: pathogen transmission from managed bees, habitat loss, pesticides, interspecific competition with non-native species and climate change (Colla et al., 2006; Otterstatter & Thomson, 2008; Cameron et al., 2011; Szabo et al., 2012; Goulson et al., 2015; Kerr et al., 2015; McMahon et al., 2015; Papanikolaou et al., 2017; Cameron & Sadd, 2020; Soroye, Newbold & Kerr, 2020). As Bombus is a genus of relatively large bees, sometimes with distinct physical features, experts are often able to provide a confident identification to species level from photos, although it does depend on the species and angle and quality of the photo (Lye et al., 2011; Suzuki-Ohno et al., 2017; Falk et al., 2019).

The community science program, Bumble Bee Watch, was launched as a website (www.bumblebeewatch.org) in March 2014. An iOS app (computer program for mobile devices such as phones and tablets) was added in July 2017, and an Android app was added in July 2018. Users (members of the public who participate in this program) submit photo observations of bumble bees with known dates and geographic locations throughout Canada and the United States. Users are not required to have any prior experience with or training in bumble bee identification to participate in the program, and can range in age, ability and location (MacPhail, Gibson & Colla, 2020). Photo documentation reduces the need for traditional monitoring protocols and equipment, including physical collection and curation of collected specimens (Silvertown, 2009; Lye et al., 2011; Lebuhn et al., 2012; Kobori et al., 2016), although photos are not always sufficient to identify some bumble bee species due to either the photo quality or the physical features of the specimen that need to be examined (Lye et al., 2011; Suzuki-Ohno et al., 2017; Falk et al., 2019).

Bumble Bee Watch is an example of expert assisted community science, where submissions from community scientists are reviewed by an expert in the field before entry into a species database (Le Féon et al., 2011; Gardiner et al., 2012). For Bumble Bee Watch, experts (aka verifiers) are individuals with significant expertise in the identification of bumble bees. Verifiers currently include some of the top bumble bee experts in North America, and all new verifiers must be referred to and vouched for by experts known to the current administrators (e.g., RH of the Xerces Society, SC of York University).

Community science monitoring projects benefit from accurate identifications by participants, which reduces verification time from experts, speeding up feedback to participants and enhancing educational value. Yet accuracy can vary amongst participants and programs. Examining accuracy in the Bumble Bee Watch program can be useful in assessing the over- and/or under-reporting of various bumble bee species compared to researcher collected data. These bumble bee records are highly valuable to conservation biologists who may use data for common species as a comparative background for looking at the status of at-risk species.

Assessing the accuracy of initial identification of both common (least concern) and at-risk (vulnerable through extinct) bumble bee species (as determined by the IUCN Red List of Threatened Species (IUCN) (2019)) is useful for Bumble Bee Watch to help assess the identification skill levels of participants, the need for expert verification, and the most common species misidentifications. Overall, a high accuracy of initial identification from users could suggest one or more of the following: (1) bumble bees are relatively easy to identify; (2) common bumble bee species are easy to identify; (3) experts and users make similar mistakes during identification; (4) users are taking advantage of the bumble bee identification materials provided by the program to have a high matching ability; (5) the identification key on the website and smart filter on the app is intuitive and easy to use; (6) users are experienced in bumble bee identification (e.g., enthusiasts, researchers, participants in targeted outreach programs); and/or (7) education and outreach programs are increasing the skill and abilities of users. Low accuracy of initial identification rates could suggest that: (1) bumble bees are hard to identify by novices and that expert assisted community science is required; (2) there is a need for program changes and/or additional training materials; (3) there are potential bias(es) by users; and/or (4) photographs are not clear enough or do not provide enough detail for verification. Accuracy of identification data can also be used to assess user interest and bias (Dickinson et al., 2012). Therefore, understanding what influences user accuracy could aid in the development of a long-term monitoring protocol for bumble bees in North America.

Effective long-term pollinator conservation requires collection of continuous, broad-scale data on native pollinator communities (Grixti et al., 2009; Cameron et al., 2011; Lebuhn et al., 2012; Goulson et al., 2015; Hatfield et al., 2015a, 2015b; Strange & Tripodi, 2019). Conservation biologists are dependent on accurate spatial and temporal data to determine the status and extinction risk of bumble bee species (Rodrigues et al., 2006; Mace et al., 2008; Cardoso et al., 2011; Colla et al., 2012; Hatfield et al., 2015a; Colla, 2016; MacPhail, Richardson & Colla, 2019). Since community science increases the ability for long-term monitoring programs to be developed, and community science data have recently been used to help assess many species (Howard & Davis, 2008; Kremen, Ullmann & Thorp, 2011; Lye et al., 2011; Suzuki-Ohno et al., 2017; United States Fish & Wildlife Service (USFWS), 2016a, 2016b, 2017a, 2017b, 2019a, 2019b, 2019c, 2019d; Committee on the Status of Endangered Wildlife in Canada (COSEWIC), 2018; MacPhail, Richardson & Colla, 2019), there is potential for Bumble Bee Watch data to contribute to long-term monitoring and management decisions for at-risk species. Community science data can also be used to monitor all species for trends in abundance, potentially providing an early warning before catastrophic collapses occur, such as have been seen in several North American bumble bees (Colla & Packer, 2008; Cameron et al., 2011; Colla et al., 2012; Bartomeus et al., 2013; MacPhail, Richardson & Colla, 2019; Mathiasson & Rehan, 2019; Richardson et al., 2019; Cameron & Sadd, 2020).

A recent meta-analysis showed that no significant difference, a correlation greater than 0.5, or a minimum percent agreement of 80% between community scientists and professional scientists is considered acceptable in terms of equivalency (Aceves-Bueno et al., 2017), although this does not take into account repercussions such as misidentification of specific species (such as those that are rare, endangered or invasive) (Austen et al., 2016; Stribling et al., 2008). Here, we use the reviewed database available from Bumble Bee Watch to assess the accuracy rate of data submitted by community scientists as assessed by bumble bee experts (The Xerces Society for Invertebrate Conservation et al., 2019).

Experts and users may have different perceptions on the ease or difficulty in identifying specific bumble bee species. Identifying differences in these perceptions could help those conducting outreach and training target their efforts and resources appropriately. To accomplish this, we completed a user and expert online survey to investigate the perceived ease of identification for each species and compared that to the actual observed accuracy in the program.

As the users who participate in the Bumble Bee Watch program have a range of bumble bee identification experience ranging from complete novice (majority of users) to expert (small number of users) (MacPhail, Gibson & Colla, 2020), there could be user bias or influence in the percent agreement between the user submitted name and expert verified one for some species. For example, if one user submitted most of the data for a specific species, and they are an individual with high identification accuracy, this could skew the percent agreement results as compared to species with identifications contributed by users with an average skill level. The reverse could also be true for a user with a lower than average identification accuracy.

We hypothesized (1) that there are differences in user identification accuracy amongst species; (2) that the agreement (relationship) between expert and user identifications is not due to chance alone; (3) that users have a higher accuracy of identification for common species than at-risk species; (4) that users and experts would have different perceptions of ease of identification; and (5) that the user perception of ease of identification would match their actual observed accuracy of identifications.

Methods

Results

Discussion

The accuracy of initial identification is important for determining the utility and quality of community science-collected data (Gardiner et al., 2012). We found that records submitted by Bumble Bee Watch community scientists have an overall percent agreement of 52.76% (range 0–100%, average 53.20%), when comparing the number of correct user submitted records (as determined by expert review) to the total number of user submitted records (i.e., Uc/Ut × 100). This compares closely to the overall “recorder success” for BeeWatch at 59% (range 4–84%, average 60%) and Blooms for Bees at 40% (range 0–100%, average 21%) (Falk et al., 2019), “recorder accuracy” and “recorder identification ability” for BeeWatch at 59% (range 51–64%) (Comont & Ashbrook, 2017), as well as “precision” at 44% (range 0–86%) for Big Bumblebee Discovery (Roy et al., 2016).

Our overall veracity (proportion of all verified records correctly identified by users) (i.e., Uc/E × 100) was 52.76%, (range 0–100%, average 55.86%). This compares to an overall value of 49% (range of 42–78%, average 45%) in Bee Watch and 44% (range of 0–100%, average 39%) in Blooms for Bees (Falk et al., 2019) and 63% (range 0–83%, average 45%) in Big Bumblebee Discovery (Roy et al., 2016) (note that Comont & Ashbrook (2017) did not investigate this metric), suggesting that Bumble Bee Watch participants miss fewer identifications than in other programs.

While we used two different ways of calculating accuracy, percent agreement and veracity, we found that there was no significant difference between them over all species combined, although there were large differences observed for some species (e.g., observers captured 97% of all verified B. affinis Cresson, 1863 records in their submissions, although their percent agreement of identifications was low with only 44% of their total B. affinis submissions being correct).

While the interpretation of Cohen’s Kappa statistic is not consistent across researchers, our Kappa values do suggest that the percent agreement and veracity values calculated for a number of species in our study should be interpreted with caution as the agreement between users and experts is likely a result of chance. For instance, of the 41 species we calculated the statistic for in our study, 9 had Kappa values of less than 0.40, which is considered only a slight or fair agreement by Landis & Koch (1977) and no or minimal agreement by McHugh (2012), and another 17 species had values less than 0.60, which while considered a moderate agreement by Landis & Koch (1977), is considered a weak agreement by McHugh (2012). These low values of agreement (high involvement of chance) may be due to the small sample size observed with some species, and/or the small number of correct observations submitted by users.

While ten of the 42 species in our analyses (B. bifarius, B. borealis Kirby, 1837, B. cryptarum, B. huntii, B. impatiens, B. mixtus Cresson, 1878, B. perplexus Cresson, 1863, B. ternarius Say, 1837, B. vosnesenskii Radoszkowski, 1862, and B. vosnesenskii or caliginosus) were at or above the suggested level of 80% percent agreement that would eliminate or reduce the need for expert review (Aceves-Bueno et al., 2017), the remaining 32 species were below this level. Falk et al. (2019) had two (of 22) species above the 80% threshold in the BeeWatch program and one (of 24) species in the Blooms for Bees programs, while Roy et al. (2016) had two (out of 5 color groups) above the 80% threshold for the Big Bumblebee Discovery program. It is not uncommon for community science programs to be under the 80% threshold, as Aceves-Bueno et al. (2017) found that only 52% of programs were at or above that value. This does, however, reinforce the need for expert-review in our program and others that involve bumble bee identification by community scientists.

Critically endangered and endangered species were more likely to be misidentified in our study than species considered to be at least risk of extinction when percent agreement was evaluated, but the opposite trend was seen with veracity, where, except for between endangered and critically endangered species, levels of veracity increased as risk of extinction increased, although there was a low sample size in the higher risk of extinction levels. The low percent agreement could be because some at-risk species are more cryptic or difficult to identify as compared to other at-risk species and common species (i.e., uncommonness may or may not result in difficulty in identification). There is also increasingly more guidance and education about how to identify some rare species, and many of these species (e.g., B. affinis, B. occidentalis Greene, 1858) are targeted in searches and programs for their conservation; while these targeted searches may increase the total veracity by capturing more of the actual occurrences in the initial identifications, it may also result in users misidentifying more observations due to a desire to find the rare species or because they recognize that species name in the list of options, therefore dropping the percent agreement (number of correct observations compared to all submitted observations of that species).

We anticipated that experts in bumble bee biology, particularly those with extensive identification experience, would be better able to rank the difficulty of identifying species by photos than users, which we did not clearly see in this study. We also anticipated that species that are perceived as easy to identify would have a higher identification accuracy. Although there was a significantly different relationship with expert rank of ease of identification and actual observed percent agreement of identification rates in this article (i.e., the expert prediction did not match the observed percent agreement), the veracity was not different from the expert perception (i.e., the expert view was similar to the veracity observed over all records). This difference in findings could be related to users misidentifying a number of species in their submitted identifications and thus skewing the percent agreement results as compared to the overall perception of experts. There was no difference with users and observed percent agreement or veracity, although the predicted range rarely met the observed range of percent agreement. It is also possible that experts do not fully understand how difficult users find the identification process and the perceptions of the two groups could be quite different and the findings only relevant to that specific group.

Experts generally ranked species as easier to identify than users. This may be because experts have a lot more experience and training, look for features that users do not, and know more “exceptions” to the typical or standard appearance, and thus have less confusion or hesitation over identifications than users may have. The respondents to the user survey were not entirely representative of contributors to Bumble Bee Watch overall, with the respondents tending to submit more records than users overall (MacPhail, Gibson & Colla, 2020). It is possible that the rankings would be different than that given by a truly representative sample of contributors, particularly those who only submitted one record. However, many users may not actually know enough to be willing to offer a ranking; in our user survey, the number of respondents who ranked a species ranged from 39 to 209 respondents per species out of the 342 respondents in total. Similarly, a low number of experts contributed to ranks of some species in the expert survey, ranging from 2 to 13 respondents per species out of 15 total respondents. It is possible that if a larger sample of both users and experts were obtained, results closer to observed may be obtained. Additionally, no photographs or illustrations were provided for the species in the survey, so users may have had difficulty assigning a rank as they were unsure what that bee looked like; future investigations should provide standardized photos or illustrations (e.g., color pattern guides as used in Williams et al. (2014)). In the future, experts could be specifically asked their perceptions on how easy or hard it is for the average user to identify the species, rather than their own perceptions.

The range in numbers of user submitted names per expert assigned species, and vise versa, was extensive. Mimicry and cryptic coloration patterns may cause greater difficulty in accurate species identification than species with consistent patterns. Identification guides do show the variations present in some species, with B. rufocinctus being the most diverse (Williams et al., 2014). Additionally, some species may have had more incorrect species names because they are a wide-ranging species and thus more potential for confusion with other co-occurring species in different parts of its range. These are two areas that could be investigated further.

As noted above, some species can be hard to identify, with specific photo angles needed to showcase identifying characteristics, and/or with characteristics best seen under a microscope using a physical specimen. In some cases, experts may still be able to identify the specific species without a microscope using just the photos provided, often based on gestalt or intuition built on past experience; this may also explain why experts in our study perceived similar-looking species (e.g., B. sandersoni and B. vagans Smith, 1854) as being at different levels of identification difficulty (levels 5 and 3, respectively).

Bumble Bee Watch uses two species complexes or groups to help in situations when the observation can be narrowed down to one of a few similar-looking species, but confirmation of the exact species cannot be made. These include B. vagans, B. sandersoni and B. perplexus (the two-striped group) and B. vosnesenskii and B. caliginosus (the yellow-faced group). When the percent agreement is calculated with the allowance that any of the user-submitted species is correct if it is verified as being either that species or one of the species in the associated complex, the percent agreement increases, sometimes drastically, as compared to strictly considering the single species in isolation.

For example, of the 516 records that had been submitted as B. vagans, 333 were verified as either being B. vagans, B. sandersoni, B. perplexus, or part of the complex, leading to a potential 64.53% percent agreement if any of those responses were to be considered correct, as compared to original comparison of 239 being verified as B. vagans alone (46.32% percent agreement). For B. sandersoni, the percent agreement increases to 38.58% from the original 3.15% with the extra allowance, and for B. perplexus, 86.44% from the original 83.02%. Similarly, for B. vosnesenskii, the percent agreement increases to 96.20% from 79.95% if you include all records verified as B. vosnesenskii, B. caliginosus, or the complex, and for B. caliginosus, 95.26% from 18.53%. This suggests that, at least for some hard to identify species, users may not be as “wrong” or inaccurate as the main calculation shows (i.e., they were “close” to the correct identification), an additional area for consideration.

The Bumble Bee Watch platform has a system of checks in place that only allow for species known to be in that provincial/territorial/state jurisdiction to be entered, reducing the number of potential incorrect species options for species. While the program’s interactive identification key and filter also allow for greater accuracy, it also introduces a potential source of error related to the identification of male specimens: males do not always follow the same colour patterns as females, but the program’s keys and filters focus solely on the female patterns.

To increase the percent agreement and veracity of community science-collected data, professional scientists may install quality control and assurance methods (Kosmala et al., 2016; Freitag, Meyer & Whiteman, 2016; Aceves-Bueno et al., 2017). Bumble Bee Watch could update its various existing filters and checkpoints, or what have also been referred to as “automated error checking capabilities” (Crall et al., 2011), or “smart filter” systems (Freitag, Meyer & Whiteman, 2016), throughout the identification key for at-risk species, males of species that differ in appearance from the females, and for other species that are frequently misidentified. A similar validation method is used by Project FeederWatch, a community science program where a system of semi-automated, novel filters for each US state or Canadian province and territory dictates the submission of records by users (Bonter & Cooper, 2012). The Project FeederWatch program uses historical data to help detect potential errors in bird identification, for example, records submitted in locations outside of known ranges (Bonter & Cooper, 2012). This system had an initial accuracy of 97.7% by participants over three consecutive years (Bonter & Cooper, 2012). Bumble Bee Watch currently has a data validation pathway during identification to restrict based on range, but could add in prompts with tips and checks to help users become aware of potential mimics and to discern the differences in the frequently misidentified species.

For our analyses, we removed records that could not be determined to species by experts; this was often due to the submission of poor-quality photos or photos missing key features/angles needed for identification. It is possible that these unknown bumble bee species had in fact been correctly identified by the user but the records were not verifiable. To increase the amount of verifiable records, additional fact sheets and tutorials can be provided. When users have a better understanding of the specific features needed for identification of each bumble bee species, they can take photos that allow for easier and more accurate expert identifications (Suzuki-Ohno et al., 2017). Continued participation in the program could also have users increase their accuracy, particularly when they have previously submitted less than ten records (Aceves-Bueno et al., 2017; Falk et al., 2019). Supporting the user experience would increase retention and participation rates, which would improve the dataset.

While observer bias could influence percent agreement, particularly when the number of observers and/or the number of records are small, it is unlikely to be a problem in our study for most species. The vast majority of the 4,912 users only submitted one or two photos, and their overall contribution was also low per species. There are a few exceptions where observer bias would likely play a role, such as for B. bohemicus and B. suckleyi as only 1 user submitted verified records of each, as well as B. kirbiellus with 3 users and B. crotchii with 5 users. While B. frigidus had 12 users who submitted verified records, the most frequent user had submitted 40% of the records, which could influence the accuracy if (s)he was a high or low skill level for identification.

An assumption behind Cohen’s Kappa is each observer (in our case users and experts) independently classify their observations. From an analysis of our data (not presented in the text), 673 of the 22,159 total reviewed records (3.04%) were self-verified (i.e., submitted by and reviewed by the same individual), which would violate the assumption. However, since it is such a small value, and the verifications do not usually occur at the same time as the original submissions, we do not believe it would cause an impact on the analyses.

A current challenge with the Bumble Bee Watch program, and one other bumble bee community science programs face (Comont & Ashbrook, 2017; Falk et al., 2019), is the delay that may exist in the time it takes a record to be reviewed by an expert and the user receiving confirmation of the identification; this was noted as one of the most common drawbacks to the program in a recent user survey (MacPhail, Gibson & Colla, 2020). While each record currently is reviewed by experts regardless as to its probable accuracy, the current structure of the verification side of the database allows for accurate identifications to be reviewed and generalized feedback (often just a brief thank you) to be given much faster than when the identification needs to be changed or personalized feedback given (although currently even the generalized feedback needs to be expert generated versus automatically generated as is the case with BeeWatch (Falk et al., 2019)). Increasing the initial user accuracy on Bumble Bee Watch therefore means record validation may proceed more quickly and efficiently, as experts would spend less time correcting an identification and providing feedback to participants about why the identification is wrong, and more time processing submissions, thus providing a faster turnaround time from the time a participant submits a record to the time they get a confirmation. If the percent agreement for a species is reliably high, over time it may be possible for experts to skip the review process for those records and/or verify a sub-sample only, thus further freeing up resources to focus on the more commonly misidentified species.

An assumption of Bumble Bee Watch is that the expert reviewers are always “correct”. However, experts are not always correct or consistent (Stribling et al., 2008; Kosmala et al., 2016; Austen et al., 2016, 2018; Suzuki-Ohno et al., 2017). Misidentifications may have little impact but could have serious repercussions, particularly related to noting species declines, conserving endangered species, tracking invasive or harmful species, and other related environmental protections and conservation activities (Austen et al., 2016; Stribling et al., 2008). It is therefore important to make sure taxonomic identifications are validated (Packer et al., 2018). More investigations into the accuracy and consistency of expert identifications of bumble bees, particularly from field photos, should be conducted, and additional quality control procedures implemented in the Bumble Bee Watch database.

The Bumble Bee Watch program has thus far recorded 39 unique species from across North America. There are 46 species listed as being found in North America by Williams et al. (2014): the missing ones include B. distinguendus Morawitz, 1869, B. franklini, B. hyperboreus Schönherr, 1809, B. jonellus, B. neoboreus Sladen, 1919, B. polaris Curtis, 1835, and B. variabilis. It is not surprising that these species are missing. Bombus franklini is suspected to be extinct, previously occupying an extremely narrow range in southern Oregon to northern California (Williams et al., 2014). Bombus variabilis is one of the rarest bumble bees in North America and has been extremely infrequently seen (Williams et al., 2014). All the other missing species are only found in the tundra/taiga in the far north (Williams et al., 2014), where there are few humans let alone participants with the Bumble Bee Watch program. That said, there may be records of these species still awaiting review on Bumble Bee Watch, or that have previously been verified as an unknown bumble bee species, as these species are frequently difficult to identify, particularly from photos. We have already encountered a few photos from the far north that we cannot comfortably identify: in some cases a tentative identification was discussed in the comments (e.g., bee record 3,547 is a tentative B. hyperboreus and bee record 4,922 is a tentative B. polaris) while others were generally identified simply as “sp” (e.g., bee records 15,468, 15,540). The encouragement and allowance of more photos or videos to be submitted per record (the current maximum is 3 photos), and the additional guidance of best features to photograph per species, may allow further identifications to occur in the future. Users submitting images of these potential species could also be contacted in order to obtain more photos and/or physical specimens for confident identifications to be made.

Bumble Bee Watch data has already been shown to add important information for species conservation status assessments. A next step is comparing user and expert collected data to see if they show the same trends for more species; if this is the case, then community scientists may be capturing an accurate representation of the wild bumble bee community, which are essential baseline data for conservation purposes. Examining accuracy can also lead to increased understanding of participants’ motivations, for example if submissions of a species (and related misidentifications) peak in correlation to federal listing activity, or relatedly, if at-risk species are reported more frequently than are found. There were 45 provinces/territories/states with less than 100 records submitted and reviewed in this data set (Table S1). Promotion of the program in jurisdictions with low participation, particularly in the areas with less than 5 records as of the date of our data export (Alaska, District of Columbia, Rhode Island, South Dakota, West Virginia, and the Yukon) can also help to increase the coverage and value of the program.

Conclusion

The goal of Bumble Bee Watch is to track and conserve the bumble bees of North America. It has been successful in collecting tens of thousands of species records over broad temporal and geographic scales, including records of at-risk species, and the data has already fed into status assessments and other conservation tools and actions. With an urgency in the lack of local and global pollinator species data available (Potts et al., 2010), projects such as Bumble Bee Watch are filling a time-sensitive gap. For community science programs to produce scientifically rigorous data for conservation policy and management, data must undergo quality assurance protocols. Our work found that for Bumble Bee Watch, the accuracy of user identification varies greatly depending on the species, and expert review is needed for most records, but at-risk species are not always over-reported as compared to common ones.

Work to increase the accuracy of species identification in community science may benefit the users, experts, and project overall. Further investigations into the trends and causes of overall initial accuracy is needed. New educational materials and various checkpoints in the pathways to identification should be developed to assist in identification: this could speed up or potentially even eliminate the need for expert review for at least some species, and help users increase their knowledge and identification skills.

Future research may also focus on examining the connection between government activity and/or media attention (e.g., listing of a species as endangered, regional program spotlights) and increases in community scientist participation. Relatedly, it is also important to determine how best to recruit and maintain the interest of a diverse group of users from across North America, to ensure the Bumble Bee Watch program increases its capacity to capture a detailed picture of our bumble bees.

Supplemental Information