Evaluating the effectiveness of participatory science dog teams to detect devitalized Spotted Lanternfly (Lycorma delicatula) egg masses

Participants

Approval to conduct this study was granted by the Virginia Tech Institutional Animal Care and Use Committee (IACUC #23-025) and the Virginia Tech Institutional Review Board (IRB #23-257). Through Facebook posts, we recruited potential dog-handler teams with experience in either sport scent detection competition or other detection work (e.g., search and rescue, bed bug detection). We invited the handlers to attend an informational webinar, which they could attend live or watch via recording. After watching the webinar, those interested in participating completed a Google form in which they self-reported their own and their dog’s scent detection experience, whether they wanted to be considered as a handler and/or a trainer, and their geographic location. From the participants who completed the Google form, we created groups of handlers (owners who would train and test their dogs) with an overseeing trainer based on geographic proximity. All participants signed a written consent form, agreeing to participate in the study. The trainers served as the primary point of contact between the research team and the dog-handler teams, receiving and distributing materials, organizing training sessions, and coordinating evaluations. Many of the handlers and trainers had a prior teacher/student relationship. In all teams, the trainer also participated as a handler for at least one dog in the study. We enrolled all participants in a Google Classroom space, where they could access the forms, questionnaires, and study updates. We also invited all participants to join an optional private Facebook group, which facilitated a sense of community and allowed teams to ask questions about any aspects of the study.

Questionnaires and training log

Each handler was asked to complete five online questionnaires. Links to the questionnaires were provided to handlers in the Google Classroom; handlers were encouraged to complete them at the time of enrollment; however, many handlers completed them at various times throughout the study. Handlers were also encouraged to record their training weekly via a Google form. The form included the handler’s and dog’s names for identification, the number of training sessions per week, the average length of each training session, and the intended objective of the training. The results of the surveys and training logs will be analyzed in a separate paper, which will discuss the predictive factors for teams’ success in this study.

Training aids

We harvested approximately 1,500 grams of SLF egg masses over two consecutive winters at the Virginia Tech Agricultural Research and Extension Center in Winchester, Virginia. The eggs were scraped from the substrate using a plastic scraper and directly transferred into glass jars, each holding 30 to 50 grams of eggs. The sealed glass jars were stored in a −80 °C freezer for at least 30 days to ensure the eggs were devitalized, rendering them inert and suitable for shipping outside the quarantine zone (Essler et al., 2021). We cut, folded, and crimped 304-grade stainless steel 80-mesh into 2.5 cm × 2.5 cm packets to create the training aid packets for distribution to the teams. The training aid packets were designed to provide a consistent odor sample for the teams to train with while ensuring no SLF eggs were dispersed into the environment (Kane, Aviles-Rosa & Hall, 2023). We washed the empty packets in methanol to remove manufacturing oils and other contaminants. Then, we baked them in a 200° Celsius oven to evaporate the remaining methanol and allowed the packets to air cool. We added 0.5 g of devitalized eggs to each packet and then folded and crimped the top flap to secure the eggs inside, creating the Spotted Lanternfly training aid (SLF training aid), as shown in Fig. 2. The SLF training aids were placed in sealed Mylar bags and stored in a residential freezer at −20 °C until shipped to the teams. We also isolated a subset of the cleaned packets without any enclosed eggs to serve as matched blanks (MB) for training and testing, ensuring that dogs were alerting to SLF egg-specific odor stimuli. We placed the MB in sealed Mylar bags and stored them separately at room temperature to ensure zero odor transfer. We shipped training packs overnight to the trainers. Each pack contained six SLF training aids and one MB per handler. We also included instructions on storing the SLF training aids in sealed Mylar bags in a freezer when not in use, as well as guidelines on recording the number of hours the SLF training aids were out of the freezer for training purposes. Handlers were instructed to use a new SLF training aid after 40 h of use. Additional shipments of SLF training aids were made as needed.

Training

We provided no instructions or requirements on how the training should be conducted, allowing the trainers to work with their groups as usual. The handler completed a weekly training log for each dog participating, which was submitted via a Google form. Handlers reported the number of training sessions they finished, the average length of each training session, the training method used, and the focus of the training sessions. Trainers communicated to the research team when their teams were ready to begin evaluations, and then the research team coordinated with them to establish a testing time and location.

Odor recognition trial

We conducted the odor recognition test (ORT) at locations identified by the trainers. In some instances, these locations were familiar to the dogs; in others, they were novel.

Setting. All ORTs were conducted in a space with a minimum of 7 m × 7 m of unobstructed open area, inside or outside. The Evaluator assessed the space for safety and the absence of significant odors such as air fresheners or other distractions such as accessible food. The Evaluator created the trial array by placing numbered cards (1–6) on the floor and securing them with tape, 1 m apart, in a semicircle configuration (Fig. 3). These cards designated where the boxes would be placed during the trials. We placed two GoPro cameras, a minimum of two m away from the array, on the outer edge, proximal to positions three and six, to record all trials.

Evaluator and observers. For each ORT, a specific person was designated the Evaluator, responsible for administering the ORT. In some cases, the Evaluator was one of the researchers, and in other cases, this role was filled by the trainer or a designee if the trainer was handling the dog. Additional observers were present for all ORTs; only the Evaluator could access the testing sheets. All observers remained quiet and behind the Evaluator during the trials. During ORT trials, no other dogs were allowed in the space. The Evaluator was positioned at the front of the room and, during the trial (from the beginning of the search until the handler notified the Evaluator of their findings), maintained their back toward the team conducting the test. This procedure ensured the trials were consistently double-masked. The Evaluator was responsible for reviewing the protocol with each testing team, placing the boxes in the correct position during each trial, and recording the time and outcomes of each trial on the test sheet.

Boxes and odors. During the setup time on the ORT day, the Evaluator prepared two sets (A and B) of seven cardboard boxes each; each box measured 10 cm × 10 cm × 5 cm. We drilled nine 6.35-mm holes in the top flap, arranged in a 3 × 3 pattern, and labeled each box with a unique code on the underside to designate its contents. One box contained the SLF training aid, and the other six boxes contained distractor odors as detailed: one contained the MB, one contained locally sourced tree bark, one contained locally sourced grass, one contained a medical glove, and two remained empty. Between trials, the boxes were placed in designated spaces on a tray with the holes oriented down. To further isolate the SLF egg odor, the SLF training aid box was placed with holes down, and at a minimum of 15 cm space between it and the other boxes. We prepared three additional training boxes for the teams to use during the acclimation period before the first trial commenced. One acclimation box contained the SLF training aid, one contained the MB, and one was empty. The boxes were clearly labeled on top, so the handlers could easily identify the correct one. Additionally, empty boxes were available to exchange during the trials if a box became damaged or contaminated. The Evaluator decided when to exchange a box.

ORT setup. Each ORT began with the Evaluator orienting the handler, without the dog present, to the room, the numbered spaces on the floor, and how the handler should signal an alert by the dog to the Evaluator. The Evaluator also reviewed the protocol, including the number of boxes per trial, the number of target odors, the number of distracting odors, and time limits, with the handler. Afterward, the handler was allowed to ask questions. When ready, the handler returned to the ORT area with their dog on a leash and was given a 2-minute acclimation period. During this time, the three marked training boxes, one with an SLF training aid, one with MB, and one empty, were placed outside of the array area (no other boxes were in place at this time). The handler was instructed that they could use the acclimation time however they wanted. For example, they could cue and reinforce the correct choice, perform a practice search of the three boxes, or use the time to allow the dog to become comfortable in the space by engaging in other activities that provide reinforcement. After the acclimation period, the handler and dog moved to the inter-trial team space, a minimum of 5 m away from the array, and had water available for the dog throughout the test. The handler and their dog stood in the inter-trial team space, their backs to the array. Once the handler was facing away, the Evaluator removed the acclimation boxes, placed them with the hole-side down on a tray outside the array, and prepared for the test.

ORT test. Each ORT test was made up of 10 trials. Each trial contained zero or one SLF training aid and five or six distractor odors. For each trial, the placement of the specific odors was predetermined, and we created four different 10-trial test versions. The four test versions were constructed using a computer randomizing program. Before the acclimation time, the handler drew a card to determine which version they would run; however, they were unaware of the version number. If the handler was running multiple dogs, the version they ran the previous time was removed from the options for their subsequent dog. The Evaluator placed the boxes from set A at the designated positions on the floor. Boxes were always placed in positions one through six sequentially and then picked up in the same order to ensure that no specific odor was ever handled last or first. Once the trial boxes were in place, the team was instructed to begin the trial. The open end of the horseshoe array was considered the start line, located between boxes one and six. The handler and dog could choose which direction to work in and whether they would be on or off-leash. The handlers could carry the dog’s reinforcer (usually treats or a toy) with them and were permitted to provide it within the array, however, and whenever they chose. Teams had 90 s per trial, which started when the handler cued the dog to search. The trial ended when the handler called out “alert”, or the box number for an odor find, “clear” if they determined no target odor was present, or when 90 s had elapsed. The handler was responsible for determining if the dog showed an alert. After the trial, the evaluator provided the correct outcome, regardless of whether the team succeeded. The team then returned to the inter-trial space, and the Evaluator reset the array for the subsequent trial until 10 trials were complete or the handler opted to withdraw for any reason. Between trials five and six, box set A was replaced with box set B to ensure neither the dog nor the handler was learning the specific odor or look of the SLF training aid box in set A. After a team completed their ORT, all boxes were inspected and replaced if they were deemed contaminated or visually identifiable. The floor was checked for food or other debris and cleaned as appropriate. The time between testing teams was at least 10 min to allow for reset and preparation for the next team.

Measurement. The Evaluator recorded the handler’s call for each trial, indicating whether it was correct, along with the time elapsed from the start of the trial to when the handler called out “find” or “clear”, or if 90 s elapsed with no determination made by the handler (no response). Video recorded on the GoPro camera was available to verify times. To pass the ORT, the team must correctly complete at least eight out of ten trials. A team that failed to meet the 80 percent requirement was given two additional opportunities to retake the test, either on the same date or at a future date. If the team retested, the version(s) they had run previously were removed from the options.

Field evaluation

Teams that passed the ORT could attempt the field evaluation (FE). As with the ORT, the FE was conducted at locations identified by the trainers. In some instances, the dogs were familiar with these locations; in others, they were novel.

Setting. The FE was conducted in an exterior or semi-exterior space (open barn, picnic area, lumber yard, etc.) (Fig. 4) that was a minimum 25 m × 25 m area, with safe access to the entire area. The Evaluator assessed the space for safety and evidence of SLF egg masses. We placed cones or flagging tape to designate the corners of the search area. We recorded using two GoPro cameras; one was positioned to have visibility over the evaluation area, and the other was worn on a chest harness by the handler.

Evaluator and observers. The Evaluator functioned just as they did for the ORT. In some cases, the Evaluator was one of the researchers; in other cases, this role was filled by the trainer or their designee. On all FE trials, additional observers were present; however, only the Evaluator knew the hide locations. Observers were positioned a minimum of 10 m outside the search area and remained quiet during the trial. No other dogs were allowed in or around the search area during FE trials. The Evaluator remained outside the search area but close enough to communicate with the handler. The Evaluator had their back turned while the team searched to ensure the trial was double-masked. The Evaluator was responsible for reviewing the protocol with each testing team, placing the SLF-training aids and MB in the search area, and recording times and outcomes of the FE on the test sheet.

FE odor setup. We placed the SLF-training aid and MB in a 44 mL glass jar with no lid to prevent odor transfer to the surrounding substrates. We could reuse an area for a different dog team by doing so. Between three and five SLF-training aids were placed in the search area; however, the team and all observers were masked to the number and placement of the hides. The Evaluator determined SLF-training aid and MB placement in the search area, considering weather conditions and the availability of suitable concealment locations, with care taken to prevent overlapping odors from adjacent hides. One hide was placed at ground level, one at a maximum height of 1 m, and the others at heights equal to or between these extremes. All hide locations were placed so the handler and the dog did not have to climb on, over, or crawl under anything. The SLF-training aids were concealed in the environment out of the handler’s plain sight. The Evaluator created a sketch of the search area, numbering the SLF-training aid hide and MB locations, which correlated with the hide numbers on the test sheet and allowed for easy recording of times during the search. If the handler ran multiple dogs, a different search area was used for subsequent dogs. The hides were in place for a minimum of 15 min and a maximum of 30 min before the team searched the area.

FE procedures Each FE began with the Evaluator orienting the handler, without the dog present, to the area, including identifying and defining boundaries from outside of the search area and reviewing the protocol with the team, including range of number of targets, minimum and maximum height of targets, and time limit. The handler was instructed that when they called an alert, they had to define the location of the odor within a 25 cm radius around the hide’s actual location to be correct. Handlers were given free choice on how they did this, for example, verbally declaring “alert” and pointing at the location. The handler was given 5 min to view the area, assess the conditions, and ask clarifying questions. They were not permitted to enter the designated search area during this time. When ready, the handler returned to the FE area with their dog on a leash; the handler could approach any search area boundary, work in any direction, and work on or off the leash. The handlers could carry the dog’s reinforcer with them and were permitted to provide it within the search area, however, and whenever they deemed appropriate.

FE test. The FE was a one-trial event with a total time of 300 s, although there was no penalty for the handler calling an end to the search at any time. The Evaluator began the timer when the dog or the handler stepped inside the search area. The handler was responsible for determining if the dog showed an alert. When a handler called an “alert” and identified the area, the Evaluator verbally confirmed whether the location was correct with either a “yes” or a “no”. An incorrect location was counted as a false alert. At the end of 300 s, the Evaluator called “time”, and the search ended. Any alerts called after the time were not counted, even if the dog had been at the location prior to the period ending. After the search ended, the Evaluator reviewed the score sheet with the handler. The SLF training aids and MB were removed, and the area was left unused for a minimum of 30 min before setting up a new, different FE. No other cleaning or changes were made except to remove identified safety hazards.

Measurement. The Evaluator recorded the wind speed and temperature before each FE trial. The time elapsed from the beginning of the search to the handler’s alert call at each location was also recorded. Any alerts called at locations where no SLF training aids were placed were inspected for the presence of SLF egg masses. If SLF egg masses were found, the alert was reclassified from false to correct. To pass the FE, the team was required to locate the total number of hides minus one (n − 1), where (n) = 3, 4, or 5 and could not have more than one false alert at a location without SLF-training aid. Teams failing to meet these criteria were allowed two additional opportunities to retake the test, either on the same date or at a future date. A different search area was used for retests to prevent familiarity with the prior setup.

Analysis

For the ORT and FE, we recorded the dogs’ responses to each trial as either a true positive (TP), false negative (FN), correct rejection (CR), or false positive (FP). Table 1 illustrates a standard confusion matrix, showing how responses are classified in relation to the target’s presence. Trials containing SLF egg masses are referred to as “hot” trials, while trials with no SLF egg masses are classified as “blank” trials.

Incorrect responses were either false negatives or false positives. A False Negative (Hot Trial Error): The team failed to correctly identify the target during a hot trial by calling an all-clear or selecting an incorrect odor. A False Positive (Blank Trial Error): The team incorrectly identified a non-target odor as the target during a blank trial (Wickens, 2002) We use the following formulas to calculate sensitivity, specificity (ORT only), and precision in the results. Specificity can only be determined when the distracting odors are controlled, as in the ORT array. In the FE, distracting odors include all environmental elements except the target. ${\displaystyle \text{Sensitivity}=\frac{\text{TP}}{\text{TP}+\text{FN}}\text{Specificity}=\frac{\text{CR}}{\text{CR}+\text{FP}}\text{Precision}=\frac{\text{TP}}{\text{TP}+\text{FP}}}$ ${\displaystyle \text{Accuracy}=\frac{\text{TP}+\text{CR}}{\text{Total Trials}}}$

Participants

The initial call for participants generated 1,033 responses via a Google form, with 427 (41.3%) respondents self-identifying as having prior odor detection training with their dog. From these, 110 teams were selected and organized into 12 groups based on geography, proximity, and existing working relationships to form Cohort 1. An additional 72 teams were selected and organized into six groups using the same metrics to form Cohort 2. In total, 182 teams were enrolled, comprising 150 handlers and 182 dogs. Among the handlers, 121 enrolled with one dog, 27 enrolled with two dogs, one with three dogs, and one with four dogs. Of the 182 enrolled dogs, 76 were male, 60 were female, and 46 did not report sex. For dogs with known or reported ages, the youngest was 1 year old, the oldest was 11 years, and the mean age was 5.7 years. Table 2 shows the distribution of breeds. All dogs that attempted at least one ORT had their breed, age, and sex reported.

Overall ORT and FE results

Of the 182 teams enrolled, 78 (42.9%) dropped out before making their first attempt at the ORT. Of the remaining 104 teams (57.1%) that attempted the ORT, 77 (42.3% of the original 182 enrolled, 74.0% of the 104 that attempted at least one ORT) successfully passed the ORT. Each team had three attempts to complete the ORT; 52 (50.0%) teams completed the ORT on attempt one, 20 (19.2%) on attempt two, and five (4.8%) on attempt three for a total of 153 ORT’s. Teams that dropped before or between ORT attempts most frequently cited time commitments and scheduling, although not all gave a reason. During 23 ORT tests, the team did not complete all 10 trials. The decision not to complete all 10 trials was at the handler’s discretion, and the reason was not recorded. These attempts, recorded as “Did Not Complete”, were counted as failed attempts at the ORT. All attempts, whether successful or unsuccessful, are included in the following analysis (Fig. 5). The Did Not Complete trials were omitted from analysis beyond Fig. 5, and all further analyses involve the 130 completed ORTs.

Of the 77 teams that passed the ORT, 66 (85.7%) teams attempted the FE, and 56 (72.7%) successfully passed the FE. Each team had three attempts to complete the FE. Forty-six teams completed the FE on attempt one, six on attempt two, and four on attempt three, for a total of 84 FE trials.

ORT and FE sensitivity, specificity, and precision

For the 104 teams (total of 130 ORT trials) that completed at least one ORT, we calculated the sensitivity, specificity, and precision (Table 3). For the 66 teams (a total of 84 FE trials) that completed at least one FE, we calculated the sensitivity and precision (Table 3). Calculations used are provided in the analysis section.

ORT version and trial effects

The ORT included four versions of odor placement randomization, each containing between one and three blank trials. We investigated whether dogs’ performance varied across versions. Figure 6 displays the scores for each ORT version. Versions one and two included two blank trials, version three had three blank trials, and version four had one blank trial. Using a Kruskal–Wallis test, we found no significant differences in scores, H(3) = 7.226, p = .0650, between test versions (Fig. 6).

Of the 1,300 individual trials (130 ORTs with 10 trials each), 901 (76%) trials were correct identifications, either as a true positive (the dog selected the box containing SLF-training aid during a hot trial) or as a correct rejection (the dog made no selection during a blank trial). Of the 281 incorrect responses (23%), 172 (61%) were false negatives (failure to select the correct box in a hot trial), and 102 (36%) were false positives (selection of an incorrect box during a blank trial). Five trials (0.4%) resulted in no response (timed out). Using a Kruskal–Wallis across trial numbers, we found no significant difference in the proportion of correct responses, H(10)=10.15 p = .0.3382, between trial numbers (Fig. 7).

ORT incorrect responses

Of the 1,300 individual trials, 1,032 (79.38%) were hot trials (the SLF target odor was present). Of these 1,032 hot trials, teams gave 186 (18%) false negative responses. Of those trials with a false negative outcome, 55 (29.6%) were called “all-clear” by the handler. Included in the false negative responses are four trials in which the dogs made no response to any odor within the 90-second trial time (i.e., they timed out). The remaining 131 responses were to one of the distracting odors (Fig. 8). The remaining 268 (20.6%) individual trials were blank trials. Of these 268 blank trials, teams gave 118 (44.02%) false positive responses to one of the distracting odors. Included in the false positive responses is one trial where the team timed out (gave no response) (Fig. 8). In each ORT trial, there were either five or six distracting odors. The distribution of the distracting odors varied by the version number. We calculated each team’s allocation of responses to each distracting odor in 10 trials, separated by hot trials versus blank trials as: ${\displaystyle \frac{\text{Number of Responses to Odor X}}{\text{Total Number of Presentations of Odor X}}\times 100\text{\%}.}$

ORT time searching

We recorded each team’s time spent searching the array, from when they crossed the start line to when the handler notified the Evaluator of their determination. Any team that timed out was recorded as a fail, with a search time of 90 s (the maximum allowable time per trial). For analysis, we separated the trials into four categories: hot trial correct response (Hot Trial Pass), hot trial incorrect response (Hot Trial Fail), blank trial correct response (Blank Trial Pass), and blank trial incorrect response (Blank Trial Fail). The median values (s) and inter-quartile range were as follows: Hot Trial Pass (Med = 14, IQR = 87), Hot Trial Fail (Med = 24, IQR = 88), Blank Trial Pass (Med = 28, IQR = 82), and Blank Trial Fail (Med = 28.5, IQR = 86). A Kruskal–Wallis test revealed a significant effect of trial type, H(4) = 17831, p < 0.0001. Dunn’s post hoc test indicated that the time spent searching was significantly greater during failed hot trials compared to passed hot trials (p < 0.0001), significantly greater during passed blank trials compared to passed hot trials (p < 0.0001), and failed blank trials compared to passed hot trials (p < 0.0001) (Fig. 9).

FE effects of wind and temperature

For each FE trial, we recorded the temperature (°C) and wind speed (km/h). We used a multiple logistic regression to evaluate wind speed and temperature as predictors of the likelihood of passing the FE. The dependent variable was the outcome of the FE (0 = fail, 1 = pass). The model included wind speed and temperature as independent variables. The model did not significantly predict FE outcomes, as neither wind speed (β = 0.021, SE = 0.035, 95% CI [−0.046–0.095]) nor temperature (β = 0.011, SE = 0.030, 95% CI [−0.048–0.071]) were statistically significant. The odds ratios for wind (OR = 1.021, 95% CI [0.955–1.099]) and temperature (OR = 1.011, 95% CI [0.954–1.074]) indicate minimal influence of these variables on passing outcomes. Model diagnostics further suggest poor predictive performance. The area under the receiver operating characteristic (ROC) curve was 0.590 (SE = 0.071, 95% CI [0.451–0.730], p = .179), indicating that the model performed only slightly better than chance.

Analysis

For the LE-ORT, all trials included the target; we recorded the dogs’ responses to each trial as either a true positive (TP) or a false negative (FN). Table 1 illustrates a standard confusion matrix, showing how responses are classified in relation to the target’s presence. A passing score required 80% correct responses. The calculations used are provided in Experiment 1, analysis section.

Overall LE ORT

Of the 56 teams that completed the ORT and FE in Experiment 1, 24 were available when live SLF eggs were accessible. Of the 24 teams that attempted the LE-ORT, 22 teams (91.7%) passed. Each team had three attempts to complete the LE-ORT. 11 teams passed the LE-ORT on attempt one, eight on attempt two, and four on attempt three, for a total of 29 LE-ORTs. Figure 10 shows the cumulative number of teams passing. During 14 LE-ORT attempts, the team did not complete all five trials. These attempts, recorded as “Did Not Complete”, were counted as a failed attempt at the ORT but were excluded from subsequent analysis. All attempts, whether successful or unsuccessful, are included in the following analysis.

LE-ORT sensitivity

We calculated the sensitivity of the 24 teams that completed at least one LE-ORT. The sensitivity for the LE-ORT was 0.8.

LE-ORT time searching

A Mann–Whitney U test was conducted to compare the time spent searching between passing and failing teams in hot trials. The results indicated no significant difference in searching time between passing and failing trials. U = 1268, p = 0.0632 (Fig. 11).