Making sense of fossils and artefacts: a review of best practices for the design of a successful workflow for machine learning-assisted citizen science projects

General characterisation

A general characterisation of the reviewed articles (Table S1) illustrates the novelty and rapid development of the CS-ML approach. The first publication which meets our criteria is from 2005, with the majority published since 2020 (Fig. 4). Furthermore, the reviewed articles were published in a wide variety of journals. The articles are spread thinly over journals; no well-recognized journal exists for the subject.

The most frequently used keywords in the reviewed articles (Table 3) largely match our systematic search criteria (e.g., “citizen science”) or fall within the same category (e.g., “crowdsourcing”). However, recurring keywords such as “biodiversity” and “management” do highlight the early successful adoption of the CS-ML approach in biodiversity research and conservation management (e.g., McClure et al., 2020). With research domain differences in mind, the applications, best practices and challenges of CS and ML reported in these articles should be informative for our innovative workflow for a CS-ML approach in palaeontology and archaeology (see Discussion).

The CS-ML approach: potential and pitfalls

An integrated CS-ML approach offers opportunities beyond the opportunities CS and ML present on their own. CS-ML projects can greatly improve the speed of data acquisition and validation as well as task allocation, allowing real-time action (Green et al., 2020; McClure et al., 2020; Trouille, Lintott & Fortson, 2019). CS-ML data validation can achieve higher accuracy than data validation by CS or ML alone (Green et al., 2020; Lotfian, Ingensand & Brovelli, 2021; McClure et al., 2020). The discovery potential of a dataset is increased with a CS-ML approach due to the complementary serendipitous discovery potential of CS and the complex pattern discovery potential of ML (McClure et al., 2020). With rapidly increasing efforts in collection digitisation and digital data generation in both palaeontology and archaeology, the combined effort of CS and ML can help recognise new/unique objects (e.g., Verschoof-van der Vaart & Lambers, 2021). Furthermore, the CS-ML approach has great public engagement potential (Green et al., 2020; McClure et al., 2020) by generating real-time feedback and promoting interdisciplinary engagement (Lotfian, Ingensand & Brovelli, 2021) (further discussed below). Finally, the potential efficiency of the CS-ML approach presents a cost-effective method when it comes to resources (McClure et al., 2020).

However, the CS-ML approach also poses challenges that need to be addressed. Inaccuracies might arise from low quality data and otherwise biased data (see below). A general lack of trust or scepticism towards CS data leads to a low scientific publication output (McClure et al., 2020). To the same effect, the “black box” problem of ML models (i.e., the inexplicability of the inner workings of the neural networks), makes interpreting results less transparent, unreproducible and hinders engagement (McClure et al., 2020). Engagement can be limited by citizen scientists feeling like the ML model is fully automating their input (Lotfian, Ingensand & Brovelli, 2021). Finally, the threshold for starting a CS-ML project might be high due to the dependency on the large effort of training volunteers, the effort of designing and implementing the project and the required hardware and knowhow (McClure et al., 2020). Moreover, these prerequisites require sufficient financial resources (McClure et al., 2020). These general pros, cons and “lessons learned” for the CS-ML approach, which have emerged over recent years from fields such as biodiversity monitoring, inform our CS-ML approach in palaeontology and archaeology (see Discussion).

Phased CS-ML approach

The proposed CS-ML workflow consists of four phases: (I) preparation, (II) execution, (III) implementation and (IV) reiteration phase (Fig. 5). Each phase has its own objectives, challenges, milestones and evaluation moments/criteria. In the preparation phase (I) the objective is to establish an outline of the CS-ML project. A major challenge is to establish support (e.g., institutional embedding, financial support and community support) for the duration of the project and the implementation afterwards. As CS-ML projects inevitably require communication over very different domains and with different participants, an advisory board with members from the different domains might be useful for the project manager to strengthen communications. With a well-supported project outline, the preparation phase can be concluded with an evaluation in which possible concerns from contributors and other stakeholders are discussed and incorporated in the next phase. The execution phase (II) (Fig. 5) consists of three parts that are enacted simultaneously: (IIa) data, (IIb) ML model and (IIc) app/infrastructure. The main objective is creating high quality datasets and ML models in an iterative process to identify the ideal data collecting strategy as well as the best performing model architecture and parameters. These strategies feed into the design of the app and infrastructure for public engagement. A major challenge is to reduce biases and ensure data quality. The execution phase is concluded when the online CS platform is ready to launch. The experience of the contributors and the performance of the ML models can then be evaluated. The main objective of the implementation phase (III) is to generate new data and use these for research and outreach. Finally, the reiteration phase (IV), when new data and feedback are periodically used to retrain and update ML models, apps and infrastructure, runs parallel to the implementation phase. The major challenge for both implementation and reiteration phases is the sustained use and support by contributors and other stakeholders, even beyond the project duration. Periodic milestones and evaluations can be based on the scientific output and use of the CS platform.

CS-ML approach objectives and challenges

Within the CS-ML approach we have identified further objectives and challenges that can be organized in five subject domains: CS tasks, ML development, research, stakeholder engagement and app development. Each subject domain has different objectives and challenges that evolve during the successive phases of the CS-ML approach (Fig. 6).

The main objectives of the CS tasks subject domain, are the sustained participation (beyond duration of the project), motivation and self-efficacy of citizen scientists and involve both tasks performed by different CS groups as well as the related organisational and managerial tasks. The main objective during the preparation phase (I) is to outline the CS participation and to initiate CS engagement. By already involving citizen scientists in the planning phase, project managers can incorporate their wants and needs in the design of the project. A main challenge here is to address potential scepticism related to the use of ML and the role of CS in scientific research. During the execution phase (II), citizen scientists are involved in collecting and annotating data, in identifying case studies and in validating ML model predictions. It is important to realise that these tasks are time consuming and that it is a challenge to properly mitigate data biases and varying data quality (e.g., Koch et al., 2022). The objectives related to these challenges are for citizen scientists to provide feedback and to express doubt while performing tasks as well as to create a CS platform that is user friendly and designed for maximum data quality through clear instructions and opportunities for feedback. These can include instructions and feedback on how to orientate the object and how to take pictures for example. To motivate citizen scientists, it should be clearly communicated that their work contributes to the development of ML models and that the citizen science community is properly attributed when releasing models or publishing work in which the models are used. During the implementation phase (III), the main CS task objective is to provide training and information. The challenge is to initiate and maintain engagement with all potential CS groups by increasing motivation and self-efficacy (e.g., Soul et al., 2018; Bourgeois et al., 2024). From our experiences with palaeontological and archaeological collectors, building and maintaining a thriving CS community requires large time investment, attention and recurring participation meetings. Whenever a CS community becomes larger inevitably collaboration issues will arise. A liaison officer role is required to organise meetings, notice signs of discomfort among participants and address these in a timely manner. These responsibilities can also include handling social media accounts related to the project. In the reiteration phase (IV), the main objective is to incorporate user feedback in the continued development of the CS platform by which motivation and self-efficacy can be increased.

For the second subject domain, the ML development, the objective is to deploy a ML model trained on CS data that can be used by the general public. In the preparation phase (I), the objective is to explore the existing ML tools and hardware that are available and to decide on using or converting an existing ML pipeline or to build a new ML pipeline. Sufficient institutional support with regards to finances, resources and knowhow is key. During the execution phase (II), the collected dataset can be made more robust with class balancing, data augmentation or data synthesis as palaeontological and archaeological data are generally scarce (e.g., Bickler, 2021). This way biases can be reduced, and varying quality of the data can be improved. To do so requires domain knowledge and a strong collaboration between computer scientists and domain experts, i.e., palaeontologists and archaeologists. ML models are trained, tested and model performance is evaluated which provides insights into missing or weak parts of the dataset. Both the ML models and the dataset can be improved in an iterative process. Incorporating metadata in the training process can further reduce biases (e.g., Bird et al., 2014; Gooliaff & Hodges, 2018; Santos-Fernandez et al., 2021). A challenge here is to standardise the metadata and to incorporate the metadata collecting in the data collection strategy. Aside from the main ML models, additional models can be trained for task allocation and for providing real-time feedback for the CS platform users. Incorporating these models in the app and infrastructure can further reduce biases and increase data quality. In the implementation phase (III), the objective is to provide live ML models that can be used in the app. A major challenge is to explain the inner workings of the model to users (e.g., Hassija et al., 2024). In the reiteration phase (IV), the model is also analysed, looking for gaps in the dataset and looking for innovative methods to improve model performance. As ML methods develop rapidly, it is a key challenge to keep up with these innovations (i.e., keeping code up to date and making sure dependencies are working) and periodically rebuild the ML models and infrastructure to guarantee compatibility and state-of-the-art model performance.

In the third subject domain, research, the overall objective is new scientific discoveries and insights. Additionally, domain knowledge support for the CS and ML tasks as well as outreach are important objectives. During the preparation phase (I), the specific research objectives are established. The challenge here is to evaluate if the CS-ML approach actually suits these research objectives. This includes an evaluation of the suitability of CS involvement in, e.g., fieldwork with regards to the applicable rules and regulations. In the execution phase (II), research data tasks focus on training citizen scientists, standardising (meta)data collection and annotation and identifying case studies. The major challenge is again to deal with biases and varying data quality. Together, these tasks are time consuming and need to be planned out ahead of time. Validation of ML model predictions focusses on more rare and complex cases. Standardised naming conventions, i.e., thesauri, need to be employed for consistency. Also, for research purposes it is challenging but key to express doubt while performing these tasks to reduce biases in the data and ML models. Additional subject information can be provided for the app for outreach purposes, as well as input on the design of a digital reference collection and CS collection registration system. This is done with input from different user groups in mind. With the app and infrastructure up and running in the implementation phase (III), the main objective of research becomes new discoveries and insights resulting from the project. A challenge here is to fully explain interpretations from the CS-ML approach with its possible biases and to generate scientific output. In the reiteration phase (IV), gaps in the dataset are identified and new sources of data must be found and incorporated in the dataset.

The fourth subject domain, general stakeholder engagement, aims at engaging in a wider societal discourse and to incite sustained real-time (inter-)action. For this purpose, the relevant stakeholders and what they might want to get out of the project need to be identified in the preparation phase (I). An array of approaches for stakeholder identification and engagement exists, e.g., The BiodivERsA Stakeholder Engagement Handbook (Durham et al., 2014). In the execution phase (II), the main objective is to create a FAIR data infrastructure to make sure the data can be (re)used for different projects beyond the scope or topic of the original project. CS stakeholders that invested time in creating/validating datasets should be acknowledged. The implementation phase (III) includes the development of educational material and different outreach packages. Although challenging, a FAIR data infrastructure can incite real-time action in and beyond the scope of the CS-ML project. In the reiteration phase (IV), feedback from stakeholders can be incorporated in the data management and in improving the platform design. Furthermore, additional (meta)data can be collected if this can improve (interdisciplinary) engagement and research.

The objective for the fifth subject domain, app and infrastructure development, is a durable and adaptable infrastructure with an app that is user friendly and leads to high quality, FAIR data suitable for a wide variety of scientific research. During the preparation phase (I), the objective is to explore the requirements of the infrastructure to host the FAIR data, ML models and engagement with researchers and other stakeholders. For such an infrastructure to be durable, sufficient institutional embedding is required. During the execution phase (II), the app and infrastructure are built. A possible challenge that might arise are contradicting infrastructure requirements for the aims of different stakeholder groups (e.g., with regards to privacy and licencing of CS data). In the implementation phase (III), the main objective is to maintain the app and infrastructure, which again requires sufficient institutional embedding. Finally, in the reiteration phase (IV) the aim is to identify and design new features for the app and update the infrastructure when needed.

Roles and task allocation

As CS-ML projects are inherently complex, and involve many actors, a clear definition of roles and tasks is paramount to a successful collaboration. Here, we formulate tasks based on the identified objectives and challenges and allocate the tasks to contributors in different roles (Fig. 7). Tasks can be assigned to multiple roles, individual contributors can perform multiple roles and multiple contributors can in turn perform a single role. Based on the results of our review, combined with the experience of the authors, we propose six categories of contributors/participants, viz. user/contributor, volunteer expert, citizen and professional researcher, computer scientist, app/infra developer and the project manager. We split the general group of citizen scientists into user/contributor and the volunteer expert. Although both fall under the CS umbrella, they may perform different tasks because their skill and engagement levels differ. The citizen- and professional researcher, computer scientist and app/infra developer each are domain experts, and their roles largely correspond to the research, machine learning development and app/infra development subject domains respectively. However, their tasks will sometimes overlap as input or support from different domains may be required. The project manager is involved with all subject domains in the project and is responsible for keeping track of general progress and the different objectives and challenges. Here, we also group the previously mentioned liaison officer role with the project manager role.

App and infrastructure

An online CS platform requires a durable and adaptable infrastructure to reach a CS-ML project’s full scientific and public engagement potential. Specific app and infrastructure designs depend on the wants and needs of particular CS-ML project participants. Here, we summarise findings from the reviewed articles in a generalised app and infrastructure workflow and relate the different aspects of the workflow to the previously identified CS-ML approach subject domains (Fig. 8). Our generalised workflow consists of a mobile/web app, set of web services for deploying ML models and storing data, a database web portal and a linked open data service.

The mobile/web app is the main front end of the workflow and is designed for engagement with citizen scientists and other stakeholders. The app aims at a user-friendly environment with clear instructions for consistent high-quality data. It may contain information on the project and participation guidelines as well as communication and feedback functions to interact with other contributors and the project managers (e.g., chat, forum, mail, newsletters, messages). Furthermore, it contains the ML tools for e.g., image classification, and a portal for volunteer experts to validate the data and ML model predictions. Finally, the app contains different tools to engage with collections and data that is gathered through the app. These tools can include a personal collection registration system, a digital reference collection, data overviews such as graphs and maps and additional subject information. The app is where contributors from all CS-ML approach subject domains interact and engage with the project.

The set of web services are the backbone of the infrastructure where data and ML models are stored and through which the data flows between different front-end portals. Depending on the individual project needs, the web services can include APIs for storing new observations, (reference) database and ML model servers. For the web service infrastructure design, input is required from different approach domains such as research and ML development. As designing and building an infrastructure from scratch is expensive and time consuming, it might be worth considering making use of existing, large infrastructures. An additional benefit from a shared, large infrastructure is that there might be pre-existing audiences that can be introduced to your project.

The database web portal and linked open data service do function as research tools with which the data can be analysed, compared to other datasets and used to develop new tools. In the database web portal, citizen scientists, researchers and other stakeholders can go through, filter and visualise data from the platform database. These tools can be used to gain general insights, produce research or create educational material. The linked open data service is mainly geared towards researchers, data scientists and software developers that want to combine the data from different databases to create further applications or conduct further data research.

Specific challenges in object-based CS-ML approaches: lessons from the LegaSea project

Finally, we discuss aspects of the implementation of our proposed CS-ML workflow in practice with a case study, the NWO-ENW-funded LegaSea project. LegaSea aims to identify and characterise late-Quaternary vertebrate communities from rich beach finds through a CS-ML approach and is currently in the execution phase. Late-Quaternary fossils and artefacts are found in strata underlying the seafloor in the southern North Sea, the area known as ‘Doggerland’ (Amkreutz & Van der Vaart-Verschoof, 2021). These strata provide sand used for beach nourishments in the Netherlands that include the abundant fossils and artefacts. The publicly accessible sand nourished beaches attract lots of CS activity, outreach and scientific output (e.g., Amkreutz & Van der Vaart-Verschoof, 2021; Kuitems et al., 2015; Moerdijk et al., 2010; Mol, 2016; Mol et al., 2006; Mol & Bakker, 2022). Active collaboration between citizen scientists, researchers and institutes is organised through societies and working groups (e.g., Working Group Pleistocene Mammals, WPZ, and Working Group Tertiary and Quaternary Geology, WTKG) as well as through online platforms (e.g., Oervondstchecker; https://www.oervondstchecker.nl/) and social media groups (e.g., ‘Strandfossielen’ on Facebook). The Dutch CS community is very diverse and ranges from participants that operate on a professional level to novice collectors with little experience. Engagement must be tailored to specific subgroups of citizen scientists. Furthermore, CS (field) days are organised regularly by societies, institutes and museums such as the Dutch National Museum of Antiquities and Futureland.

In the preparation phase of the LegaSea project, citizen scientists, researchers and computer scientists that are engaged with the research subject and/or host institute were consulted on the project aims and outline. This way the main objectives and challenges for the CS tasks, research and ML development were addressed. There are several CS communities, networks and initiatives working on Quaternary beach finds with active collaborations with professional researchers, but these communities lack an overarching platform with a robust and open data framework which would benefit the collaboration between researchers and citizen scientists. Within the Netherlands fossil finds are not regulated. However, archaeological artefacts are subject of a multitude of heritage laws such as formulated in the Valletta Convention (Council of Europe, 1992) and the Dutch Heritage Act (Government of the Netherlands, 2016) that aim to preserve archaeological heritage and only allow excavation under specific circumstances and under strict conditions (Van Kolfschoten, 2006). For the beach finds that are ex-situ and do not derive from land, most of these laws are not applicable other than the requirement to administer archaeological finds. Only for human remains (including fossil human remains) do strict regulations exist for reporting material to the authorities. Discouraging beach collecting through overregulation would be counterproductive as these objects will disappear when not collected. However, application of our CS-ML workflow to in-situ on land sites will need further steps to incorporate legal requirements for such activities. This is beyond the scope of LegaSea. A pre-existing ML infrastructure at the host institute, designed and used for biodiversity monitoring, could be used and adapted in the LegaSea project. This ‘pipeline’ allowed for a quick exploration and piloting phase which informed the data collection strategy in the execution phase of the project.

Both the stakeholder engagement and the app and infrastructure development were not yet outlined in the preparation phase of the project. First, the specific needs and requirements form the target CS communities had to be explored and established. Within the LegaSea project we organized a CS participant board and discussed with them their uses, requirements and desires for ML-based support and an online platform. Another stakeholder research relevant for the project is currently being carried out under the umbrella of the NWO-SWG funded research program ‘Resurfacing Doggerland’, which targets the post-glacial human occupation history of the southern North Sea, engages with CS and uses in part beach finds of both artefacts and fossils. Citizen scientists and other stakeholders were asked about their previous experiences working with scientists and online CS platforms as well as their wants and needs for a future online CS platform. The findings are presented and discussed in the ‘Doggerland Heritage Community’ (DOGHER-C) report (S van der Vaart-Verschoof, 2025, in preparation) and provide valuable insights in the different experiences of the citizen scientists and other stakeholders. A common approach is desirable as these objects derive from the same context and provide relevant and complementary information for both archaeology and palaeontology (Fig. 1).

So far, the main emphasis of the execution phase of LegaSea has been on data collection, validation and pre-processing. To increase public engagement, part of the dataset was photographed at a public venue at the host institute over a period of 2 months. Further complementary datasets were collected from other institutes in the Netherlands, as well as from CS collections. With the help of citizen scientists and a volunteer at the host institute, there is a continuous generation of new data. As both data quality and quantity affect the model performance, the quality of data from different sources is currently being investigated as part of the workflow by comparing model performances on photographs taken under ideal conditions and photographs uploaded to an online CS platform (https://www.oervondstchecker.nl/). Furthermore, we are currently assessing whether various data augmentation techniques can improve model performance. A major difference between observation-based biodiversity and object-based archaeological and palaeontological CS-ML approaches is the nature of metadata. Important data in the former are location and time of observation and furthermore context data such as habitat or environmental parameters. Important additional data in the latter are data of the stratigraphic source (geological unit, age, interpreted palaeoenvironment). In the case of ex-situ beach finds these require geological research in the extraction areas (e.g., Niekus et al., 2023) and often come with uncertainties such as diffuse age-data (e.g., Busschers et al., 2014). Providing such context data often is beyond the remit of individual CS (but there are exceptions, e.g., Mol, 2016) and requires professional support or management.

The LegaSea project has not yet reached the implementation and reiteration phases, but a start has been made with publishing the ML models from the project. Doing so early in the project allows us to motivate and galvanise citizen scientists and stakeholders to participate in the project and to provide further (data) input and feedback consistent with an Open Science approach. This early implementation benefits from the pre-existing infrastructure at the host institute. Further implementation and continuation of the project results require additional, structural financial support and institutional embedding. Exploration of the possibilities requires an approach where all stakeholders are involved and feel like the project is mutually beneficial. Our CS-ML approach addresses those needs and makes a successful project possible.