Evaluation of a multi-institution mastoidectomy performance instrument
- Published
- Accepted
- Subject Areas
- Otorhinolaryngology, Science and Medical Education, Statistics
- Keywords
- assessment, mastoidectomy, virtual reality, simulation
- Copyright
- © 2017 Kerwin et al.
- Licence
- This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ Preprints) and either DOI or URL of the article must be cited.
- Cite this article
- 2017. Evaluation of a multi-institution mastoidectomy performance instrument. PeerJ Preprints 5:e2954v1 https://doi.org/10.7287/peerj.preprints.2954v1
Abstract
Objective: The objective of this work is to obtain validity evidence for an evaluation instrument used to assess the performance level of a mastoidectomy. The instrument has been previously described and had been formulated by a multi-institutional consortium.
Design: Mastoidectomies were performed on a virtual temporal bone system and then rated by experts using a previously described 15 element task-based checklist. Based on the results, a second, similar checklist was created and a second round of rating was performed.
Setting: Twelve otolaryngological surgical training programs in the United States. Participants 66 individuals with a variety of temporal bone dissection experience, from medical students to attending physicians. Raters were attending surgeons from 12 different institutions.
Results: Intraclass correlation (ICC) scores varied greatly between items in the checklist with some being low and some being high. Percentage agreement scores were similar to previous rating instruments. There is strong evidence that a high score on the task- based checklist is necessary for a rater to consider a mastoidectomy to be performed at the level of an expert but a high score is not a sufficient condition.
Conclusions: Rewording of the instrument items to focus on safety does not result in increased reliability of the instrument. The strong result of the Necessary Condition Analysis suggests that going beyond simple correlation measures can give extra insight into grading results. Additionally, we suggest using a multiple point scale instead of a binary pass/fail question combined with descriptive mastery levels.
Author Comment
This is a preprint submission to PeerJ Preprints.