Abstract The University of California, San Francisco School of Dentistry wanted to determine if a predental school manual dexterity test predicts: 1) subsequent grades in preclinical restorative courses, and 2) faculty perceptions of satisfactory performance in these skills that would indicate the student is ready to advance to the clinic. The study population was comprised of all 244 applicants admitted to UCSF School of Dentistry’s D.D.S. Program from Classes of 2000 to 2002 and who matriculated into the program. The manual dexterity test (MDT) consisted of a two-hour block-carving test. Three preclinical faculty, three clinical faculty, and two basic science faculty graded the blocks. Even after instruction and calibration, faculty varied greatly in their grading (intra-rater reliability kappa statistics ranging from 0.34 to 1.00). Two of three preclinical raters gave No Passes for the MDT in 9.8 percent of the incoming, first-year dental students.
Of these twenty-three students, only four (17 percent) were in the lower 10 percent of their classes according to their five preclinical restorative laboratory courses after two years, and four (33 percent) were among the twelve students the three preclinical laboratory directors identified as laboratory cautions. The MDT did not significantly (p=0.342) predict students in the bottom 10 percent after five restorative preclinical laboratory courses, above and beyond current admissions criteria.
Among current admissions criteria, PAT score was the only item at least moderately correlated with preclinical average percentile class rank (Spearman correlation = 0.34). In conclusion, the MDT did not appear to add information to the current admissions criteria. Keywords: • • • • • • • • • •. Methods The study sample comprised all applicants admitted to UCSF School of Dentistry’s D.D.S.
Program from 1996 through 1998 (dental graduating classes of 2000, 2001, and 2002) and who matriculated into the program. During the first dental morphology laboratory session in their first week of class, these students were asked to participate in this study, the protocol for which had been reviewed and granted an exempt waiver by UCSF’s institutional review board, the Committee on Human Research.
The consent form for the study communicated the study aims, informed students that their identities would be masked from the faculty, and stated that the outcome of the block-carving would not in any way influence students’ subsequent course grades or progress through the curriculum. An administrator (HP) created the database with information from the dental admissions office. The database included the undergraduate grade point average (GPA), the science grade point average (SGPA), and the DAT and PAT scores for all students admitted to UCSF School of Dentistry in 1996, 1997, and 1998 (Classes of 2000, 2001, and 2002). Students were assigned unique study ID numbers to protect confidential student identifiers such as name or student ID number. Block Carving.
Each student was given a plaster block, a blade holder, two interchangeable blades, a millimeter ruler, a pencil, a printed handout showing the form to be carved, written instructions to guide them in the process, and the four criteria by which the block would be graded as Pass or No Pass (Figure 1). Students were shown a ten-minute videotape demonstrating the carving step-by-step. Students then had two hours in which to complete their carvings in accordance with the four criteria.
American Society for Testing and Materials. And Monitoring >MTO Qualified Labs >MTO Manual - Full Lab Manual MTO. Laboratory, laboratories. MATERIAL SPECIFICATION FOR AGGREGATES - SURFACE TREATMENT. MTO Laboratory Testing Manual. Meera Vijay Tv Serial 2010. Legible and signed by a testing laboratory.
The blocks were prepared, distributed, and collected by a laboratory technician, who assigned each a block number, recorded student names and block numbers, and stored the blocks until the end of the study, when all three classes’ blocks were graded. An administrator (HP) in the dean’s office kept the code keys and students’ names. In all, 256 students completed the block-carving exercise. After students in each of the three classes completed the carving, the blocks were renumbered using randomly generated three-digit labels.
During this process, the original numbers assigned at the time of the carving exercise were painted out, as were any names students mistakenly wrote on the blocks. Once all the blocks were assigned a random three-digit identification label, grading sheets were prepared, and the blocks were placed on five laboratory benches for grading. Twelve of the 256 blocks could not be graded due to various problems including two pairs of blocks with the same number, one block with a student’s name crossed out and replaced by another name, and five blocks with ID numbers for which no student name was recorded, thus leaving a total of 244 blocks that were graded. Considering our original purpose of identifying a small number of students who might fail the dexterity test, this number of unaccounted blocks raised a concern that some key students who might have failed the MDT were missed. Calibration and Grading. The blocks were graded Pass/No Pass on the basis of criteria described in the handout distributed to students at the time of the carving exercise.
Before the blocks were graded, three senior faculty members, who are directors of preclinical laboratory courses, participated in a calibration session overseen by an administrator. Pre-graded blocks, which Dr.
Donald Beglau from the School of Dentistry at Loma Linda University loaned for this purpose, were used for the calibration. Loma Linda uses a four-level grading process, rating blocks as “Highly Acceptable,” “Acceptable,” “Marginally Acceptable,” or “Unacceptable.” The set of twelve blocks used in this calibration exercise included three blocks from each of the four categories according to the Loma Linda faculty. UCSF graders did not know the number of blocks in each category. After independently assigning Pass/No Pass grades to each of the twelve blocks, the three graders compared their results, learned the assessments of the Loma Linda faculty consensus, and refined their application of the agreed-upon grading criteria.
Later that same day, the three faculty graded the 244 blocks that the three classes of dental students had carved for this study. During the grading of the blocks, each instructor began on a different laboratory bench; no talking or consultation occurred until all graders had finished grading all blocks. Grading took approximately sixty minutes.
The grading sheets were reviewed to ensure their legibility and to make sure that each block had received a grade. Not passing was operationally defined as two of the three preclinical faculty assigning grades of No Pass. Separate calibration and grading sessions were also conducted for three senior clinical faculty members who are experienced graders in the clinics. Thus, six senior faculty members graded each block as Pass/No Pass.
Intra-Examiner Grading. To test intra-examiner (within examiner) grading reliability, three grading sessions at one-week intervals were conducted. Eight faculty members participated in these sessions: the three preclinical laboratory and the three clinical graders as before, plus two senior faculty from the basic sciences (neuroscience, physiology) who are not dentists and were included in this phase of the study as a comparison group. The basic science faculty were included because some faculty members wondered if the experienced laboratory and clinical graders were being too critical in what was intended to be a less complex screening for serious errors.
To test this idea, the basic sciences faculty who had no previous background or experience in dental techniques were included. At each of these grading sessions, the eight instructors first reviewed the grading criteria and completed a calibration exercise, independently grading twelve blocks as Pass/No Pass. The same calibration exercise was used as previously described for the faculty who graded the 244 blocks but with different blocks.
One of authors (WB) selected thirty-six calibration blocks from the original 244 blocks, with a different set of twelve used for each of the three calibration sessions. Each set included a spectrum of quality from unacceptable to flawed to excellent, according to that senior faculty member (Figure 2). After grading the calibration blocks, the participants shared the grades, discussed discrepancies, and reached consensus on how to apply the grading criteria. The eight faculty members then independently graded a sample of sixty carved blocks; this same sample of sixty was used throughout all three sessions to test intra-examiner reliability over time.
Grade sheets were scanned for legibility and completeness and, as before, no talking or consultation occurred during the grading sessions. The sample of sixty blocks used to assess intra-examiner reliability was selected to reflect the distribution of quality of the 244 blocks, rather than at random.
For example, all six graders passed 156 (63.9 percent) of the blocks. Therefore, four categories of blocks based on the number of No Pass grades each block had received were used.
A senior clinician (WB) who was not otherwise involved in the grading selected representative blocks from each of the four categories so that the percentage of blocks from each category in the sample corresponded to the percentage of blocks in that category in the total population. The sixty-block sample included thirty-eight blocks from the first category (0 No Passes), thirteen blocks from the second category (1–2 No Passes), four blocks from the third category (3–4 No Passes), and five blocks from the fourth category (5–6 No Passes). Preclinical Laboratory Grades. As one quantitative measure of success in mastering the manual dexterity skills required in dentistry, the cumulative raw numerical scores from practical examinations completed in each of the five required first- and second-year restorative laboratory courses were used. Although the content and educational objectives of these courses remained consistent over the period of the study, the actual graded exercises, scoring, and course titles varied from year to year. Therefore, for the purposes of this study, cumulative numerical scores were ranked within each of the three classes of students, and within-class percentile ranks were assigned for each course.
The dental students from three dental classes completed five restorative laboratory courses for a total of sixteen units over two years. The total time in these five courses was 580 credit hours. Grades were determined as raw scores for each of the three classes, then ranked as percentile scores for that class of students in each course. A mean percentile class rank for each student was used as an overall measure of student performance. Scores were then divided to determine the lowest 10 percent of the students.
Faculty Perceptions and Laboratory Cautions. In addition to the quantitative data collected from laboratory courses, we used an established UCSF preclinical laboratory procedure (not initiated for this study) to include qualitative faculty perceptions in this study. To achieve this, the three faculty members who are course directors in the preclinical laboratory courses met to identify retrospectively (after students had completed the first two years) those students they believed might have difficulty in their clinical courses. This identification process involved each faculty member listing the students whose skills they felt were highly questionable; the faculty discussed their selections and conferred to reach consensus on the final identification.
During the process, the faculty utilized their grading records as well as class photographs, to help them accurately recall the students and their performance. The final consensus list identified twelve students from the 244 total students. These identifications were labeled “laboratory cautions” to signify concern about the students’ readiness to perform restorative care for patients in the third- and fourth-year clinics. Statistical Analysis. The hypothesis tested was that students who failed the manual dexterity test would be in the bottom 10 percent of their class for five preclinical restorative classes.
Similarly, we also tested the hypothesis that the manual dexterity test significantly predicted the bottom 10 percent after accounting for admissions criteria. Reliability was tested within and among MDT graders using chance adjusted agreement Kappa statistics. Spearman rank correlations evaluated linear relationships with a scatterplot matrix and local polynomial regression (loess) smoothing to examine nonlinearity. Admission scores including GPA, SGPA, DAT, and PAT were summarized with median and interquartile range (middle 50 percent) as the distributions were skewed.
One degree of freedom (df) chi-square trend tests compared number of MDT failures with GPA, SGPA, DAT, and PAT scores. Logistic regression was used to test the relationship between MDT failure (2/3 preclinical faculty) with percentile rank score ≤10 percent and with laboratory cautions after adjusting for SGPA, PAT score, and two indicators for class year.
(Due to collinearity, SGPA and PAT were used without GPA and DAT, which were less related to average percentile rank score.). Pass/No Pass Grading of 244 Chalk Blocks by Three Preclinical Faculty. The three preclinical faculty graded the 244 blocks during a sixty-minute session in one day.
The faculty varied on the number of blocks that received No Pass grades. However, 187 (77 percent) of the 244 blocks were seen as acceptable to all three preclinical faculty (Table 3) with one of the three (“B”) faculty accounting for twenty-nine of the thirty-three students identified by only one faculty member as not passing. Unanimous agreement on non-passing blocks was reached with fourteen (6 percent) of the 244 blocks. Two of the three examiners assigned non-passing grades to twenty-four (10 percent) of the 244 blocks. Inter-rater reliability Kappa statistics among the three pre-clinical faculty were 0.36, 0.52, and 0.64, indicating “fair,” “moderate,” and “substantial” agreement, respectively. Pass/No Pass Grading of 244 Chalk Blocks by Three Clinical Faculty.
The three senior clinical faculty members graded the 244 blocks during a sixty-minute session in one day that included training and discussion before judging (Table 3). The clinical faculty varied on the number of blocks that received No Pass grades. However, 171 (70 percent) of the 244 blocks were seen as acceptable to all three preclinical faculty. Forty-three (18 percent) of the blocks were judged by only one faculty member as not passing with one member (“E”) accounting for thirty-nine of them. Unanimous agreement on non-passing blocks was reached with sixteen (7 percent) of the 244 blocks. Two of three examiners gave No Pass to thirty (13 percent) of the blocks.
Class Grades of 236 Students in Five Restorative Laboratory Technique Courses. Two hundred and thirty-six students had class grades and block-carving test results that could be merged.
Nineteen (8.9 percent) of the 213 students who were above 10 percent in the class did not pass the block-carving test (Table 4). Four of the twenty-three students in the lowest 10 percent of the class passed the block-carving test, resulting in negative predictive value (NPV) of 17.4 percent and positive predictive value (PPV) of 91.1 percent. Subjective Evaluation (Laboratory Cautions) of Dental Student Progress in Preclinical Courses. The preclinical faculty normally evaluate students at the end of second year to identify students with a laboratory caution indicating their skills are not sufficient to proceed to the clinic (Table 5).
The faculty identified twelve students from the 244 whom they felt did not have sufficient clinical skills to proceed into the clinic for the third and fourth years. The MDT identified four of those twelve, resulting in NPV of 33.3 percent and PPV of 91.6 percent. Admission Criteria including GPA, SGPA, DAT, and PAT. The admission criteria of GPA, SGPA, and DAT scores were not significantly associated (p≥0.195) with the number of No Passes on the MDT (Table 6). The number of failures was significantly related to PAT (1 df chi-square test p=0.042).
Theoretically, GPA and SGPA range from 1.0 to 4.0, while DAT and PAT scores range from 0 to 30. However, among UCSF students, the range was much narrower. The scatterplot matrix (Figure 3) shows the univariate distributions of, and relationships among, average percentile ranking (AvgPctRk), GPA, SGPA, DAT and PAT—two at a time. In the upper left corner, average percentile ranking (the mean of five uniformly distributed variables) is somewhere between uniformly and normally distributed. The remaining panels in the first row show the relationships between average percentile rank with GPA, SGPA, DAT, and PAT, respectively. Using Spearman correlation co-efficients, only PAT had a moderate (r s=0.34) correlation with average percentile ranking (GPA: 0.06; SGPA: 0.08; DAT: 0.16). As expected, GPA and SGPA were highly correlated (r s=0.90).
Comparison of students who did not pass the manual dexterity test (dark dots) to those who did pass (gray dots) as assessed by five criteria used in admissions Note: Notice that some relationships have a strong linear correlation with Spearman correlation coefficients (rs) values. Abbreviations: AvgPctRk, mean percentile class rank of each student in five preclinical laboratory courses over two years; GPA, grade point average in undergraduate school; SGPA, science grade point average in undergraduate school; DAT, Dental Aptitude Test overall score; PAT, Perceptual Aptitude Test score.
Predicting Performance in Preclinical Restorative Courses. Among admissions criteria, only PAT score was suggestive of being significantly related to average percentile rank ≤10 percent (odds ratio (OR)=0.82, 95 percent confidence interval (CI): 0.67 – 1.01, p=0.063) with higher scores having lower chance of being in the bottom 10 percent. Class year, SGPA (OR=0.44, 95 percent CI: 0.14 – 1.43, p=0.157) and MDT (2/3 No Passes) (OR=1.79, 95 percent CI: 0.53 – 6.06, p=0.342) were not statistically significant. Faculty laboratory caution status significantly predicted the bottom 10 percent (OR=10.9, 95 percent CI: 2.78 – 43.1, p. Discussion These data indicated the manual dexterity test (MDT) identifies about 10 percent of the incoming first-year dental students as not passing the chalk test (2 of 3 No Passes). Of these students who received a No Pass score on a one-time manual dexterity block carving test for two hours, only four of them were in the lower 10 percent of the class defined in the five preclinical restorative laboratory courses after two years. They did not have significantly lower undergraduate grade point averages.
However, the students who failed the MDT had significantly lower PAT scores, raising an interesting question as to their perceptual as well as motor abilities. The fact that only four (17.4 percent) of the twenty-three students who failed the MDT were in the lower 10 percent of their class by five preclinical laboratory courses suggests that the overwhelming majority of students could learn the manual dexterity skills needed for dentistry. A more valuable tool for identifying students who may have difficulty might be the qualitative assessment by the directors of the three preclinical courses who had the students over two years. Personal faculty bias and subjectivity enter this evaluation, but faculty members are aware of considerations that extend beyond traditional course grading. These considerations include the student’s competency and personality and the faculty member’s perception of a student’s potential as a future dentist.
Approximately one-third of students who received laboratory cautions failed the MDT. It is important to note that faculty identification of students likely to struggle in the clinic occurred before MDT grading. A second set of important questions relates to the criteria faculty use to evaluate the MDT. We chose not to have unanimous decision denote an MDT failure, but accepted a two-thirds majority decision as our criteria.
Faculty vary in their judgment even with established criteria, after calibration and comparison of criteria. The fact that the two non-dental faculty could determine those blocks that were not acceptable about as well as dentist faculty indicates that experience was not a factor. Interestingly, the pre-clinical faculty member (“B”) and clinical faculty member (“E”) who identified the most failures in the assessments actually had the fewest during the calibration, so they may have overcompensated after calibration. If manual dexterity tests are used, our results indicate there needs to be some accounting of individual faculty member variation and much more stringent training and calibration to improve reliability. In comparing the admissions criteria to students’ first two years of five preclinical laboratory courses, there is little correlation between the entering GPA, SGPA, and DAT with the percentile ranking in the student class.
The preclinical laboratory percentile rank correlated better (but modestly) with the PAT than with any other admissions criterion. Indian Ethnic Music Download here. In conclusion, the manual dexterity test using carved chalk identifies some students who will have difficulty in preclinical restorative courses if the faculty criterion of laboratory cautions is used as the outcome, but not the grading system for the courses.
The courses may be designed to pass students or repeat procedures until students do pass. The faculty assessment (laboratory caution) that evaluates factors other than course performance seems to have some value in identifying students who have difficulty in manual dexterity. If an MDT becomes a part of the admissions process, it would have to be used judiciously and economically. The minimum criterion at UCSF for PAT scores is usually around fourteen.
Students with lower PAT scores might be candidates for voluntary manual dexterity testing. Using other criteria such as GPA and DAT did not identify students who tested in the bottom of preclinical courses.
If the MDT was used for all applicants who receive interviews (200–250 per year), it is possible that more students in an applicant pool might fail the MDT. One potential limitation is that since the MDT in this study did not count towards class grades, students may not have tried their best on the test.
However, because the MDT was administered during the first week of dental school, when students are enthusiastic, this possibility was very low. Although the study subjects were matriculating students, not applicants, this study showed that a manual dexterity test did not significantly add to the current UCSF admissions criteria.
The MDT did not increase positive predictive value as was desired.