SUMMARY OBJECTIVES: The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test. METHODS: This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student’s scores as the dependent variable. RESULTS: The comparisons of the students’ scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools’ average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students’ scores (p<0.001) and 0.9% of the 6–year students, without statistical difference (p=0.085). CONCLUSIONS: The differences between the students’ scores cannot be attributed to the time when the schools paused the clerkship rotations.

ABSTRACT BACKGROUND: Progress tests are longitudinal assessments of students’ knowledge based on successive tests. Calibration of the test difficulty is challenging, especially because of the tendency of item-writers to overestimate students’ performance. The relationships between the levels of Bloom’s taxonomy, the ability of test judges to predict the difficulty of test items and the real psychometric properties of test items have been insufficiently studied. OBJECTIVE: To investigate the psychometric properties of items according to their classification in Bloom’s taxonomy and judges’ estimates, through an adaptation of the Angoff method. DESIGN AND SETTING: Prospective observational study using secondary data from students’ performance in a progress test applied to ten medical schools, mainly in the state of São Paulo, Brazil. METHODS: We compared the expected and real difficulty of items used in a progress test. The items were classified according to Bloom’s taxonomy. Psychometric properties were assessed based on their taxonomy and fields of knowledge. RESULTS: There was a 54% match between the panel of experts’ expectations and the real difficulty of items. Items that were expected to be easy had mean difficulty that was significantly lower than that of items that were expected to be medium (P < 0.05) or difficult (P < 0.01). Items with high-level taxonomy had higher discrimination indices than low-level items (P = 0.026). We did not find any significant differences between the fields in terms of difficulty and discrimination. CONCLUSIONS: Our study demonstrated that items with high-level taxonomy performed better in discrimination indices and that a panel of experts may develop coherent reasoning regarding the difficulty of items.

Este estudo teve por objetivo averiguar as potencialidades do Teste de Progresso (TP) no auxílio à gestão acadêmica. Foram analisados os desempenhos dos estudantes do curso de Medicina no TP aplicado em 2008 e reaplicado sem modificações em 2011. Os resultados demonstraram que, no TP de 2008, a progressão ocorreu a cada dois anos de curso, e no TP 2011 esta progressão iniciou-se somente a partir da terceira série. Nas áreas de Saúde Coletiva, Clínica Cirúrgica e Áreas Básicas, não houve acúmulo de conhecimentos gradativo desde o início do curso, o que merece reflexões por parte dos gestores do currículo.

This study aimed to investigate the potential of progress testing to help academic management. To this end, the performances of medical students in the Progress Test applied in 2008 and reapplied without modifications in 2011 were analysed. The results showed that in the Progress Test of 2008 progression occurred once every two years of the course and in TP 2011 progression started only from the third year onwards. In the areas of Public Health, Surgery and Basic Sciences there was no gradual accumulation of knowledge since the beginning of the course, which merits reflection on behalf of the curriculum managers.