Original Publication
Open Access

Development and Implementation of Multi-source Assessment Tools for ACGME Residents and Fellows

Published: July 9, 2014 | 10.15766/mep_2374-8265.9839

Included in this publication:

  • Evaluation and Feedback Subcommittee.doc
  • Faculty Eval of Trainee-Short Version-Expectations Scale.pdf
  • Faculty Eval of Trainee-Standard Version-Expectations Scale.pdf
  • Faculty Evaluation of Trainee Appendix-Expect Scale.pdf
  • Instructions for Template 360 Evaluation Forms.pdf
  • Instructor's Guide.docx
  • Other Healthcare Professional Eval of Trainee.pdf
  • Peer Evaluation of Trainee.pdf
  • Trainee Self-Assessment.pdf

To view all publication components, extract (i.e., unzip) them from the downloaded .zip file.


Editor's Note: This publication predates our implementation of the Educational Summary Report in 2016 and thus displays a different format than newer publications.

Abstract

Effective evaluation is a cornerstone of medical education. The Accreditation Council for Graduate Medical Education (ACGME) requires that trainees are assessed and provided feedback for each rotation, and that competency-based, multi-source evaluation is incorporated. National data from the ACGME’s 2013 resident survey indicates dissatisfaction with faculty feedback among 27% of residents; this finding has been consistent over several years.

In 2011, the Partners HealthCare System (including Massachusetts General and Brigham and Women’s hospitals) appointed a 16-member education subcommittee to enhance evaluation and feedback in the system’s 100+ accredited GME programs. The Evaluation and Feedback Subcommittee (EFS) included program directors and other educators from multiple specialties, institutional GME leadership, and trainees. This undertaking reflected concern about the high number of evaluation-related citations among sponsored programs, and suboptimal ratings of evaluation/feedback on trainee surveys and in internal reviews. An Education Specialist was hired to help lead and implement this effort.

The EFS analyzed the barriers to achieving consistent, effective and timely evaluation and feedback. The group identified development of template multi-source evaluation tools and faculty development as key resource needs.

The Education Specialist presented results of a literature review and of interviews with educational leaders from major teaching institutions across the country. In addition, assessment tools from various Partners training programs and other teaching institutions were examined. The EFS concluded that ideal, competency-based 360-degree assessment tools were not generally available.

The following considerations were the focus of thoughtful discussion and debate: Content, evaluators, number of items, type of rating scale, and the requirement for text comments. Key decisions were made after extensive debate and consultation with stakeholder groups outside of the committee. The system-side education committee then endorsed the resulting assessment tools and they were distributed to all training programs through the on-line system, New Innovations.

Five evaluator roles were identified, and a template evaluation form was developed for each: Self (resident or fellow); Peer; Faculty; Other health professional (e.g. nurse); Patient. [Note: Regulatory barriers to patient surveys regarding resident performance were encountered; these will be described and the tool will be published separately].

Content included relates to knowledge, skills and behavior relevant to all specialties. This will be particularly important as specialty-based “milestones” are determined at the national level.

The resulting assessment tools provide multi-source, competency-based evaluation for assessment of ACGME trainees. They have been provided to the approximately 100 ACGME-accredited GME programs at Massachusetts General and Brigham and Women’s hospitals and are currently in use by 36 programs.

Twenty-four faculty development sessions have been provided since 2011. These sessions introduce the assessment tools with tips for implementation and evidence-based feedback and evaluation strategies.

Competency-based 360 assessment tools combined with faculty development may improve the quality and consistency of feedback and evaluation of trainees. These assessment tools provide the framework for ongoing formative verbal feedback, end of rotation formative evaluation as required by the ACGME, and serve to inform the biannual specialty-specific Milestone assessments.

Effective evaluation is a cornerstone of medical education. The Accreditation Council for Graduate Medical Education (ACGME) requires that trainees are assessed and provided feedback for each rotation, and that competency-based, multi-source evaluation is incorporated. National data from the ACGME’s 2013 resident survey indicates dissatisfaction with faculty feedback among 27% of residents; this finding has been consistent over several years. ACGME migration to milestones as part of the Next Accreditation System (NAS) has helped focus increased GME Director and faculty attention on assessment and underscores the potential usefulness of thoughtfully developed standardized evaluations. Semi-annual reporting of individual achievement of specified milestones must rely upon ongoing formative (e.g. end of rotation) assessment utilizing a variety of tools. Indeed, the ACGME continues to require formative, competency-based multi-source assessment at the conclusion of each rotation or educational assignment. This work represents a collection of multi-source assessment tools (e.g. evaluation of trainee by faculty, peer, other healthcare professional, self- assessment) intended for formative competency-based feedback and evaluation. They meet ACGME requirements for formative competency-based evaluation at the end of each rotation and are intended for multi-source assessment in both ACGME and non-ACGME residency and fellowship programs.

Educational Objectives

  1. Utilize formative competency-based, multi-source evaluation of residents and fellows in ACGME- accredited Graduate Medical Education (GME) programs to inform summative milestone assessment.
  2. Provide formative multi-source assessment tools to ACGME programs not yet reporting milestones.
  3. Provide multi-source assessment tools to non-ACGME programs in an effort to standardize an approach to multi-source evaluation tools as the cornerstone for end-of-rotation written evaluation, and as a vehicle for facilitating verbal feedback.
  4. Provide standard templates for multi-source trainee evaluations that can be used across specialties and customized, as needed.
  5. Utilize an institutionally-approved assessment tool as a key element in faculty development focused on assessment.
  6. Create an opportunity to benchmark performance and facilitate comparisons across programs and specialties.

Author Information

  • Mary Ellen Goldhamer, MD, MPH: Massachusetts General Hospital and Harvard Medical School
  • Keith Baker, MD, PhD: Massachusetts General Hospital and Harvard Medical School
  • Anne Rigg: Partners Healthcare
  • Debra Weinstein, MD: Massachusetts General Hospital and Harvard Medical School

Disclosures
None to report.

Funding/Support
None to report.

Prior Presentations
Goldhamer ME, Baker KH, Rigg A, Weinstein DF. Development and implementation of multi-source assessment tools for ACGME residents and fellows. Presented at: Association of American Medical Colleges Annual Meeting; November 2013; Philadelphia, PA.



Citation

Goldhamer M, Baker K, Rigg A, Weinstein D. Development and implementation of multi-source assessment tools for ACGME residents and fellows. MedEdPORTAL. 2014;10:9839. https://doi.org/10.15766/mep_2374-8265.9839