Original Publication
Open Access

Critical Synthesis Package: Assessment of Professional Behaviors (APB)

Published: September 29, 2014 | 10.15766/mep_2374-8265.9902

Included in this publication:

  • Assessment of Professional Behaviors.pdf
  • Critical Analysis of the Assessment of Professional Behaviors (APB).docx

To view all publication components, extract (i.e., unzip) them from the downloaded .zip file.


Editor's Note: This publication predates our implementation of the Educational Summary Report in 2016 and thus displays a different format than newer publications.

Abstract

This Critical Synthesis Package contains (1) a Critical Analysis of the psychometric properties and application to health sciences education for the Assessment of Professional Behaviors Instrument (APB); (2) a copy of the Assessment of Professional Behaviors Instrument developed by the National Board of Medical Examiners and (3) two sample reports from the NBME based on instrument data.

The Assessment of Professional Behaviors program is a multisource feedback program designed to evaluate the observable behaviors of residents/fellows during their clinical training. The core component of the program is a multisource feedback instrument, Assessment of Professional Behaviors, which is the focus of this critical analysis.

There are two versions of the multisource feedback instrument, designed to be administered electronically, one to the subjects (self-assessment) and the other to those observing their behaviors (faculty, nurses, peers, etc.). Both versions evaluate 23 behaviors, rated using a 5-point frequency scale as well as two free text fields intended as room to provide specific critiques and/or praise. Behaviors assessed as part of formative assessment include interpersonal and communication skills, systems based practice, professionalism, and practice-based learning and improvement.

It is important that in the later part of the 10-year development process, the NBME, focused on training raters, including the use of online videos about providing feedback to trainees. This training is essential to the interpretation of the data and the feedback provided to the trainees to assure a realistic synthesis of data and applicability to their level of training. The research on this instrument has resulted in mixed outcomes related to response process validity and difficulty in conducting successful rater training sessions. Observers’ skills in providing useful written feedback may be amenable to improvement through better modeling, training, and practice. Training that generates awareness of the elements of effective feedback may lead to improved written feedback specific to professionalism.

The 10-year development timeline for the APB is impressive and certainly demonstrates a rigorous process to attempt to increase the validity of the content. It is important in the later part of this 10-year process the NBME and collaborators focused on training raters and incorporating use of online videos to train raters who provide feedback to trainees. This training is essential to the interpretation of the data and the feedback provided to the trainees to assure a realistic synthesis of data and applicability to their level of training, as the research to date on the instrument has resulted in mixed outcomes related to response process validity and difficulty in conducting successful rater trainer sessions. Observers’ skill in providing useful written feedback may be amenable to improvement through better modeling, training, and practice. Findings highlight an opportunity that exists to make written feedback more effective. Training that generates awareness of the elements of effective feedback may lead to improved written feedback specific to professionalism.

The NBME reports the developmental work the NBME performed to create the APB indicated that the NBME’s existing IT systems could not support APB implementation in an effective, scalable, sustainable manner. Consequently the NBME decided that the best way to utilize its investment in developing the APB would be to share the resulting instrument and guidance with the medical education community. Additionally, many programs that used the APB instrument had an interest in editing it to be specific to their specialty or program.  The NBME reports it is currently working with the American Board of Pediatrics and the Association of Pediatric Program Directors to develop observational assessments that can be used to measure Pediatric milestones.  Some of the APB items and training materials have been reused for this purpose.

Although assessing professionalism poses many challenges, gauging and detecting changes in professionalism is impossible without measurement. Instead of creating new professionalism assessments, existing assessments should be improved. Also, more studies on the predictive validity of assessments and their use as part of formative evaluation systems are recommended. For Faculty, one of the benefits of using the instrument is to help structure their feedback to learners.

There is little evidence of reported measures that are effective in assessing attitudes towards professionalism in medicine as a whole. Likewise, there is scant evidence of interventions that influence attitude change over a period of time. Future studies should take into account the need to measure more global attitudes rather than attitudes towards specific issues in professionalism and the need to track attitudes throughout the curriculum.

 The APB program requires professional development to implement and the actual APB instrument requires rater training and piloting within an institution before actual use with a learner population. The Feedback Provider training videos that were included in the submission of the Assessment of Professionals Behaviors program to DREAM provide guidance in using feedback reports to conduct feedback sessions with learners. This program requires institutional/department commitment to assess competency of physicians in training to achieve a quality outcome, which will require allocation of time.

Educational Objectives

  1. To describe the purpose and basic properties of the APB, including number of items and scales, and psychometric properties.
  2. To describe the application of the APB to the field of health sciences education;.
  3. To evaluate the relative strengths and weaknesses of the APB.
  4. To provide the APB and supplemental materials to aid in its administration.

Author Information

  • Alice Fornari, EdD: Hofstra North Shore - LIJ School of Medicine
  • Salaahuddin Akbar: NorthShore-LIJ Health System
  • Samantha Tyler: Union College

Disclosures
None to report.

Funding/Support
None to report.



Citation

Fornari A, Akbar S, Tyler S. Critical synthesis package: assessment of professional behaviors (APB). MedEdPORTAL. 2014;10:9902. https://doi.org/10.15766/mep_2374-8265.9902