Original Publication
Open Access

Online Faculty Development for Implementation and Use of Student Portfolios

Published: August 5, 2016 | 10.15766/mep_2374-8265.10436

Appendices

  • All Modules
  • Sample Questions - Small-Group Discussion.docx
  • Sample Questions - Portfolio Development.docx
  • User Instructions Guide.docx
  • Feedback Form.docx

All appendices are peer reviewed as integral parts of the Original Publication.

To view all publication components, extract (i.e., unzip) them from the downloaded .zip file.


Abstract

Introduction: Over the past decade, portfolios have gained popularity in medical education as tools to evaluate and provide feedback about learning and completion of professionally authentic tasks. Though faculty development has been noted to be key for successful portfolios, there are few available resources. As part of an interinstitutional collaborative project, we have developed online faculty development modules that provide pedagogical information about portfolios, practical advice, and resources from the available literature. Methods: The materials associated with this publication include downloadable modules, which take approximately 45 minutes to complete and can be paused at any time, and sample questions to facilitate small-group discussion with faculty either in the planning stage of portfolios or as part of program evaluation of an institution’s portfolios. Results: A survey taken by faculty from four medical schools after completion of the modules showed that they were well received, with 41% of participants stating that they were very knowledgeable after undertaking the modules compared to 11% before undertaking the modules. Faculty reported increased interest in the topic and increased confidence in their ability to undertake planning for development of portfolios at their institution and considered using the modules as a mandatory curriculum for portfolio coaches at their institutions. Discussion: We suggest that these modules be used for individual self-development, as part of faculty development sessions for portfolio coaches and mentors, or to provide faculty with background information about portfolios during the planning phase of portfolios at an institution.


Educational Objectives

After completing the modules, users will be able to:

  1. Describe a competency-based portfolio.
  2. List the typical contents of competency-based portfolios.
  3. Discuss assessment strategies for portfolios.
  4. Explain factors influencing the successful implementation of portfolios.

Figures and Tables

Figure 1. Level of knowledge before and after module completion.
Figure 2. Feedback on modules.

Introduction

Traditionally, a portfolio is a portable collection of work by artists. In education, portfolios have been described as a compilation of student work assembled to create a lasting archive of academic work products, to evaluate academic achievement, and to determine whether students have met learning standards by monitoring learner development.1 In recent years, portfolios have also been used to document both reflective thinking, in order to educate self-critical doctors,2 and professional growth of a learner through a collection of material.3 Electronic portfolios in particular can have a dossier function that allows tracking of a student’s development of competency over time4 and provides a visual representation of aggregated information about students. Driessen et al. documented that when portfolios are well implemented and triangulated with other assessment methodologies, they can support the professional development of learners.5,6

A national survey in the USA with 71 out of 141 medical school respondents documented that 47% of schools utilized portfolios. Ninety-seven percent of schools that reported using portfolios at their institution stated that there was room for improvement. Success factors identified included engaged portfolio coaches/mentor and faculty development.7

Successful implementation of competency-based portfolios requires “a well-informed mentor.”3 A well-informed mentor has a clear understanding of the educational value of portfolios5 and is able to provide feedback to students.8 One of the major constraints in faculty development is faculty availability. In this era of pressure to generate revenue through clinical practice, it is difficult to pull faculty out of their busy clinical schedules for workshops.9,10

Online faculty development modules are in high demand for their ability to be viewed at the convenience of the faculty member.11 The most effective online modules are those that develop a community of practice and learning on the job.12 These on-demand faculty development modules incorporate online professional development best practices by supporting faculty in their role as portfolio mentors while they are serving as portfolio mentors. The modules introduce faculty to the rationale behind portfolios and prepare them to successfully undertake their role as portfolio mentor. The modules can also be used by institutions that have implemented portfolios to generate a discussion about program evaluation and by institutions that may be planning to implement portfolios to develop faculty understanding of portfolios.

Our target audience includes health and allied health professionals dealing with portfolios; early-, mid-, and late-career faculty; portfolio coaches; and faculty development deans.

Faculty learners can view this presentation independently on a personal computer, laptop, or handheld device. No background knowledge about portfolios is required prior to undertaking these modules.

Methods

Module Development Process
The Southern Group on Educational Affairs (SGEA) encourages interinstitutional collaborative educational research projects supported through the Medical Education Scholarship, Research, and Evaluation (MESRE) grant. Faculty members from four institutions—University of Florida College of Medicine, Vanderbilt University School of Medicine, Baylor College of Medicine, and University of Central Florida College of Medicine—who were part of the SGEA Portfolio Interest Group came together to develop online faculty development modules as part of an awarded SGEA MESRE grant. Institutional review board approval was obtained from the University of Florida College of Medicine in 2014.

Several of us who have educational expertise with portfolios, including Zareen Zaidi, Ashleigh Wright, Toufeeq Ahmed, Amy Fleming, and Maureen Novak, individually undertook extensive literature searches on the use of portfolios in health professions education. We used this literature to identify topics for faculty development, asking ourselves, “What would faculty need to know about portfolios if they have limited prior knowledge about portfolios? How can we provide resources and succinct review of literature to those who have a basic understanding of portfolios?” All of us from the four universities participated in conference calls to discuss key areas of importance that we had identified as being grounded in the literature. Four major areas for focus were identified: purpose of portfolios, portfolio contents in medical education, using portfolios for formative and summative feedback, and factors contributing to the success of portfolios. We decided to develop individual modules for each of these focus areas.

Our author group includes two biomedical informatics experts (Monica Bailey and Toufeeq Ahmed) who advised about the use of information technology to develop the modules. We decided to use Adobe Captivate 8, which is an authoring tool used for creating e-learning content file formats that can be played with media players or uploaded to video-hosting websites. The interface allows recording of narrated PowerPoint presentations. We worked in small groups to prepare four PowerPoint presentations with narration scripts grounded in the literature for each of the modules. The presentation was then reviewed and edited by Maureeen Novak, the senior author. Final versions of the modules were pilot tested with faculty, and feedback was used to improve the modules prior to the recording of the narration by an education informatics expert using Adobe Captivate 8.

Using the Modules
The four modules can be used as a faculty development opportunity for those involved in implementation of portfolios, either as an instructional designer or as a mentor. The modules (Appendix A) can be downloaded from the MedEdPORTAL website. It takes approximately 10-12 minutes to listen to each module or 45 minutes in total for all four modules. Users can stop the video and return to the content at any time.

Setting
Other than individual faculty using the modules for self-development, we suggest a few ways the modules can be used:

  1. For institutions that have portfolios, faculty acting as portfolio coaches or mentors can be sent the MedEdPORTAL module link via email. On completion of the modules, a facilitated small-group discussion can help faculty explore methods to improve their portfolios. Appendix B provides sample questions to facilitate the small-group discussions.
  2. For institutions planning to develop portfolios, key stakeholders including students, faculty, and administrators can be sent the MedEdPORTAL module link via email. On completion of the modules, a facilitated group discussion can be conducted on the feasibility and process of developing portfolios at the group members’ institutions. Appendix C provides sample questions to facilitate a discussion about institutional needs and feasibility of portfolios.

Instructions to Download Modules
You will need to be connected to the internet to download the modules. Appendix D provides instructions on how to download the modules in Appendix A. Once the files have been downloaded, you can pause the modules as needed and return to them at your convenience.

Results

We studied the effectiveness of the modules through a feedback survey questionnaire distributed to faculty undertaking the modules; the survey used Qualtrics LLC. Appendix E provides the feedback survey questions. Eighteen faculty completed the survey (three assistant professors, eight associate professors, five professors, and two technical support/information technology staff members in the areas of medicine, radiology, surgery, psychiatry, pediatrics, biomedical informatics, family medicine, and pharmacology). All respondents did not answer all questions. The majority of the respondents—13 out of 18 (72%)—were involved mainly in undergraduate medical education, three out of 18 (17%) in graduate medical education, one out of 18 (6%) in continuing medical education, and one out of 18 (6%) stated other.

Respondents were asked about perception of their baseline knowledge of portfolios before undertaking the modules and after undertaking the modules. On a scale of 1-4 (no knowledge, basic knowledge, moderate knowledge, or very knowledgeable), five out of 18 (28%) reported only basic knowledge before undertaking the modules, whereas none reported only basic knowledge after the modules. Two out of 18 (11%) reported that they were very knowledgeable before the modules, and seven out of 17 (41%) reported they were very knowledgeable after the modules (see Figure 1). Overall satisfaction with the modules was reported on a scale of 1-5 (ranging from very poor to very good), with a mean response of 4. Most reported satisfaction was good (11 out of 18, 61%), while two out of 18 (11%) reported satisfaction as fair and five out of 18 (28%) reported very good satisfaction. None rated the modules as very poor or poor.

Figure 1. Level of knowledge before and after module completion.

Faculty strongly agreed (11 out of 18, 61%) or agreed (six out of 18, 33%) with the statement “This topic is an important topic for the professional development of portfolio advisors.” One respondent neither agreed nor disagreed. Additionally, when asked if a module was helpful in their faculty roles, seven out of 18 (39%) strongly agreed, and nine out of 18 (50%) agreed, while one neither agreed nor disagreed and only one disagreed (both 6%). Finally, when asked if the module content was organized logically, 10 out of 18 (56%) strongly agreed, while six out of 18 (33%) agreed. One neither agreed nor disagreed, and one respondent strongly disagreed (see Figure 2).

Figure 2. Feedback on modules.

Narrative feedback was encouraged from those who viewed the modules. Overall, the feedback was positive. Many respondents stated an increased desire to use and discuss portfolios in the educational setting. Participant comments below reflect planned actions after reviewing the modules:

  • “Try to incorporate more narrative assessment by faculty for all portfolio submissions.”
  • “Use a portfolio for both formative and summative feedback.”
  • “Have questions to ask in our monthly portfolio coach meeting for clarification of our program.”
  • “Monitor a portfolio in an ongoing manner.”
  • “Take a more active role in giving the students feedback.”
  • “Promote electronic portfolios at my institution.”

Others noted increased confidence in creating and using portfolios and better understanding of the contents of portfolios, including the following:

  • “Better understanding of formative and summative assessments.”
  • “Better knowledge of the difference between formative and summative data.”
  • “Increased understanding of types of artifacts suitable for portfolio.”

One participant explicitly stated that he/she “plans to distribute the module for all educators in the department and consider making it mandatory.” Another stated that after completing the modules, he/she “viewed medical students’ reflective writings differently.” Other feedback showed that many who viewed the modules felt inspired to undertake further learning about portfolios, including reviewing the papers cited in the modules, finding out how to use portfolios better at their institutions, and discussing the topic with other portfolio mentors. A participant commented that the modules were long and it was difficult to complete watching them in one session.

Discussion

Portfolios have been widely recognized as an effective tool to document individual learners’ development over time, but there is a lack of resources for faculty development purposes in this area. Faculty development has been noted to be key for successful implementation of portfolios.7,13 Faculty must not view portfolios as just another innovation but need to understand their educational value and own the project.14

Faculty development activities are meant to assist faculty in their numerous diverse roles.15 Though faculty development can be powerful, health care provision demands on faculty often leave little time for other pursuits. Given the competing demands on faculty time, there have been calls for new approaches to faculty development.16 Our online modules aim to provide faculty with a resource that can be used by individuals or in a group setting. The modules offer flexibility to faculty with busy schedules and can be reviewed when convenient.

The modules were well received by faculty, who reported an increase in knowledge regarding portfolios and commented about the usefulness of having the online modules available as a resource. We recommend that those responsible for faculty development in the portfolio area use this resource in combination with small-group discussion with their faculty; it can also be used to stimulate discussion among faculty at institutions planning to undertake portfolios or to help with program evaluation of portfolios.

Medical schools will benefit from utilization of this portfolio education series. By learning how to describe a competency-based portfolio and list the typical contents, faculty will better know what to expect in interacting with portfolios, and this will encourage meaningful implementation. By discussing assessment strategies for portfolios and explaining factors for portfolio success, faculty will be encouraged to develop and implement portfolios for educational use. Participating in these online modules will also encourage faculty at an institution to open up a discussion about portfolio development and use at that institution.

Though our team worked cohesively to develop the modules, we recommend that others planning similar modules work with informatics experts at the start of the project to lay out unifying features for the modules. In other words, develop a consensus regarding the interactive features all the modules should contain.

The modules are best completed in one go, which requires 45 minutes. Some faculty may feel this is too long a time. However, they can pause the video and come back to it later. Though we have provided sample questions to stimulate group discussions (Appendices B & C), we do not provide results of such experiences; rather, we present data from individual faculty members’ experiences undertaking the modules. In the future, we hope to provide more results from our small-group discussions. For others developing similar modules with Adobe Captivate 8, we would suggest using the more interactive tools available, including multiple-choice questions with built-in responses.


Author Information

Zareen Zaidi, MD: Associate Professor, Department of Medicine, University of Florida College of Medicine

Ashleigh Wright, MD: Assistant Professor, Department of Medicine, University of Florida College of Medicine

Monica Bailey: Assistant Director for Faculty Development, University of Florida College of Medicine

Toufeeq Ahmed, PhD: Director of Education Informatics and Assistant Professor of Biomedical Informatics, Vanderbilt University School of Medicine

Amy Fleming, MD: Associate Professor, Department of Pediatrics and Associate Dean for Medical Student Affairs, Vanderbilt University School of Medicine

Crystal Wright, MD: Assistant Professor, Department of Anesthesiology, Baylor College of Medicine

Maureen Novak, MD: Professor, Department of Pediatrics and Associate Dean for Medical Education, University of Florida College of Medicine


Acknowledgments
We thank Sarah Krentz, MS, educational consultant, Vanderbilt University School of Medicine, for her assistance in recording the narration using Adobe Captivate 8.

Disclosures
None to report.

Funding/Support
This work was supported by a Medical Education Scholarship, Research, and Evaluation grant distributed by the Southern Group on Educational Affairs and by the Gatorade Trust through funds distributed by the Department of Medicine, University of Florida College of Medicine, Gainesville, FL.

Ethical Approval
This publication contains data obtained from human subjects and received ethical approval.


References

  1. Arter JA, Spandel V. Using portfolios of student work in instruction and assessment. Educ Meas Issues Pract. 1992;11(1):36-44. http://dx.doi.org/10.1111/j.1745-3992.1992.tb00230.x
  2. Driessen EW. Educating the Self-Critical Doctor: Using a Portfolio to Stimulate and Assess Medical Students’ Reflection [doctoral thesis]. Maastricht, the Netherlands: Maastricht University; 2008.
  3. Tochel C, Haig A, Hesketh A, et al. The effectiveness of portfolios for post-graduate assessment and education: BEME Guide No 12. Med Teach. 2009;31(4):299-318. http://dx.doi.org/10.1080/01421590902883056
  4. Van Der Vleuten CP, Schuwirth LW, Driessen EW, Govaerts MJ, Heeneman S. Twelve tips for programmatic assessment. Med Teach. 2015;37(7):641-646. http://dx.doi.org/10.3109/0142159X.2014.973388
  5. Driessen E, Van Tartwijk J, Van Der Vleuten C, Wass V. Portfolios in medical education: why do they meet with mixed success? A systematic review. Med Educ. 2007;41(12):1224-1233. http://dx.doi.org/10.1111/j.1365-2923.2007.02944.x
  6. Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide No. 45. Med Teach. 2009;31(9):790-801. http://dx.doi.org/10.1080/01421590903139201
  7. Chertoff J, Wright A, Novak M, et al. Status of portfolios in undergraduate medical education in the LCME accredited US medical school [published online ahead of print December 10, 2015]. Med Teach. http://dx.doi.org/10.3109/0142159X.2015.1114595
  8. Martin B, Briggs LJ. Affective and Cognitive Domains: Integration for Instruction and Research. Englewood Cliffs, NJ: Educational Technology Publications; 1986.
  9. Kollisch DO, Frasier PY, Slatt L, Storaasli M. Community preceptors’ views of a required third-year family medicine clerkship. Arch Fam Med. 1997;6(1):25-28. http://dx.doi.org/10.1001/archfami.6.1.25
  10. Fields SA, Toffler WL, Bledsoe NM. Impact of the presence of a third-year medical student on gross charges and patient volumes in 22 rural community practices. Acad Med. 1994;69(10)(suppl):S87-S89. http://dx.doi.org/10.1097/00001888-199410000-00052
  11. Milligan C, Littlejohn A. Supporting professional learning in a massive open online course. Int Rev Res Open Distance Learning. 2014;15(5):197-213.
  12. Owen H. Putting the PLE into PLD: virtual professional learning and development. J Educators Online. 2014;11(2).
  13. Driessen E, van Tartwijk J, Vermunt J, van der Vleuten C. Use of portfolios in early undergraduate medical training. Med Teach. 2003;25(1):18-23. http://dx.doi.org/10.1080/0142159021000061378
  14. Driessen EW, Van Tartwijk J, Overeem K, Vermunt JD, Van Der Vleuten CP. Conditions for successful reflective use of portfolios in undergraduate medical education. Med Educ. 2005;39(12):1230-1235. http://dx.doi.org/10.1111/j.1365-2929.2005.02337.x
  15. Steinert Y, Cruess S, Cruess R, Snell L. Faculty development for teaching and evaluating professionalism: from programme design to curriculum change. Med Educ. 2005;39(2):127-136. http://dx.doi.org/10.1111/j.1365-2929.2004.02069.x
  16. Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. 2006;28(6):497-526. http://dx.doi.org/10.1080/01421590600902976


Citation

Zaidi Z, Wright A, Bailey M, et al. Online faculty development for implementation and use of student portfolios. MedEdPORTAL. 2016;12:10436. https://doi.org/10.15766/mep_2374-8265.10436

Received: January 24, 2016

Accepted: July 5, 2016