Introduction: Prior studies have demonstrated poor guideline compliance by pediatricians, and there is no published curriculum on how to teach clinical guidelines. Furthermore, in a national survey of pediatric residency training programs conducted in 2015, only two had a formal curriculum for teaching clinical guidelines. This module provides a framework for teaching residents clinical guidelines through a modified flipped classroom approach. Associated materials include a guide for faculty facilitators, sample slides and worksheet, and pictures of the classroom setup. Methods: In this module, the guidelines for acute otitis media (AOM), obstructive sleep apnea syndrome (OSAS), and attention deficit-hyperactivity disorder (ADHD) are taught in three sessions and evaluated with a pre-/posttest assessing knowledge, attitudes, self-efficacy, and satisfaction. Each guideline is delivered in a 30-minute session, with five learners per group. Faculty training requires approximately 30 minutes of preparation. The intervention groups (n = 9 for OSAS, 10 each for AOM and ADHD) received three weekly, half-hour flipped classroom lessons. The control group (n = 19) had no formal guideline education. Results: Pre-/posttests showed a statistically significant improvement in knowledge and attitudes in the group of interns who received this educational intervention over the control group. The learners rated the sessions as highly effective. Discussion: This module provides an efficient and effective way of utilizing a modified flipped classroom approach to teach learners the correct use of clinical guidelines, a skill residents must master to provide evidence-based care. This curriculum has been successfully incorporated into our pediatric residency program.
- Discuss one key action statement in a clinical practice guideline for acute otitis media (AOM), obstructive sleep apnea syndrome (OSAS), and attention deficit-hyperactivity disorder (ADHD).
- Describe one relevant clinical example for their assigned key action statement.
- Teach back a minimum of one key point of the key action statement to their small group.
- Create a diagnostic or treatment algorithm for the management of AOM, OSAS, and ADHD using relevant clinical practice guidelines.
Studies show that although adherence to and use of clinical practice guidelines vary widely in pediatric practice, these guidelines are generally used less frequently than similar ones in adult medicine. In Flores, Lee, Bauchner, and Kastner’s evaluation of 1,088 pediatricians, practice guidelines were used by only 35%, despite the fact that 89% of respondents agreed that practice guidelines improve outcomes and that a vast majority (94%) desired to improve quality.1 In another study, Forrest and colleagues evaluated the use of a clinical decision support system. Before their intervention, it was found that only 15% of acute otitis media (AOM) and 5% of otitis media with effusion guidelines were followed.2 Rushton, Fant, and Clark described similar results when evaluating the use of the American Academy of Pediatrics (AAP) diagnostic criteria for attention deficit-hyperactivity disorder (ADHD), reporting that only 25.8% of clinicians indicated routine use of all four diagnostic components in the survey.3 Little evidence has been produced and few explanations made as to why pediatricians neither follow nor adhere to clinical practice guidelines. Meanwhile, there is no published curriculum on how to teach clinical guidelines. Furthermore, in a national survey of pediatric residency training programs we conducted in 2015, only two indicated they had a formal curriculum for teaching the guidelines. This is a true gap in resident education, and we hypothesize that if residents are taught how to read and apply the clinical guidelines during their training, they may be more likely to follow them when they go into practice.
While medical educators and physicians struggle to teach and implement clinical practice guidelines, there is a movement to change the way medicine is taught that offers a tangible solution. Medical educators currently advocate moving away from the traditional classroom model of didactics and towards a flipped classroom approach (also called blended learning or an inverted classroom), where the learners engage with the material beforehand (prework) and participate in small active-learning groups during a facilitated session.4-7 The flipped classroom has been shown to be equally as effective, if not more so, in knowledge acquisition across multiple settings (e.g., medical school core courses such as biochemistry and core clerkships like surgery).8-12 Additionally, learners tend to prefer this model of self-directed learning over the traditional classroom.12,13
Residency training, however, poses a challenge to the flipped classroom model due to the intense clinical demands on trainees that limit their ability and willingness to complete the required prework.14 In an effort to ensure pediatric residents are learning essential clinical practice guidelines during their training, we propose a hybrid flipped classroom model that incorporates the prework directly into the small active-learning groups. Because the prework is built into the facilitated session, residents are able to briefly engage with the material on their own and then work in a small group to apply, create, and analyze the clinical practice guidelines to develop their own diagnostic or treatment algorithm for the condition.
Hybrid Flipped Classroom Intervention Development
Guideline selection: We developed a pilot study to evaluate the effectiveness of three 30-minute hybrid flipped classrooms to teach pediatric interns three clinical practice guidelines. The clinical practice guidelines selected for this study were those for AOM, obstructive sleep apnea syndrome (OSAS), and ADHD.15-17 Through expert consensus, we distilled five key learning points for each guideline from the guidelines’ key action statements (Table 1). This was done by having three of the authors independently review the three guidelines and select the most pertinent key action statements to teach in each session. Mediation by the senior author was used to address discrepancies. These learning points were then made the focus of the modified flipped classroom sessions.
|Learning Point||Key Action Statements|
|Obstructive Sleep Apnea Syndrome||Acute Otitis Media||Attention Deficit-Hyperactivity Disorder|
|2||2A + 2B||1B + 1C||2|
|3||3 + 4||3A + 3B + 3C + 3D||3|
|4||5A + 5B||4A + 4B + 4C||5A + 5B + 5C|
|5||6 + 7||5A + 5B||6|
Intervention design: The intervention groups (n = 9 for OSAS, 10 each for AOM and ADHD) received weekly, half-hour flipped classroom lessons for the three guidelines, while the control group (n = 19) had no formal education about guidelines. The intervention consisted of a 30-minute structured small-group activity led by a facilitator. The facilitator led the session with a simple PowerPoint (Appendix A for OSAS, Appendix B for AOM, or Appendix C for ADHD) to advance the groups through each portion of the intervention.
Intervention Implementation Guide
Classroom setup: Prior to the session, the room is arranged with tables and chairs forming groups of five, with a printed copy of the clinical practice guidelines of that session’s selected topic (OSAS, AOM, or ADHD) at each seat, as demonstrated in Appendix D.15-17 Each printed copy of the clinical practice guideline has a post-it note with an assigned learning point derived from the key action statements (e.g., Learning Point 1 for Group Member 1, Learning Point 2 for Group Member 2, etc.). As stated previously, the five assigned learning points were identified earlier through expert consensus and reflect the content on the pre/post assessment test. Each small group receives one guideline summary sheet (Appendix E), a piece of chart paper, and markers.
Introduction and objectives: All three sessions begin with the facilitator reviewing the learning objectives and introducing the clinical practice guideline for the session using the presentation found in Appendix A, B, or C. Facilitators should conduct the presentation only through slide seven for Appendix A, slide six for Appendix B, and slide seven for Appendix C. The remaining slides for each presentation are used later in the session. Typically, the facilitator should set aside 2 minutes for this session. Please note that during the students’ initial session, the pretest (Appendix F) is administered first to assess student presession knowledge of clinical guidelines for OSAS, AOM, and ADHD. The facilitator should allow an additional 5-10 minutes to administer this pretest during the first session.
Prework: Residents are instructed to quietly and individually read their assigned learning point. They are prompted to identify the main point they think is most relevant for all pediatric residents to know and develop a relevant clinical example. This is the prework portion of the flipped classroom model that we adapted to occur during our hybrid sessions. The facilitator allots 3 minutes for this portion.
Teach back: The facilitator assigns 12 minutes for this portion. Residents are asked to teach back the main point of their assigned learning point to their group in approximately 2 minutes (in sequential order of learning guideline). A scribe is assigned to record the main points for the group on the guideline summary sheet (Appendix E). The residents are encouraged to highlight key tables, figures, or charts while explaining their relevance. This generates small-group discussion as the remaining group members can ask clarifying questions and compare or contrast their prior knowledge to the action statement. The process is repeated roughly every 2 minutes until each group member has taught his or her assigned learning point to the small group.
Create diagnostic flowchart/algorithm: During the remaining 10 minutes of the session, each group compiles the content gathered on the guideline summary sheet (Appendix E) into a diagnostic and treatment algorithm or flowchart, a sample of which can be found in Appendix G. Additional examples are found in Appendices A-C, and these may be revisited if needed. The facilitators can circulate the room but should refrain from interjecting unless a group is off task and needs to be redirected. The purpose of this time-constrained portion is to challenge group members to apply the knowledge they have just learned to create a tool they can reference in the future. If the group needs prompting, possible questions can include “How will you use this information next time you are in clinic?”, “What component was most challenging or interesting?”, and “What is the first step to diagnosing this condition?” The end products may vary in their design and structure (e.g., a flowchart vs. a table vs. a diagram), but the content should reflect the learning points.
Check for understanding—apply to case: The facilitator asks the large group a series of clinical questions that each small group must answer using its diagnostic flowchart/algorithm. These questions are provided in the facilitator PowerPoints and begin on slide eight of Appendix A, slide seven of Appendix B, and slide eight of Appendix C. This allows the facilitator to reinforce the main learning points and find any major gaps in the flowcharts/algorithms before the end of the session. Any misinformation can be quickly corrected and retaught to ensure the residents do not leave with incorrect information. Three minutes are generally allotted for this component. If the students are on their final session, they are given the posttest evaluation (Appendix F). Students should be given an additional 5-10 minutes to complete the posttest.
A deidentified 32-question test (Appendix F) was developed to assess pediatric interns’ knowledge, attitudes, and general use of clinical practice guidelines both prior to and after they had experienced the three sessions. For the purpose of the pilot study, the test was administered to 29 pediatric interns (PGY1) during Week 1 (preintervention) and repeated on Week 4 (postintervention). Ten interns received the intervention, and 19 received the control (no-guidelines curriculum). A single investigator graded all tests using a scoring rubric. Evaluation of the quality of the facilitation, content, and structure of the three flipped classroom sessions was obtained at Week 4 (postintervention) using Likert scales and open-ended feedback for improvement (Appendix H). The study was Stanford IRB exempt.
All of our pediatric interns (N = 29, 22 female, seven male) were included in the study. Descriptive statistics and unpaired, two-tailed t tests (p < .05) were used for analyses. Before the intervention, three out of 29 interns indicated on the pretest (Appendix F) that they had received teaching on OSAS, 13 out of 29 on AOM, and eight out of 29 on ADHD, respectively. Only 20 of 29 interns knew there were any clinical practice guidelines published by the AAP, and these 20 were able to name on average only three of the 10 guidelines. On a 5-point Likert scale (1 = very important, 5 = not important), 28 interns rated the importance of a pediatrician knowing and using the guidelines as very important; the other intern rated these as somewhat important.
The interns had an average knowledge score of 44% on the pretest. The score breakdown by guideline was 45% for OSAS, 11% for AOM, and 79% for ADHD. A convenience sample was used to select the intervention group based on the interns who were at the main training site (Lucile Packard Children’s Hospital). The number of interns in each intervention group was nine for OSAS, 10 for AOM, and 10 for ADHD.
When the posttest was adminsistered, the intervention interns had a statistically significant increase in knowledge over the control interns: 21% versus 10% for OSAS (p = .03), 16% versus 4% for AOM (p = .009), and 21% versus −5% for ADHD (p = .005; Table 2). There was a statistically significant increase in self-efficacy posttest versus pretest but no difference between intervention and control groups.
|Disorder||Control Groupa||Intervention Group||p Value|
|Obstructive sleep apnea syndromeb||9.92||20.96||0.03|
|Acute otitis mediac||4.11||16.13||0.009|
|Attention deficit-hyperactivity disorderd||−5.26||21.05||0.005|
All participants filled out anonymous evaluations (Appendix H) after the session to assess facilitation, handouts, comfort using the guideline, and overall session. A 5-point Likert scale (1 = poor, 5 = excellent) was used. Overall, the sessions were rated 4.4 for OSAS, 4.4 for AOM, and 4.2 for ADHD. The sessions received unanimously high marks for all categories (facilitation, handouts, and comfort; Table 3).
|Obstructive sleep apnea syndrome||4.5||4.375||3.875||4.375|
|Acute otitis media||4.375||4.5||3.75||4.375|
|Attention deficit-hyperactivity disorder||4.18||4||3.45||4.18|
We developed and implemented a modified flipped classroom approach to teach clinical guidelines, which successfully improved intern knowledge and attitudes when compared with standard education. These sessions were well received. In fact, our PGY2 and PGY3 residents insisted the sessions be taught to all levels of learners. Furthermore, it was noticed that the interns were incorporating the guideline knowledge into their actual patient care in clinic and on the wards and at times knew the guidelines better than the PGY2 and PGY3 residents. This created further interest among the PGY2 and PGY3 residents, which led to us offering this module to all levels in later months. The curriculum was expanded to include all 10 published AAP guidelines, and it is now a core component of our primary care conference series. In addition, Drs. Peterson, Louden, and Gribben presented this work at both the Association of Pediatric Program Directors Chief Resident Forum and the Academic Pediatric Association Education Committee’s national meeting in 2016.
Although this curriculum was created to teach clinical guidelines, the teaching methods can also be adapted to teach any topic to medical students, residents, and fellows. This work successfully showed that a modified flipped classroom approach can be used for busy clinical learners who might not have sufficient time to prepare ahead of class. The interactive nature of the sessions allowed learners to solidify their baseline knowledge and apply it to real clinical scenarios. In this way, the key concepts of the clinical guideline were reinforced at least four times throughout the session.
A few revisions have been made to the curriculum since inception. These include the creation of advanced and modified sessions with slight variations. For example, in the hyperbilirubinemia guidelines, each group works on one action statement and reports out (a jigsaw technique).18 For AOM, skills stations have been added as a form of case-based application, and as the guidelines are discussed, residents match the category of tympanic membrane disease with pictures of tympanic membranes. Regarding challenges in implementing the curriculum, although there was initially student discomfort with learning via a new model and some uncertainty as to the appropriate level of participation, the students quickly learned how to participate. In addition, given the short duration of our morning report (30 minutes), we found that the facilitator needs to stick closely to the time line so as not to get off schedule.
Despite the curriculum’s rigorous design and implementation, a few limitations do exist. A convenience sample of interns at a particular site was studied, and although the curriculum was required of all interns at that site, residents could self-select into receiving or not receiving it. Moreover, this was a small pilot study at a single institution. Follow-up studies should investigate the sustainability of the increased knowledge and attitudes. Our educational intervention provides a robust framework for teaching how to use clinical guidelines and demonstrates an interactive teaching method (the modified flipped classroom) that is a more feasible construct than the true flipped classroom for busy clinical learners.
None to report.
None to report.
Peterson J, Louden DT, Gribben V, Blankenburg R. A “flipped classroom” model: teaching interns clinical guidelines. Poster presented at: Pediatric Academic Societies Annual Meeting; April 30-May 3, 2016; Baltimore, MD.
Peterson J, Louden DT, Gribben V, Blankenburg R. A “flipped classroom” model: teaching interns clinical guidelines. Poster presented at: Lucile Packard Children’s Hospital Stanford Research Day; April 2016; Palo Alto, CA.
Peterson J, Louden DT, Gribben V, Blankenburg R. A “flipped classroom” model: teaching interns clinical guidelines. Poster presented at: Association of Pediatric Program Directors Annual Spring Meeting; March 30-April 2, 2016; New Orleans, LA.
Peterson J, Louden DT, Gribben V, Blankenburg R. Teaching residents clinical practice guidelines using a flipped classroom. Presented at: Pediatric Grand Rounds at the Lucile Packard Children’s Hospital; June 2016; Palo Alto, CA.
Peterson J, Louden DT, Gribben V, Blankenburg R. Teaching residents clinical practice guidelines using a flipped classroom. Presented at: Pediatric Grand Rounds at the Santa Clara Valley Medical Center; June 2016; San Jose, CA.
Peterson J, Louden DT, Gribben V, Blankenburg R. A “flipped classroom” model: teaching interns clinical guidelines. Presented at: APA Education Committee Meeting at the Pediatric Academic Societies Annual Meeting; April 30-May 3, 2016; Baltimore, MD.
Peterson J, Louden DT, Gribben V, Blankenburg R. A “flipped classroom” model: teaching interns clinical guidelines. Presented at: Chief Resident Forum at the Association of Pediatric Program Directors Annual Spring Meeting; March 30-April 2, 2016; New Orleans, LA.
Reported as not applicable.
- Flores G, Lee M, Bauchner H, Kastner B. Pediatricians’ attitudes, beliefs, and practices regarding clinical practice guidelines: a national survey. Pediatrics. 2000;105(3):496-501.
- Forrest CB, Fiks AG, Bailey LC, et al. Improving adherence to otitis media guidelines with clinical decision support and physician feedback. Pediatrics. 2013;131(4):e1071-e1081. https://doi.org/10.1542/peds.2012-1988
- Rushton JL, Fant KE, Clark SJ. Use of practice guidelines in the primary care of children with attention-deficit/hyperactivity disorder. Pediatrics. 2004;114(1):e23-e28. https://doi.org/10.1542/peds.114.1.e23
- Prober CG, Khan S. Medical education reimagined: a call to action. Acad Med. 2013;88(10):1407-1410. https://doi.org/10.1097/ACM.0b013e3182a368bd
- Vogel L. Educators propose “flipping” medical training. CMAJ. 2012;184(12):E625-E626. https://doi.org/10.1503/cmaj.109-4212
- Tolks D, Schäfer C, Raupach T, et al. An introduction to the inverted/flipped classroom model in education and advanced training in medicine and in the healthcare professions. GMS J Med Educ. 2016;33(3):Doc46. https://doi.org/10.3205/zma001045
- Bergmann J, Sams A. Flip Your Classroom: Reach Every Student in Every Class Every Day. Arlington, VA: International Society for Technology in Education; 2012.
- Prober CG, Heath C. Lecture halls without lectures—a proposal for medical education. N Engl J Med. 2012;366(18):1657-1659. https://doi.org/10.1056/NEJMp1202451
- Morton DA, Colbert-Getz JM. Measuring the impact of the flipped anatomy classroom: the importance of categorizing an assessment by Bloom’s taxonomy [published online ahead of print July 18, 2016]. Anat Sci Educ. https://doi.org/10.1002/ase.1635
- Liebert CA, Mazer L, Merrell SB, Lin DT, Lau JN. Student perceptions of a simulation-based flipped classroom for the surgery clerkship: a mixed-methods study. Surgery. 2016;160(3):591-598. https://doi.org/10.1016/j.surg.2016.03.034
- U.S. Department of Education, Office of Planning, Evaluation, and Policy Development. Evaluation of Evidence-Based Practices in Online Learning: A Meta-Analysis and Review of Online Learning Studies. Washington, DC: U.S. Department of Education; 2010.
- McLaughlin JE, Roth MT, Glatt DM, et al. The flipped classroom: a course redesign to foster learning and engagement in a health professions school. Acad Med. 2014;89(2):236-243. https://doi.org/10.1097/ACM.0000000000000086
- Tan E, Brainard A, Larkin GL. Acceptability of the flipped classroom approach for in-house teaching in emergency medicine. Emerg Med Australas. 2015;27(5):453-459. https://doi.org/10.1111/1742-6723.12454
- Tainter CR, Wong NL, Cudemus-Deseda GA, Bittner EA. The “flipped classroom” model for teaching in the intensive care unit: rationale, practical considerations, and an example of successful implementation. J Intensive Care Med. 2017;32(3):187-196. https://doi.org/10.1177/0885066616632156
- Lieberthal AS, Carroll AE, Chonmaitree T, et al. The diagnosis and management of acute otitis media. Pediatrics. 2013;131(3):e964-e999. https://doi.org/10.1542/peds.2012-3488
- Marcus CL, Brooks LJ, Draper KA, et al. Diagnosis and management of childhood obstructive sleep apnea syndrome. Pediatrics. 2012;130(3):576-584. https://doi.org/10.1542/peds.2012-1672
- Wolraich ML. The new attention deficit hyperactivity disorder clinical practice guidelines published by the American Academy of Pediatrics. J Dev Behav Pediatr. 2012;33(1):76-77. https://doi.org/10.1097/DBP.0b013e318241eadf
- Aronson E, Blaney N, Stephin C, Sikes J, Snapp M. The Jigsaw Classroom. Beverly Hills, CA: Sage Publishing; 1978.
This is an open-access publication distributed under the terms of the Creative Commons Attribution-NonCommercial-Share Alike license.
Received: October 5, 2016
Accepted: January 20, 2017