Effectiveness of flipped classroom techniques in an advanced laboratory physics course

We report preliminary observations of changes in responses to student surveys over a five year period in an advanced laboratory course for third-year physics majors at the Massachusetts Institute of Technology. This period spanned the introduction of curriculum reforms which included the use flipped classroom techniques — facilitated by the OpenEdX platform — for those aspects of the course material which had previously been taught by direct instruction, such as data analysis techniques and basic laboratory instrumentation. Not all variables of the classroom environment were controlled during the study period, so flipped classroom techniques cannot be identified with full confidence as the cause of the measured changes. Survey data was collected using the E-CLASS and institutionally administered subject evaluations. Improvements were observed in some metrics of interest to the course’s strategic goals — notably in students’ self-reported hours per week spent on coursework and in the overall rating of the course — while negative or null results were observed in other metrics of interest.


I. INTRODUCTION
Since 2010, the MIT Department of Physics has been engaged in a curriculum reform project for its upper division advanced laboratory course in modern physics, the so-called "Junior Lab". These efforts have benefited from coinciding with a period of renewed national interest in laboratory teaching by both the physics education and education research communities [1-4, for example]. The principle intervention in Junior Lab has been the rearrangement of course material in the Fall 2012 and 2013 semesters to allow for the introduction of flipped classroom teaching techniques for those portions of the course where such techniques makes sense. While not fully controlled for all variables, this situation constituted a natural experiment on the effectiveness of flipped classroom methods in an advanced laboratory environment. Statistics in the form of student survey responses (institutional end-ofsemester surveys, as well as the E-CLASS [5][6][7] for part of the study period) were kept throughout the study period, allowing for comparison before and after the intervention.
Somewhat novelly, we used the OpenEdX [8,9] platform -designed for massive open online courses (MOOCs) -to flip content in a laboratory class. This proposal may not immediately make sense, since the course under study is neither "massive", nor "open", and the very nature of laboratory work precludes a fully "online" offering. Nevertheless, it was found adequate to the task.

A. Description of Junior Lab
Junior Lab is a two semester laboratory sequence taken primarily by third year physics majors. It is a stand-alone course (that is, not supplementary to a corresponding lecture course) * spatrick@mit.edu in experimental physics, with a majority of the available experiments being drawn from the canon of modern physics. As shown in TABLE I, the experiments span special relativity, experimental foundations of quantum mechanics, atomic structure and optics, statistical mechanics, nuclear and particle physics, and other more contemporary topics. Students, who work in pairs with coaching from instructional staff (including faculty), spend between 4 and 6 three-hour lab sessions on each experiment. Each experiment is preceded by a short set of questions on the required preparatory reading and is followed by a report on the performance, analysis, and results of the experiment. The report consists of a four-page Physical Review style written summary and a 15-minute conferencestyle presentation to the students' faculty instructor.
While nominally a course in modern experimental physics, the primary focus of Junior Lab is on the professional and personal development of the student as a scientist. This includes laboratory technique and topics in modern physics, but extends beyond that to skills such as oral and written communication methods, the troubleshooting process, professional scientific attitude, data analysis, and reasoning about uncertainty. Unlike similar courses at many US institutions, Junior Lab does not necessarily serve as a gateway to undergraduate research, as the majority of MIT physics majors will have already been engaged in faculty research projects well before the third year via the MIT UROP program [10]. Junior Lab is intended to provide a structured environment for learning authentic but generalized modes of professional practice, in complement to the less structured setting and more specialized modes encountered in research experiences.
Due to the idiosyncrasies of the physics major requirements at MIT, the first semester of Junior Lab (subject number 8.13) is required for most physics majors, while the second semester (subject number 8.14) is not required for most physics majors.    One goal of the reform efforts in 8.13 is to increase enrollment in 8.14, which follows it in the curriculum but has been optional since the introduction in 2001 of a "flexible" physics track -contrasted with a traditional "focused" physics track. The data above show that while the physics major overall is quite healthy in terms of enrollments, the recent changes in the 8.13 curriculum have had negligible effect on 8.14 enrollments, at best stabilizing a previously falling trend. Trend lines on the plot are cubic smoothing splines. has dropped precipitously, while the enrollment in 8.13 has remained high. Note, all students on the flexible track are still required to take an advanced laboratory course, but some students are allowed to fulfill the requirement with a sufficiently rigorous offering from another academic department in lieu of 8.13. In contrast, the traditional or "focused" physics major track requires both 8.13 and 8.14.

B. Project Motivation and Goals
There are two main goals of the curriculum reform project. The first goal is institutionally driven: to increase enrollment in subject 8.14. The second goal is more broadly educational: to improve the Junior Lab learning environment and learning outcomes. In regards to the first goal, a belief widely expressed (anecdotally) in the student community before the start of the study period was that the most common reason for choosing not to enroll in 8.14 was the intensity of the student experience in 8.13. While the learning in 8.13 was often highly valued by students (again, anecdotally expressed), the 8.13 experience itself would sometimes be described as demoralizing, unproductively time consuming, and emotionally or academically overwhelming. This suggested that one possible approach to increasing 8.14 enrollment would be to improve the quality of the 8.13 experience.
Correspondingly, Junior Lab instructional staff often noted inefficiencies of delivery in certain parts of the 8.13 course material which seemingly lead to levels of student frustration and time-on-task far beyond that which could be reasonably suspected as contributing to positive learning gains. This again motivated focusing on improvements to 8.13.
Specifically, it was commonly regarded by the instructors, based on extensive but anecdotal observations of students, that the majority of unproductive time-on-task and frustration was due to a lack of preparation in a few key technical areas -such as the mechanics of preparing a professionalstyle manuscript or interacting with data analysis softwarewhich could be cleared up with a relatively small amount of instruction. Written and sometimes oral instruction (short lectures) was already provided in all such content areas, but no structured practice was present in the curriculum. Further, the students' incoming tacit knowledge of these areas was apparently insufficient to enable knowledge transfer from the provided instruction to hands-on applications. This suggested that a set of structured exercises in key topic areas occurring early in the semester could have a large effect on outcomes.
However, another common observation of the instructors was that some fraction of student time was wasted in the lab itself due to students arriving unprepared to do that day's work, despite the existence of required exercises which quizzed the students on their understanding of the relevant reference materials. Seemingly, students could do the assigned preparatory work without realizing their own lack of understanding of that work. A hypothesis was generated that if students come to lab primed and prepared to learn, they will make more effective use of lab time and contact hours with expert teachers, leading to less total hours spent on coursework, better learning outcomes, more expert attitudes, and higher satisfaction with and enrollment in the experimental physics track.
All of this points towards using flipped classroom techniques in which students are presented with course materials before class, receive instant feedback on their understanding of that material through automatically graded questions, come to class ready to engage the expert instructor with meaningful discussion of their own misunderstandings, participate in active learning activities guided by the instructor's intermittent coaching, and finally follow up those activities with reflective homework projects that cement learning and promote expert attitudes about experimental physics.
This discussion also suggests some utility would be had not only in measuring the direct project outcomes, such as enrollment numbers, but also some metrics relevant to the hypothesis that students are not enrolling in 8.14 due to their dislike of 8.13, despite adequate learning gains in 8.13. We attempt to get at such questions using the E-CLASS survey and specialized questions on institutional end-of-semester surveys which go beyond an overall satisfaction rating.

II. METHODS
"Flipped classroom" is a modern jargon term for the old fashioned notion of requiring students to "do the reading" before coming to class. The intent of the method is to optimize the limited classroom hours available for the high level of learning associated with student-teacher dialog and face-toface engagement with a subject matter expert. Often, application of the jargon implies the use of online learning technologies to deliver course content as video lectures or written materials, possibly with associated autograded problems to test understanding. The classroom activities which follow typically employ similarly modern active learning strategies.
In the present study, we used MIT's instance of the OpenEdX MOOC platform to enable flipped classroom techniques, but we "flip the classroom, not the lab". That is, we change the delivery method for those content areas where we suspect lectures work poorly or where lab time could be better utilized, but leave the majority of the coursework unchanged. Hands-on laboratory work and professional-style communication are maintained as the core of the learning experience. (Note, despite using a MOOC platform, enrollment was restricted to students enrolled in the residential course.) There are two ways in which flipped learning was used: • introductory lessons on fundamentals (data analysis, equipment basics, and communication technique) during the first month of the semester, and • online lab manuals and preparatory questions to be completed before each experiment throughout the semester.
Subject 8.13 was offered every fall semester during the study period (Fall 2010 -Spring 2015), as well as Spring 2012 -2015. Spring enrollments were roughly 10 students per semester, while fall enrollments were closer to 60 students per semester. In the semesters prior to Fall 2012, 8.13 used no online content delivery. Direct instruction in laboratory fundamentals included only short lectures on data analysis during the first few lab periods. The introductory period, including the above lectures and three short preliminary experiments, lasted three weeks and was followed by four longer experiments. Each of the experiments, both preliminary and long, included a lab manual (a PDF document) with preparatory questions which were to be completed in written form and handed in for grading by graduate student teaching assistants.
In Fall 2012 and Spring 2013, the introductory course materials were expanded to include more in-class exercises and homework over four weeks, followed by three longer experiments. Then, for all semesters Fall 2013-Spring 2015, the introductory exercises were delivered in a flipped format of assignment sequences including introductory videos, autograded online questions, in-class active exercises, and follow up homework projects. In addition, each lab manual was divided into a set of OpenEdX web pages and all preparatory questions were transformed into autograded problems. Subject 8.14 was offered every spring semester during the study period. There is no introductory material in 8.14. Students perform four longer experiments, one of which is an exploratory project of their own design. The three standard experiments include preparatory questions as in 8.13. In Spring 2015, the 8.14 lab manuals and questions were moved into the online format.
Effects of the reforms were measured by several metrics: • Enrollment in 8.14, measured each semester as the number of registered students at the "add day" deadline.
• Standard end-of-semester subject and instructor evaluations administered anonymously online to students by the institution. Response rates consistently ranged between 60% and 70%. In this study, we only examine five of the subject evaluation questions for subject 8.13: overall subject rating; the pace of the subject; the workload of the subject; hours per week spent in the lab; and hours per week spent on coursework outside of the lab.
• A set of nine extra questions added to and administered with the standard end-of-semester survey for 8.13 which ask about student intentions for enrolling in 8.14 and the degree to which their experiences in 8.13, anticipations of 8.14, and future career plans effect their decision to enroll in 8.14. Due to a clerical error, this set of questions was not administered in the Fall 2013 semester.
• The E-CLASS survey. This was administered in each semester of 8.13 starting in Fall 2012, although results from Fall 2014 are still pending analysis.

III. RESULTS
As of this writing, only preliminary analysis of the metrics listed in the Methods section above is available. A more complete analysis will appear in a future report.  Further improvement follows the fully flipped curriculum implementation in 2013.
The E-CLASS survey provides dozens of dimensions of data about changes in how students think about experimental physics from the start to the end of the semester. As a first pass for this preliminary report, we examine the coarsest aggregate of these data: the "fraction of statements with expert-like responses". Unfortunately, even this coarse metric is not yet available for all semesters in the the study, and all but one those semesters for which it is available suffer from low statistics. , but all of these changes are at or below the 95% confidence limit. Therefore, very little can be concluded from these coarse measures at this time. The finer grained responses to all 31 questions in the E-CLASS show more encouraging gains than the aggregate score, but detailed analysis is left for a future report.

IV. CONCLUSION
Notable improvements were observed in the studentreported overall class rating and hours spent per week from before to after the 8.13 curriculum reforms in 2012 and 2013, however no discernible change is observed in 8.14 enrollment trends. A superficial examination of E-CLASS results is similarly inconclusive. However, all of these data sets contain many more dimensions which have not yet been examined. Further analysis of the available data is warranted.
While the observed improvements in 8.13 evaluation are encouraging, they cannot be definitively attributed to the introduction of flipped classroom techniques alone. Furthermore, the overall strategy of trying to increase 8.14 enrollment by improving the student experience in 8.13 does not appear to be working. Other approaches will have to be tried. Current efforts are focused on expanding the communication instruction in 8.14 to include a conference-style poster session which could be used to fulfill a "communication intensive" graduation requirement for students.

ACKNOWLEDGMENTS
This work was supported by the d'Arbeloff Fund for Excellence in Education, the MIT Office of Digital Learning, and the MIT Department of Physics. We are grateful for helpful conversations with Andrew Birkel and Margaret Pavlovich, as well as everyone involved in the teaching of Junior Lab over the past five years.