Faculty Resource Network

An academic partnership devoted to faculty development. Now in our fourth decade, we remain committed to this partnership, and to fostering connection, collaboration, and collegiality among our members.

Excerpt from the Symposium Session “Moving from Scientific Research to Education Research: What Are the Questions? How Do We Answer Them? What Have We Learned?”


Defining and Promoting Student Success
A National Symposium
November 21-22, 2008
University of San Francisco
San Francisco, California


Diane Ebert-May, Michigan State University

So basically one of the things that we’re addressing is, we’re doing all these things to improve STEM education with our students in universities and colleges and what I’ve been arguing for a number of years is the evidence of the impact of our change is thin. We would not do science and we would not make decisions in science with the kind of evidence we have in science education. So what I’m going to do this morning is get you involved in asking the questions we want answered about this reform and take you through that in a stepwise sort of way so that when we leave today, we’ll end up with the researchable questions you want to ask about your students, and I know many of you are administrators so you’re likely not in the classroom, so when we do these activities, think about it from your faculty’s perspective. You’ll see how that plays out in a couple seconds. So what we want to find out is, what are the questions we need to answer about our students’ learning and our teaching, and how are we going to address these questions?

Part of science that’s so challenging is, we have a cool question – what’s our method going to be? And that’s part of what motivates us to do this. What have we learned about this, and how do we start to manage the assessment data? If all our assessment data on student learning is sitting on the laptop computers of our faculty, how are we, collectively as institutions accessing those data to make long- term decisions and short-term decisions? So it’s a very interesting question about data when it comes to student data, very different from the human genome for example, so that’s where we’re going to go. When I was thinking about this meeting I thought, why don’t we do a little history about why we’re starting to think very seriously about research and assessment in undergraduate education. So, in the sixties is when we began to have these research and development centers in our colleges and universities. A lot of stuff had been happening in K-12, but we hadn’t really seen it percolating up into the higher education level. And then in the seventies is when faculty development centers started to appear on our campuses. So how many of you have those at your institutions? (pause) So yeah, we have faculty development centers. They’re there, and we were kind of building these infrastructures. How many of your scientists use those centers? (pause) Okay, so some of you. But then in the eighties, when A Nation at Risk was published, the A- word became apparent – assessment. A Nation at Risk said, “Show us the data about your students.” And that was a lot about K-12 but it was a lot of higher education too.

And the question is, “what kind of data do we have about student learning?” Not just self-report opinions about that, but what did they really learn? How are we accountable? And then in the nineties we began seeing reports like Boyer saying, okay we have the scholarship of discovery for scientists, and we get that. We create the knowledge. But there is a scholarship of teaching. This scholarship can count for promotion and tenure. This scholarship can be peer reviewed and it can be funded. And this is news to most scientists because we’re trained as scientists. So we started to get permission to do this and we started to see other NRC reports like How People Learn and the Carnegie institute doing all their things, and the AAAS had many, many national reports and policies to suggest that there’s some scholarship involved in teaching that we can push a little bit on and probably push a lot on. And then finally in the 2000s we had some very fine reports about how students learn, knowing what students know. So we’ve been influenced, at least on the national level, by some pretty seminal policy studies that I would give any scientist in my department, and I’m in the plant biology department, and if you know anything about Michigan State, we do plants, we’re big on plants, but I could give my colleagues any of those reports and they would say, “Oh that’s interesting.” They get those. So we’re starting to think about research. Okay, and these are some of the reports – 2010, you’ve all seen that, the integration of chemistry, physics, biology, mathematics, and I know many of you aren’t scientists because you’ve all come through different avenues, but we’re really talking about STEM education here. And this one’s very interesting, Advancing Scientific Research in Education, and what does that kind of research look like? So we have this big national push let’s say, with people talking about it at the high level. How does that translate into your own individual faculty?

So here’s what I want you to do – you have this very blank-looking pieces of paper in front of you, and I’m going to collect these, but you don’t have to sign them, they’re not going to be graded. You will need an implement to write with, but I want you to write in these boxes for a second, and if you’re completely away from the classroom, speak on behalf of your faculty members. And the question I have is, “Are you or would you be willing to conduct research on student learning?” And some of you are already doing this, I know, but why or why not? Fill that in and then I’ll give you a minute to talk with your neighbors……. SO just take a minute to talk to your neighbors about why you would do this.

…I teach very large classes at Michigan State, and this is a very typical class for me, I teach a class of 250, introductory Biology and it’s a very interactive class, and while you were all talking, I went around with my iPhone and calculated all your results. I’m just kidding, these were from other faculty from the Summer Institute program which Howard Hughes funded that I do at the University of Wisconsin with Joe Handlesman, but these are from other faculty, I ask faculty around the country, so at least you get an idea of what your peers are thinking. So this one, yeah, I’ve considered doing this, because the only feedback I have is student evaluations, and they’re very subjective. So faculty are thinking along that way, and these are scientists again, that’s the population I’m sampling. Another one, yeah I never considered it because I’m a plant biologist, so why should I study teaching? Wasn’t on the radar, and no I wouldn’t, don’t they have a center for that sort of thing? This is the classic I’m the biologist teaching biology, but someone else ought to study that.

In order to think about this kind of research, we need to think about three things – we need to think about the teacher, the students, and the environment. So let’s look at the variables we’re dealing with when we look at this kind of research. We know this – that individuals learn differently, we know our students communicate in multiple ways, we know that students perform skills in different ways, and we know that they conceptualize ideas in different ways – we know a lot about learning. We also know that diversity affects learning. So that’s what we know about students.

What do we know about teachers? What do we know about us, people in the classroom? How do teachers teach? Well, this is how we teach, and those of you who have seen these cartoons before, if you were with us in Atlanta, we have this notion in the professor-dom that we have this blank slate, that these students come to us at 18 years old with no knowledge, they’ve merely been existing for the last 18 years and the job of the professor is to fill the head of this empty vessel with knowledge. But we know this doesn’t work because there’s leakage. Teaching and telling doesn’t work. Students have to do something with this knowledge, they have to construct their knowledge, so we build these active learning classrooms with students teaching students, interacting actively, inquiring if you’re doing science, whatever you’re doing there’s a job for the teachers, there’s a job for the students and there’s a lot of synergistic stuff going on in an interactive classroom, and I’m preaching to the choir here, but we have to know what the variables we’re working with are, so that will guide our questions.

We’ve got so far the teachers and the students. Let’s talk about the environments we’re working with here. I’m an ecologist so I’m very big on environments, and these are very complex systems – we’re leading up to the complex system. And this could be a very general learning environment. This an AP Biology high school class – these kids are going to be Freshman at your schools next year, they’re very excited about Biology. But I throw that in just to show you, you could walk into that environment and assume there’s not a whole lot going on there, at least at that point in time. Whereas you could walk into my 500 person classroom on day 2 in veterinary medicine because that’s the only place that will hold that many students and there are a bunch of kids on the floor playing with termites and you think, Oh my gosh will these kids learn anything? How am I going to get into med school, she’s messing with my brain. But in fact we have pretty good evidence that it works and so you’ll see now big undergraduate classrooms, and the reason I use these big classes as an exemplar is that it’s really easy to scale down. Our issue in big state universities, where most of our kids in the country go to school – giant schools with 45 thousand, we put these kids into big classes. So if we can figure out what to do in big classes, and if you have fifty, wonderful. Then it’s easy.

And we’re beginning to see scale up classrooms, have any of you seen these scale up classrooms at MIT, and North Carolina State, University of Minnesota. This is a classroom with 122 students in plant biology, and what you’ll notice which is striking here – this is a table of nine students, they have a microphone, each table has one of these flat screen computer displays, and no lecturing goes on. This is Robin Wright and Rob Brooker who do this kind of stuff, and they’re doing this for a class of 244, and this is for bio-majors, and they’re called scale up classrooms if you Google it, and it’s about the design. We spend millions of bucks making bad classrooms. And we just did it at Michigan State, I lost the battle. There’s a beautiful design for this and we’re seeing these pop up all over, there’s one in Florida, I’m not sure, I know University of Minnesota, NC State for physics, and MIT, there’s one in Florida that’s escaping me right now, what is it? Miami, Miami’s got one. People are looking at this and saying, this is cheap relative to what we do. This is a cheap investment.

The key is to ask the questions about the learning. Why is educational research so challenging? I am a tundra ecologist, which means that I go to a mountain and I look at plants, and I’ve been looking at the same plants for the last 35 years. They don’t move, they’re there every year, they don’t come from dysfunctional families, they’re perennials, they have no baggage, they’re happy to see me every spring, not really but I like to think they are. That research is really difficult, but why is educational research so difficult and challenging for those of us who haven’t been trained in that. Some of us have but most of us haven’t. Well there’s an old story in one of my favorite NRC reports that just came out, they told the story about six blind men and they wanted to touch an elephant. And they touched this elephant and they told an artist, “This is what I’m feeling.” So the artist represented the structure of an elephant. And as you can see this is what the artist came up with in this tale and you can see that the elephant is a very complex system and the structure is, probably this elephant is not going to function very well, unless each of these structures interacts and communicates in ways to make that elephant function. So in fact, this complex system of the classroom as we begin to do research on the area requires lots of different people with different knowledge and abilities, because we’re dealing with a very complex system. So if we take this directly to science, because I’m always wondering, how do I do my science and how do I do my education research. Here’s a very complex system, this is from the global change, this is a system that asks questions about climate change. Oh my gosh, you don’t even have to read it because it’s too small, but we’re dealing with human systems, we’re dealing with earth systems and all these interactions. So in order to answer the question, is climate changing, we need lots and lots of people exchanging ideas, using their interdisciplinary approaches to figure out questions we want to ask about climate change. And in fact, when we study humans, we’re dealing with a more complex system, I would argue. And that’s our classroom. So that then requires really interdisciplinary groups of people beginning to go after the kind of questions we want answered about our students, because we all have the students. I may have them for 80 minutes in biology and you may have them for 80 minutes in history and someone else might have them in chemistry, and so it’s very complex.

So if you want to, I’m a modeler here, so I built the classroom system here to say, okay where are these research questions going to come from? Alright so this is the easy part of it. We’re aiming to build learning-centered classrooms, the learner is the center of the universe, and the teacher’s got a role, the teacher directs learning- centered classrooms, so let’s look at the variables about our research that we need to think about. We’re talking about the top one collaborative, so we’re dealing with student-teacher interaction and student-student interaction, so that’s a variable. Continual – I was arguing yesterday that a point-in-time kind of research, what I learn in my classroom in 15 weeks is helpful, but it’s not going to tell me about the four years of that student. We don’t do longitudinal studies in higher education of our undergraduates very well. So we need to look at this time issue – it needs to be scholarly. If I’m doing educational research and I’m in the biology department, and I get reviewed based on my scholarship, and what are the traits of scholarship? Grants and publications. Then I’d better have grants and publications, and it better be scholarly. And it’s got to be context specific, because we all live in disciplinary kinds of worlds. Practical and relevant – that’s helpful. Can my research help early career faculty research better? So there’s the complex system, the elephant to the real system of our classroom.

Now, on Q2 of your paper, because I’ve just lectured enough, and you can do this with a partner, depending on what your discipline is, you should have a scientist around you, if you’re not a scientist. What I’d like to do, just as a scientist, is list the steps you take to do your research. Just write them down. List the process…… Okay, do we have one here whose handwriting I can read? Alright thank you, and you are Deborah? And you’re an advanced class so we’re moving right along. Had I a visualizer, and if you were in my large class I would have this tool and put Deborah’s work under here. And by the way, what was I doing, Jean, while you were busy what was I doing?…Yeah I was just cruising around. Do you realize that’s one of the hardest things for faculty to do? I will remind you again that to leave that safe place up there and mingle among the unwashed is one of the scariest things for faculty to do, and sometimes we don’t think about that, but that’s on the side. Okay, so Deborah said this is what she would do if she were to conduct scientific research and this is in general. Identify a topic of interest, go to the literature. Define the variables. Pose a hypothesis. Test the hypothesis. I saw things there, communicates in the back when you do these sorts of things. Yeah you all know how to go about doing science, and we could fuss about that, but we’re not going to, we have the general process that the kids memorized in fifth grade from the book and never got to do it. But then the next question is, on the right-hand side, if you were going to move into educational research, what would it look like, so take a minute to do that. Now you get to conduct research on teaching and learning. Look at your left side and build your right side. How would you go about conducting research in teaching and learning? Take a minute. Anything different?……. Is anybody finding anything different in general?….. Yeah probably not, they probably look about the same. So the point of that little exercise, when you do this with a room full of scientists, is that you know how to do your work in your field, you’re going to go into teaching and learning, and again this is how I approach it as a scientist because I didn’t come through my career as a social scientist with my college education or anything like that, so I’m a quantitative researcher. This is what makes sense to faculty. Oh it’s the same thing? It’s just like that comment on the faculty saying, Oh I thought there was a center to do that? What do you think that do to study teaching and learning? You’ve got to basically ask your questions, you’ve go to consult the literature, you’ve got to come up with some sorts of design and just knowing that scientists can actually start thinking about the way they do their daily work with the human subject, becomes enlightening for most of them. They never thought about that before. So, what we see is that all of us want to be reflective of teachers, and the majority of people teaching our undergraduates are thinking that way as well and they want to figure out why their students aren’t working. So basically, a teacher- scholar going to the scholarship part of the model that we’re talking about, basically takes the next step and figures out why their students aren’t learning, and it could be in the realm of a classroom, or it could be much more formal realm of a funded research projects where they’re actually doing these sorts of things. So what we’re asking with this whole research thing is to get the people who are teaching our students day in and day out to find out what their students’ work tells them about their learning, because this is where the questions are going to be generated from. How do we determine if in fact the innovations we’re using work? Because you’ve all heard that too. Well sure this interactive learning is fine, but I can tell them that much information in thirty minutes. And the question is what does “work” mean?

So we begin by making observations and listening to our students. When you move from assessment to research, you’re basically going to fill in box three of your paper. So based on that and what you know about teaching and the observations that you’ve made, what I’d like you to do, and this is the fun part, propose a researchable question that you would ask about students in this day and age. So talk about it in your groups, come up with a question based on what you know, and I’ll give you a minute to do that…….. (Unclear question from man in audience) Good chatting, we could continue this, but right now we’re going to do a little report out from the field. Since I have lots of tools here, I’ve got this microphone, let’s get some ideas now,

based on all the observations you’ve made about students, teachers and environments in your learning communities whatever they are, wherever they are, what are some of the questions that we need to answer. So you are, what is your name? Celeste, okay.

Celeste: Will students exhibit more proficient and cohesive writing in the intermediate ESL class at the university in one semester by using the writing process from day one and through a 15 week program?

Okay, so what is she testing?

Another woman: A particular writing process.

Right, so you’re doing research about a process and you have to come up with a certain design. Okay that’s a testable question, we can go after that. Do we have another one? Okay and you are Lynette.

Lynette: Did my active teaching methods respond to the variety of learning styles of my students?

Okay so we have two variables there – active learning, and learning styles, whatever those may be. Alright those are testable questions, so you’re getting the drift. And again, when we talk about these sorts of things, and we could spend a lot of time doing this, but now once we get our question, based on whether it’s the observations or the literature, or whatever we’re doing, then we have to decide on some sort of design. That’s what all your little papers said, what’s the research design?

So now I’m going to give you a one-minute slide on research design in education. We have to figure this out. I’m sorry, let me back off, and you are Jackie. Jackie is an experimental psychologist, but remember, we’re going back to that elephant with all these disciplines in very complex systems, and I want to bring to you, because if educational research is going to have any meat and potatoes about it, it has to be theoretically based. We don’t just go around saying, “whoa, my students learned really well this semester.” That’s all fine and good, we’re happy about that, but show me the data. Now here are the theoretical frameworks that I’ve been able to identify in my group that will influence each and every question that you all are doing. Okay, computer science – artificial intelligence, I have a post doc who’s doing this, this is model building. The AI people doing this, do I really like to read AI literature? No, but my post doc does. But artificial intelligence is a big thing, we’re talking about learning. Psychology, okay? Not only descriptive ecology, but there is learning and developing theories, behaviorism, experimental psychology, okay, so that’s another discipline we borrow from when we’re doing educational research. Philosophy – the whole concept of constructivism of how students learn is coming from philosophy. So this is a wonderful interdisciplinary thing we’ve got in our elephant here. Interdisciplinary – systems theory – it’s its own discipline now, certainly in the science. Sociology – where does all that stuff about this cooperative learning come from? It comes from the social sciences, but specifically sociology, and that’s when we get into consideration of ethnicity and culture and gender and all of these sorts of things. So we need sociologists around. And then of course our own hardcore scientific disciplines – biology, physics, math, chemistry. These are the theoretical frameworks that will underlie the importance of your research question, not necessarily all of them, but the two that I’ve heard, very much so. So that’s why in this complex system we need these collaborative groups of faculty to begin to get at these hard questions. All right, so here’s the research design, very quickly. I like to look at it like this – we have the quantitative kind of research we do as scientists and we have the qualitative research. I’m not making value judgments whatsoever, I’m saying these are the tools we have available in the big picture.

So quantitative approaches tend to be experimental designs, qualitative approaches tend to be case studies, ethno-graphic studies, oral interviews – those are all fine and good. What the heck is design research? Anybody know? Design research? Who does design research? In the disciplines, who does design research? Do we have any engineers in here? That’s why we don’t have an answer to the question – where are all the engineers? Okay design research is what engineers do. I want to make a better microphone. That’s what engineers do. And then they make a better microphone. They might go into empirically testing a microphone, but somebody says, “Make me a fuel-efficient car.” Okay, that’s design research. I would argue that a lot of research we do in education is design research. I want to make a better module to teach evolution to my students. I want to do a better writing process to teach them writing. We just have to know what we’re doing to justify our existence to the people who are going to look at our data and say, “Is this meaningful?” So I would argue in education we’re doing a lot of design research because we’re asking, we want to design a better curriculum, we want to design a better use of technology, and then we might get into quantitative experiment, you know, more quantitative things, and we might use qualitative data, but those are the kind of research designs I’m seeing in the literature that we’re doing.

Okay data collection – you’ve got the question, you’ve got the design, what kind of evidence are you going to gather? Well, let’s talk about what we have. These look very testy to me, and they are. So in our courses we do a lot of different data collection and assessments, everything from multiple choice and true-or-false, and surveys, you know, student surveys – these are all data. So when faculty say, “Oh I don’t do assessment.” You don’t? Well what is the basis for the grade you put on that kid’s transcript? “Well tests.” Well those are assessments. Start to think about this – the key is you have to identify what kind of data are going to answer your questions and we also know that some kinds of data are easier to gather than others. But their potential for assessing learning is less. So in other words, a multiple choice question for 500 students in my biology class is a pretty easy way to collect data, but it’s got limited value, in terms of what it’s telling me about my students’ learning – it’s just a piece, whereas oral interviews are impossible to do with 500 students, but they’re very rich and can tell me lots more about students’ thinking, and so there’s a mixture of those sorts of things. And again, you ask your question, you’ve got a theoretical basis for it, you come up with your design, now you’ve got to collect data. So that’s one, and then there are other types of data that we’re not thinking about necessarily. When I’m walking around talking to you, I’m getting to know you better just by walking around and talking to you. So these are real formative kinds of data, but you can use these things – demographics, learning the names of your students, that’s a data point, it’s a very formative kind of data thing, but again we can do lots of different sorts of data. So you’ve got your theoretical basis, your question, your design, I’m going to collect data.

Now we’ve got to report the results because this is what my scholarship is, and your faculty, no matter where you are, have to do scholarly work. If they’re teaching four courses a semester, that’s an important job, but I’d like to see them asking meaningful questions about their students’ learning, and helping us all understand things. That’s a scholarly approach to teaching, and there are other ways of doing this. In my group, and I encourage faculty if we’re doing research on science and math and engineering, this is what scientists read. These are their journals. So we’re putting our results in scientific journals so scientists will see them, because we’re doing our research to influence scientists who teach the masses of our courses to our students. Again, and the different disciplines can go in other journals as well. So we’re reporting the results. Okay, so here’s the summary slide, and this is from the book doing scientific, uh the NRC report by Town, Wise, and Winters (eds.) 2005 so that in quality science research and education we have to have significant questions that you can answer empirically. I think those are important; we need to come up with the questions. We need to link the research to theory, and I just gave you the theoretical basis for it, and this is good science. We need to use methods that prevent direct investigation, and this I’m going to show, I was telling Clarissa about the study, I’m going to show the results of a major study we did on faculty. So we have to have evidence that is direct, we need to have a coherent, explicit chain of reasoning, that’s good science too. This is everything, I mean who’s against this? It’s like being against pumpkin pie on Thanksgiving. You know, you might like pecan, but we’re not against pie, okay? And then we need to replicate and generalize, so what we find out at Michigan State, it’s critical that you be able to see this. And yes, you’re at a different school, in a different state, in a different part of the country, with a different group of people, but what’s generalizable about the work we do that will help all of our students learn? We need to think about that, and we need to peer review this. And remember, the notion of a classroom as a shut-door community where we don’t have people engaging that, can you imagine if science was a shut door community, and I’m not going to tell you anything about my science? You know, that’s just not how it works, so when we talk about scientific teaching, I’m arguing that education needs to move this way too.

So I’m going to give you two exemplars, and I’m right on time, about the questions, and here’s the big thing – we’re asking faculty to change. Because we’re implying that if they change, will students do better? Because if everybody’s satisfied with the way our students are learning, we’ll just go have coffee. We’ll just leave and cancel this conference and go watch the football game, which is starting right about now. No, okay, let me remind you, we’re doing teachers, students, and environment – those are the units of analysis, but let’s look at how, if teachers change, and I’ll just give you an example from the course level. So we can do research about the curricula, you have to be clear about what you’re researching about. A course, people, but I’m going to give you a piece of research on a course. We want to change the learner in a course, so we’re back to this model, and we’re going to do this. We want students to understand complex systems, so biology is all about complex systems now. And understanding a complex system is very difficult for students, you know, how do they even connect things like Mendelian genetics to evolution and we’re just dealing with some simple complex systems. How do we teach students to do this, and do they have to wait till graduate school to do this, and we argue absolutely not. So we’re asking questions about what’s an effective pedagogy for helping students learn complex systems, because this is the country of biology and if they can’t think in complexity, I don’t know, we don’t necessarily teach that.

All right, so we went to the artificial intelligence and here’s evolution. This is just evolution. Don’t read that. But it’s very, very complex, and that’s only one field of biology. How do you get a kid to understand that? Well what we tend to do is break it down into its little minutia parts and memorize trivia, and it doesn’t help. So we want them to understand that complex system, and so what we’re arguing, by

figuring out how to teach students how to deal with complex systems, maybe we’re promoting learning transfer. How many of you are into, you know, can you transfer information from one course to another, from one year to then next – that’s the big question. Can we focus on getting students from memorizing facts to actually thinking about connections. Isn’t that an outcome we want for our students? Can they make connections within their bachelor’s degree? And of course systems research is the hottest topic in biology, so that makes sense for those of us who are biologists. And by way of, I’m going to show you something, and I threw this slide in based on what George said yesterday – focus learning from facts to connections. We did a study, and I’m pulling this information out of a study, so you don’t have the context, but I got exams from 100 faculty at research universities on their introductory biology course. And I assigned a Bloom’s writing to each of the items – do we all know Bloom’s taxonomy? Well we’ll explain Bloom’s taxonomy to Phyllis. Okay, you’re on Bloom’s. What is it?

Phyllis: It’s a tool to promote higher order thinking, from the lowest level identified defined, to a higher order of thinking, analyze, generalize, and synthesize information.

Okay, so it’s a way of taking a look at, let’s say an exam question and saying, “Is that a knowledge question? Is that an application question? Is that a synthesis question?” Okay so what kind of assessments are we giving our students in this county that equals their grade? Okay, 100 faculty, all their exams, 12, 000 items that I Bloomed. What do you think the average Bloom rating is for introductory biology across the country? Huh?

Man: 1.1

1.1? You’re really close. Okay, here they are, this is 12,000 items of how we’re assessing students across the country at bunches of universities, ladies and gentlemen, and I almost feel like, oh my god. This is what kids are being asked to do. This is out of context, yes there were projects, yes there were other things going on, but this is what the exams are looking like.

And I go backwards. I want them to solve complex systems. There’s my goal for my students, I want students in my courses to be able to make models. I want them to do mathematical modeling, I want them to use computer simulations, I want them to connect ideas. But across the country, these are the kinds of things being asked, so I get them as graduate students, and I say, “You can’t do what?” So those are kinds of data, so a big research area we’re getting into is how do we teach students how to make models? So then, the other study, I’ve got another example of research that’s ongoing, this is the last little piece I’m doing, is we’re asking faculty to change, so we can do research on faculty change, right? And that’s an interesting one, and so basically the research questions – do faculty change their teaching in response to all these professional development things we’re doing? What kind of data do we usually have from faculty development programs? What kind of data do we collect? Participation, and what else do we get from them after a workshop? Surveys – self- report data: that was the best workshop, I’m jazzed, right? So that’s cool.

So then we ask the question – do all these millions of dollars we spend on faculty development really result in change in the classroom? How are we going to get their? How am I going to collect data on change in the classroom? I can look at student work, sure. Clarissa? Thank you! I can go into the classroom. Videotape them. So we basically went in there after two major things, the first project which is our faculty institutes reforming science teaching and the summer institutes at Wisconsin. Very high-end long-term professional development program, so we ask the question, do faculty change? So I had volunteers and we went in and we videotaped them in their classroom four times, and analyzed 350 videotapes of faculty teaching introductory science classes across this country. It was a fascinating study, and so basically the professional development was what we all value, you learn to design, implement and assess active learning, based on the principles of scientific learning, which Tracy mentioned, and we did large enrollment courses. And I promise to finish by ten o- clock, we’re almost done here. Okay and just to show you, now I’m going to show you what the day will look like, just a little bit, the study that we’re putting into science next week, and you’re going to ask questions. Who were the faculty? There they are – male, female, tenured, non-tenured. Data about the subjects. Institution types – they are, you have the different Carnegie-types classifications and you can see that lots of them were from research universities, but we had community colleges, bachelors, masters level, so we had a fairly decent cross section of faculty and we ended up with enough information from 37 faculty programs, from another 38, and got two to four videos from each of them in their classroom up to two years after participation, and the two workshops were virtually identical. Here, nothing before, and you ask a really good question. If you were to study me before I started teaching, when would you have needed that videotape? (Inaudible answer from audience.) Or when I was a graduate student. Because I started teaching- We can’t isolate, we can’t control them so well. I’m going to show you what it looks like. I can argue that most faculty in this country start their teaching by doing this – straight lecture. They just hope they can get through the day. And RTOP is a very cool, validated and reliable instrument developed at Arizona State called Reformed Teaching Observation Protocol (RTOP). It’s basically the elements we want to see in a classroom, based on someone who’s had development that we do active inquiry- based learning. We should see this in the classroom. So a straight lecture, you end up with a score between zero and thirty, and this is just to show you the categories, which is important. The use of lecture with some engagement, which is where they shut up for a while and use pointers and other pedagogies. And then significant student engagement is a three. And we rated these tapes and we had a reliable way to rate them, with two people rating each tape. It’s a very powerful tool. Alright, I’m going to show you the mix. Of the faculty after the workshop, 24 to 25 percent started at straight lecture, and stayed with straight lecture when we went into their classroom, regardless of what their surveys suggest – Oh yeah I do active learning. Some of them started at level one, these are the people who started at level one, did lecturing with some student involvement and stayed there. Some of them went down, regressed, some of them went up and so forth.

So you can begin to see faculty change over time and to use this data to figure out what a professional development program should look like and sound like. So it’s a powerful study. And the last slide is to show you, this is what I was saying yesterday about forcing these complex models, we’re dealing with a complex system. Doing pre and post analysis with a T test is not convincing to me. It’s not bad, but I think we need to start forcing much more sophisticated methods of analysis. These are all the variables we identified to influence faculty person’s teaching score. The performance of time and research, the tenure, the years teaching, the course size, your knowledge of active learning, your practice of active learning. That’s a complex model of what influences our faculty teaching. I’m not going into statistics here, but pick those variables that influence somebody’s score. The variable that came out of this, which one or ones do you think it is? Class size, tenure status, anything else?

Knowledge, experience, none of those came up. I’ll show you what came up – years teaching, and it was a negative correlation. So this notion of who was getting the highest score on their teaching, it was the pre-tenure, early-career faculty. Yes! Do you love it or what? And this is convincing enough so what’s next? Now we’re doing this work with post-docs. Our professional development is at CCLI phase 3, we are now dealing with two hundred post-docs around the country, and you will see this because you know post-docs who will need to apply to this. So all I want to leave you with is the thought – who is going to ask the questions in science education about teachers, and students, and our environment? We are. It’s going to be the people in the discipline and we must collaborate so we can combine all of our insights and techniques and expertise to try to understand this complex creature. This is a structure that’s very well-defined, but we need to understand the whole elephant. We need to ask good questions about our teaching and learning. We need to integrate this, and that will help all students learn science.