Educational Consulting, Learning, Pedagogy

Understanding Course Evaluations

At the end of almost every college course, almost all colleges and universities in the United States have students fill out a student course evaluation, in which students fill out a form that gives the school their feedback about the class, the instruction, and the textbook. There are a recent interesting studies out examining their effectiveness, including one study out of UC Berkeley evaluating the validity of student course evaluations in measuring teaching effectiveness. The results are similar to the results of the many other studies conducted in the past: student course evaluations are not reliable indicators of teacher effectiveness:

Student ratings of teaching have been used, studied, and debated for almost a century. This article examines student ratings of teaching from a statistical perspective. The common practice of relying on averages of student teaching evaluation scores as the primary measure of teaching effectiveness for promotion and tenure decisions should be abandoned for substantive and statistical reasons: There is strong evidence that student responses to questions of “effectiveness” do  not measure teaching effectiveness. Response rates and response variability  matter. And comparing averages of categorical responses, even if the categories  are represented by numbers, makes little sense. Student ratings of teaching are valuable when they ask the right questions, report response rates and score distributions, and are balanced by a variety of other sources and methods to evaluate teaching.

What do student course evaluations measure, then? The authors of this study summarize the findings of previous studies here:

  • Student teaching evaluation scores are highly correlated with students’ grade expectations (Marsh and Cooper 1980; Short et al. 2012; Worthington 2002). WHAT THIS MEANS:
    • If you’re an instructor and want high course evaluations, pass out As like candy.
    • Adjunct instructors, having the least job security and the most job retention anxiety, are most likely to inflate grades to get high course evaluations.
    • Net result: over-reliance on adjunct instructors and on student course evaluations to evaluate teachers leads to grade inflation and low course rigor; i.e., poor educational quality.
  • Effectiveness scores and enjoyment scores  are related. In a pilot of online course evaluations in the UC Berkeley Department of Statistics in Fall 2012, among the 1486 students who rated the instructor’s overall effectiveness and their enjoyment of the  course on a 7-point scale, the correlation between instructor effectiveness and course enjoyment was 0.75, and the correlation between course effectiveness and course enjoyment was 0.8.
    • WHAT THIS MEANS: If students enjoyed the course, they will rate it highly. But enjoyment by itself isn’t a measure of learning. The instructor may just be a good performer.
    • Conversely, lack of enjoyment doesn’t mean the student didn’t learn. The types of assessments and activities that promote long term retention, in fact, lead to low course evaluations. The practices that students like the least actually help them learn and retain the most. See the link right above.
  • Students’ ratings of instructors  can be predicted from the students’ reaction to 30 seconds of silent video of the instructor: first impressions may dictate end-of-course evaluation scores, and physical attractiveness matters (Ambady and Rosenthal 1993).
    • WHAT THIS MEANS: student course evaluations are, more than anything else, superficial measures of instructor popularity rather than teaching effectiveness.
  • Gender, ethnicity, and the instructor’s age matter (Anderson and Miller 1997;  Basow 1995; Cramer and  Alexitch 2000; Marsh and Dunkin 1992;  Wachtel 1998; Weinberg et al. 2007; Worthington 2002).
    • WHAT THIS MEANS: student course evaluations are, at worst, racist, elitist, ageist, and sexist superficial measures of instructor popularity.

So how do we rate teaching effectiveness? I’d recommend the following:

  • Worry less about evaluating the teacher for promotion and focus on gauging effectiveness for the sake of seeking out the most effective strategies for that specific student population.
  • Rely in part on peer evaluations — teachers in the field conducting this evaluation. Field specific knowledge matters, as teaching isn’t just a matter of technique, but of careful selection of content.
  • We still do want to hear from students, of course, so use course evaluation tools that focus on teaching effectiveness, such as those provided by the IDEA Center.

Just for the record, I’ve always been an engaging instructor who generally gets high course evaluations, so I’m not worried about myself here. I am, however, worried about how effectively students are being educated. Reliance on student course evaluations, at present, is working against educational quality.

You can read the study below:

Cost of Degree, Educational Consulting, Majors and Areas of Study, Pedagogy, Understanding the Market

Podcast: James Rovira and the Anazoa Educational Project on Punkonomics

I’ve blogged quite a bit about a number of higher education topics, but what’s my own vision for higher education? I spent some time with Dr. Beni Balak, Professor of Economics at Rollins College, on the phone for his Punkonomics podcast to discuss the Anazoa Educational Project and my vision for higher education. Follow the link to listen to the podcast.

Educational Consulting, Learning, Majors and Areas of Study, Return on Investment, Understanding the Market

Podcast: Are College Students Being Prepared for the Workforce?

Have you enjoyed my blogging about workforce preparation and college education? If you like listening to podcasts, I was interviewed by Tim Muma about that topic on LJN Radio.

Educational Consulting, Learning, Machines, Pedagogy, Technology

Why You Should Take Notes by Hand

Because of the way that human beings interact with laptops, studies indicate that students who take notes on laptops don’t learn nearly as much as those who write out their notes on paper.

This learning differential doesn’t exist only because students are distracted while working on their laptops. It’s actually the use of the laptop itself. Students taking notes on a laptop attempt to capture everything that’s being said, so that they’re acting more like passive recipients of information — like stenographers — rather than thinking about the lecture.

On the other hand, students who take notes on paper have to think about what they’re writing down because they can’t possibly capture everything. That means they’re more cognitively engaged with the lecture material than the laptop note taker. Even a week later, students who took notes on paper scored higher on tests for both conceptual and factual content than laptop note takers.

But in addition to this difference, students taking notes on laptops are indeed distracted by other things on their laptops: according to other studies, 40% of the time students are looking at non-course related material while in class if they’re using a laptop, like Facebook, email, and chats.

These bad practices disseminate throughout educational institution. Because students aren’t learning as much, they complain about the quality of their education (a result noted in one study). School administrators listen to these student complaints and attempt to address outmoded instructional methods.

To appear innovative, they then spend a lot of money on educational technology that puts learning onto a screen. Schools then have to spend millions of dollars on this tech so have to adjunctify the faculty pool, which further degrades instructional quality. The problem is not that adjunct instructors are bad instructors, but that they are badly paid and badly overworked.

As a result, we have a higher educational system that everyone says is “broken” because of “outmoded instructional methods” but that no one thought was “broken” until relatively recently (say, the last fifteen to twenty years).

The real fix: shut off the laptop and take notes on paper. Just read the study.

Some great points made during a LinkedIn discussion about these ideas:

  • Handwriting on a tablet may well be a good middle way between typing on a computer and handwriting notes on a pad and paper, if you can get a good app for that. I haven’t had any luck, but this tech is continually evolving. I get the impression others have. I use an iPad Air.
  • There is neuroscience supporting the idea that your brain processes things differently when handwriting as opposed to typing, so this difference may also be related to how our brains and bodies work together. In fact, different areas of the brain are activated with printing out by hand compared to writing in cursive, so even different types of handwriting matter.
  • The study is just about one specific activity — note taking — so of course wouldn’t necessarily apply to group work and other tasks that require more engagement than passive recording of notes on a keyboard.
  • There are always exceptions. Some students need the support provided by electronic devices when note taking. Let’s just be careful not to define the rule by its exceptions.

Avoiding Student Loan Debt, Educational Consulting, Majors and Areas of Study, Understanding the Market

Understanding College Advising

If you’re a student, you probably have a number of people advising you: teachers, guidance counselors, college admissions people, your parents… the list could go on. I’d like to help students understand what advising — and educational consulting — can and can’t do for you.

All advisors should listen to the student’s own long and short term goals, and they should ask leading questions to help the student clarify them. They should recommend a variety of paths to the student to reach those goals, but be honest, informed, and realistic about these different paths. Advisors should also provide materials to the student to help the student make better informed decisions, and they should engage in advising with the student’s best interests in mind.

But a good advisor won’t feel obligated to validate all of the student’s goals or ideas. If a student is really committed to humanities Ph.D. study or law school, for example, the advisor should still inform the student of the realities of these programs of study, not tell the student what he or she wants to hear. Being honest about the realities of a path may be discouraging to the student, but the student still needs that information to make an informed decision.

A good advisor will also be honest about the student’s demonstrated abilities so far in their educational careers. If a student’s grades are weak in math, science, or English, a good advisor will be honest with the student about the barriers those grades might present. However, a good advisor will also know that a student’s grades do not define the student as a person or necessarily summarize their potential. In other words, good advisors know that not all barriers are insurmountable.

A good advisor won’t do anything other than advising: good advisors give students facts about the field, the market, and educational options, but they don’t try to make students’ decisions for them. They also don’t give students advice intended to benefit the student’s educational institution above the student. Every B.A. program would love to say 90% of their graduates were accepted for Ph.D. study, but that doesn’t mean that 90% of their graduates should be pursuing Ph.D. study. Good advising, in other words, is never anything other than a supplement to the student’s own decision-making process. It is not supposed to or be able to take the place of the student’s own decision making.

If you’re a student, you should know that your decisions are ultimately your own. You make them and then you live with the consequences. Because these are ultimately your decisions, you should be aggressive in pursuing information that will help you make the most informed decisions possible. Get everything that you can from your advisor and then seek out other information as well. Listen to your advisors, even if you disagree with them, rather than demand to be told certain things. And, listen to a number of advisors. Don’t get your information only from one source.

You should also think generally about what you most want. Do you mainly want to make a living? Or do you mainly want to perform fulfilling work? Are you willing to make a bit less money to be more fulfilled in the kind of work that you do?

There are no right answers to these questions. Some people pursue work in high-paying fields and then burn out and make expensive mid-career shifts to more fulfilling fields. Some people pursue fulfillment but have a hard time making a decent living. Ideally, of course, we would all work in fulfilling jobs that pay well, whether we work as employees, own our own businesses, or do creative, freelance work.

We all also need to understand that the ability to do work that is both fulfilling and very profitable is dependent upon many arbitrary factors. At the least, it is dependent upon the random intersections of what this society chooses to reward financially, your own abilities, and your own interests. Just don’t mistake profitability for inherent value: scientific or engineering work generates patents and/or high end products (like bridges, tanks, and computers), so produces a lot of money, and there aren’t enough people around with math skills at that high a level, so the employee pool is small.

Someone who produces something that can be packaged and sold at high volume can also make a lot of money: one hit single can pay a lot. But while small employee pools, high end products, and mass produced products drive up the profitability of a line of work, an engineer or singer is not inherently more valuable, socially, than a middle school math or music teacher. You can’t have engineers and singers without math and music teachers. If we lost every pro basketball player in the world, the world wouldn’t be that bad off–maybe it’d even be better off in some ways. But if we lost all of our music and math teachers, that would be a long term disaster for the human race.

What might that ideal spot of wage earning and job fulfillment look like for you? No advisor can answer that question. No one can tell you what you want. Advising can only point you in a direction that leads you to your goals, so no advising will be better than your own knowledge of your own goals. Bright Futures Educational Consulting is here to help you define your goals first, and once you’ve done that, it will show you the best paths to help you meet them. Contact us for more information.