Hedgehogs, foxes, and academic portfolios

This week I am participating in an academic portfolio institute facilitated by Peter Seldin, Beth Miller, and a number of faculty members at my institution. Good stuff! Together with several colleagues, I am now revising my academic dossier, re-writing my teaching philosophy, description of teaching innovations, summaries of research projects, highlights of my service work, and the like. The point is to find common threads in one’s work, the overall purpose that drives what we do, and use this to guide the next steps of our careers. You can read about it in Seldin and Miller’s book. Since I am starting a new position this summer, as assistant director for career planning at my institution’s faculty development center, I see the institute as a welcome opportunity to think about what’ll come next.

So I’ve been busy writing my way through my past career, working on the common vision behind it all. And it got me thinking, which is a good thing: Does there have to be one overarching vision, a common purpose, a grand plan behind all the different things I’ve been doing? Honest question! I can see the ayes as well as the nays. (And, yes, I know that voice votes use “ayes” and “nos”, but I think this text needs more “y”s.)

Aye: It is good to know that I’m doing what I’m doing for a purpose, and that the purpose is one that I agree with. (Maybe even one of my choice!) It is good to be able to say yes to things because they belong to what I’m doing and no to things because they don’t belong to what I’m doing. It’ll make life easier. Also, knowing my professional purpose will make it easier to decide what the things are that I want to do before retiring (yeah right, retirement!), and to get them done. Maybe I’ll even become a peak performing faculty!!

Nay: This is so not me. I’m an empirical political scientist who has specialized on U.S. courts, and do you know the topic of the article that I recently read and found really exciting? Comparative political theory! On my commute home, I listened to Philippe Jaroussky’s recording of Verlaine songs, and that was the most important thing I think I came across today (sorry, Seldin, Miller, and colleagues ). Actually, the most important thing wasn’t Jaroussky but something I’m not going to write about beyond the note that it overshadowed everything else. And that’s how life is: it is not streamlined according to purpose or plan, it is multi-centered and whirling apart!

But then. We need meaning. We want to see connections. Scientists value parsimony. Religions seek the unifying vision, force, or deity. We like consistency in our ethics and politics.

Isaiah Berlin, in an essay on Tolstoy, cites the poet Archilochus: “The fox knows many things, but the hedgehog knows one big thing.” (Here is the first section of the essay, which states its premise.) According to Berlin, “hedgehogs” reduce their view of the world to one system, a few central principles, while “foxes” accept the variety, confusion, and contradiction of life. Berlin puts it much more nicely than I can summarize:

For there exists a great chasm between those, on one side, who relate everything to a single central vision, one system, less or more coherent or articulate, in terms of which they understand, think and feel – a single, universal, organising principle in terms of which alone all that they are and say has significance – and, on the other side, those who pursue many ends, often unrelated and even contradictory, connected, if at all, only in some de facto way, for some psychological or physiological cause, related to no moral or aesthetic principle. (p. 2)

Berlin clearly sympathizes with the foxes and distrusts the hedgehogs. Ronald Dworkin, in one of his last books, Justice for Hedgehogs, provides a counterpoint and argues for a liberal perspective that establishes a philosophical foundation connecting ethics, justice, and law.

I don’t really have a conclusion here. No big picture that connects everything and makes meaning here. Sorry!

Computer use reduces average grade from C- to low C-

The higher ed internets and social medias are positively abuzz these days with the Carter et al. paper on the effects of computer use in introductory econ at West Point. I've finally gotten around to reading the paper, and I like the study! It is extremely well-designed: it is situated in a real-life classroom setting, not a lab like other computer use studies; it uses real-life exams, not some ungraded quiz that students could care less about; it randomly assigns computer use policies in small course sections of large classes that use a common textbook and a common final exam; it compares course sections taught by different instructors and course sections taught by the same instructor; it includes important controls, does robustness checks with different types of standard errors, and covers almost any objection that I could think of. It's exactly how a quantitative, outcome-focused study in the Scholarship of Teaching and Learning should be conducted. It should be required reading in SoTL workshops.

But I don't buy the substantive conclusion drawn by the authors: that they have shown that the use of laptops and/or other electronic devices had a substantive impact on learning. Here is why.

Well, first, here is why not: I don't have a problem with the whole correlation/causation thing in this study. Sure, correlation is not causation, but the authors do a good job (including the use of 2SLS) to exclude alternative causal stories to the extent possible. I also don't have a problem with the conclusion that the authors found a statistically significant effect of computer/el.dev. use. I'm as critical of significance tests as anybody who has used Bayesian analysis, but I buy that Carter et al. find an effect that is clearly significant according to econometric practice.

My problem is that the statistically significant effect is small. (Yes, I know that the authors go out of their way to argue that it's large - more on that below.) Carter et al. present various versions of their analysis, most with a pretty consistent effect of around -0.2. Since the dependent variable is standardized, this means that permitting laptop or electronic device use was associated with a ceteris paribus reduction of the final exam grade (multiple choice and short answer portions) by .2 of a standard deviation. On a 100-point scale, that standard deviation was 9 points. In other words, the possibility of computer use in class reduced the average exam grade by less than two points out of 100 hundred.

Let's put this in grade terms. The average final exam score in the classes under observation was 72 in the multiple choice and short answer portions. Electronics use in the classroom was equivalent to reducing that score to a 70.2.

News flash: computer use in the classroom reduces a C- to a low C-.

Significance is not substance. This looks pretty insubstantial to me. But the authors make the argument that the effect is in fact meaningful, even large, and they do so by comparing it to changes of other factors identified in other studies. The currency of comparison is the standard deviation: Carter et al. point to the fact that "Aronson, Barrow, and Sander (2007) find that one standard deviation improvement in teacher quality increases test scores by 0.15 standard deviations" in a study of high-school students; as a result, they conclude, permitting laptops in class is worse than reducing teacher quality. This assumes, of course, that a standard deviation in a West Point intro to econ test is equivalent to a standard deviation of a high school test score. What's the apple and what's the orange here?

There's another aspect of the Carter et al. study that makes me wary of its conclusion that it shows a reduction of learning due to laptops in the classroom. What worries me is the 72-point average that I mentioned above. You put highly motivated young people who were selected on the basis of academic merit and test scores into a small class taught by an expert with a graduate degree - and at the end of the semester the average score is a C-? That's puzzling! But it's not uncommon (I should take another look at my own classes!), and I think, based on my own experience but no actual data analysis, that there are three likely explanations for this. First, the final exam may not be well-aligned with what was taught in the course. Most frequently, this happens when the exam questions are about material that was not emphasized but mentioned somewhere in the class material, or when exam questions are not about the core of the facts and concepts but about marginal details associated with them. A popular type of such questions asks about the precise value of some number that was used to exemplify a course concept in the textbook. Such exams measure not only to what extent students have understood and can use the course material but also to what extent they can memorize the details. A second reason for such a low final exam score could be that the exam consists of a large number of questions, so that a fair number of students lose points because they don't get through the exam. In that case, the exam measures to what extent students are skilled at working quickly with a particular exam format, in addition to measuring learning. A third possibility is that the instructors aim for a particular grade curve. It could be that they want only a certain percentage of top students to continue in the major, or they may want to identify top students for particular honors or other outstanding treatment. These reasons are not mutually exclusive and, depending on the goals of the instructor and the educational program, they may be legitimate. But they suggest that the exam scores may have limited measurement validity as indicators of learning. In other words, computer use may (slightly) reduce something, but it is not necessarily learning, or not all learning.

The last point raises an interesting question about an intriguing detail of the Carter et al. study: While computer use in the classroom had a significant effect on multiple choice and short answer scores, this effect was missing with regards to the essay portion of the exam; instead, the instructor effect was much stronger. The authors, in response, dismissed the essay grade as a bad indicator of student learning. Could it be, though, that the opposite was true and that at least some instructors used the essay grade to correct for the low standardized scores of students that they knew, from classroom experience, office hours, and the like, were bad test-takers but had learned a lot?

In any case, I am generally happy about the results found by Carter at al.: Once you control for all the other factors that influence student performance, it barely matters whether instructors permit electronic devices in the classroom or not. Considering the many problems associated with prohibiting computers in the classroom, this is good news! Of course, the results do not tell us how instructors and students can use all the tools, electronic and otherwise, at their disposal to actually increase their learning, but that's the next conversation to be had.

Discussing discussions

Over the last year, I noticed in several of my classes that students were more reluctant than usual to participate in discussions. Sure, there had always been students who were more willing to voice their thoughts than others, but this time the silences were more frequent, in small classes as well as in large ones. This may have been a fluke – and one class proved the exception to the rule – but close listening to how students responded to my prompts for more participation gave me the impression that something was actually going on: Students were more afraid than usual of exposing themselves intellectually to the scrutiny of others.

It makes sense that (not only) students have become more reluctant to speak up: Over the last years, the most visible forms of discussion have become cesspools of hostility and public shaming: The insults hurled at Reddit OPs have become proverbial; Twitter blowups are ugly and personal even among people who are nominally adults; and a thoughtless post on a semi-private forum like Facebook may end up making the rounds as a screenshot. As smartphones are only inches away from student hands, a comment in a 10-student class may very well go around the world.

I don't think we could ever expect class discussion to come easily. After all, Karp and Yoel's work, which found that the majority of students delegate participation to a small number of their more engaged fellows, is almost 40 years old! If most students refused to participate in the staid social environment of 40 years ago, how can we expect them to jump at the opportunity today? At the same time, it is becoming increasingly clear that effective learning requires active engagement with what we learn. How can we create a classroom climate that fosters student participation, that makes it more easy and desirable for students to interact? And how can we make sure students learn how to engage and interact productively?

I don't have it all figured out yet. Luckily there's a bit of literature out there; among freely available resources, I found Blunt and Napolitano's paper on "Leading Classroom Discussions" useful, as well as Todd Finley's piece for edutopia. The Faculty Focus resources on class discussion (which are mainly by Maryellen Weimer) are very useful as well. For those with access to JSTOR or other subscription services, there is Goodman's excellent 1995 College Teaching paper on "Difficult Dialogues".

What I have found important and useful is to start the semester with a class discussion on discussions. Luckily, since I am teaching political science, this discussion usually fits into the course topic, as deliberation is central to many political processes.

The central prompt for the discussion on discussions looks somewhat like this:

Many learning activities in this course involve critical discussions that ask you to exchange ideas with your fellow students. This being a political science course, many of these ideas can be controversial and may lead to disagreement. Disagreement is a good thing, because it helps us learn. However, if disagreement is expressed in ways that upset us and that just leads to fights, we are likely to learn very little. Therefore, we need ground rules that make sure our discussions remain civil and focused on the things we want to learn.

To create discussion ground rules for this course, list three rules that you find important to create civil discussions.

In a small class, I let students write their proposed rules, then collect them on the board and have students discuss the rationale for the rules, which rules are most or least likely to be effective, and why they might work or not. At the end, I'll propose additional rules that I find important and that are missing from the list (here are some possible ground rules, proposed by the good people at Carleton College). I type up the list and circulate it among the students. In a large class, I first break up the discussion into small groups, then have the groups report to the whole class on giant post-its, whiteboards, or the like. Online, I require students to post their lists and respond to at least one other student's list by highlighting what they think are the most and least effective rules.

While this type of discussion on discussions is only a fairly small step, I find it useful, as it usually goes beyond how-to rules on discussion. A recent instance was a discussion in which students quickly agreed that class discussion should be purely rational and arguments should only be based on facts and data. This, in turn, led to a discussion on the role of affective elements in human interactions as well as the question whether the class objectives (or any important objectives, for that matter) could be addressed on a purely factual basis.

But there are more things that need to be done to foster widespread and productive class discussion. The actual substantive discussions later on have to be on important and engaging questions that help students learn; students need activities to reflect on the discussion, to debrief themselves, and to draw conclusions about what they've learned, and they need instructor feedback on whether they are on the right track. They also should have opportunities to break the discussion rules and be called out for it, since otherwise the rules are quickly forgotten. I'm not sure how to get all of this to work, but I think I'm partly on the way. The resources linked to above provide some useful starting points.

Write an inviting syllabus, not trigger warnings

The debate on trigger warnings seems to have run its course for now, with many thoughtful (if you agreed) or outrageous (if you disagreed) contributions. While I agree with many of the thoughtful contributions (see, for example, Ari Kohen, Tressie McMillan Cottom, and David Perry), I'm not too happy about the overall frame of the debate, it's focus on warning students:

Some of the (thoughtful) debate contributors have pointed to the learning hurdles encountered by students with invisible disabilities such as PTSD and the fact that we have to help them overcome these hurdles. While trigger warnings might prevent the encounter with unexpected triggers, the triggers constitute only a portion of these hurdles. Another thoughtful point in the debate is that we should not exclude students who experience discomfort and stress with difficult and conflictual course topics, even if this discomfort does not rise to the level of trauma or other psychological distress. But trigger warnings can very easily exclude those students ("You have been warned: Take this course only if you are worthy of hard questions!"). As a result, trigger warnings are only a partial solution to one problem and make another problem worse.

Instead of adding a trigger warning to the lengthy (thanks, accreditors!) list of rules, instructions, penalties, grading scales etc. that we have to include on our syllabi, why not write a syllabus that helps create an open and inviting course climate? Instead of warning students off, why not invite those who fear conflictual or difficult course material to talk to you about ways to handle the difficult material and succeed in the course? Such a conversation, hopefully at the beginning of the course, is more likely than a trigger warning to encourage students to face the questions that they find threatening and to work with the instructor to do so successfully. In addition, instructors can encourage students to contact disability services and possibly get an access plan if they suspect that their fears are connected to a disability that should be accommodated.

Making your class inviting opens a number of boxes, particularly if you think in terms of inclusive learning, so here are just a few pointers to things that can be done fairly easily. Harnish et al., in an article on Creating the Foundation for a Warm Classroom Climate, focus largely on the type of language instructors use in their syllabi and how they frame the rationale for assignments and explain their own interest in the course topic. They provide a useful list of things that instructors can check when they create their syllabi.

Related to this, with a stronger emphasis on the disciplinary focus of the course, is Ken Bain's Promising Syllabus model, which he summarizes in a short paper (pdf) as well as in his book, What the Best College Teachers Do. Bain's syllabus model encourages instructor and students to enter into a conversation about what students will learn, why they should do so, and how they can document success. If this is successful, the exchange between instructor and student should include a conversation about possible affective and psychological hurdles that the students face.

Among the suggestions most applicable to trigger-warning style situations is one by Christina Petersen, David Langley, and Cheryl Neudauer, in a presentation at the 2013 PODNetwork conference. Petersen and her colleagues propose the inclusion of a paragraph that explicitly invites students to talk to the instructor when they feel overwhelmed and promises that the instructor will try to find a solution with them that helps them succeed. The presentation material used by Petersen and her colleagues, which includes example text for such a paragraph, can be found at the wikiPODia website.