The higher ed internets and social medias are positively abuzz these days with the Carter et al. paper on the effects of computer use in introductory econ at West Point. I've finally gotten around to reading the paper, and I like the study! It is extremely well-designed: it is situated in a real-life classroom setting, not a lab like other computer use studies; it uses real-life exams, not some ungraded quiz that students could care less about; it randomly assigns computer use policies in small course sections of large classes that use a common textbook and a common final exam; it compares course sections taught by different instructors and course sections taught by the same instructor; it includes important controls, does robustness checks with different types of standard errors, and covers almost any objection that I could think of. It's exactly how a quantitative, outcome-focused study in the Scholarship of Teaching and Learning should be conducted. It should be required reading in SoTL workshops.
But I don't buy the substantive conclusion drawn by the authors: that they have shown that the use of laptops and/or other electronic devices had a substantive impact on learning. Here is why.
Well, first, here is why not: I don't have a problem with the whole correlation/causation thing in this study. Sure, correlation is not causation, but the authors do a good job (including the use of 2SLS) to exclude alternative causal stories to the extent possible. I also don't have a problem with the conclusion that the authors found a statistically significant effect of computer/el.dev. use. I'm as critical of significance tests as anybody who has used Bayesian analysis, but I buy that Carter et al. find an effect that is clearly significant according to econometric practice.
My problem is that the statistically significant effect is small. (Yes, I know that the authors go out of their way to argue that it's large - more on that below.) Carter et al. present various versions of their analysis, most with a pretty consistent effect of around -0.2. Since the dependent variable is standardized, this means that permitting laptop or electronic device use was associated with a ceteris paribus reduction of the final exam grade (multiple choice and short answer portions) by .2 of a standard deviation. On a 100-point scale, that standard deviation was 9 points. In other words, the possibility of computer use in class reduced the average exam grade by less than two points out of 100 hundred.
Let's put this in grade terms. The average final exam score in the classes under observation was 72 in the multiple choice and short answer portions. Electronics use in the classroom was equivalent to reducing that score to a 70.2.
News flash: computer use in the classroom reduces a C- to a low C-.
Significance is not substance. This looks pretty insubstantial to me. But the authors make the argument that the effect is in fact meaningful, even large, and they do so by comparing it to changes of other factors identified in other studies. The currency of comparison is the standard deviation: Carter et al. point to the fact that "Aronson, Barrow, and Sander (2007) find that one standard deviation improvement in teacher quality increases test scores by 0.15 standard deviations" in a study of high-school students; as a result, they conclude, permitting laptops in class is worse than reducing teacher quality. This assumes, of course, that a standard deviation in a West Point intro to econ test is equivalent to a standard deviation of a high school test score. What's the apple and what's the orange here?
There's another aspect of the Carter et al. study that makes me wary of its conclusion that it shows a reduction of learning due to laptops in the classroom. What worries me is the 72-point average that I mentioned above. You put highly motivated young people who were selected on the basis of academic merit and test scores into a small class taught by an expert with a graduate degree - and at the end of the semester the average score is a C-? That's puzzling! But it's not uncommon (I should take another look at my own classes!), and I think, based on my own experience but no actual data analysis, that there are three likely explanations for this. First, the final exam may not be well-aligned with what was taught in the course. Most frequently, this happens when the exam questions are about material that was not emphasized but mentioned somewhere in the class material, or when exam questions are not about the core of the facts and concepts but about marginal details associated with them. A popular type of such questions asks about the precise value of some number that was used to exemplify a course concept in the textbook. Such exams measure not only to what extent students have understood and can use the course material but also to what extent they can memorize the details. A second reason for such a low final exam score could be that the exam consists of a large number of questions, so that a fair number of students lose points because they don't get through the exam. In that case, the exam measures to what extent students are skilled at working quickly with a particular exam format, in addition to measuring learning. A third possibility is that the instructors aim for a particular grade curve. It could be that they want only a certain percentage of top students to continue in the major, or they may want to identify top students for particular honors or other outstanding treatment. These reasons are not mutually exclusive and, depending on the goals of the instructor and the educational program, they may be legitimate. But they suggest that the exam scores may have limited measurement validity as indicators of learning. In other words, computer use may (slightly) reduce something, but it is not necessarily learning, or not all learning.
The last point raises an interesting question about an intriguing detail of the Carter et al. study: While computer use in the classroom had a significant effect on multiple choice and short answer scores, this effect was missing with regards to the essay portion of the exam; instead, the instructor effect was much stronger. The authors, in response, dismissed the essay grade as a bad indicator of student learning. Could it be, though, that the opposite was true and that at least some instructors used the essay grade to correct for the low standardized scores of students that they knew, from classroom experience, office hours, and the like, were bad test-takers but had learned a lot?
In any case, I am generally happy about the results found by Carter at al.: Once you control for all the other factors that influence student performance, it barely matters whether instructors permit electronic devices in the classroom or not. Considering the many problems associated with prohibiting computers in the classroom, this is good news! Of course, the results do not tell us how instructors and students can use all the tools, electronic and otherwise, at their disposal to actually increase their learning, but that's the next conversation to be had.
Sunday, May 15, 2016
Thursday, July 24, 2014
Discussing discussions
Over the last year, I noticed in several of my classes that students were more reluctant than usual to participate in discussions. Sure, there had always been students who were more willing to voice their thoughts than others, but this time the silences were more frequent, in small classes as well as in large ones. This may have been a fluke – and one class proved the exception to the rule – but close listening to how students responded to my prompts for more participation gave me the impression that something was actually going on: Students were more afraid than usual of exposing themselves intellectually to the scrutiny of others.
It makes sense that (not only) students have become more reluctant to speak up: Over the last years, the most visible forms of discussion have become cesspools of hostility and public shaming: The insults hurled at Reddit OPs have become proverbial; Twitter blowups are ugly and personal even among people who are nominally adults; and a thoughtless post on a semi-private forum like Facebook may end up making the rounds as a screenshot. As smartphones are only inches away from student hands, a comment in a 10-student class may very well go around the world.
I don't think we could ever expect class discussion to come easily. After all, Karp and Yoel's work, which found that the majority of students delegate participation to a small number of their more engaged fellows, is almost 40 years old! If most students refused to participate in the staid social environment of 40 years ago, how can we expect them to jump at the opportunity today? At the same time, it is becoming increasingly clear that effective learning requires active engagement with what we learn. How can we create a classroom climate that fosters student participation, that makes it more easy and desirable for students to interact? And how can we make sure students learn how to engage and interact productively?
I don't have it all figured out yet. Luckily there's a bit of literature out there; among freely available resources, I found Blunt and Napolitano's paper on "Leading Classroom Discussions" useful, as well as Todd Finley's piece for edutopia. The Faculty Focus resources on class discussion (which are mainly by Maryellen Weimer) are very useful as well. For those with access to JSTOR or other subscription services, there is Goodman's excellent 1995 College Teaching paper on "Difficult Dialogues".
What I have found important and useful is to start the semester with a class discussion on discussions. Luckily, since I am teaching political science, this discussion usually fits into the course topic, as deliberation is central to many political processes.
The central prompt for the discussion on discussions looks somewhat like this:
While this type of discussion on discussions is only a fairly small step, I find it useful, as it usually goes beyond how-to rules on discussion. A recent instance was a discussion in which students quickly agreed that class discussion should be purely rational and arguments should only be based on facts and data. This, in turn, led to a discussion on the role of affective elements in human interactions as well as the question whether the class objectives (or any important objectives, for that matter) could be addressed on a purely factual basis.
But there are more things that need to be done to foster widespread and productive class discussion. The actual substantive discussions later on have to be on important and engaging questions that help students learn; students need activities to reflect on the discussion, to debrief themselves, and to draw conclusions about what they've learned, and they need instructor feedback on whether they are on the right track. They also should have opportunities to break the discussion rules and be called out for it, since otherwise the rules are quickly forgotten. I'm not sure how to get all of this to work, but I think I'm partly on the way. The resources linked to above provide some useful starting points.
It makes sense that (not only) students have become more reluctant to speak up: Over the last years, the most visible forms of discussion have become cesspools of hostility and public shaming: The insults hurled at Reddit OPs have become proverbial; Twitter blowups are ugly and personal even among people who are nominally adults; and a thoughtless post on a semi-private forum like Facebook may end up making the rounds as a screenshot. As smartphones are only inches away from student hands, a comment in a 10-student class may very well go around the world.
I don't think we could ever expect class discussion to come easily. After all, Karp and Yoel's work, which found that the majority of students delegate participation to a small number of their more engaged fellows, is almost 40 years old! If most students refused to participate in the staid social environment of 40 years ago, how can we expect them to jump at the opportunity today? At the same time, it is becoming increasingly clear that effective learning requires active engagement with what we learn. How can we create a classroom climate that fosters student participation, that makes it more easy and desirable for students to interact? And how can we make sure students learn how to engage and interact productively?
I don't have it all figured out yet. Luckily there's a bit of literature out there; among freely available resources, I found Blunt and Napolitano's paper on "Leading Classroom Discussions" useful, as well as Todd Finley's piece for edutopia. The Faculty Focus resources on class discussion (which are mainly by Maryellen Weimer) are very useful as well. For those with access to JSTOR or other subscription services, there is Goodman's excellent 1995 College Teaching paper on "Difficult Dialogues".
What I have found important and useful is to start the semester with a class discussion on discussions. Luckily, since I am teaching political science, this discussion usually fits into the course topic, as deliberation is central to many political processes.
The central prompt for the discussion on discussions looks somewhat like this:
Many learning activities in this course involve critical discussions that ask you to exchange ideas with your fellow students. This being a political science course, many of these ideas can be controversial and may lead to disagreement. Disagreement is a good thing, because it helps us learn. However, if disagreement is expressed in ways that upset us and that just leads to fights, we are likely to learn very little. Therefore, we need ground rules that make sure our discussions remain civil and focused on the things we want to learn.
To create discussion ground rules for this course, list three rules that you find important to create civil discussions.In a small class, I let students write their proposed rules, then collect them on the board and have students discuss the rationale for the rules, which rules are most or least likely to be effective, and why they might work or not. At the end, I'll propose additional rules that I find important and that are missing from the list (here are some possible ground rules, proposed by the good people at Carleton College). I type up the list and circulate it among the students. In a large class, I first break up the discussion into small groups, then have the groups report to the whole class on giant post-its, whiteboards, or the like. Online, I require students to post their lists and respond to at least one other student's list by highlighting what they think are the most and least effective rules.
While this type of discussion on discussions is only a fairly small step, I find it useful, as it usually goes beyond how-to rules on discussion. A recent instance was a discussion in which students quickly agreed that class discussion should be purely rational and arguments should only be based on facts and data. This, in turn, led to a discussion on the role of affective elements in human interactions as well as the question whether the class objectives (or any important objectives, for that matter) could be addressed on a purely factual basis.
But there are more things that need to be done to foster widespread and productive class discussion. The actual substantive discussions later on have to be on important and engaging questions that help students learn; students need activities to reflect on the discussion, to debrief themselves, and to draw conclusions about what they've learned, and they need instructor feedback on whether they are on the right track. They also should have opportunities to break the discussion rules and be called out for it, since otherwise the rules are quickly forgotten. I'm not sure how to get all of this to work, but I think I'm partly on the way. The resources linked to above provide some useful starting points.
Sunday, May 25, 2014
Write an inviting syllabus, not trigger warnings
The debate on trigger warnings seems to have run its course for now, with many thoughtful (if you agreed) or outrageous (if you disagreed) contributions. While I agree with many of the thoughtful contributions (see, for example, Ari Kohen, Tressie McMillan Cottom, and David Perry), I'm not too happy about the overall frame of the debate, it's focus on warning students:
Some of the (thoughtful) debate contributors have pointed to the learning hurdles encountered by students with invisible disabilities such as PTSD and the fact that we have to help them overcome these hurdles. While trigger warnings might prevent the encounter with unexpected triggers, the triggers constitute only a portion of these hurdles. Another thoughtful point in the debate is that we should not exclude students who experience discomfort and stress with difficult and conflictual course topics, even if this discomfort does not rise to the level of trauma or other psychological distress. But trigger warnings can very easily exclude those students ("You have been warned: Take this course only if you are worthy of hard questions!"). As a result, trigger warnings are only a partial solution to one problem and make another problem worse.
Instead of adding a trigger warning to the lengthy (thanks, accreditors!) list of rules, instructions, penalties, grading scales etc. that we have to include on our syllabi, why not write a syllabus that helps create an open and inviting course climate? Instead of warning students off, why not invite those who fear conflictual or difficult course material to talk to you about ways to handle the difficult material and succeed in the course? Such a conversation, hopefully at the beginning of the course, is more likely than a trigger warning to encourage students to face the questions that they find threatening and to work with the instructor to do so successfully. In addition, instructors can encourage students to contact disability services and possibly get an access plan if they suspect that their fears are connected to a disability that should be accommodated.
Making your class inviting opens a number of boxes, particularly if you think in terms of inclusive learning, so here are just a few pointers to things that can be done fairly easily. Harnish et al., in an article on Creating the Foundation for a Warm Classroom Climate, focus largely on the type of language instructors use in their syllabi and how they frame the rationale for assignments and explain their own interest in the course topic. They provide a useful list of things that instructors can check when they create their syllabi.
Related to this, with a stronger emphasis on the disciplinary focus of the course, is Ken Bain's Promising Syllabus model, which he summarizes in a short paper (pdf) as well as in his book, What the Best College Teachers Do. Bain's syllabus model encourages instructor and students to enter into a conversation about what students will learn, why they should do so, and how they can document success. If this is successful, the exchange between instructor and student should include a conversation about possible affective and psychological hurdles that the students face.
Among the suggestions most applicable to trigger-warning style situations is one by Christina Petersen, David Langley, and Cheryl Neudauer, in a presentation at the 2013 PODNetwork conference. Petersen and her colleagues propose the inclusion of a paragraph that explicitly invites students to talk to the instructor when they feel overwhelmed and promises that the instructor will try to find a solution with them that helps them succeed. The presentation material used by Petersen and her colleagues, which includes example text for such a paragraph, can be found at the wikiPODia website.
Some of the (thoughtful) debate contributors have pointed to the learning hurdles encountered by students with invisible disabilities such as PTSD and the fact that we have to help them overcome these hurdles. While trigger warnings might prevent the encounter with unexpected triggers, the triggers constitute only a portion of these hurdles. Another thoughtful point in the debate is that we should not exclude students who experience discomfort and stress with difficult and conflictual course topics, even if this discomfort does not rise to the level of trauma or other psychological distress. But trigger warnings can very easily exclude those students ("You have been warned: Take this course only if you are worthy of hard questions!"). As a result, trigger warnings are only a partial solution to one problem and make another problem worse.
Instead of adding a trigger warning to the lengthy (thanks, accreditors!) list of rules, instructions, penalties, grading scales etc. that we have to include on our syllabi, why not write a syllabus that helps create an open and inviting course climate? Instead of warning students off, why not invite those who fear conflictual or difficult course material to talk to you about ways to handle the difficult material and succeed in the course? Such a conversation, hopefully at the beginning of the course, is more likely than a trigger warning to encourage students to face the questions that they find threatening and to work with the instructor to do so successfully. In addition, instructors can encourage students to contact disability services and possibly get an access plan if they suspect that their fears are connected to a disability that should be accommodated.
Making your class inviting opens a number of boxes, particularly if you think in terms of inclusive learning, so here are just a few pointers to things that can be done fairly easily. Harnish et al., in an article on Creating the Foundation for a Warm Classroom Climate, focus largely on the type of language instructors use in their syllabi and how they frame the rationale for assignments and explain their own interest in the course topic. They provide a useful list of things that instructors can check when they create their syllabi.
Related to this, with a stronger emphasis on the disciplinary focus of the course, is Ken Bain's Promising Syllabus model, which he summarizes in a short paper (pdf) as well as in his book, What the Best College Teachers Do. Bain's syllabus model encourages instructor and students to enter into a conversation about what students will learn, why they should do so, and how they can document success. If this is successful, the exchange between instructor and student should include a conversation about possible affective and psychological hurdles that the students face.
Among the suggestions most applicable to trigger-warning style situations is one by Christina Petersen, David Langley, and Cheryl Neudauer, in a presentation at the 2013 PODNetwork conference. Petersen and her colleagues propose the inclusion of a paragraph that explicitly invites students to talk to the instructor when they feel overwhelmed and promises that the instructor will try to find a solution with them that helps them succeed. The presentation material used by Petersen and her colleagues, which includes example text for such a paragraph, can be found at the wikiPODia website.
Friday, November 29, 2013
Will filibuster reform lead to more partisan judges?
Observers agree that last week's filibuster reform was a big deal, but nobody knows yet in which way it is going to be big. Speculation abounda - a good example is Judge Wilkinson's argument that collegial interaction in the courts will suffer as future judges do not have to gain bipartisan approval (his column, by the way, includes a beautiful invocation of collegiality in the courts, which all should read).
I am skeptical of this conclusion, for a number of reasons. While the filibuster has been abolished for all practical purposes, there are other mechanisms for minority party involvement and obstruction. As a result, presidents still have incentives to nominate judges that gain at least some bipartisan approval.
First, the Senate's rule change is not a complete demise of the filibuster – it merely reduces the number of votes needed to invoke cloture. As a result, the majority party can now invoke cloture whenever almost all of its members support a nomination. It does not need Republican votes to do so. But according to Senate rules, a successful cloture vote does not necessary lead to an immediate vote on the nomination: it can still be followed by up to 30 hours of debate. In other words, the filibuster has been turned from a block to a delay that can still be costly to the majority's political agenda. If the president can reduce Senate debate to under 30 hours by appointing judges that accommodate minority party concerns, he is likely to do so, at least in many instances.
Second, the "end" of the filibuster for presidential nominations is not the end of the blue slip process, as several commentators noted in recent days (see here, here, or here). When the president nominates a lower-court judge, the chair of the Senate Judiciary committee sends blue forms to the two senators of the nominee's state. If the two home-state senators do not return the blue forms, or return them with negative comments, the Senate judiciary chair traditionally blocks hearings on the nomination, particularly if the objecting senator is from the president's party. Judiciary chairs have differed on whether they let blue slips from opposition senators block nominations; the current Judiciary chair, Patrick Leahy (Vt), has vowed to honor Republican blue slips and requires positive blue slips from both home-state senators before he proceeds with hearings on the nominations.
This begs the question why Senate Democrats, now that they got rid of the filibuster, wouldn't get rid of the blue slip process as well. I think the answer lies in who controls the blue slip process, compared to the filibuster. The cloture vote, which is the tool to end a filibuster, is controlled by the Senate as a whole – 16 senators have to request cloture, and the president pro tempore then holds a cloture vote. The blue slip process, on the other hand, is in the hands of the Judiciary Committee chair. And the Judiciary chair may have institutional incentives beyond those of the majority party leadership, or even the average majority party member. Leahy may find that he can do his job much more effectively if he has a minimum level of goodwill from the Republican members of his committee, with whom he interacts more closely than with other Senate Republicans. As a result, it is at least thinkable that Leahy will continue to honor negative (or missing) blue slips from Republican home-state senators, particularly if the objections raised by those senators are reasonable.
So I think that it is quite likely that judicial appointments will be less partisan than some of the doom sayers expect, and they will not be more partisan than previously appointed judges: The appointing president's and the home-state senators' ideologies have been pretty good predictors of how federal judges will decide in cases with politically definable outcomes &ndsh even before the filibuster change.
I am skeptical of this conclusion, for a number of reasons. While the filibuster has been abolished for all practical purposes, there are other mechanisms for minority party involvement and obstruction. As a result, presidents still have incentives to nominate judges that gain at least some bipartisan approval.
First, the Senate's rule change is not a complete demise of the filibuster – it merely reduces the number of votes needed to invoke cloture. As a result, the majority party can now invoke cloture whenever almost all of its members support a nomination. It does not need Republican votes to do so. But according to Senate rules, a successful cloture vote does not necessary lead to an immediate vote on the nomination: it can still be followed by up to 30 hours of debate. In other words, the filibuster has been turned from a block to a delay that can still be costly to the majority's political agenda. If the president can reduce Senate debate to under 30 hours by appointing judges that accommodate minority party concerns, he is likely to do so, at least in many instances.
Second, the "end" of the filibuster for presidential nominations is not the end of the blue slip process, as several commentators noted in recent days (see here, here, or here). When the president nominates a lower-court judge, the chair of the Senate Judiciary committee sends blue forms to the two senators of the nominee's state. If the two home-state senators do not return the blue forms, or return them with negative comments, the Senate judiciary chair traditionally blocks hearings on the nomination, particularly if the objecting senator is from the president's party. Judiciary chairs have differed on whether they let blue slips from opposition senators block nominations; the current Judiciary chair, Patrick Leahy (Vt), has vowed to honor Republican blue slips and requires positive blue slips from both home-state senators before he proceeds with hearings on the nominations.
This begs the question why Senate Democrats, now that they got rid of the filibuster, wouldn't get rid of the blue slip process as well. I think the answer lies in who controls the blue slip process, compared to the filibuster. The cloture vote, which is the tool to end a filibuster, is controlled by the Senate as a whole – 16 senators have to request cloture, and the president pro tempore then holds a cloture vote. The blue slip process, on the other hand, is in the hands of the Judiciary Committee chair. And the Judiciary chair may have institutional incentives beyond those of the majority party leadership, or even the average majority party member. Leahy may find that he can do his job much more effectively if he has a minimum level of goodwill from the Republican members of his committee, with whom he interacts more closely than with other Senate Republicans. As a result, it is at least thinkable that Leahy will continue to honor negative (or missing) blue slips from Republican home-state senators, particularly if the objections raised by those senators are reasonable.
So I think that it is quite likely that judicial appointments will be less partisan than some of the doom sayers expect, and they will not be more partisan than previously appointed judges: The appointing president's and the home-state senators' ideologies have been pretty good predictors of how federal judges will decide in cases with politically definable outcomes &ndsh even before the filibuster change.
Subscribe to:
Posts (Atom)