Saturday, Dec 15, 2018

Light Rain, 52 °F


Featured Stories

Encouraging critical thinking via a prediction tournament: the Ohio University experience

The following article was submitted to Compass by: 

  • Glenn Dutcher, Department of Economics, Ohio University
  • Robert Frank, Dean, College of Arts & Sciences, Ohio University
  • Daniel Karney, Department of Economics, Ohio University
  • Daniel Phillips, Department of Physics & Astronomy, Ohio University
  • Adam Siegel, Cultivate Labs

“Critical thinking” is a hard skill to define and perhaps an even harder one to teach.  Yet, developing “critical thinking” is a stated goal in programs of study across a vast range of post-secondary academic institutions and a diverse spectrum of college majors.

Typically, classes within those majors spearhead the critical-thinking-development task as particular skills are honed within the context of a discipline. This is an attractive approach because students can use their discipline-specific knowledge as fodder for practicing critical thought.  However, the danger is that the student’s specialized disciplinary knowledge leads them to assume–implicitly or explicitly–a narrow range of applicability for the critical-thinking skills they have labored to acquire.

Indeed, many students find it difficult to apply these skills to problems they meet in the wider world. The structure of college degrees can exacerbate this problem: coursework tends to become concentrated in the major as students advance in a program of study and thus they have only limited in-class opportunities to use their critical-thinking skills broadly.

Systematically helping a wide range of students across a college campus develop their general critical thinking thus represents a key challenge in American higher education.

At Ohio University, we decided to run a University-wide prediction tournament to bring campus-wide focus to the skills that make a good forecaster – skills which, we believe, also make for a good critical thinker. To our knowledge this is the first time a university has run a campus-wide prediction tournament.

We were inspired to try this approach after reading Dr. Philip Tetlock’s Superforecasting (2015). Tetlock’s book, based on his U.S.-government funded research on crowd-sourced forecasting, explores the possibility of developing general strategies and best practices for forecasting – practices that can be applied across contexts as diverse as politics, science, economics, and popular culture. Tetlock argues that certain intellectual traits or skills are found in people who are consistently good at forecasting; they are curious to seek out new data and have the ability to integrate that data into an applicable framework. Also important are the difficult tasks of perpetual self-reflection and acknowledgment of one’s biases and knowledge limitations. Such intellectual humility means that good forecasters are willing to modify their opinions or beliefs in the light of new information. This set of good-forecaster qualities mimics many descriptions of “critical thinking.”

The first Ohio University prediction tournament ran for two months in Fall 2016. Questions covering a range of topics were posed to students, often weeks before the answer to the prediction was known. Students were asked to predict that event’s outcome and their level of confidence in their prediction. They were also encouraged to update their prediction when new information became available. Evaluation was set up so that students were rewarded for a combination of accuracy, early arrival at the right answer, and well-calibrated predictions (i.e., reliability). The particular weights of these components are at the discretion of the tournament organizers.

Questions in our tournament involved topics such as the 2016 election, Nobel-prize recipients, Ohio University football and volleyball games, the price of gasoline, first snowfall in Athens, OH, the number of hurricanes in hurricane season, and the Booker-prize winner. The diverse set of topics ensured that disciplinary expertise was not any great advantage in the tournament.

To run the tournament, Ohio University partnered with Cultivate Labs, a startup based in Chicago, IL. Cultivate provides a prediction-tournament platform and consulting services. They work mainly with companies that use internal tournaments to predict key metrics, competitive activity, and other critical business information. Their platform gave us a broad suite of tools to create new questions whenever we wanted, analyze the data that came from forecasting activity, and, importantly, keep students engaged by sending reminders and other prompts to make regular forecasts.

However, we did not just rely on Cultivate’s technology to spur interest. First, we endeavored to make the questions we asked compelling to our student audience. We also purchased advertising time on screens in our Student-Union building, put up posters around campus, encouraged other faculty members to tell their students, and sent out a campus-wide email to all students announcing the tournament. We also hosted two on-campus events where we invited experts from outside the university to speak about forecasting and prediction tournaments.  Finally, we provided a financial incentive by awarding the top-five predictors from each week a $15 gift card.  A weekly email to all participants publicly congratulated the winners and reminded everyone about the prizes.  

In our first semester running the tournament, 174 students signed up and made at least one prediction. From a marketing perspective, the most successful way of generating initial interest was the campus-wide email announcement. The number of students making a prediction in the first week was 158. This dropped off to an average near 30 for each of the remaining 6 weeks. Anecdotally, the students who remained engaged said they did so to try and win the gift-card prizes and because the questions themselves were interesting.

Overall, we were pleased with the tournament and eager to make improvements for our next “season”, which will take place in Spring 2017. Given the drop-off in the number of active students after the first week, our next challenge is to design better strategies to maintain engagement. Our data suggests that those not initially successful in the tournament eventually dropped out.  In Season Two, we will implement a new prize structure that encourages those who are not good forecasters initially to remain active.  We will also give students more individualized information about their performance. We believe this will simultaneously help them become better at predicting and encourage further participation–much the way people who review their personal health dashboards are motivated to meet their daily step count.

Given our early success and the changes we will make for Season Two, we are confident future prediction tournaments will improve the critical-thinking skills of a sizable segment of our student body. Our ultimate goal is to enable students to apply the critical-thinking skills they sharpen in the tournament across a broad array of academic and non-academic settings. Ohio University will then be delivering on the promise which all colleges and universities make to their students: that their education will make them critical thinkers.