What’s in a Name? Think-a-Loud Protocol

By Dawn Woods, RME Elementary Mathematics Coordinator, and Marilea Jungman, RME Project Specialist

In January, Research in Mathematics (RME) conducted think-aloud protocols with second, third and fourth grade students at Nebbie Williams Elementary School in Rockwall ISD. The school volunteered to participate in the think-aloud protocol study that was designed to enrich the development of the ESTAR Universal Screener.

Background: What is the ESTAR Universal Screener?

The Elementary Students in Texas Algebra Ready, or ESTAR, is the latest intitative within the Texas Algebra Ready (TXAR framework) to support elementary students in the state of Texas to achieve a high level of preparedness in mathematics. The ESTAR Universal Screener is being designed to help educators identify students who may need additional support in becoming algebra-ready in the elementary grades and will be aligned with algebra-readiness knowledge and skills articulated in the revised Texas Response to the Curriculum Focal Points.

This document, based on the revised TEKS adopted in April 2012, identified critical areas of mathematics instruction in a framework for sequencing and developing curricula at each grade level. This document provides the content of the ESTAR Universal Screener and will be organized around foundational, bridging, and target knowledge and skill levels and simultaneously includes items written to target four levels of cognitive complexity - research indicates 4 areas critical for mathematics success: procedural understanding, conceptual understanding, strategic competence, and adaptive reasoning. Data generated from the screener will be reported in a format that helps teachers make informed decisions about the content and structure of mathematics instruction in the classroom.

Why is the ESTAR Universal Screener important?
Although performance standards are in process of being established for the State of Texas Assessments of Academic Readiness (STAAR), data from 2012 indicate that 3rd grade students responded, on average, to only 30 of 49 mathematics items correctly (61%) while 4th grade students responded, on average, to only 32 of 48 (or 66%) of mathematics items correctly (Texas Education Agency, 2012). These data speak to a need for early identification of students who may be struggling to learn critical mathematics content. One of the research steps in developing the Universal Screener is to conduct student interviews, also known as think-aloud protocols.

What is a Think-Aloud Protocol?
The purpose of a student interview or think-aloud protocol is to transform a student’s covert thinking process into an observable behavior so that the thinking process can be documented and analyzed (van Someren, Barnard, & Sandberg, 1994). Basically, we ask the student to work through a small number of math items appropriate to their grade level, and to “think-aloud” as they work. This concurrent data capture is maximized through the notes and reflections of the interviewer, the use of audio/visual, and a field observer dedicated to recording the student’s thoughts, hesitations, and gestures verbatim.

Once the student solves a math item, the interviewer asks the student to reflect on his or her thinking process after the task is completed. This is called retrospective data collection. Here, the interviewer uses questioning, prompting or dialogues to encourage the student to talk about his or her thoughts about the math item. The repetitive nature in the questioning allows student’s initial thoughts to be repositioned, and in many instances, a student alights on the correct answer after a first-round wrong choice.

We use the student interviews to verify and provide validity evidence of misconceptions captured in item designs as well as learn how metacognition (Flavell, 1979) plays a role in the planning and strategies students use in mathematical problem solving. Furthermore, student interviews also provide valuable information about students’ sense of self-efficacy, which may be correlated to student’s academic achievement (Hackett & Betz, 1989) and predicts later success for elementary school students (Bandura, 1997; Joet, Usher, & Bressoux, 2011).

Summing It Up
Think-aloud protocol is a valuable qualitative research method that enables researchers to uncover and map thinking processes. As we begin to analyze our data from our project at Nebbie Williams Elementary School, we hope to not only provide validity to our items for the ESTAR project, but gain valuable insight into how students think about math and how metacognition and self-efficacy play a role in students mathematics achievement.

Thank you again to the wonderful teachers that helped us with our project!

3rd Grade
Ms. Jennifer McCurry and Ms. Melody Carrilo

4th Grade
Ms. Christine Gregory and Ms. Lana Edwards

2nd Grade
Dr. Marcella J. Hodges and Ms. Kathleen Elam

Bandura, A. (1997). Self-efficacy: The exercise of control. U.S.A.: Macmillan.  

Flavell, J. H. (1979) Metacognition and cognitive monitoring: A new area of cognitive-developmental inquiry. American Psychologist 34(10). 906-911.  

Hackett, G. & Betz, N. E. (1989) An exploration of the mathematics self-efficacy/mathematics performance correspondence. Journal for Research in Mathematics Education 20(3). 261-273.  

van Someren, M. W., Barnard, Y. F., & Sandberg, J. A.C. (1994). The think aloud method: A practical guide to modeling cognitive processes. London: University of Amsterdam, Department of Social Science Informatics. 

Tweet