Analyzing Assessment Items

By Dr. Pooja Shivraj, RME Educational Assessment Researcher

Much of the work we do at Research in Mathematics Education involves the development of assessments used by educators to identify students who may be struggling with algebra-readiness knowledge and skills, so that teachers can provide additional instructional support. The research process we use is rigorous and begins with an assessment blueprint, then item writing, internal reviews and external expert reviews, followed by a pilot test and finally the development of the test forms. The pilot test is given to a large number of students in order to determine the validity of the assessment items. Our researchers receive the results of the pilot and perform an extensive statistical analysis to determine if an item is good, psychometrically speaking.

The point of obtaining item statistics is to develop a pool of items that function well from which future tests can be designed. There are two kinds of analyses that can be performed: a Classical Test Theory (CTT) analysis, which is sample-dependent and non-model based, or an Item Response Theory (IRT) analysis, which is sample-independent and model-based. Regardless of the type of analysis performed, three primary statistics are used to determine if an item is psychometrically good. The ranges listed below are the acceptable norms found in the literature.

(1) The item should have a strong correlation between each item score and the total score. In other words, the correlation should show that the test-takers choosing the correct answer on the item are likely to receive a higher score. This statistic is measured by the point-biserial correlation (CTT) or the point-measure correlation (IRT). A good item would have a point-biserial correlation of >0.2 or a point-measure correlation of >0.25.
(2) The difficulty of the item, measured by the proportion of students answering the item correctly (CTT), should be between 30% to 80% of the test-takers. In IRT, the difficulty parameter, b, should be between -4 and +4.

An item characteristic curve depicting the discrimination parameter
(a) and the difficulty parameter (b) in an IRT model

(3) The discrimination of the item, also measured by the point-biserial correlation (CTT) should be higher for the correct response than the distractors. In IRT, the discrimination parameter, a, should be between 0.5 and 1.5. The greater the discrimination, the better the item discriminates between lower ability and higher ability students.

What can you do with items that don't function well? For the items that don't function well, reviewing the data would be the first step. Are the items functioning poorly because the majority of students are choosing the correct answer? Is one distractor not being chosen at all? Are the majority of students choosing a single distractor more often than other options? These data would all be red flags. The next step would be to review the content of all the items that don't function well, especially the items that were flagged in the previous step. What about the content led students to choose or not choose a particular response choice?

Using this process of analyzing data, reviewing items, and adjusting the content of the items, a pool of items that function well can be developed for use in the future.

Note: Many other statistics (e.g., fit statistics in IRT like Chi square, infit, outfit, etc.) could be used to determine if an item functions well in addition to the ones described above that could also provide information at the test level. Please feel free to email me if you would like more information at pshivraj@smu.edu.

Benjamin Banneker Week

By Brea Ratliff, RME Secondary Mathematics Coordinator

For many students, mathematics is viewed as a faceless, and sometimes meaningless, course of study, but learning more about the fascinating and prodigious minds who have shaped the subject can be inspiring. In the face of doubt, criticism, failure, and even seemingly impossible circumstances, many great men and women have been intellectual trailblazers whose extraordinary contributions to society are a testament to the power and importance of teaching mathematical processes and critical thinking. One such individual was African-American mathematician, author, scientist, agriculturalist, astronomer, publisher, and urban planner, Benjamin Banneker.

Image from
http://www.bnl.gov/bera/activities
/globe/banneker.htm

Benjamin Banneker was born outside of Baltimore, Maryland on November 9, 1731. He was born a free black, and was generally self-taught through most of his young adult life. Banneker began to display his brilliance as an engineer while he was a young man; first through his often noted affinity toward solving puzzles, and later through his mathematically-perfect creation of the first clock made entirely of hand carved wooden parts and pinions (Washington Interdependence Council, 2014). This clock, which Banneker built after carefully studying a borrowed pocket watch, accurately kept time for decades.

Benjamin Banneker’s love for learning encouraged him to begin studying astronomy and advanced mathematics from sets of books loaned to him by a neighbor. As a result of these studies, he was able to accurately predict solar and lunar eclipses, and became the author of an internationally published almanac, which contained his many scientific and mathematical calculations. The international recognition of his almanac also served as a springboard for Banneker to become a recognized proponent for the abolishment of slavery. He famously composed a letter addressed to Thomas Jefferson, in which he insisted black Americans possess the same intellectual ability and should be afforded the same opportunities as white Americans (Chamberlain, 2012). This letter led to an ongoing correspondence between the two men, and led to Banneker receiving a considerable amount of support by abolitionist groups in Maryland and Pennsylvania (Biography, 2014).

Banneker was also selected to assist Major Pierre L’Enfant to survey and develop the city plans for our nation’s capital, which was later named the District of Columbia. After L’Enfant abruptly quit the project, Benjamin Banneker was able to reproduce the plans – from memory - for the entire city in just 2 days. These plans provided the layout for the streets, buildings, and monuments that still exist in Washington D.C. (Chamberlain, 2012).

During the week of November 9th through the 15th, individuals and groups across the nation will honor the many contributions of this great mathematician by celebrating “Benjamin Banneker Week”. The Benjamin Banneker Association, an organization dedicated to mathematics education advocacy by providing support and leadership for educators and students in order to ensure equity exists for all students, is sponsoring a mathematical task competition to continue his legacy.

Schools, libraries, community and professional organizations, or interested citizens are urged to make mathematics a significant part of children’s lives by coordinating a Benjamin Banneker Celebration event in their communities. Visit the Benjamin Banneker Day website (www.benjaminbannekerday.weebly.com) to learn more about Benjamin Banneker, and how you and your community can participate in this year’s celebration.

Benjamin Banneker: A Memorial to America’s First Black Man of Science (2014). Retrieved Oct 13, 2014 from http://www.bannekermemorial.org/history.htm

Benjamin Banneker. (2014). The Biography.com website. Retrieved Oct 13, 2014, from http://www.biography.com/people/benjamin-banneker-9198038.

Chamberlain, G. (2012) Benjamin Banneker – The Black Inventor Online Museum. Retrieved Oct 13, 2014 from http://blackinventor.com/benjamin-banneker/

Bringing the Associative Property of Multiplication to Life

By Cassandra Hatfield, RME Assessment Coordinator, and Megan Hancock, Graduate Research Assistant

The Institute of Education Science (IES) Practice Guide for Improving Mathematical Problem Solving in Grades 4 through 8 Recommendation five states that it is important to “help students recognize and articulate mathematical concepts and notation” (Woodward et al., 2012). One way to carry out this recommendation is to “ask students to explain each step used to solve a problem in a worked example” and “help students make sense of algebraic notation” (Woodward et al., 2012).

The Associative Property of Multiplication will illustrate this recommendation by going beyond a procedural skill and making connections conceptually that support the symbolic notation. Our goal is to give evidence that the Associative Property of Multiplication can be taught through multiple representations. Through our research we found that some representations are mathematically accurate, but may not provide students with a compelling reason to use this property.

When developing the concept of volume of rectangular prisms, decomposing the rectangular prism into layers allows students to make the connection with content they are already familiar with, arrays and area. This decomposition also exemplifies the Associative Property of Multiplication. Here are some examples of how the rectangular prism shown above can be decomposed in different ways.

 

• A: 2 × (6 × 4)
• B: (2 × 6) × 4
• C: Supports commutative property of multiplication too
• 2 × 6 × 4; 2 × 4 × 6; (2 × 4) × 6

By designing activities and lessons that support the decomposition of rectangular prisms into different layers, teachers can support students in making sense of the notation of Associative Property of Multiplication, A x (B x C) = (A x B) x C, and finding the volume of rectangular prisms. Explorations like this also support teachers in holding students accountable for understanding the notation because students can use the different models to support their explanation of their understanding.

Woodward, J., Beckmann, S., Driscoll, M., Franke, M., Herzig, P., Jitendra, A., Koedinger, K. R., & Ogbuehi, P. (2012). Improving mathematical problem solving in grades 4 through 8: A practice guide (NCEE 2012-4055). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from http:// ies.ed.gov/ncee/wwc/publications_reviews.aspx#pubsearch/.