How are paper-pencil and digital test achievements different? A systematic review of the experiences from PISA, TIMSS and PIRLS

Abstract

The National Assessment of Basic Competencies (NABC) is the largest student assessment operating in Hungary. The assessment has been measuring students in public education for twenty years. It assesses mathematical literacy and reading comprehension previously in two ( 6th and 8th), now in three grades (6th, 8th and 10th ). NABC was implemented for the first time with an electronic data collection in 2022. This is following the trends of international large scale student assessments (ILSA). PISA, TIMSS and PIRLS, in which Hungary typically participates since their beginning, have been changed to computer-based assessments during recent measurement cycles. An aim of the assessments above is to compare countries or grades and to report trends. To ensure the continuity, it is necessary to examine the possible difference between the two delivery methods, which is called mode effect. Since NABC typically follows the methodology of the ILSAs, we assumed that mode effect studies related to them could be a good source of information for the domestic assessment. Evidence can be in the own documents of the assessments or in independent analyses. Our research question was what scientific evidence can be obtained in connection with the PISA, TIMSS and PIRLS assessments in the domestic and international scientific space or in the documents of the ILSAs regarding the differences between paper-based and computer-based delivery. The documentation of the assessments typically showed a mode effect of 10-20 points. To ensure the continuity of trends mode effect was treated uniformly in forming trends, except for a few countries or items. However, critical studies point out that the mode effect may differ from country to country, leading to a false conclusion drawn from trends.