Yang Song, Yunkai Xiao, Jonathan Stephens, Emma Ruesch, Sean Roginski, Lucas Layman
Yang Song, Yunkai Xiao, Jonathan Stephens, Emma Ruesch, Sean Roginski, and Lucas Layman. 2020. Suitability of SCS1 as a Pre CS2 Assessment Instrument: A Comparison with Short Deliberate practice Questions. In 2020 ACM Southeast Conference (ACMSE 2020), April 2 4, 2020, Tampa, FL, USA. ACM, New York, NY, USA, 2 pages. https://doi. org/10.1145/3374135.3385277
Publication year: 2020

In an entry-level programming course, the instructor needs to assess students’ prior knowledge or to evaluate their learning at important milestones. The Second CS1 Assessment (SCS1) is one of the best-known validated tests of programming knowledge. It is a multiple-choice test written in pseudo-code, covering concepts commonly presented in Computer Science 1 (CS1) courses. However, SCS1 is known to be an unwieldy assessment – the questions tend to be difficult and thereby may not provide good discrimination between students with average competencies against the weak ones. In our project, we created a set of short deliberate-practice questions for CS1 topics and used them to evaluate students’ prior knowledge at the beginning of a Computer Science 2 (CS2) course. This set of deliberate-practice questions consists of 127 multiple-choice questions.

Both SCS1 questions and our short deliberate practice questions were given to CS2 students as pre CS2 assessments on an online practice system called HawkQB. Using Item Response Theory (IRT), we analyzed students’ responses in both sets of questions as pre CS2 assessments to examine whether the SCS1 questions are suitable to be used as a pre-CS2 assessment instrument. We found that the SCS1 questions are of greater difficulty and low discrimination compare with our short deliberate-practice questions based on the data from students in four sections of CS2 in fall 2019. Considering the time gap between CS1 and CS2 may vary, we question the suitability of SCS1 as a pre-CS2 assessment used in a formal test setup in the first or second week of CS2 courses.