Message: TAC Materials

TAC Materials
FromSireno, LisaDate  Monday, July 3, 2017 10:25 AM
To
Neale, Chris;Preis, Stacey;Strand, Jocelyn;Henningsen, Blaine
Cc
SubjectTAC Materials

Hello,

 

See attached for the materials Questar has submitted for our TAC’s review prior to Wednesday’s discussion.

 

Lisa

 

Lisa Sireno | Standards and Assessment Administrator | Office of College and Career Readiness

Missouri Department of Elementary and Secondary Education | 573-751-3545 | dese.mo.gov

 


Deriving Scale Scores for Students with Unscorable Algebra I Performance Event (PE) Responses Presented to DESE | June 29, 2017 1. Introduction This document describes how scale scores were derived for students whose Session 2 Algebra I performance event (PE) responses were rendered unscorable due to technological issues and includes additional analyses performed at the request of the Missouri Department of Elementary and Secondary Education (DESE) and Technical Advisory Committee (TAC). There were 176 students affected by the technology issue. The DESE requested that Questar move forward to produce student reports to prevent reporting delays. Questar statistically linked student performance on Session 1 of the Algebra I test to student performance on the full Algebra I test. Session 1 consisted of 40 machine-scored items (40 points), and Session 2 consisted of one hand-scored PE (10 points) for a maximum score of 50 points. The statistical linking process placed the scores earned on Session 1 onto the scale used to report scores on the full test. Thus, the linked Session 1 scale scores are comparable to scale scores from the total test. Due to score reliability and content differences, the linked scale scores will not be considered completely exchangeable with the full test scale scores. In other words, it will not be a matter of indifference to individual students whether their scale scores are based on Session 1 only or on the full test. Additional analyses were conducted at the request of DESE and TAC during the June 2, 2017 conference call. The additional results include raw score summary statistics for the general population (i.e., students not affected by the PE issue) and the students affected by the PE issue, correlations of the scores from the two test sections, and actual and projected PE score results for the non-affected students. 2. Test Design Table 2.1 provides the content representation of Session 1, Session 2, and the full test. The Session 2 points are distributed across the Algebra and Functions reporting categories. Page 1 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Table 2.1. Content Representation of the Algebra Test Reporting Category Session 1 Session 2 Full Test #Points % #Points % #Points % Algebra Functions Numbers & Operations Statistics/Probability 17 15 3 5 42.5 37.5 7.5 12.5 6 4 ---- 60.0 40.0 ---- 23 19 3 5 46.0 38.0 6.0 10.0 Total 40 100.0 10 100.0 50 100.0 3. Linking Procedure Two facts are important for understanding the linking plan: 1. The Session 1 items cover all reporting categories and represent 80 percent of the Algebra 1 test. 2. The Session 1 items are part of the Algebra 1 calibrated item pool. Because of these facts, a Session 1 raw-to-scale score (RSS) conversion table can be derived through a well-known process known as fixed-common-item parameter (FCIP) linking using the following steps. Step 1. Obtain student ability estimates (i.e., the raw-to-logit score table) for Session 1 by anchoring all Session 1 items to their banked values. The result is a raw-to-logit score table generated by Winsteps. Each student’s Rasch logit-ability score can be obtained from this table based on his or her number-correct score. Step 2. Perform inverse mapping of theta values from Session 1 to the Algebra 1 RSS table to obtain a projected total test score for students without scorable PE responses. Based on this concordance table, students without PE scores will be assigned total raw scores, scale scores, and conditional standard errors of measurement (CSEMs) based on the RSS table for the full test. Step 3. Calculate Session 2 raw scores by subtracting the projected total test raw scores from the Session 1 raw scores. 4. Results The results were independently replicated by two Questar psychometricians. Table 4.1 presents the Algebra I RSS table. Table 4.2 shows the alignment of the raw-to-theta values for Session 1 and the full test, as well as the difference of raw score, theta, and standard error (SE) values. Page 2 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Table 4.1. Algebra I RSS Conversion Table Raw Score Theta SE Scale Score CSEM Performance Level Theta Cut 0 -5.1536 1.8519 100 40 Below Basic 1 -3.8844 1.0442 107 22 Below Basic 2 -3.1048 0.7633 124 16 Below Basic 3 -2.6201 0.6399 134 14 Below Basic 4 -2.2591 0.5663 142 12 Below Basic 5 -1.9676 0.5160 148 11 Below Basic 6 -1.7208 0.4790 153 10 Below Basic 7 -1.5054 0.4503 158 10 Below Basic 8 -1.3132 0.4272 162 9 Below Basic 9 -1.1388 0.4084 166 9 Below Basic 10 -0.9786 0.3926 169 8 Below Basic 11 -0.8298 0.3792 172 8 Below Basic 12 -0.6904 0.3677 175 8 Below Basic 13 -0.5589 0.3578 178 8 Below Basic 14 -0.4340 0.3492 181 7 Below Basic 15 -0.3147 0.3418 183 7 Below Basic 16 -0.2002 0.3352 186 7 Below Basic 17 -0.0897 0.3295 188 7 Basic -0.1359 18 0.0171 0.3244 190 7 Basic 19 0.1209 0.3201 193 7 Basic 20 0.2221 0.3162 195 7 Basic 21 0.3211 0.3129 197 7 Basic 22 0.4181 0.3101 200 7 Proficient 0.4615 23 0.5134 0.3076 201 7 Proficient 24 0.6074 0.3056 203 7 Proficient 25 0.7002 0.3039 205 7 Proficient 26 0.7921 0.3025 207 6 Proficient 27 0.8833 0.3015 209 6 Proficient 28 0.9740 0.3009 211 6 Proficient 29 1.0644 0.3006 213 6 Proficient 30 1.1548 0.3008 215 6 Proficient 31 1.2455 0.3014 217 6 Proficient 32 1.3367 0.3027 219 6 Proficient 33 1.4289 0.3046 221 7 Proficient 34 1.5224 0.3073 223 7 Proficient 35 1.6180 0.3110 225 7 Advanced 1.6282 36 1.7162 0.3159 227 7 Advanced 37 1.8178 0.3221 229 7 Advanced 38 1.9240 0.3299 231 7 Advanced Page 3 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Raw Score Theta SE Scale Score CSEM Performance Level Theta Cut 39 2.0360 0.3398 234 7 Advanced 40 2.1555 0.3520 236 8 Advanced 41 2.2847 0.3673 239 8 Advanced 42 2.4264 0.3862 242 8 Advanced 43 2.5846 0.4100 245 9 Advanced 44 2.7649 0.4403 249 9 Advanced 45 2.9757 0.4796 250 10 Advanced 46 3.2306 0.5327 250 11 Advanced 47 3.5544 0.6098 250 13 Advanced 48 4.0013 0.7378 250 16 Advanced 49 4.7419 1.0251 250 22 Advanced 50 5.9831 1.8409 250 39 Advanced Note. Cut scores are highlighted in yellow. Table 4.2. Alignment of Session 1 and Algebra I Raw Score to Theta Scores Session 1 Algebra I Full Test Difference* Raw Score Theta SE Raw Score Theta SE Raw Score Theta SE 0 -5.1280 1 -3.8552 2 -3.0704 3 -2.5803 4 -2.2138 5 -1.9164 6 -1.6636 7 -1.4417 8 -1.2427 9 -1.0610 10 -0.8929 11 -0.7357 12 -0.5872 13 -0.4458 14 -0.3101 15 -0.1792 16 -0.0521 17 0.0720 18 0.1938 19 0.3139 20 0.4330 1.8533 1.0467 0.7668 0.6441 0.5713 0.5217 0.4854 0.4576 0.4355 0.4176 0.4028 0.3906 0.3804 0.3719 0.3648 0.3590 0.3542 0.3505 0.3477 0.3457 0.3446 0 -5.1536 1.8519 1 -3.8844 1.0442 2 -3.1048 0.7633 3 -2.6201 0.6399 4 -2.2591 0.5663 5 -1.9676 0.5160 6 -1.7208 0.4790 7 -1.5054 0.4503 8 -1.3132 0.4272 9 -1.1388 0.4084 11 -0.8298 0.3792 12 -0.6904 0.3677 13 -0.5589 0.3578 14 -0.4340 0.3492 15 -0.3147 0.3418 16 -0.2002 0.3352 17 -0.0897 0.3295 19 0.1209 0.3201 20 0.2221 0.3162 21 0.3211 0.3129 22 0.4181 0.3101 0 0.0256 0 0.0292 0 0.0344 0 0.0398 0 0.0453 0 0.0512 0 0.0572 0 0.0637 0 0.0705 0 0.0778 -1 -0.0631 -1 -0.0453 -1 -0.0283 -1 -0.0118 -1 0.0046 -1 0.0210 -1 0.0376 -2 -0.0489 -2 -0.0283 -2 -0.0072 -2 0.0149 0.0014 0.0025 0.0035 0.0042 0.0050 0.0057 0.0064 0.0073 0.0082 0.0091 0.0236 0.0229 0.0226 0.0227 0.0230 0.0238 0.0247 0.0304 0.0315 0.0328 0.0345 Page 4 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Session 1 Algebra I Full Test Difference* Raw Score Theta SE Raw Score Theta SE Raw Score Theta SE 21 0.5516 0.3443 23 0.5134 0.3076 -2 0.0382 0.0367 22 0.6703 0.3449 25 0.7002 0.3039 -3 -0.0299 0.0410 23 0.7897 0.3463 26 0.7921 0.3025 -3 -0.0024 0.0438 24 0.9104 0.3487 27 0.8833 0.3015 -3 0.0271 0.0472 25 1.0331 0.3520 29 1.0644 0.3006 -4 -0.0314 0.0514 26 1.1584 0.3564 30 1.1548 0.3008 -4 0.0036 0.0556 27 1.2874 0.3620 31 1.2455 0.3014 -4 0.0419 0.0606 28 1.4209 0.3691 33 1.4289 0.3046 -5 -0.0080 0.0645 29 1.5602 0.3777 34 1.5224 0.3073 -5 0.0378 0.0704 30 1.7068 0.3884 36 1.7162 0.3159 -6 -0.0094 0.0725 31 1.8627 0.4015 37 1.8178 0.3221 -6 0.0449 0.0794 32 2.0303 0.4178 39 2.0360 0.3398 -7 -0.0057 0.0780 33 2.2132 0.4383 40 2.1555 0.3520 -7 0.0577 0.0863 34 2.4166 0.4646 42 2.4264 0.3862 -8 -0.0098 0.0784 35 2.6482 0.4995 43 2.5846 0.4100 -8 0.0636 0.0895 36 2.9211 0.5478 45 2.9757 0.4796 -9 -0.0546 0.0682 37 3.2594 0.6201 46 3.2306 0.5327 -9 0.0288 0.0874 38 3.7167 0.7435 47 3.5544 0.6098 -9 0.1623 0.1337 39 4.4637 1.0272 49 4.7419 1.0251 -10 -0.2782 0.0021 40 5.7070 1.8413 50 5.9831 1.8409 -10 -0.2761 0.0004 *Difference is calculated by Session 1 minus Algebra I full test values. Note. Cut scores are highlighted in yellow. Table 4.3 shows the number and percent of students at each achievement level for the general population and the students with malformed PE responses. The group of students affected by the malformed PE responses was lower-performing than the students not affected by the PE readability issue, as indicated by the percent of students achieving the Proficient and Advanced level (17.05 versus 60.40, respectively). Page 5 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Table 4.3. Achievement Level Distribution—Spring 2017, Algebra I Content Area Achievement Level General Population* Malformed PE Students Percent Difference* Freq. % Freq. % Algebra I Below Basic Basic Proficient Advanced Below Basic + Basic Proficient + Advanced 12,392 20.46 11,601 19.15 25,069 41.38 11,519 19.01 23,993 39.60 36,588 60.40 102 57.95 44 25.00 28 15.91 2 1.14 146 82.95 30 17.05 -37.49 -5.85 25.47 17.87 -43.35 43.35 Total 60,581 100 176 100 -- *General population includes students not affected by the malformed PE responses. **Difference is calculated by the population minus students with the malformed PE responses. Table 4.4 shows descriptive statistics for the general population and the PE-affected students for Session 1, Session 2, and total raw score. Results indicate that performance on Session 1 and the total test for the PE-affected students is about one standard deviation lower than the general population. The mean of 3 out of 10 for the general population shows that the PE task is difficult for the students. Table 4.4. Raw Score Descriptive Statistics by Group Session Group N Minimum Maximum Mean Std. Deviation 1 General Population PE-Affected Students 60,564 176 0 5 40 33 22.20 14.92 7.57 5.40 2 General Population PE-Affected Students* 60,460 176 0 0 10 7 3.04 1.43 2.61 1.17 Total Score General Population PE-Affected Students* 60,581 176 0 5 50 40 25.23 16.35 9.44 6.50 Note: General population does not include the PE-affected students. * The linking procedure was used for these results. The correlation coefficients between the test sessions and total test score are shown in Tables 4.5 and 4.6 for the general population and the PE-affected students, respectively. The correlations between Session 1 and the total test are high for both group of students (i.e., greater than 0.97). The correlations between Session 2 and the total test scores are higher for the PE-affected students (r=0.953) than the general population (r=0.780). Page 6 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Table 4.5. Correlation Coefficients of Test Sessions for the General Population #Points Session 1 Session 2 Total Test Session 1 40 1.000 0.627 0.976 Session 2 10 0.627 1.000 0.780 Total Test 50 0.976 0.780 1.000 Table 4.6. Correlation Coefficients of Test Sessions for the PE-Affected Students #Points Session 1 Session 2 Total Test Session 1 40 1.000 0.931 0.998 Session 2 10 0.931 1.000 0.953 Total Test 50 0.998 0.953 1.000 The method of calculating Session 2 scores based on Session 1 scores was applied to the general population. The correlation between the actual Session 2 scores and the projected Session 2 scores is 0.634. The difference distribution between the PE score and the projected PE score is shown in Table 4.7. The PE difference scores range from 9 to -10. At the extremes of the distribution range, two students obtained scores of 9 on the PE and scores of 0 for the projected score based on the 40 point Session 1 results (PE difference score of 9) and one student obtained a score of 0 on the PE but the projected score was 10 based on his/her Session 1 result (PE difference score of -10). About 78 percent of the difference scores fell between 2 and -2 and 89 percent of difference scores fell between 3 and -3. The results presented graphically in Figure 1 approximate a normal distribution. Finally, Figure 2 provides a scatter plot of the predicted Session 2 score on the actual Session 2 score for the general population. The bulk of the observations are in the lower left hand corner (low scores on both the x and y-axis). As suggested in Table 4.7, very few students had x, y pairs that differed more than 4 points. Page 7 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Table 4.7. Frequency Distribution of PE Score Difference PE Score Difference* Frequency Percent Cumulative Frequency Cumulative Percent 9 2 0.00 2 0.00 8 28 0.05 30 0.05 7 110 0.18 140 0.23 6 330 0.55 470 0.78 5 786 1.30 1,256 2.08 4 1,499 2.48 2,755 4.56 3 2,908 4.81 5,663 9.37 2 5,077 8.40 10,740 17.77 1 8,612 14.25 19,352 32.02 0 12,612 20.87 31,964 52.88 -1 13,089 21.66 45,053 74.54 -2 7,701 12.74 52,754 87.28 -3 3,857 6.38 56,611 93.66 -4 2,149 3.56 58,760 97.22 -5 1,017 1.68 59,777 98.90 -6 444 0.73 60,221 99.63 -7 162 0.27 60,383 99.90 -8 50 0.08 60,433 99.98 -9 9 0.01 60,442 100.00 -10 1 0.00 60,443 100.00 * Score difference is calculated by actual minus projected score. Page 8 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Figure 1. PE score difference for the actual minus projected PE results Score Difference 14,000 12,000 10,000 8,000 6,000 4,000 2,000 - 9 8 7 6 5 4 3 2 1 0 -1 -2 -3 -4 -5 -6 -7 -8 -9 -10 Page 9 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses Figure 2. Projected Session 2 Scores on Actual Session 2 Scores P r o j e c t e d 0 1 2 3 4 5 6 7 8 9 10 Actual 0 1 2 3 4 5 6 7 8 9 10 Missouri Actual vs Projected Session 2 Scores Size of Bubbles are Relative to N Count 934 4549 2694 939 555 181 100 47 28 7 1 489 4309 3521 1453 996 382 251 112 45 20 2 284 2935 2959 1547 1176 557 405 245 119 55 2 104 1241 1710 1158 1012 605 385 276 146 96 15 20 499 1060 872 946 629 507 356 273 169 17 4 246 653 731 871 604 548 457 308 218 25 4 97 347 439 586 412 434 385 306 235 40 84 313 411 661 515 564 554 526 450 81 15 85 110 177 181 230 268 246 272 53 1 18 80 124 219 222 273 310 342 402 100 1 10 15 33 43 76 98 120 149 66 Note. Pearson’s r = 0.634. Size of bubbles are relative to the n-count. Page 10 of 11 Deriving Scale Scores for Unscorable Algebra I PE Responses 5. Conclusion This document outlined the procedure for deriving scale scores for students whose Session 2 Algebra I PE responses were rendered unscorable due to technological issues. The results of the analyses suggest that the approach is sound, based on the high correlations between the Session 1 and Session 2 scores and the normally distributed residuals of the projected scores and the actual scores for students that were not affected by the malformed PE responses. Page 11 of 11
MO EOC Spring 2017 Online Testing Network Slowdown MO EOC Spring 2017 Online Testing Network Slowdown Presented to DESE | June 29, 2017 1. Introduction This document presents descriptive analyses conducted to investigate the network slowdown during the Spring testing window for the Missouri End-of-Course (MO EOC) assessments. The network slowdown occurred between 8:00 a.m. and 10:00 a.m. on Tuesday, April 25, 2017 and was caused by a high volume of test takers. The slowdown affected 19,567 students across 338 (of 822) schools in 265 (of 526) districts. No item responses were lost, but two groups of students were impacted:  Group 1: Students who were actively testing when they experienced the slowdown (n = 5,664)  Group 2: Students who attempted to log in but were unsuccessful and could not begin testing (n = 13,903) The results provided here were requested by DESE and the TAC during the June 2, 2017 conference call. The analyses were conducted to further explore the impacted students. Specifically, the following analyses are presented:  Demographic representation of the students in Groups 1 and 2, non-slowdown and general population  Achievement level trends for Spring 2015, 2016, and 2017, including Groups 1 and 2 and the non-slowdown students for Spring 2017  Prior year achievement level results for Groups 1 and 2  State and school results by Group for Spring 2017  State and district results for Spring 2015, 2016, and 2017 2. MO EOC Assessment Overview For accountability purposes, districts are required to administer the MO EOC English II, Algebra I, Biology, and Government Assessments to all students prior to graduation. For students who completed the Algebra I assessment before high school, Algebra II is the required high school Mathematics assessment. The MO EOC Assessments are currently intact pre-equated forms. Two or more forms are rotated across administrations for all content areas except English I, Geometry, and Physical Science that only have one form. Table 2.1 presents the forms used for the Spring 2015, 2016, and 2017 administrations to facilitate interpretation of the data. Page 1 of 4 MO EOC Spring 2017 Online Testing Network Slowdown Table 2.1. MO EOC Spring Forms Content Area 2015 2016 2017 English II 2014H 2014G 2014H Algebra I H G H Biology A C A English I 7 7 7 Algebra II 8 7 8 Geometry 7 7 7 Government 5 1 5 American History 3 1 3 Physical Science 1 1 1 Where historical data are presented, the results for Spring 2015, 2016, and 2017 are provided. The results for Spring 2014 are not presented because test forms changed for the math and English tests (i.e., Algebra I, Algebra II, Geometry, English I, English II) beginning in the Fall 2014 and Physical Science was introduced in Fall 2014. 3. Results 3.1 Demographic Representation Table 3.1 (see the attached Excel document) shows the number and percent of students for gender and ethnicity categories for Group1, Group 2, non-slowdown students, and the population. The other demographic information (migrant status, free and reduced lunch, limited English proficient, Title I, individualized education program, accommodations) are not available until the fall when the demographic file is delivered to Questar. The last two columns show the percent difference of Groups 1 and 2 compared to the non-slowdown students. Differences that exceeded 10 percentage points were highlighted. The threshold value of 10 was selected because the small n-count groups may have results that vary considerably due to random factors. Positive values were highlighted in yellow and negative differences were highlighted in green. No gender differences were flagged. Ethnicity differences were flagged for Groups 1 and 2. Group 1 differences showed a higher percent of black and Hispanic students for Algebra II, a higher percent of multi-racial students for American History, and a lower percent of white students for both content areas. The Group 2 comparison showed a lower percent of black students for Algebra I and a higher percent of white students for Algebra I and American History. The flagged differences occurred for small sample sizes, especially for Group 1 (i.e., 92 students for Algebra I, 9 students for American History). 3.2 Achievement Level Trends Table 3.2 (see the attached Excel document) shows the achievement level data for the Spring 2015, Spring 2016, and Spring 2017 results. As indicated in Table 2.1, common forms were used for the Spring 2015 and 2017 administrations for all content areas, and MO EOC Spring 2017 Online Testing Network Slowdown common forms across all three Spring administrations were used for English I, Geometry, and Physical Science. A consistent pattern of performance across content areas was not observed for the students who experienced network slowdown. Considering the percent differences at the Proficient + Advanced level, Group 1 had a higher percent of students for five content areas and Group 2 had a higher percent of students for three content areas compared to the non-slowdown students. Consistent with the demographic data, a 10 percentage point difference threshold for flagging was applied. Looking at the percent of students in the Proficient and Advanced performance level (Proficient + Advanced), there was a lower percent of Group 1 students compared to the non-slowdown students for American History (n = 9) and Geometry (n = 164). None of the Group 2 percent differences from the non-slowdown students were flagged using the 10 percent difference threshold for the Proficient + Advanced performance level. 3.3 Prior Achievement for Slowdown Groups The prior year’s performance level results of the slowdown students, Groups 1 and 2, are presented beside the Spring 2017 results in Table 3.3. Student records were searched for the Summer and Fall 2015 and Spring 2016 administrations within the same subject matter: for the math subjects (Algebra I, Algebra II, Geometry), English subjects (English I, English II), science (Biology, Physical Science), and social science (American History, Government). For instance, the prior year’s results for Geometry Group 2 consist of 161 Algebra I and 18 Algebra II (for a total n = 179). Very few records from last year were found for some content areas (e.g., Algebra I, English I), likely because the pre-requisite course is at the Grade 8 level. Over 60 percent of the cases had matches to prior year results for Algebra II, Group 2, and Geometry Groups 1 and 2. The percentage of students achieving the Proficient + Advanced level was within about 1 percent for Geometry and within about 5 percent for Algebra 2. Based on these results, there does not appear to be a decline of performance for this year. 3.4 State and School Results for Spring 2017 Table 3.3 (see the attached Excel document) shows the number and percent of students that achieved the Proficient + Advanced level by school. The data for the state is presented in the first row for each subject. 3.5 State and District Results Across Spring Administrations Table 3.4 (see the attached Excel document) shows the number and percent of students that achieved the Proficient + Advanced level by district across the Spring 2015, 2016, and 2017 administrations. The data for the state is presented in the first row for each subject. MO EOC Spring 2017 Online Testing Network Slowdown 4. Summary and Conclusion The results contained in this document are intended to shed light on the demographic characteristics and performance level results of the students who experienced the network slowdown. Based on the data presented, the demographic results and the performance level results indicated no consistent pattern for Group 1 and Group 2 compared to the students who did not experience the network slowdown. Results from the prior year indicate that students were performing similarly this year. Where larger subgroup differences were seen, the n-counts were smaller, raising the potential for those differences to be due to random factors.