Message: Sunshine Request

Sunshine Request
FromHagenhoff, MargaretDate  Tuesday, October 3, 2017 8:53 AM
To
Sireno, Lisa
Cc
SubjectSunshine Request

Lisa,

 

The attached documents are ready for your review for the Sunshine Law Request.  Thanks, 

 

Margie Hagenhoff| Administrative Assistant|Office of College and Career Readiness

Phone 573.751.3545 |Fax 573.526.0812|dese.mo.gov

 


Hagenhoff, Margaret From: Sent: To: Cc: Subject: 160.526.2 Bates, Shaun Thursday, April 13, 2017 1:18 PM Hagenhoff, Sara; Sirena, Lisa Hagenhoff, Margaret RE: TAC information 2. The state board of education shall by contract enlist the assistance of such national experts to receive reports, advice and counsel on a regular basis pertaining to the validity and reliability of the statewide assessment system. The reports from such experts shall be received by the state board of education. Within six months prior to implementation of or modification or revision to the statewide assessment system, the commissioner of education shall inform the president pro tempore of the senate and the speaker of the house of representatives about the procedures to implement, modify, or revise the statewide assessment system, including a report related to the reliability and validity of the assessment instruments, and the general assembly may, within the next sixty legislative days, veto such implementation, modification, or revision by concurrent resolution adopted by majority vote of both the senate and the house of representatives. From: Hagenhoff, Sara Sent: Thursday, April 13, 2017 12:50 PM To: Bates, Shaun; Sirena, Lisa Cc: Hagenhoff, Margaret Subject: TAC information Importance: High I'm looking for the statute that requires us to have a Technical Advisory Committee. Can any of you direct me to that? We need to send some documentation to OA to back up the members being appointed by the Commissioner, not bid out. Any help is appreciated and the sooner, the better ... Thanks! Sara Hagenhoff I Director of Procurement I Financial and Administrative Services I 573-751-4463 I dese.mo.gov 1
Name Email Address Phone Number Address City State Zip code Doar, Dr. Bertha Bertha.Doar@slQs.org 314-345-2360 18225 Prickly Pear Court Pacific MO 63069 Egan, Karla karlaegan@gmail.com 660-631-0843 120 North Brunswick Marshall MO 65340 Linn, Dr. Robert Robert.Linn@colorado.edu 970-316-0032 315 N. LaGrange Road, #839 LaGrange Park IL 65026 Mertz, Dr. Ronald devoemertz@sbcglobal.net 314-628-9376 165 Bon Chateau Drive Town and Country MO 63141 Plake, Dr. Barbara bQlakel@unl.edu 719-221-5196 PO Box4658 Buena Vista co 81211 Porter, Dr. Andrew andyQ@gse.uQenn.edu 215-898-7014 4601 Cedar Avenue Philadelphia PA 19143 Roeber, Dr. Ed edroeber@aol.com 517-575-0125 PO Box 519 Haslett Ml 48840 Winter, Dr. Phoebe Qhoebe.winter@outlook.com 804-272-0996 2319 Traymore Road Richmond VA 23235
Office of College and Career Readiness July 11, 2017 Dr. Bertha Doar 18225 Prickly Pear Court Pacific, MO 63069 Dear Dr. Doar: Blaine Henningsen, Ed.D. • Assistant Commissioner 205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC) for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your fellow TAC members to provide the agency and its contractors with ongoing psychometric advice and counsel. We know that your collective and individual input ensures a testing program of the highest quality. This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year. The Department will reimburse you for reasonable and necessary travel expenses in accordance with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The honorarium covers preparation for the meeting as well as attendance at each meeting. Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness, directs the TAC's work If you have questions, please contact her at (573) 751-3545. On behalf of the Missouri State Board of Education, Department staff, educators and students who benefit from MAP results, I thank you for your contributions to our assessment program. Sincerely, ~ Ef o...:rse-Blaine Henningsen Assistant Commissioner Office of College and Career Readiness c: Lisa Sireno Andy Martin Phone 573-751-3545 • occr@dese.mo.gov ' 6 Missouri i i§I5tUCAArroN ,~ Office of College and Career Readiness July 11, 2017 Dr. Karla Egan 120 North Brunswick Avenue Marshall MO 65304 Dear Dr. Egan: Blaine Henningsen, Ed.D. • Assistant Commissioner 205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC) for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your fellow TAC members to provide the agency and its contractors with ongoing psychometric advice and counsel. We know that your collective and individual input ensures a testing program of the highest quality. This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year. The Department will reimburse you for reasonable and necessary travel expenses in accordance with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The honorarium covers preparation for the meeting as well as attendance at each meeting. Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness, directs the TAC's work. If you have questions, please contact her at (573) 751-3545. On behalf of the Missouri State Board of Education, Department staff, educators and students who benefit from MAP results, I thank you for your contributions to our assessment program. Sincerely, Blaine Henningsen Assistant Commissioner Office of College and Career Readiness c: Lisa Sireno Andy Martin Phone 573-751-3545 • occr@dese.mo.gov Blaine Henningsen, Ed.D. • Assistant Commissioner Office of College and Career Readiness 205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov July 11, 2017 Dr. Ronald Mertz 165 Bon Chateau Drive Town and Country, MO 63141 Dear Dr. Mertz: Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC) for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your fellow TAC members to provide the agency and its contractors with ongoing psychometric advice and counsel. We know that your collective and individual input ensures a testing program of the highest quality. This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year. The Department will reimburse you for reasonable and necessary travel expenses in accordance with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The honorarium covers preparation for the meeting as well as attendance at each meeting. Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness, directs the TAC's work. If you have questions, please contact her at (573) 751-3545. On behalf of the Missouri State Board of Education, Department staff, educators and students who benefit from MAP results, I thank you for your contributions to our assessment program. Sincerely, ~ Ela..-:r~ Blaine Henningsen Assistant Commissioner Office of College and Career Readiness c: Lisa Sireno Andy Martin Phone 573-751-3545 • occr@dese.mo.gov Office of College and Career Readiness July 11, 2017 Dr. Barbara Plake PO Box4658 Buena Vista, CO 81211 Dear Dr. Plake: Blaine Henningsen, Ed.O. • Assistant Commissioner 205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC) for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your fellow TAC members to provide the agency and its contractors with ongoing psychometric advice and counsel. We know that your collective and individual input ensures a testing program of the highest quality. This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year. The Department will reimburse you for reasonable and necessary travel expenses in accordance with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The honorarium covers preparation for the meeting as well as attendance at each meeting. Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness, directs the TAC's work. If you have questions, please contact her at (573) 751-3545. On behalf of the Missouri State Board of Education, Department staff, educators and students who benefit from MAP results, I thank you for your contributions to our assessment program. Sincerely, ~ da...:rse.-Blaine Henningsen Assistant Commissioner Office of College and Career Readiness c: Lisa Sireno Andy Martin Phone 573-751-3545 • occr@dese.mo.gov Office of College and Career Readiness July 18, 2017 Dr. Andrew Porter 4601 Cedar Avenue Philadelphia, PA 19143 Dear Dr. Porter: Blaine Henningsen, Ed.D. • Assistant Commissioner 205 Jefferson Street, P.O. Box480 • Jefferson City, MO 65102-0480 • dese.mo.gov Thank you for your willingness to continue serving as chair of the Technical Advisory Committee (TAC) for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We value your previous work, and we depend on you and your fellow TAC members to provide the agency and its contractors with ongoing psychometric advice and counsel. We know that your leadership ensures a testing program of the highest quality. This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year. We ask that you provide a brief written report of the previous year's discussions prior to March 1, 2018. The Department will reimburse you for reasonable and necessary travel expenses in accordance with State of Missouri travel guidelines. You will receive an honorarium of $3,000 per day for facilitating each meeting, providing consultation prior to and following each meeting, and writing a report following each meeting. If the meeting is a conference call or web-based meeting, you will receive an honorarium of $375 per meeting hour for facilitating each meeting, providing consultation prior to and following each meeting, and writing a report following each meeting. Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness, will direct the TAC's work. If you have questions, please contact her at (573) 751-3545. On behalf of the Missouri State Board of Education, Department staff, educators and students who benefit from MAP results, I thank you for your contributions to our assessment program. Sincerely, ~ tlo....:r~ Blaine Henningsen Assistant Commissioner Office of College and Career Readiness c: Lisa Sireno, Coordinator Andy Martin, Finance Director Phone 573-751-3545 • occr@dese.mo.gov Office of College and Career Readiness July 11, 2017 Dr. Ed Roeber POBox519 Haslett, MI 48840 Dear Dr. Roeber: Blaine Henningsen, Ed.D. • Assistant Commissioner 205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65.102-0480 • dese.mo.gov Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC) for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your fellow TAC members to provide the agency and its contractors with ongoing psychometric advice and counsel. We know that your collective and individual input ensures a testing program of the highest quality. This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St. Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year. The Department will reimburse you for reasonable and necessary travel expenses in accordance with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The honorarium covers preparation for the meeting as well as attendance at each meeting. Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness, directs the TAC's work If you have questions, please contact her at (573) 751-3545. On behalf of the Missouri State Board of Education, Department staff, educators and students who benefit from MAP results, I thank you for your contributions to our assessment program. Sincerely, ~ da....:rse-a1aine Henningsen Assistant Commissioner Office of College and Career Readiness c: Lisa Sireno Andy Martin Phone 573-751-3545 • occr@dese.mo.gov Office of College and Career Readiness July 11, 2017 Dr. Phoebe Winter 2319 Traymore Road Richmond, VA 23235 Dear Dr. Winter: Blaine Henningsen, Ed.D. • Assistant Commissioner 205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC) for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your fellow TAC members to provide the agency and its contractors with ongoing psychometric advice and counsel. We know that your collective and individual input ensures a testing program of the highest quality. This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year. The Department will reimburse you for reasonable and necessary travel expenses in accordance with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The honorarium covers preparation for the meeting as well as attendance at each meeting. Lisa Sire no, Standards and Assessment Administrator, Office of College and Career Readiness, directs the TAC's work If you have questions, please contact her at (573) 751-3545. On behalf of the Missouri State Board of Education, Department staff, educators and students who benefit from MAP results, I thank you for your contributions to our assessment program. Sincerely, ~do...:r~ Blaine Henningsen Assistant Commissioner Office of College and Career Readiness c: Lisa Sireno Andy Martin Phone 573-751-3545 • occr@dese.mo.gov
1 7/5/2017 To: Lisa Sireno and Shaun Bates From: Andy Porter, Chair, Missouri Technical Advisory Committee Subject: Minutes of Missouri TAC Meeting on July 5, 2017 The Missouri Technical Advisory Committee met by telephone conference call on July 5, 2017. The conference call was a follow-up to a conference call on June 2, 2017 where members of the TAC asked Questar for additional information and that additional information was supplied prior to the July 5 conference call. Members of the TAC in attendance were Bertha Doar, director of assessment, St. Louis Missouri Public Schools; Ron Mertz, St. Louis public schools, retired; Barbara Plake, University of Nebraska, retired; Andy Porter, chair of TAC, University of Pennsylvania; Ed Roeber, independent consultant; and Phoebe Winter, independent consultant; DESE: Lisa Sireno, Shaun Bates; Questar: Katie McLarty and Scott Bishop. Discussions focused on two topics from the spring 2017 administration of end of course tests. Topic one concerned end of course testing on April 25 when heavy use of the system resulted in bandwidth limitations. Topic two concerned a software malfunction for the algebra I performance event. Impact of bandwidth limitations on April 25 Across all subjects, 14,666 students were recognized as either logged on and taking an assessment but with slow response time in moving from one item to the next or attempting to log on and unsuccessful. In the text that follows these will be referred to as groups 1, and 2. The question presented to the TAC was what course of action should be taken by DESE for district and school accountability. At the June 2, meeting the TAC preferred removing the affected students for one set of results for districts and schools and use the obtained results as another set of results; for each district and school, for accountability purposes use the better of their two results. The TAC asked for additional results to see how representative of students statewide were the students in group 1 and group 2. Based on the results produced by Questar and distributed in advance of the call, the TAC decided that only group 1 should be included in the reductions since group 2 membership was representative of statewide results. Software malfunction for the algebra I 2 For 172 students, the performance event software for algebra I created unscorable responses. These unscorable responses were triggered by students entering a select few characters which for unknown reasons created gibberish for their entire answer.. The Questar proposal was to statistically link student performance from session one which was not affected to student performance for the full test for those not affected. Then, to use the statistical link to produce the predicted results for the affected students. One of the handouts showed that the affected students were very low performing students. The TAC concluded that using statistically estimated scores for the affected students in session 2 was appropriate and justified, conditioned on the correlation between predicted scores and actual scores for the unaffected students being high. DESE requested that Questar produce this correlation and in addition a distribution of the absolute value of the deviations of predicted and actual scores. The TAC reviewed these results and found them to be sufficient to support using the statistical predicted scores. In particular the correlation between session 1 and session 2 was .63.
1 8/26/2017 To: Shaun Bates and Lisa Sireno From: Andy Porter, Chair, Missouri Technical Advisory Committee Subject: Minutes of Missouri TAC Meeting on August 17 and 18, 2017 The Missouri Technical Advisory Committee met at the Renaissance St. Louis Airport Hotel from 10:00 AM to 5:00 PM, August 17, 2017, and from 7:30 AM to 12:30 PM, August 18, 2017. Members of the TAC in attendance were Bertha Doar, director of assessment, St. Louis Missouri Public Schools; Karla Egan, independent consultant; Ron Mertz, St. Louis Public Schools, retired; Barbara Plake, University of Nebraska, retired; Andy Porter, chair of TAC, University of Pennsylvania; Ed Roeber, independent consultant; and Phoebe Winter, independent consultant; In attendance from DESE: Lisa Sireno, Shaun Bates, John Kitchens, Commissioner Margie Vandeven, Deputy Commissioner Stacey Preis, Assistant Commissioners Blaine Henningsen, and Chris Neale. In attendance from DRC: Lindy Wienand, Joanna Tomkowicz, Sara Brazzle, Rick Mercado. In attendance from Questar: Adam Johnson, Mike Woods, Sandra Durden, Katie McClarty and Scott Bishop. In attendance from MetaMetrics: Ellie Sanford-Moore. Contractors did not join the meeting until lunch on the first day. Missouri update Lisa Sireno provided the TAC with an update on assessment activity in Missouri. The Missouri Show Me Standards provide process and content standards for the state. The Missouri Learning Standards address Grade-Level and course content. New fine arts standards will soon be drafted, but no assessments of them are planned at the present time. The personal finance standards will soon be revised and a new assessment is planned for 2019/20. Missouri adopted new learning standards for mathematics, English language arts (ELA), social studies and science in 2016. The Missouri peer review submission to the U.S. Department of Education was largely approved for Algebra I, English II and grade-level Science as well as MAP-A in mathematics and ELA. MAP-A continues to be provided by Dynamic Learning Maps (DLM) for grades 3 through 8 and 11 in ELA and mathematics and grades 5, 8 and 11 in science. The state continues regular assessments in ELA and mathematics for grades 3 through 8 and in science for grades 5 and 8. Data Recognition Corporation (DRC) is developing interim assessments for Missouri. End-of-Course (EOC) testing continues. Sireno noted that this past year, approximately 20% of the eighth grade students took the Algebra I EOC. Missouri continues to use the WIDA consortium for assessing English language competence. For the past three years the state has done census assessment using the ACT for 11 th graders. Recently the state cut $4 million from the assessment budget (out of the approximately $18 million budget provided by the state, supplemented by $7 million from the federal government). 2 The current plan is to address this shortfall by discontinuing the state-funded ACT census assessment at 11 th grade. There will be new assessments for Grade-Level testing in ELA and mathematics and the EOC assessment in 2017/18. There will be new assessments for fifth and eighth grade Science and the EOC assessment in 2018/19 and in social studies for 2019/20 but only in high school. DESE has published several resource documents for state educators concerning the new standards. Sireno cited as challenges to assessment in Missouri the frequency and size of changes to state standards and the impact on assessments. The number and frequency of changes has created a great deal of confusion among K-12 educators in the state. There have also been challenges to the EOC assessments, a topic of discussion for later in the meeting. At the end of the conversation about challenges, the TAC noted that many of the materials for the current meeting were received too late to be read in advance and that this practice needs to be corrected. Grade-Level assessments technical report early tables for 2016 /17 Materials were received in advance of the meeting and Joanna Tomkowicz from DRC walked the TAC through the results for ELA, mathematics and science. In addition to reporting percent of students at various proficiency levels and mean scale scores, there were also results on interrater reliability and differential item functioning (DIF). When investigating subgroup differences, there continue to be large differences between average performance for black students and white students and general education students versus students with disabilities. Tomkowicz noted that no items were dropped from use based on the DIF analyses. DESE reminded the TAC that all items were reviewed in advance of their initial use by appropriate state content committees. The TAC wondered whether it would be useful to consider bringing in some “advocacy groups” to look at DIF flagged items. The reliabilities of the assessments for subgroups were investigated and reported as good to excellent despite the possibility of restriction in range for some of the subgroups in terms of their performance on the assessments. The TAC thanked Tomkowicz and DRC for the quality of their work and was pleased to see that the Grade-Level assessments continue to function well in Missouri. Grade-Level assessment data forensics Reports were circulated to members of the TAC in advance of the meeting on analyses of answer changes, test time and item time. In addition, there were analyses reported of the possibility of students copying answers from one another. Overall there was very little evidence of results flagged as suspicious and possibly needing investigation except for test time. Tomkowicz noted that for the forensics analyses of time, the amount of time for an individual student is not necessarily the real time because students didn’t always remember to log out. The TAC did note that the reports refer to flagging in terms of numbers of standard deviations when in fact the flags were identified in terms of number of standard errors. 3 This should be corrected. In the future, DRC will investigate the possibility of unusually large gains in student achievement from one year to the next. These results will also be analyzed by crossing them with wrong to right answer change analyses. The TAC complimented DESE and DRC for their attention to data forensics in detecting the possibility of cheating. Further, this commitment and the resulting analyses are surely serving as a deterrent to any cheating in the state. Sireno reported that DESE is developing follow-up plans in response to data forensics flags of classrooms and schools. The TAC requested that they have an opportunity to review these follow-up plans when available. The TAC recommends that the forensics analyses also look to see if there are any school districts that receive a surprising number of flagged classrooms/schools and that data analysis be added to the current analyses. The TAC noted that flagged/classrooms might sometimes be an indication of administration problems and those possibilities should be investigated as well. Embedded field tests in spring 2017 In spring 2017, 393 ELA items and 612 math items were field tested. Each item was taken by approximately 3,000 to 4,000 students. For ELA, 73 items were flagged largely for low or negative correlations with total score. For mathematics, 203 items were flagged primarily because they were too difficult. All flagged items were reviewed by DESE employees and dropped or modified as appropriate. Multi-select items Multi-select items are in a multiple-choice format, but instead of only one right answer, several of the answers provided below the item stem can be correct and appropriately chosen. While there is not a great deal of evidence in support, some believe that these multi-select items can test content in the standards that is difficult to test with other item formats. A sheet of rules for writing these types of items was distributed at the meeting, and discussed in detail; revisions were suggested by the TAC. Generally, the TAC favored a single set of rules to apply to all three subjects: ELA, math and science. Further, the TAC favored simplifying the rules where possible. Providing math formula sheets with the assessments Thus far in Missouri assessments, sheets providing a variety of mathematical formulas that are accessible to the student while taking the exam have not been provided. Nevertheless, there are some who would prefer that formula sheets be provided. The TAC asked how the state content standards address this issue and was assured that the standards were silent on the matter. The TAC recommends that the state continue to not provide formula sheets for mathematics assessments. If there should come a time when an item is written which is meant for the formula to be provided, that formula can be provided in the item itself. If that happens, care should be taken the formula provided does not change what is tested by any of the other items on the test form. 2017/18 ELA and mathematics vertical scaling plan 4 The TAC was provided in advance of the meeting an email from Joanna Tomkowicz to Lisa Sireno and Shaun Bates describing DRC’s plans for vertical scaling for the spring 2018 MAP assessments in ELA and mathematics developed to be aligned to the new Missouri learning standards. DRC proposes to use a set of common items between spring 2017 and spring 2018 operational tests in every grade ELA and mathematics to serve as anchor items in developing the new vertical scales. In addition, sets of items from the grade above and the grade below will be administered to samples of students taking on Grade-Level operational tests to facilitate between- grade assessment linking. A hybrid approach to building vertical scales as recommended by the TAC in a previous meeting will be used. In this approach, concurrent calibration results are treated as an initial scale the operational assessments. The initial scales are then equated to the existing ELA and mathematics scales using the common anchor items between the two administrations. Concurrent calibration of test data for all grades and content areas will then be conducted to build the vertical scales. The metric of the new vertical scales will be established to have means and standard deviations substantially different from the previous scales to signal that these are new assessments for new standards and comparisons of results to earlier years should not be made. The TAC complimented DRC on their plans for building the new vertical scales. The TAC did recommend that vertical linking items be carefully selected so that they reflect the best available items for spanning alignment of content between the two adjacent grades. The TAC would like to see the results for building the vertical scale using a) items from above and below grades, b) items from only above grades and c) items from only below grades before deciding which approach should be used. Interim assessments DESE has asked DRC to develop an interim assessment for grades 3 through 8 in ELA and mathematics and for grades 5 and 8 in science to be ready for the 2017/18 year. These interim assessments consist of one form per subject and grade and are also being called “pretests.” The TAC asked what the goals were for these pretests and was told they would be used as practice tests that mimic the operational tests in format and length. The practice tests could be taken any time and as many times as desired between the period November and June. The plan is to calibrate the practice test to the operational test using first-time test-taker data only. Results would be reported on the same scale as the operational test and end of year proficiencies would be reported. The TAC liked the plans, but recommends that the tests be called “practice tests,” not “interim assessments” nor “pretests.” Vertical articulation between middle and high school assessments DESE has asked its two contractors (DRC and Questar) to work together to better articulate high school assessments and Grade-Level assessments when new standards are set for each. The two contractors jointly talked to the TAC about their plans, saying that a formal proposal for vertical 5 articulation will be presented to DESE and the TAC at the next meeting. The goals of the vertical articulation are to have a K-12 assessment system where the proficiency impact data is coordinated between the assessment systems for K-8 and high school. Further, where there are jumps in the impact data, the articulation should provide an explanation for why those jumps make sense. Essentially, the idea is that if a student is proficient at one Grade-Level he or she is ready to be proficient at the next higher Grade-Level. With those goals in mind, the two contractors plan to set high school standards first. Maybe they will use ACT benchmarks and impact data to inform the high school standard-setting. Then, Grade-Level proficiency standards would be set within the context of the previously set EOC standards. At one point in the discussion, a rationale for higher percents proficient for EOCs was offered, noting that the focus of the course and the assessment for a single course might cover a narrower domain than the domain for an entire Grade-Level for a subject. These narrower domains might be more easily taught and mastered. The TAC noted that the contractors use different IRT models and that this might influence the selection of a response probability for standard-setting. The TAC further noted that there can be difficulties for standard-setting when a test is centered on the standards but not well centered on actual student achievement levels. The TAC concluded that while it is all right to have different response probability values for grades 3-8 and high school, the same response probabilities should be used across subjects and grades for grades 3-8 and across courses for End-of-Course tests in high school. The TAC was informed by DESE that they are working on a policy piece addressing these issues; and the TAC recommends that that policy piece be included in the proposal from the two contractors addressing vertical articulation of the two assessment systems. EOC 2016 /17 results DESE Commissioner Margie Vandeven joined the TAC for a discussion of whether the 2016/17 results for Algebra I and English II are valid for use in the state accountability program. The discussion was a continuation from the TAC conference call held in July. The concern is that student performance in these two subjects dropped substantially from the previous year. In its July conference call, the TAC concluded that this was a form effect, at least in part, rather than a true drop in performance. The TAC noted that when the same form was used two years prior as was used in 2017, performance was comparable (although somewhat higher). While it is common to use different forms over years for EOC tests, the forms were not built to ensure that they are psychometrically on the same scale. Importantly, 2017 is the last year for using the non-equated forms; in the future, EOC assessment forms will be carefully built so that the forms are not only reliable and valid but also comparable. At this point in time, results of the 2016/17 assessments have been reported to students, but their use for accountability has not yet happened. The interpretation of results of the other EOC assessments is not being called into question. The TAC was informed that a performance index is built based on student proficiency levels with 1 point for below basic, 3 points for basic, 4 points for proficient and 5 points for advanced. Using this index, three measures are used for accountability: progress, value-added and status. The TAC noted that neither value-added nor progress requires that forms be comparable between 2016 and 2017 since value-added is metric free and progress is measured such that it reflects 6 only change between two years ago and the current year (i.e., 2015 and 2017); for those two years the same form was used. The problem then is status. Discussion identified six possible actions: 1. Drop the 2016 results and use only 2015 and 2017 for which the same form was used. 2. Hold schools and districts harmless, meaning that if a school or district does better in 2017, on results as reported they get credit for the improvement, otherwise they get the same accountability levels as for the prior year. 3. Cut scores could be adjusted by, say, a single raw score point to make the results between 2016 and 2017 appear more comparable. 4. Use equipercentile procedures, putting the form used in 2017 on the same scale as the form used in 2016 5. Accept the results as reported to students and use the results for accountability accordingly. 6. Exclude the results for Algebra I and English II in accountability and don’t report those results. Of the above six possibilities, five members of the TAC favored the sixth option, not using the results for accountability and not reporting them. Two members of the TAC favored the first option; drop the 2016 results and use only 2015 and 2017 for which the same form was used. By the end of the meeting, however, all seven members of the TAC favored the sixth option. The TAC recommends that in the future, every effort be made to analyze the data before the results are reported to students so any problems can be detected and corrected. Small-scale pilot update Questar is developing new writing prompts and performance tasks. In the previous meeting, the TAC recommended that these be pilot tested and in a later TAC meeting, the TAC reviewed plans for a pilot test. At the time of that review, TAC strongly recommended that the pilot be online as is the operational test. At the current meeting, Questar reported that the pilot will take place on September 3; the pilot will be online and approximately 200 students reflecting a range of achievement levels will be used to pilot each prompt and performance task. There will be a different sample of students for each prompt and task. The TAC complimented Questar on these plans. Stand-alone field test for science Questar plans to build four forms each for Biology and Physical Science: two operational forms, one pre-test form, and one breach form for each assessment. Apparently, 95% of the students in the course will take the field test. There was some discussion as to whether or not students would be comparably motivated to do well on the field test as they will be motivated on the 7 new operational test. There was some discussion as to whether or not items in the field test could include items from the item pool to estimate the size of motivation effects. Ultimately, the TAC was supportive of Questar’s plans for the field test. Lexile® and Quantile® measures Ellie Sanford from MetaMetrics led the TAC in a discussion of how Missouri might report student achievement in reading in Lexiles® and in mathematics in Quantiles®. DESE is interested in exploring these possibilities and asked that the TAC provide their thoughts on the pros and cons of doing this. The motivation for the initiative is to provide students and parents with new ways of thinking about student achievement levels and appropriate future instruction. These reporting metrics are based on the concept of prerequisites. Knowing what prerequisites students have mastered provides suggestions as to where their next instruction should be. Sanford mentioned that one possibility was to embed MetaMetrics items in Missouri assessments so that the Missouri assessments could be reported on the MetaMetrics scales. Discussion revealed that currently many districts in Missouri are using other tests than those given by the state that report results in Lexiles® and Quantiles®. The TAC wondered whether it would be possible to use these data to link to state assessment results and put the state assessment results on the MetaMetrics scales rather than having the additional student burden of adding items to the state assessments to accomplish that end. The TAC was uncertain as to DESE’s questions concerning MetaMetrics and its products. DESE will prepare questions for the TAC to address on these issues at a future meeting. In closing The next meeting of the TAC is scheduled for December 7 and 8, 2017, and the meeting after that for March 8 and 9, 2018. Yet a third meeting was scheduled for August 2 and 3, 2018.
1 7/28/2017 To: Lisa Sireno and Shaun Bates From: Andy Porter, Chair, Missouri Technical Advisory Committee Subject: Minutes of Missouri TAC Meeting on July 27, 2017 The Missouri Technical Advisory Committee met by telephone conference call on July 27, 2017. Members of the TAC in attendance were Bertha Doar, director of assessment, St. Louis Missouri Public Schools; Karla Egan, independent consultant; Ron Mertz, St. Louis public schools, retired; Andy Porter, chair of TAC, University of Pennsylvania; and Ed Roeber, independent consultant; DESE: Lisa Sireno, Shaun Bates, Debbie Jameson, Commissioner Margie Vandeven, Blaine Henningsen, Jocelyn Strand, Jason Young, Janet Duncan, Angie Riner-Mooney, Jeff Falter, Chris Neale and Lisa Scroggs; Questar: Elliot Dunn, Adam Johnson, Katie McClarty, Tess Dawber and Scott Bishop. Discussion focused on End-of-Course test results for spring 2017 and their validity for use in student, school and district accountability. In Missouri, there are End-of-Course assessments for nine subjects: algebra I, algebra II, American history, biology, English I, English II, geometry, government and physical science. For most subjects, the assessments are given three times a year: summer, fall and spring. By far the largest number of students is assessed at the spring assessment. While the number of years of assessment results vary by subject, all subjects were assessed at least in 2014/15, 2015/16 and 2016/17. For 2 of the 9 assessed subjects, there was a substantial drop in the percent of students found to be proficient or above from 2016 to 2017. In algebra I, the percent proficient or above in the spring of 2016 was 7 percentage points higher than in spring of 2017. Similarly for English II, the percent proficient or above in the spring of 2016 was 9.5 percentage points higher than in spring of 2017. For the other six subjects, the change from 2016 to 2017 for spring assessments varied from a 1 percentage point increase to a 3.4 percentage point drop. Comparing student performance over years is complicated by the fact that for some subjects different forms of the assessment are used in different years and for some subjects the number of students tested differs from year-to-year. Thus, changes in performance from one year to the next could be due to a change in test form and/or a different cohort composition from year to year. The best data for making year-to-year comparisons is the spring assessment results because that is when by far the largest number of students was tested with the same form. Lisa Sireno provided the framework for discussion. Advice was sought on using the End-of-Course assessment results in spring 2017 for accountability purposes. In particular the results for algebra I and English II were of concern due to the drop in percentage of students judged to be proficient or better. In each subject, the same test form was used in spring of 2017 as was 2 used in spring of 2015, but a different form was used in spring of 2016. In algebra I, the spring results for 2015, 2016 and 2017 were 62.8%, 67.4% and 60.4% judged proficient or better and for English II the comparable results were 74.5%, 80.8%, 71.3%. When comparing 2015 to 2017 where the same forms were used for each assessment, the drop in performance was much more modest than from 2016 to 2017 where a different form was used. While the number of students tested in spring 2017 was less by 1000 students or so than the number tested in 2015, the TAC judged that the primary reason for a drop in performance was due to the form used in spring of 2015 and spring of 2017 being easier than the form used in spring of 2016 for each of these two subjects. The question became whether or not the results should be used for accountability purposes as obtained or whether some adjustment should be made to the results for 2017. Lowering the cut score for proficient or above one raw score point would result in 4 percentage points increase in students judged proficient or better in English II and 3.7 percentage points increase for algebra I. Because the state intends to do an early release of accountability data for the year, a decision was needed now. The TAC judged that in neither of the two subjects was the drop in performance from spring 2016 to spring 2017 affected by a change in the number of students tested. Thus, the drop in performance is likely due to either a true drop in student achievement and/or a difference in the difficulty of the form used between the two springs. The TAC noted that there was a small drop in performance from 2015 spring to 2017 spring when the same form was used at each assessment point. While the TAC concluded there is no strong psychometric solution to adjusting for the almost certain form affect, in fairness to students, schools and districts, some adjustment seemed warranted. Thus, the TAC recommended dropping the proficient cut score by one raw score point for each subject for spring and summer 2017 results (if the same test form is used in summer 2017 as was used in spring 2017). The TAC noted that the End-of-Course forms would not be used again after the summer 2017. The TAC strongly recommends that in developing new forms for the End-of-Course assessments, a design be used so that each form shares a sufficient number of common items with each other form so that forms can be linked one to another thus yielding comparable scale scores across forms.
1 6/7/2017 To: Lisa Sireno and Shaun Bates From: Andy Porter, Chair, Missouri Technical Advisory Committee Subject: Minutes of Missouri TAC Meeting on June 2, 2017 The Missouri Technical Advisory Committee met by telephone conference call on June 2, 2017. Members of the TAC in attendance were Bertha Doar, director of assessment, St. Louis Missouri Public Schools; Karla Egan, independent consultant; Ron Mertz, St. Louis public schools, retired; Barbara Plake, University of Nebraska, retired; Andy Porter, chair of TAC, University of Pennsylvania; Ed Roeber, independent consultant; and Phoebe Winter, independent consultant; DESE: Lisa Sireno, Shaun Bates Debbie Jameson and Lisa Scroggs; Questar: Elliot Dunn, Adam Johnson, Katie McLarty, Scott Bishop and Josh Borton. Discussions focused on three issues from the spring 2017 administration of end of course tests: English I and II, algebra I and II, biology, geometry, government, American history, and physical science. Performance on English II, algebra I or II, biology and government are used for accountability purposes. Issue one concerned end of course testing on April 25, when heavy use of the system resulted in bandwidth limitations. Issue two concerned a software malfunction for the algebra I performance event. Issue three concerned the occurrence of students being timed out while responding to the English I and II writing tasks. 1. Impact of bandwidth limitations on April 25 Across all subjects, 14,666 students were recognized as either logged on and taking an assessment but with slow response time in moving from one item to the next or attempting to log on and being unsuccessful. At the time of the TAC meeting, the numbers in each of these two groups were unknown, but the number of students logged on and experiencing delays in moving from one item to the other is believed to be much the smaller group. Additional students of unknown numbers were unable to log on at all and therefore took the test at another time. In the text that follows these will be referred to as groups 1,2 and 3. Questar reported that the bandwidth problem was fixed within 90 minutes and that no student responses were lost. Testing windows were extended for approximately two dozen districts that requested an extension. The question presented to the TAC was what course of action should be taken by DESE for accountability purposes. The discussion determined that student accountability was not a problem, since student scores on these tests are not used for student accountability. They can be used by teachers as one piece of information to determine a student’s course grade, but this 2 is a teacher option and teachers did know which students were affected by the “slow down”. Thus, discussion focused on purposes of district and school accountability. The TAC identified four options: 1. Use the obtained results as is for accountability purposes; 2. Do not use the results for accountability purposes; 3. Use some statistical procedure to adjust the results and use those adjusted results for accountability purposes; 4. Remove the affected students for one set of results for districts and schools and use the obtained results as another set of results; for each district and school, for accountability purposes use the better of their two results No members of the TAC were in favor of statistical adjustment. Not using the results for accountability was not an attractive option, especially since EOC results have recently not been available for accountability. Ultimately the TAC favored using for accountability option 4, the better results from the data as obtained and the data after affected students were removed. There was considerable discussion as to whether to remove only those students from group 1, those logged on and experiencing a serious slowdown in moving from one item to the next, or to also remove those from group 2, those that experienced the frustration of trying to log on and not being able to. Unfortunately, the membership of group 3, those who couldn’t even register an attempt to log on is unknown. The TAC recognized that the frustration experienced by students in each of these three groups could have bearing on their performance on the affected test for them. Questar estimates that approximately half of the students in the state were possibly affected. Though Questar presented contrasts in results for the various effects in groups from previous years and the current year, it was impossible to interpret these comparisons because the representativeness of the affected groups to students statewide was unknown. Questar estimated that they would have the analysis file necessary to make these representativeness determinations on or before June 19. Some members of the TAC believed that only group 1 members might be used in the preferred option for accountability purposes if the students in group 2 were representative to statewide students. Other members of the TAC felt that both groups 1 and 2 should be used in the preferred option for accountability purposes regardless. The TAC will meet again virtually once Questar has produced the requested analyses to determine the representativeness of students in the affected groups. Not only did the TAC recommend that Questar produce results on representativeness in terms of prior-year test scores and demographics but also results on the percent of students in each affected group in aggregate and the distribution of percent of affected students in each group for schools. Questar was charged with providing an analysis plan to DESE by close of business the following Monday (the TAC meeting was on Friday). 3 2. Software malfunction for the algebra I performance event For 172 students, the performance event software for algebra I created unscorable responses. These unscorable responses were triggered by students entering a select few characters that for unknown reasons created gibberish for their entire answer. While this problem has been fixed for future administrations, the question to the TAC was what to do about the affected students. Questar presented the TAC in advance of the meeting two short papers addressing this issue. The Questar proposal was to statistically link student performance from session one which was not affected to student performance for the full test for those not affected. Then, to use the statistical link to produce the predicted results for the affected students. One of the handouts showed that the affected students tended to be very low performing students. The plan from Questar was to report student performance for both session 1 (having no performance event) and session 2 (the performance event worth 10 points). The TAC concluded that the statistically estimated scores for the affected students in session 2 was appropriate and justified, conditioned on the correlation between predicted scores and actual scores for the unaffected students being high. DESE requested that Questar produce this correlation and in addition a distribution of the absolute value of the deviations of predicted and actual scores. The TAC will review these results and make a recommendation as to whether to go ahead with using the statistical predicted scores. 3. Students being timed out for the English I and II writing tasks For some reason, students who spent a long time entering text in response to the writing tasks on the English I and II end of course assessments were timed out of the testing session. The students could sign back in to complete the test, but still they experienced frustration. Unfortunately, the number of students and who the students are who were affected is unknown. This problem occurred across platforms and is seemingly widespread yet known only anecdotally. The TAC recommended that in the future students be given a warning when they are about to be kicked off of the system. Questar thought this may solve the problem for the future. As for the recent administration, the TAC had no solution to offer as the affected students are not known.
August 31, 2017 1 Talking Points for 2017 EOCs Main Message: The results of the 2017 EOCs in English II and Algebra I do not produce year-to-year comparable results. The department doesn’t plan to release statewide results from these particular assessments. We will hold the testing vendor, Questar, accountable for not producing usable assessment results.  DESE’s TAC – Technical Advisory Committee – concluded that the results of the Algebra I and English II tests do not produce year-to -year comparable results.  School districts and charter LEAs have already received Questar’s individual student reports (ISRs). Consistent with previous guidance from DESE, educators should use raw scores from those assessments. Accordingly, student record labels may be used as appropriate.  Districts may also use student End-of-Course (EOC) scores for determining A+ scholarship eligibility.  For students who took the Algebra I and English II EOCs, the participation requirement has been met. Students do not need to retake these tests.  We are extremely frustrated with our vendor’s inability to deliver usable results. Our students and teachers work hard throughout the year, and they are counting on us to get this right.  We are working with our attorney to pursue our legal options. Timeline of Events  When the department received 2017 statewide assessment results in late July, staff immediately alerted Questar, Missouri’s EOC test vendor, about unexpected changes in the results compared to test administrations in previous years.  DESE staff then convened the state’s Technical Advisory Committee (TAC) – made up of nationally recognized experts in assessment – to review the results.  At its August 18 meeting, the TAC concluded that there were year-to -year comparability issues for the EOCs for Algebra I and English II. The TAC did not express concern about any other EOCs.  These tests have now been retired. Missouri educators have been developing mathematics and English language arts assessments—both Grade-Level Assessments and EOCs—that will be administered beginning this year. APR Talking Points  In developing the APR, DESE will take the following steps to identify the best solution for this challenging situation: August 31, 2017 2 o District APRs will not include Algebra I and English II EOCs in the status and progress calculations. o Status targets and progress calculations will be adjusted to accommodate the loss of the two EOCs. o Priority is on district and charter LEA APRs. Decisions regarding building-level APRs will be made at a later date. Timeline The department’s next steps will include the following: 1. MSIP TAC will convene on September 12. 2. Comprehensive Guide revisions will be made during the week of September 18. 3. Public Hearing on MSIP Comprehensive Guide will be held on October 4. Event Date Achievement data to school districts and charter schools during week of 9/5 Appeals and data corrections window closes for EOCs 9/22 Further timeline information will be published as soon as it is available. What are the department’s next steps regarding assessments? We will convene a work group to review and make recommendations regarding ways to improve test administration and delivery of results. Topics will include such things as  a timeline for the release of scores to districts,  the length of testing windows, and  technology variations among districts and charter LEAs.
September 1, 2017 Dear Parents: A top priority of educators is to help prepare Missouri children for success, not only in school but also for life after graduation. We know you are proud of your children, and we are too. We want to inform you about some developments resulting from the 2017 End-of-Course assessments in Algebra I and English II. The technical advisory committee (TAC) retained by the Department of Elementary and Secondary Education (DESE) to review assessment results each year determined that the 2017 Algebra I and English II assessments are not suitable for establishing assessment performance trends. Upon the recommendation of the TAC, DESE will exclude data from these assessments in school district and charter school annual performance reports (APRs). The APRs that we provide to school districts and charter schools look at multiple factors, including state assessments in English language arts, mathematics, science, and social studies; a host of college and career readiness indicators; attendance rates; and graduation rates. For each of these indicators, multiple years of data are used. Your child’s class grades and any local decisions about A+ eligibility, as it pertains to these two assessments, are not at risk. I appreciate your understanding as we work through this unfortunate event. Thank you for supporting Missouri public education and your local school district. Sincerely, Margie Vandeven Commissioner of Education
AGENDA Missouri Technical Advisory Committee July 15, 2016 9:00 a.m. – 1:00 p.m. Conference call/web meeting 9:00 a.m. Topic 1: Grade-Level Mathematics and English Language Arts Cut Scores Hello and introductions (DESE, TAC, DRC) Presentation (DRC) Commentary from observers (Karla Egan) Discussion How to join: JOIN WEBEX MEETING https://datarecognitioncorpaudio.webex.com/datarecognitioncorpaudio/j.php?MTID =m12cd39acdb8511ad35e4b5e44e80986b Meeting number: 282 892 866 Host key: 896239 Meeting password: Eftbert5 JOIN BY PHONE Dial-in: 1-800-549-9519 P-code: 16250282 Break 11:00 a.m. Topic 2: EOC Spring Biology Form Performance Hello and introductions (DESE, TAC, Questar) Presentation (Questar) Discussion How to join: https://global.gotomeeting.com/join/207782485 You can also dial in using your phone. United States : +1 (224) 501-3212 Access Code: 207-782-485 First GoToMeeting? Try a test session: http://help.citrix.com/getready 1:00 p.m. Adjourn