Message: Sunshine Request

Sunshine Request
Sunshine Request

From | Hagenhoff, Margaret | Date Tuesday, October 3, 2017 8:53 AM

To | Sireno, Lisa
Cc |
Subject | Sunshine Request

State Statue 160.526.2.pdf (23 Kb ) TAC Contact Information.pdf (16 Kb ) TAC FY18 contract letters.pdf (144 Kb ) Missouri TAC report 7.5.2017.pdf (77 Kb ) Missouri TAC report 8.21.2017 FINAL.pdf (62 Kb ) Missouri_TAC_report__7.28.2017 FINAL.pdf (83 Kb ) Missouri_TAC_report__6.7.2017_ER_pw.pdf (90 Kb ) 2017 EOC Talking Points for districts.pdf (258 Kb ) EOC Parent Letter.pdf (110 Kb ) TAC Meeting Agenda 7-15-2016.pdf (70 Kb )

Lisa,

The attached documents are ready for your review for the Sunshine Law Request. Thanks,

Margie Hagenhoff| Administrative Assistant|Office of College and Career Readiness

Phone 573.751.3545 |Fax 573.526.0812|dese.mo.gov

    Hagenhoff, Margaret
    From:  Sent:  To:  Cc:
    Subject:
    160.526.2
    Bates, Shaun  Thursday, April 13, 2017 1:18 PM  Hagenhoff, Sara; Sirena, Lisa  Hagenhoff, Margaret
    RE: TAC information
    2. The state board of education shall by contract enlist the assistance of such national experts to receive reports,  advice and counsel on a regular basis pertaining to the validity and reliability of the statewide assessment system.  The reports from such experts shall be received by the state board of education. Within six months prior to  implementation of or modification or revision to the statewide assessment system, the commissioner of education  shall inform the president pro tempore of the senate and the speaker of the house of representatives about the
    procedures to implement, modify, or revise the statewide assessment system, including a report related to the  reliability and validity of the assessment instruments, and the general assembly may, within the next sixty legislative  days, veto such implementation, modification, or revision by concurrent resolution adopted by majority vote of both  the senate and the house of representatives.
    From: Hagenhoff, Sara  Sent: Thursday, April 13, 2017 12:50 PM  To: Bates, Shaun; Sirena, Lisa  Cc: Hagenhoff, Margaret  Subject: TAC information  Importance: High
    I'm looking for the statute that requires us to have a Technical Advisory Committee. Can any of you direct me to  that? We need to send some documentation to OA to back up the members being appointed by the Commissioner, not  bid out. Any help is appreciated and the sooner, the better ... Thanks!
    Sara Hagenhoff I Director of Procurement I Financial and Administrative Services I 573-751-4463 I dese.mo.gov
    1

    Name
    Email Address  Phone Number  Address
    City
    State Zip code
    Doar, Dr. Bertha  Bertha.Doar@slQs.org  314-345-2360  18225 Prickly Pear Court  Pacific
    MO  63069
    Egan, Karla  karlaegan@gmail.com  660-631-0843  120 North Brunswick
    Marshall  MO
    65340
    Linn, Dr. Robert  Robert.Linn@colorado.edu  970-316-0032  315 N. LaGrange Road, #839
    LaGrange Park  IL
    65026
    Mertz, Dr. Ronald  devoemertz@sbcglobal.net  314-628-9376  165 Bon Chateau Drive
    Town and Country  MO
    63141
    Plake, Dr. Barbara  bQlakel@unl.edu  719-221-5196  PO Box4658
    Buena Vista  co
    81211
    Porter, Dr. Andrew  andyQ@gse.uQenn.edu  215-898-7014  4601 Cedar Avenue  Philadelphia
    PA  19143
    Roeber, Dr. Ed
    edroeber@aol.com  517-575-0125  PO Box 519
    Haslett
    Ml  48840
    Winter, Dr. Phoebe  Qhoebe.winter@outlook.com  804-272-0996  2319 Traymore Road
    Richmond  VA
    23235

    Office of College and Career Readiness
    July 11, 2017
    Dr. Bertha Doar  18225 Prickly Pear Court  Pacific, MO 63069
    Dear Dr. Doar:
    Blaine Henningsen, Ed.D. • Assistant Commissioner
    205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov
    Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC)  for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your  fellow TAC members to provide the agency and its contractors with ongoing psychometric advice  and counsel. We know that your collective and individual input ensures a testing program of the  highest quality.
    This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St  Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year.
    The Department will reimburse you for reasonable and necessary travel expenses in accordance  with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per  day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will  receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The  honorarium covers preparation for the meeting as well as attendance at each meeting.
    Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness,  directs the TAC's work If you have questions, please contact her at (573) 751-3545.
    On behalf of the Missouri State Board of Education, Department staff, educators and students who  benefit from MAP results, I thank you for your contributions to our assessment program.
    Sincerely,
    ~ Ef o...:rse-Blaine Henningsen  Assistant Commissioner  Office of College and Career Readiness
    c: Lisa Sireno  Andy Martin
    Phone 573-751-3545 • occr@dese.mo.gov

    '
    6 Missouri  i i§I5tUCAArroN ,~  Office of College and Career Readiness
    July 11, 2017
    Dr. Karla Egan  120 North Brunswick Avenue  Marshall MO 65304
    Dear Dr. Egan:
    Blaine Henningsen, Ed.D. • Assistant Commissioner
    205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov
    Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC)  for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your  fellow TAC members to provide the agency and its contractors with ongoing psychometric advice  and counsel. We know that your collective and individual input ensures a testing program of the  highest quality.
    This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St  Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year.
    The Department will reimburse you for reasonable and necessary travel expenses in accordance  with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per  day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will  receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The  honorarium covers preparation for the meeting as well as attendance at each meeting.
    Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness,  directs the TAC's work. If you have questions, please contact her at (573) 751-3545.
    On behalf of the Missouri State Board of Education, Department staff, educators and students who  benefit from MAP results, I thank you for your contributions to our assessment program.
    Sincerely,
    Blaine Henningsen  Assistant Commissioner  Office of College and Career Readiness
    c: Lisa Sireno  Andy Martin
    Phone 573-751-3545 • occr@dese.mo.gov

    Blaine Henningsen, Ed.D. • Assistant Commissioner
    Office of College and Career Readiness 205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov
    July 11, 2017
    Dr. Ronald Mertz  165 Bon Chateau Drive  Town and Country, MO 63141
    Dear Dr. Mertz:
    Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC)  for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your  fellow TAC members to provide the agency and its contractors with ongoing psychometric advice  and counsel. We know that your collective and individual input ensures a testing program of the  highest quality.
    This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St  Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year.
    The Department will reimburse you for reasonable and necessary travel expenses in accordance  with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per  day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will  receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The  honorarium covers preparation for the meeting as well as attendance at each meeting.
    Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness,  directs the TAC's work. If you have questions, please contact her at (573) 751-3545.
    On behalf of the Missouri State Board of Education, Department staff, educators and students who  benefit from MAP results, I thank you for your contributions to our assessment program.
    Sincerely,
    ~ Ela..-:r~  Blaine Henningsen  Assistant Commissioner  Office of College and Career Readiness
    c: Lisa Sireno  Andy Martin
    Phone 573-751-3545 • occr@dese.mo.gov

    Office of College and Career Readiness
    July 11, 2017
    Dr. Barbara Plake  PO Box4658  Buena Vista, CO 81211
    Dear Dr. Plake:
    Blaine Henningsen, Ed.O. • Assistant Commissioner
    205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov
    Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC)  for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your  fellow TAC members to provide the agency and its contractors with ongoing psychometric advice  and counsel. We know that your collective and individual input ensures a testing program of the  highest quality.
    This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St  Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year.
    The Department will reimburse you for reasonable and necessary travel expenses in accordance  with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per  day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will  receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The  honorarium covers preparation for the meeting as well as attendance at each meeting.
    Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness,  directs the TAC's work. If you have questions, please contact her at (573) 751-3545.
    On behalf of the Missouri State Board of Education, Department staff, educators and students who  benefit from MAP results, I thank you for your contributions to our assessment program.
    Sincerely,
    ~ da...:rse.-Blaine Henningsen  Assistant Commissioner  Office of College and Career Readiness
    c: Lisa Sireno  Andy Martin
    Phone 573-751-3545 • occr@dese.mo.gov

    Office of College and Career Readiness
    July 18, 2017
    Dr. Andrew Porter  4601 Cedar Avenue  Philadelphia, PA 19143
    Dear Dr. Porter:
    Blaine Henningsen, Ed.D. • Assistant Commissioner
    205 Jefferson Street, P.O. Box480 • Jefferson City, MO 65102-0480 • dese.mo.gov
    Thank you for your willingness to continue serving as chair of the Technical Advisory Committee  (TAC) for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We value your  previous work, and we depend on you and your fellow TAC members to provide the agency and its  contractors with ongoing psychometric advice and counsel. We know that your leadership ensures  a testing program of the highest quality.
    This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St  Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year. We  ask that you provide a brief written report of the previous year's discussions prior to March 1,  2018.
    The Department will reimburse you for reasonable and necessary travel expenses in accordance  with State of Missouri travel guidelines. You will receive an honorarium of $3,000 per day for  facilitating each meeting, providing consultation prior to and following each meeting, and writing a  report following each meeting. If the meeting is a conference call or web-based meeting, you will  receive an honorarium of $375 per meeting hour for facilitating each meeting, providing  consultation prior to and following each meeting, and writing a report following each meeting.
    Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness, will  direct the TAC's work. If you have questions, please contact her at (573) 751-3545.
    On behalf of the Missouri State Board of Education, Department staff, educators and students who  benefit from MAP results, I thank you for your contributions to our assessment program.
    Sincerely,
    ~ tlo....:r~
    Blaine Henningsen  Assistant Commissioner  Office of College and Career Readiness
    c: Lisa Sireno, Coordinator  Andy Martin, Finance Director
    Phone 573-751-3545 • occr@dese.mo.gov

    Office of College and Career Readiness
    July 11, 2017
    Dr. Ed Roeber  POBox519  Haslett, MI 48840
    Dear Dr. Roeber:
    Blaine Henningsen, Ed.D. • Assistant Commissioner
    205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65.102-0480 • dese.mo.gov
    Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC)  for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your  fellow TAC members to provide the agency and its contractors with ongoing psychometric advice  and counsel. We know that your collective and individual input ensures a testing program of the  highest quality.
    This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St.  Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year.
    The Department will reimburse you for reasonable and necessary travel expenses in accordance  with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per  day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will  receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The  honorarium covers preparation for the meeting as well as attendance at each meeting.
    Lisa Sireno, Standards and Assessment Administrator, Office of College and Career Readiness,  directs the TAC's work If you have questions, please contact her at (573) 751-3545.
    On behalf of the Missouri State Board of Education, Department staff, educators and students who  benefit from MAP results, I thank you for your contributions to our assessment program.
    Sincerely,
    ~ da....:rse-a1aine Henningsen  Assistant Commissioner  Office of College and Career Readiness
    c: Lisa Sireno  Andy Martin
    Phone 573-751-3545 • occr@dese.mo.gov

    Office of College and Career Readiness
    July 11, 2017
    Dr. Phoebe Winter  2319 Traymore Road  Richmond, VA 23235
    Dear Dr. Winter:
    Blaine Henningsen, Ed.D. • Assistant Commissioner
    205 Jefferson Street, P.O. Box 480 • Jefferson City, MO 65102-0480 • dese.mo.gov
    Thank you for your willingness to serve as a member of the Technical Advisory Committee (TAC)  for the Missouri Assessment Program (MAP) for the 2018 fiscal year. We depend on you and your  fellow TAC members to provide the agency and its contractors with ongoing psychometric advice  and counsel. We know that your collective and individual input ensures a testing program of the  highest quality.
    This year, we anticipate four in-person TAC meetings, held in a hotel near Lambert Airport in St  Louis. In addition, the TAC may convene by phone or web-meeting occasionally during the year.
    The Department will reimburse you for reasonable and necessary travel expenses in accordance  with State of Missouri travel guidelines. In addition, you will receive an honorarium of $2,500 per  day you meet with the TAC. If the meeting is a conference call or web-based meeting, you will  receive an honorarium of $325.00 per meeting hour per day you meet with the TAC. The  honorarium covers preparation for the meeting as well as attendance at each meeting.
    Lisa Sire no, Standards and Assessment Administrator, Office of College and Career Readiness,  directs the TAC's work If you have questions, please contact her at (573) 751-3545.
    On behalf of the Missouri State Board of Education, Department staff, educators and students who  benefit from MAP results, I thank you for your contributions to our assessment program.
    Sincerely,
    ~do...:r~  Blaine Henningsen  Assistant Commissioner  Office of College and Career Readiness
    c: Lisa Sireno  Andy Martin
    Phone 573-751-3545 • occr@dese.mo.gov

    1

              7/5/2017
    To:  Lisa Sireno and Shaun Bates
    From:  Andy Porter, Chair, Missouri Technical Advisory Committee
    Subject:  Minutes of Missouri TAC Meeting on July 5, 2017
    The Missouri Technical Advisory Committee met by telephone conference call on July 5, 2017.  The conference call was a follow-up to a conference call on June 2, 2017 where members of the  TAC asked Questar for additional information and that additional information was supplied  prior to the July 5 conference call. Members of the TAC in attendance were Bertha Doar,  director of assessment, St. Louis Missouri Public Schools; Ron Mertz, St. Louis public schools,  retired; Barbara Plake, University of Nebraska, retired; Andy Porter, chair of TAC, University of  Pennsylvania; Ed  Roeber, independent consultant; and Phoebe Winter, independent  consultant; DESE: Lisa Sireno,  Shaun Bates; Questar: Katie McLarty and Scott Bishop.
    Discussions focused on two topics from the spring 2017 administration of end of course tests.   Topic one concerned end of course testing on April 25 when heavy use of the system resulted in  bandwidth limitations. Topic two concerned a software malfunction for the algebra I  performance event.
     Impact of bandwidth limitations on April 25
    Across all subjects, 14,666 students were recognized as either logged on and taking an  assessment but with slow response time in moving from one item to the next or attempting to  log on and unsuccessful.  In the text that follows these will be referred to as groups 1, and 2.
    The question presented to the TAC was what course of action should be taken by DESE for   district and school accountability.
    At the June 2, meeting the TAC preferred removing the affected students for one set of  results for districts and schools and use the obtained results as another set of results; for each  district and school, for accountability purposes use the better of their two results. The TAC  asked for additional results to see how representative of students statewide were the  students in group 1 and group 2. Based on the results produced by Questar and distributed in  advance of the call, the TAC decided that only group 1 should be included in the reductions  since group 2 membership was representative of statewide results.

    Software malfunction for the algebra I

    2

    For 172 students, the performance event software for algebra I created unscorable responses.  These unscorable responses were triggered by students entering a select few characters which  for unknown reasons created gibberish for their entire answer.. The Questar proposal was to  statistically link student performance from session one which was not affected to student  performance for the full test for those not affected. Then, to use the statistical link to produce  the predicted results for the affected students. One of the handouts showed that the affected  students were very low performing students.

    The TAC concluded that using statistically estimated scores for the affected students in  session 2 was appropriate and justified, conditioned on the correlation between predicted  scores and actual scores for the unaffected students being high.  DESE requested that Questar  produce this correlation and in addition a distribution of the absolute value of the deviations  of predicted and actual scores. The TAC reviewed these results and found them to be  sufficient to support using the statistical predicted scores. In particular the correlation  between session 1 and session 2 was .63.

    1
              8/26/2017
    To:  Shaun Bates and Lisa Sireno
    From:  Andy Porter, Chair, Missouri Technical Advisory Committee
    Subject:  Minutes of Missouri TAC Meeting on August 17 and 18, 2017
    The Missouri Technical Advisory Committee met at the Renaissance St. Louis Airport Hotel  from 10:00 AM to 5:00 PM, August 17, 2017, and from 7:30 AM to 12:30 PM, August 18, 2017.  Members of the TAC in attendance were Bertha Doar, director of assessment, St. Louis Missouri  Public Schools; Karla Egan, independent consultant; Ron Mertz, St. Louis Public Schools,  retired; Barbara Plake, University of Nebraska, retired; Andy Porter, chair of TAC, University of  Pennsylvania; Ed  Roeber, independent consultant; and Phoebe Winter, independent consultant;  In attendance from DESE: Lisa Sireno, Shaun Bates, John Kitchens, Commissioner Margie  Vandeven, Deputy Commissioner Stacey Preis, Assistant Commissioners Blaine Henningsen,  and Chris Neale.  In attendance from DRC:  Lindy Wienand, Joanna Tomkowicz, Sara Brazzle,  Rick Mercado.  In attendance from Questar: Adam Johnson, Mike Woods, Sandra Durden, Katie  McClarty and Scott Bishop.  In attendance from MetaMetrics: Ellie Sanford-Moore.    Contractors did not join the meeting until lunch on the first day.    Missouri update    Lisa Sireno provided the TAC with an update on assessment activity in Missouri. The Missouri  Show Me Standards provide process and content standards for the state. The Missouri Learning  Standards address Grade-Level and course content. New fine arts standards will soon be drafted,  but no assessments of them are planned at the present time. The personal finance standards will  soon be revised and a new assessment is planned for 2019/20. Missouri adopted new learning  standards for mathematics, English language arts (ELA), social studies and science in 2016. The  Missouri peer review submission to the U.S. Department of Education was largely approved for  Algebra I, English II and grade-level Science as well as MAP-A in mathematics and ELA.    MAP-A continues to be provided by Dynamic Learning Maps (DLM) for grades 3 through 8 and  11 in ELA and mathematics and grades 5, 8 and 11 in science. The state continues regular  assessments in ELA and mathematics for grades 3 through 8 and in science for grades 5 and 8.  Data Recognition Corporation (DRC) is developing interim assessments for Missouri. End-of-Course (EOC) testing continues. Sireno noted that this past year, approximately 20% of the  eighth grade students took the Algebra I EOC.     Missouri continues to use the WIDA consortium for assessing English language competence. For  the past three years the state has done census assessment using the ACT for 11
    th graders.
    Recently the state cut $4 million from the assessment budget (out of the approximately $18  million budget provided by the state, supplemented by $7 million from the federal government).

    2
    The current plan is to address this shortfall by discontinuing the state-funded ACT census  assessment at 11 th grade.
      There will be new assessments for Grade-Level testing in ELA and mathematics and the EOC  assessment in 2017/18. There will be new assessments for fifth and eighth grade Science and the  EOC assessment in 2018/19 and in social studies for 2019/20 but only in high school.    DESE has published several resource documents for state educators concerning the new  standards.    Sireno cited as challenges to assessment in Missouri the frequency and size of changes to state  standards and the impact on assessments. The number and frequency of changes has created a  great deal of confusion among K-12 educators in the state. There have also been challenges to  the EOC assessments, a topic of discussion for later in the meeting. At the end of the  conversation about challenges, the TAC noted that many of the materials for the current meeting  were received too late to be read in advance and that this practice needs to be corrected.    Grade-Level assessments technical report early tables for 2016 /17    Materials were received in advance of the meeting and Joanna Tomkowicz from DRC walked  the TAC through the results for ELA, mathematics and science. In addition to reporting percent  of students at various proficiency levels and mean scale scores, there were also results on  interrater reliability and differential item functioning (DIF). When investigating subgroup  differences, there continue to be large differences between average performance for black  students and white students and general education students versus students with disabilities.  Tomkowicz noted that no items were dropped from use based on the DIF analyses. DESE  reminded the TAC that all items were reviewed in advance of their initial use by appropriate  state content committees. The TAC wondered whether it would be useful to consider  bringing in some “advocacy groups” to look at DIF flagged items. The reliabilities of the  assessments for subgroups were investigated and reported as good to excellent despite the  possibility of restriction in range for some of the subgroups in terms of their performance on the  assessments.    The TAC thanked Tomkowicz and DRC for the quality of their work and was pleased to  see that the Grade-Level assessments continue to function well in Missouri.    Grade-Level assessment data forensics    Reports were circulated to members of the TAC in advance of the meeting on analyses of answer  changes, test time and item time. In addition, there were analyses reported of the possibility of  students copying answers from one another. Overall there was very little evidence of results  flagged as suspicious and possibly needing investigation except for test time. Tomkowicz noted  that for the forensics analyses of time, the amount of time for an individual student is not  necessarily the real time because students didn’t always remember to log out.  The TAC did note that the reports refer to flagging in terms of numbers of standard  deviations when in fact the flags were identified in terms of number of standard errors.

    3
    This should be corrected. In the future, DRC will investigate the possibility of unusually large  gains in student achievement from one year to the next. These results will also be analyzed by  crossing them with wrong to right answer change analyses.     The TAC complimented DESE and DRC for their attention to data forensics in detecting  the possibility of cheating. Further, this commitment and the resulting analyses are surely  serving as a deterrent to any cheating in the state. Sireno reported that DESE is developing  follow-up plans in response to data forensics flags of classrooms and schools. The TAC  requested that they have an opportunity to review these follow-up plans when available.  The TAC recommends that the forensics analyses also look to see if there are any school  districts that receive a surprising number of flagged classrooms/schools and that data  analysis be added to the current analyses. The TAC noted that flagged/classrooms might  sometimes be an indication of administration problems and those possibilities should be  investigated as well.    Embedded field tests in spring 2017    In spring 2017, 393 ELA items and 612 math items were field tested. Each item was taken by  approximately 3,000 to 4,000 students. For ELA, 73 items were flagged largely for low or  negative correlations with total score. For mathematics, 203 items were flagged primarily  because they were too difficult. All flagged items were reviewed by DESE employees and  dropped or modified as appropriate.    Multi-select items    Multi-select items are in a multiple-choice format, but instead of only one right answer, several  of the answers provided below the item stem can be correct and appropriately chosen. While  there is not a great deal of evidence in support, some believe that these multi-select items can test  content in the standards that is difficult to test with other item formats. A sheet of rules for  writing these types of items was distributed at the meeting, and discussed in detail; revisions  were suggested by the TAC. Generally, the TAC favored a single set of rules to apply to all three  subjects: ELA, math and science. Further, the TAC favored simplifying the rules where possible.     Providing math formula sheets with the assessments    Thus far in Missouri assessments, sheets providing a variety of mathematical formulas that are  accessible to the student while taking the exam have not been provided. Nevertheless, there are  some who would prefer that formula sheets be provided. The TAC asked how the state content  standards address this issue and was assured that the standards were silent on the matter. The  TAC recommends that the state continue to not provide formula sheets for mathematics  assessments. If there should come a time when an item is written which is meant for the  formula to be provided, that formula can be provided in the item itself. If that happens,  care should be taken the formula provided does not change what is tested by any of the  other items on the test form.
    2017/18 ELA and mathematics vertical scaling plan

    4
      The TAC was provided in advance of the meeting an email from Joanna Tomkowicz to Lisa  Sireno and Shaun Bates describing DRC’s plans for vertical scaling for the spring 2018 MAP  assessments in ELA and mathematics developed to be aligned to the new Missouri learning  standards. DRC proposes to use a set of common items between spring 2017 and spring 2018  operational tests in every grade ELA and mathematics to serve as anchor items in developing the  new vertical scales. In addition, sets of items from the grade above and the grade below will be  administered to samples of students taking on Grade-Level operational tests to facilitate  between- grade assessment linking. A hybrid approach to building vertical scales as  recommended by the TAC in a previous meeting will be used. In this approach, concurrent  calibration results are treated as an initial scale the operational assessments. The initial scales are  then equated to the existing ELA and mathematics scales using the common anchor items  between the two administrations. Concurrent calibration of test data for all grades and content  areas will then be conducted to build the vertical scales.    The metric of the new vertical scales will be established to have means and standard deviations  substantially different from the previous scales to signal that these are new assessments for new  standards and comparisons of results to earlier years should not be made.    The TAC complimented DRC on their plans for building the new vertical scales. The TAC  did recommend that vertical linking items be carefully selected so that they reflect the best  available items for spanning alignment of content between the two adjacent grades. The  TAC would like to see the results for building the vertical scale using a) items from above  and below grades, b) items from only above grades and c) items from only below grades  before deciding which approach should be used.
    Interim assessments    DESE has asked DRC to develop an interim assessment for grades 3 through 8 in ELA and  mathematics and for grades 5 and 8 in science to be ready for the 2017/18 year. These interim  assessments consist of one form per subject and grade and are also being called “pretests.” The  TAC asked what the goals were for these pretests and was told they would be used as practice  tests that mimic the operational tests in format and length. The practice tests could be taken any  time and as many times as desired between the period November and June. The plan is to  calibrate the practice test to the operational test using first-time test-taker data only. Results  would be reported on the same scale as the operational test and end of year proficiencies would  be reported.    The TAC liked the plans, but recommends that the tests be called “practice tests,” not  “interim assessments” nor “pretests.”    Vertical articulation between middle and high school assessments    DESE has asked its two contractors (DRC and Questar) to work together to better articulate high  school assessments and Grade-Level assessments when new standards are set for each. The two  contractors jointly talked to the TAC about their plans, saying that a formal proposal for vertical

    5
    articulation will be presented to DESE and the TAC at the next meeting. The goals of the vertical  articulation are to have a K-12 assessment system where the proficiency impact data is  coordinated between the assessment systems for K-8 and high school. Further, where there are  jumps in the impact data, the articulation should provide an explanation for why those jumps  make sense. Essentially, the idea is that if a student is proficient at one Grade-Level he or she is  ready to be proficient at the next higher Grade-Level.    With those goals in mind, the two contractors plan to set high school standards first. Maybe they  will use ACT benchmarks and impact data to inform the high school standard-setting. Then,  Grade-Level proficiency standards would be set within the context of the previously set EOC  standards. At one point in the discussion, a rationale for higher percents proficient for EOCs was  offered, noting that the focus of the course and the assessment for a single course might cover a  narrower domain than the domain for an entire Grade-Level for a subject. These narrower  domains might be more easily taught and mastered. The TAC noted that the contractors use  different IRT models and that this might influence the selection of a response probability  for standard-setting. The TAC further noted that there can be difficulties for standard-setting when a test is centered on the standards but not well centered on actual student  achievement levels. The TAC concluded that while it is all right to have different response  probability values for grades 3-8 and high school, the same response probabilities should  be used across subjects and grades for grades 3-8 and across courses for End-of-Course  tests in high school. The TAC was informed by DESE that they are working on a policy  piece addressing these issues; and the TAC recommends that that policy piece be included  in the proposal from the two contractors addressing vertical articulation of the two  assessment systems.    EOC 2016 /17 results    DESE Commissioner Margie Vandeven joined the TAC for a discussion of whether the 2016/17  results for Algebra I and English II are valid for use in the state accountability program. The  discussion was a continuation from the TAC conference call held in July. The concern is that  student performance in these two subjects dropped substantially from the previous year. In its  July conference call, the TAC concluded that this was a form effect, at least in part, rather than a  true drop in performance. The TAC noted that when the same form was used two years prior as  was used in 2017, performance was comparable (although somewhat higher). While it is  common to use different forms over years for EOC tests, the forms were not built to ensure that  they are psychometrically on the same scale. Importantly, 2017 is the last year for using the non-equated forms; in the future, EOC assessment forms will be carefully built so that the forms are  not only reliable and valid but also comparable. At this point in time, results of the 2016/17  assessments have been reported to students, but their use for accountability has not yet happened.  The interpretation of results of the other EOC assessments is not being called into question.    The TAC was informed that a performance index is built based on student proficiency levels  with 1 point for below basic, 3 points for basic, 4 points for proficient and 5 points for advanced.  Using this index, three measures are used for accountability: progress, value-added and status.  The TAC noted that neither value-added nor progress requires that forms be comparable between  2016 and 2017 since value-added is metric free and progress is measured such that it reflects

    6
    only change between two years ago and the current year (i.e., 2015 and 2017); for those two  years the same form was used. The problem then is status.     Discussion identified six possible actions:
    1. Drop the 2016 results and use only 2015 and 2017 for which the same form was used.
    2. Hold schools and districts harmless, meaning that if a school or district does better in  2017, on results as reported they get credit for the improvement, otherwise they get the  same accountability levels as for the prior year.
    3. Cut scores could be adjusted by, say, a single raw score point to make the results between  2016 and 2017 appear more comparable.
    4. Use equipercentile procedures, putting the form used in 2017 on the same scale as the  form used in 2016
    5. Accept the results as reported to students and use the results for accountability  accordingly.
    6. Exclude the results for Algebra I and English II in accountability and don’t report those  results.
    Of the above six possibilities, five members of the TAC favored the sixth option, not using  the results for accountability and not reporting them. Two members of the TAC favored  the first option; drop the 2016 results and use only 2015 and 2017 for which the same form  was used. By the end of the meeting, however, all seven members of the TAC favored the  sixth option.  The TAC recommends that in the future, every effort be made to analyze the  data before the results are reported to students so any problems can be detected and  corrected.    Small-scale pilot update    Questar is developing new writing prompts and performance tasks. In the previous meeting, the  TAC recommended that these be pilot tested and in a later TAC meeting, the TAC reviewed  plans for a pilot test. At the time of that review, TAC strongly recommended that the pilot be  online as is the operational test. At the current meeting, Questar reported that the pilot will take  place on September 3; the pilot will be online and approximately 200 students reflecting a range  of achievement levels will be used to pilot each prompt and performance task. There will be a  different sample of students for each prompt and task. The TAC complimented Questar on  these plans.    Stand-alone field test for science    Questar plans to build four forms each for Biology and Physical Science: two operational forms,  one pre-test form, and one breach form for each assessment. Apparently, 95% of the students in  the course will take the field test. There was some discussion as to whether or not students  would be comparably motivated to do well on the field test as they will be motivated on the

    7
    new operational test. There was some discussion as to whether or not items in the field test  could include items from the item pool to estimate the size of motivation effects. Ultimately,  the TAC was supportive of Questar’s plans for the field test.    Lexile® and Quantile® measures    Ellie Sanford from MetaMetrics led the TAC in a discussion of how Missouri might report  student achievement in reading in Lexiles® and in mathematics in Quantiles®. DESE is  interested in exploring these possibilities and asked that the TAC provide their thoughts on the  pros and cons of doing this. The motivation for the initiative is to provide students and parents  with new ways of thinking about student achievement levels and appropriate future instruction.  These reporting metrics are based on the concept of prerequisites. Knowing what prerequisites  students have mastered provides suggestions as to where their next instruction should be.     Sanford mentioned that one possibility was to embed MetaMetrics items in Missouri assessments  so that the Missouri assessments could be reported on the MetaMetrics scales. Discussion  revealed that currently many districts in Missouri are using other tests than those given by the  state that report results in Lexiles® and Quantiles®. The TAC wondered whether it would be  possible to use these data to link to state assessment results and put the state assessment  results on the MetaMetrics scales rather than having the additional student burden of  adding items to the state assessments to accomplish that end.    The TAC was uncertain as to DESE’s questions concerning MetaMetrics and its products.   DESE will prepare questions for the TAC to address on these issues at a future meeting.
    In closing    The next meeting of the TAC is scheduled for December 7 and 8, 2017, and the meeting after  that for March 8 and 9, 2018. Yet a third meeting was scheduled for August 2 and 3, 2018.

    1

              7/28/2017
    To:  Lisa Sireno and Shaun Bates
    From:  Andy Porter, Chair, Missouri Technical Advisory Committee
    Subject:  Minutes of Missouri TAC Meeting on July 27, 2017
    The Missouri Technical Advisory Committee met by telephone conference call on July 27, 2017.  Members of the TAC in attendance were Bertha Doar, director of assessment, St. Louis Missouri  Public Schools; Karla Egan, independent consultant; Ron Mertz, St. Louis public schools, retired;   Andy Porter, chair of TAC, University of Pennsylvania;  and Ed Roeber, independent  consultant;  DESE: Lisa Sireno,  Shaun Bates, Debbie Jameson, Commissioner Margie Vandeven, Blaine  Henningsen, Jocelyn Strand, Jason Young, Janet Duncan, Angie Riner-Mooney, Jeff Falter, Chris  Neale and Lisa Scroggs; Questar: Elliot Dunn, Adam Johnson, Katie McClarty, Tess Dawber and  Scott Bishop.
    Discussion focused on End-of-Course test results for spring 2017 and their validity for use in  student, school and district accountability. In Missouri, there are End-of-Course assessments for  nine subjects: algebra I, algebra II, American history, biology, English I, English II, geometry,  government and physical science. For most subjects, the assessments are given three times a  year: summer, fall and spring. By far the largest number of students is assessed at the spring  assessment. While the number of years of assessment results vary by subject, all subjects were  assessed at least in 2014/15, 2015/16 and 2016/17. For 2 of the 9 assessed subjects, there was  a substantial drop in the percent of students found to be proficient or above from 2016 to  2017. In algebra I, the percent proficient or above in the spring of 2016 was 7 percentage points  higher than in spring of 2017. Similarly for English II, the percent proficient or above in the  spring of 2016 was 9.5 percentage points higher than in spring of 2017. For the other six  subjects, the change from 2016 to 2017 for spring assessments varied from a 1 percentage  point increase to a 3.4 percentage point drop.
    Comparing student performance over years is complicated by the fact that for some subjects  different forms of the assessment are used in different years and for some subjects the number  of students tested differs from year-to-year. Thus, changes in performance from one year to  the next could be due to a change in test form and/or a different cohort composition from year  to year. The best data for making year-to-year comparisons is the spring assessment results  because that is when by far the largest number of students was tested with the same form.
    Lisa Sireno provided the framework for discussion. Advice was sought on using the End-of-Course assessment results in spring 2017 for accountability purposes. In particular the results  for algebra I and English II were of concern due to the drop in percentage of students judged to  be proficient or better. In each subject, the same test form was used in spring of 2017 as was

    2

    used in spring of 2015, but a different form was used in spring of 2016. In algebra I, the spring  results for 2015, 2016 and 2017 were 62.8%, 67.4% and 60.4% judged proficient or better and  for English II the comparable results were 74.5%, 80.8%, 71.3%. When comparing 2015 to 2017  where the same forms were used for each assessment, the drop in performance was much  more modest than from 2016 to 2017 where a different form was used. While the number of  students tested in spring 2017 was less by 1000 students or so than the number tested in 2015,  the TAC judged that the primary reason for a drop in performance was due to the form used in  spring of 2015 and spring of 2017 being easier than the form used in spring of 2016 for each of  these two subjects.
    The question became whether or not the results should be used for accountability purposes as  obtained or whether some adjustment should be made to the results for 2017. Lowering the  cut score for proficient or above one raw score point would result in 4 percentage points  increase in students judged proficient or better in English II and 3.7 percentage points increase  for algebra I. Because the state intends to do an early release of accountability data for the  year, a decision was needed now.
    The TAC judged that in neither of the two subjects was the drop in performance from spring  2016 to spring 2017 affected by a change in the number of students tested. Thus, the drop in  performance is likely due to either a true drop in student achievement and/or a difference in  the difficulty of the form used between the two springs. The TAC noted that there was a small  drop in performance from 2015 spring to 2017 spring when the same form was used at each  assessment point. While the TAC concluded there is no strong psychometric solution to  adjusting for the almost certain form affect, in fairness to students, schools and districts,  some adjustment seemed warranted. Thus, the TAC recommended dropping the proficient  cut score by one raw score point for each subject for spring and summer 2017 results (if the  same test form is used in summer 2017 as was used in spring 2017).
    The TAC noted that the End-of-Course forms would not be used again after the summer 2017.  The TAC strongly recommends that in developing new forms for the End-of-Course  assessments, a design be used so that each form shares a sufficient number of common items  with each other form so that forms can be linked one to another thus yielding comparable  scale scores across forms.

    1

              6/7/2017
    To:  Lisa Sireno and Shaun Bates
    From:  Andy Porter, Chair, Missouri Technical Advisory Committee
    Subject:  Minutes of Missouri TAC Meeting on June 2, 2017
    The Missouri Technical Advisory Committee met by telephone conference call on June 2, 2017.  Members of the TAC in attendance were Bertha Doar, director of assessment, St. Louis Missouri  Public Schools; Karla Egan, independent consultant; Ron Mertz, St. Louis public schools, retired;  Barbara Plake, University of Nebraska, retired; Andy Porter, chair of TAC, University of  Pennsylvania; Ed Roeber, independent consultant; and Phoebe Winter, independent  consultant; DESE: Lisa Sireno,  Shaun Bates Debbie Jameson and Lisa Scroggs; Questar: Elliot  Dunn, Adam Johnson, Katie McLarty, Scott Bishop and Josh Borton.
    Discussions focused on three issues from the spring 2017 administration of end of course tests:  English I and II, algebra I and II, biology, geometry, government, American history, and physical  science. Performance on English II, algebra I or II, biology and government are used for  accountability purposes. Issue one concerned end of course testing on April 25, when heavy use  of the system resulted in bandwidth limitations. Issue two concerned a software malfunction  for the algebra I performance event. Issue three concerned the occurrence of students being  timed out while responding to the English I and II writing tasks.
    1. Impact of bandwidth limitations on April 25
    Across all subjects, 14,666 students were recognized as either logged on and taking an  assessment but with slow response time in moving from one item to the next or attempting to  log on and being unsuccessful. At the time of the TAC meeting, the numbers in each of these  two groups were unknown, but the number of students logged on and experiencing delays in  moving from one item to the other is believed to be much the smaller group. Additional  students of unknown numbers were unable to log on at all and therefore took the test at  another time. In the text that follows these will be referred to as groups 1,2 and 3. Questar  reported that the bandwidth problem was fixed within 90 minutes and that no student  responses were lost. Testing windows were extended for approximately two dozen districts  that requested an extension.
    The question presented to the TAC was what course of action should be taken by DESE for  accountability purposes. The discussion determined that student accountability was not a  problem, since student scores on these tests are not used for student accountability. They can  be used by teachers as one piece of information to determine a student’s course grade, but this

    2

    is a teacher option and teachers did know which students were affected by the “slow down”.  Thus, discussion focused on purposes of district and school accountability.
    The TAC identified four options:
    1. Use the obtained results as is for accountability purposes;
    2. Do not use the results for accountability purposes;
    3. Use some statistical procedure to adjust the results and use those adjusted results for  accountability purposes;
    4. Remove the affected students for one set of results for districts and schools and use the  obtained results as another set of results; for each district and school, for accountability  purposes use the better of their two results
    No members of the TAC were in favor of statistical adjustment. Not using the results for  accountability was not an attractive option, especially since EOC results have recently not been  available for accountability. Ultimately the TAC favored using for accountability option 4, the  better results from the data as obtained and the data after affected students were removed.  There was considerable discussion as to whether to remove only those students from group  1, those logged on and experiencing a serious slowdown in moving from one item to the next,  or to also remove those from group 2, those that experienced the frustration of trying to log  on and not being able to. Unfortunately, the membership of group 3, those who couldn’t even  register an attempt to log on is unknown. The TAC recognized that the frustration experienced  by students in each of these three groups could have bearing on their performance on the  affected test for them. Questar estimates that approximately half of the students in the state  were possibly affected.
    Though Questar presented contrasts in results for the various effects in groups from previous  years and the current year, it was impossible to interpret these comparisons because the  representativeness of the affected groups to students statewide was unknown. Questar  estimated that they would have the analysis file necessary to make these representativeness  determinations on or before June 19. Some members of the TAC believed that only group 1  members might be used in the preferred option for accountability purposes if the students in  group 2 were representative to statewide students. Other members of the TAC felt that both  groups 1 and 2 should be used in the preferred option for accountability purposes regardless.  The TAC will meet again virtually once Questar has produced the requested analyses to  determine the representativeness of students in the affected groups. Not only did the TAC  recommend that Questar produce results on representativeness in terms of prior-year test  scores and demographics but also results on the percent of students in each affected group in  aggregate and the distribution of percent of affected students in each group for schools.  Questar was charged with providing an analysis plan to DESE by close of business the  following Monday (the TAC meeting was on Friday).

    3

    2.  Software malfunction for the algebra I performance event
    For 172 students, the performance event software for algebra I created unscorable responses.  These unscorable responses were triggered by students entering a select few characters that  for unknown reasons created gibberish for their entire answer. While this problem has been  fixed for future administrations, the question to the TAC was what to do about the affected  students. Questar presented the TAC in advance of the meeting two short papers addressing  this issue. The Questar proposal was to statistically link student performance from session one  which was not affected to student performance for the full test for those not affected. Then, to  use the statistical link to produce the predicted results for the affected students. One of the  handouts showed that the affected students tended to be very low performing students. The  plan from Questar was to report student performance for both session 1 (having no  performance event) and session 2 (the performance event worth 10 points).
    The TAC concluded that the statistically estimated scores for the affected students in session  2 was appropriate and justified, conditioned on the correlation between predicted scores and  actual scores for the unaffected students being high. DESE requested that Questar produce  this correlation and in addition a distribution of the absolute value of the deviations of  predicted and actual scores. The TAC will review these results and make a recommendation  as to whether to go ahead with using the statistical predicted scores.
    3. Students being timed out for the English I and II writing tasks
    For some reason, students who spent a long time entering text in response to the writing tasks  on the English I and II end of course assessments were timed out of the testing session. The  students could sign back in to complete the test, but still they experienced frustration.  Unfortunately, the number of students and who the students are who were affected is  unknown. This problem occurred across platforms and is seemingly widespread yet known only  anecdotally.
    The TAC recommended that in the future students be given a warning when they are about to  be kicked off of the system. Questar thought this may solve the problem for the future. As for  the recent administration, the TAC had no solution to offer as the affected students are not  known.

    August 31, 2017
    1

        Talking Points for 2017 EOCs
    Main Message: The results of the 2017 EOCs in English II and Algebra I do not produce year-to-year  comparable results. The department doesn’t plan to release statewide results from these particular  assessments. We will hold the testing vendor, Questar, accountable for not producing usable  assessment results.
     DESE’s TAC – Technical Advisory Committee – concluded that the results of the Algebra I and  English II tests do not produce year-to -year comparable results.
     School districts and charter LEAs have already received Questar’s individual student reports  (ISRs). Consistent with previous guidance from DESE, educators should use raw scores from  those assessments. Accordingly, student record labels may be used as appropriate.
     Districts may also use student End-of-Course (EOC) scores for determining A+ scholarship  eligibility.
     For students who took the Algebra I and English II EOCs, the participation requirement has been  met. Students do  not need to retake these tests.
     We are extremely frustrated with our vendor’s inability to  deliver usable results. Our students  and teachers work hard throughout the year, and they are counting on us to  get this right.
     We are working with our attorney to  pursue our legal options.

    Timeline of Events     When the department received 2017 statewide assessment results in late July, staff immediately  alerted Questar, Missouri’s EOC test vendor, about unexpected changes in the results compared
    to test administrations in previous years.
     DESE staff then convened the state’s Technical Advisory Committee (TAC) – made up of  nationally recognized experts in assessment – to  review the results.
     At its August 18 meeting, the TAC concluded that there were year-to -year comparability issues  for the EOCs for Algebra I and English II. The TAC did not express concern about any other EOCs.
     These tests have now been retired. Missouri educators have been developing mathematics and  English language arts assessments—both Grade-Level Assessments and EOCs—that will be   administered beginning this year.
    APR Talking Points     In developing the APR, DESE will take the following steps to identify the best solution for this  challenging situation:

    August 31, 2017
    2

    o District APRs will not include Algebra I and English II EOCs in the status and progress  calculations.
    o Status targets and progress calculations will be  adjusted to accommodate the loss of the  two EOCs.
    o Priority is on district and charter LEA APRs. Decisions regarding building-level APRs will  be made at a later date.
      Timeline    The department’s next steps will include the following:
    1. MSIP TAC will convene on September 12.  2. Comprehensive Guide revisions will be made during the week of September 18.  3. Public Hearing on MSIP Comprehensive Guide will be held on October 4.
    Event Date
    Achievement data to school districts and  charter schools during week of
    9/5
    Appeals and data corrections window closes   for EOCs  9/22
      Further timeline information will be published as soon as it is available.    What are the department’s next steps regarding assessments?    We will convene a work group to review and make recommendations regarding ways to improve test  administration and delivery of results. Topics will include such things as
     a timeline for the release of scores to districts,    the length of testing windows, and   technology variations among districts and charter LEAs.

    September 1, 2017      Dear Parents:    A top priority of educators is to help prepare Missouri children for success, not only in school  but also for life after graduation. We know you are proud of your children, and we are too.     We want to inform you about some developments resulting from the 2017 End-of-Course  assessments in Algebra I and English II. The technical advisory committee (TAC) retained by  the Department of Elementary and Secondary Education (DESE) to review assessment results  each year determined that the 2017 Algebra I and English II assessments are not suitable for  establishing assessment performance trends. Upon the recommendation of the TAC, DESE will  exclude data from these assessments in school district and charter school annual performance  reports (APRs).     The APRs that we provide to school districts and charter schools look at multiple factors,  including state assessments in English language arts, mathematics, science, and social studies;  a host of college and career readiness indicators; attendance rates; and graduation rates. For  each of these indicators, multiple years of data are used. Your child’s class grades and any local  decisions about A+ eligibility, as it pertains to these two assessments, are not at risk.    I appreciate your understanding as we work through this unfortunate event.     Thank you for supporting Missouri public education and your local school district.    Sincerely,
      Margie Vandeven  Commissioner of Education

    AGENDA
    Missouri Technical Advisory Committee
    July 15, 2016  9:00 a.m. – 1:00 p.m.
    Conference call/web meeting

    9:00 a.m. Topic 1:  Grade-Level Mathematics and English Language Arts Cut Scores    Hello and introductions (DESE, TAC, DRC)    Presentation (DRC)
      Commentary from observers (Karla Egan)    Discussion
        How to join:
    JOIN WEBEX MEETING  https://datarecognitioncorpaudio.webex.com/datarecognitioncorpaudio/j.php?MTID =m12cd39acdb8511ad35e4b5e44e80986b  Meeting number: 282 892 866  Host key: 896239  Meeting password: Eftbert5
    JOIN BY PHONE  Dial-in: 1-800-549-9519  P-code: 16250282
      Break
      11:00 a.m. Topic 2:  EOC Spring Biology Form Performance    Hello and introductions (DESE, TAC, Questar)    Presentation (Questar)
      Discussion
        How to join:
    https://global.gotomeeting.com/join/207782485
        You can also dial in using your phone.  United States : +1 (224) 501-3212
    Access Code: 207-782-485  First GoToMeeting? Try a test session: http://help.citrix.com/getready    1:00 p.m. Adjourn