Message: RE: Algebra I and English II Follow-up

RE: Algebra I and English II Follow-up
FromTeresa DawberDate  Wednesday, August 2, 2017 1:59 PM
To
Katie McClarty;Scott Bishop;Sireno, Lisa;Elliot Dunn
Cc
Bates, Shaun;Adam Johnson
SubjectRE: Algebra I and English II Follow-up

Hi Everyone,

Following up from our discussion on Monday and earlier today, please find attached an Excel file that shows the original RSS tables beside the revised RSS tables when the 3 cut scores are lowered by 1 raw score point. See worksheets called AL1 FormH Revised and EN2 Form2104H Revised. The changes result in two raw scores mapping to 200 and 225 for Algebra I and two raw scores mapping to 182, 200 and 225 for English II. The raw score frequency distributions  (raw score, count, percent) and performance level data are also in both worksheets.

 

The impact data across years and admins is presented in the worksheet called Impact data across admins. The new data with all 3 cut scores adjusted is in the last column of each table (column heading SPR 2017 Form H REVISED Cuts). The other data in the tables are the same as what was sent to the TAC for discussion. The reference document for the TAC is the file called MO EOC Year to Year Results 2017-07-25.

 

Please let me know if you have any questions.

Thanks,

Tess

 

 

From: Katie McClarty
Sent: Thursday, July 27, 2017 10:13 PM
To: Scott Bishop; Sireno, Lisa; Elliot Dunn
Cc: Bates, Shaun; Adam Johnson; Teresa Dawber
Subject: RE: Algebra I and English II Follow-up

 

I concur with Scott’s recommendation and description. I think it can be confusing to the field when two raw scores map to the same scale score, particularly at the cut, but I think that is a better solution in this case than changing multiple scale scores along the continuum.

 

Thanks,

Katie

 

Katie McClarty, Ph.D.

Chief Assessment Officer

kmcclarty@questarai.com

 

Questar_logo_email_signature

Questar Assessment, Inc.

(952) 997-0399 (Office)

(512) 417-5995 (Mobile)

(952) 997-2770 (Fax)

www.questarai.com

 

Connect with us:

LinkedIn | Facebook | Twitter

 

CONFIDENTIALITY NOTICE: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy, or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you. To reply to our email administrator directly, please send an email to Webmaster@QuestarAI.com.

 

From: Scott Bishop
Sent: Thursday, July 27, 2017 9:43 PM
To: Sireno, Lisa <Lisa.Sireno@dese.mo.gov>; Elliot Dunn <edunn@questarai.com>
Cc: Bates, Shaun <shaun.bates@dese.mo.gov>; Adam Johnson <ajohnson@questarai.com>; Teresa Dawber <tdawber@questarai.com>; Katie McClarty <kmcclarty@questarai.com>
Subject: RE: Algebra I and English II Follow-up

 

Hi again lisa & shaun:

 

There are some good friends of mine who I respect a lot that prefer not to have two RSs convert to the same scale score. (Katie is one of them.)  However, I believe it is more of a personal preference than a strict rule.  There is no psychometric rule against it that I’m aware of and many tests do so very willingly (the ACT assessment being one of them). I believe Katie might say that a concern some have is that users might have questions about why it happens. I’d use the ACT as an example that shows that is okay. Of course, I’d also be willing to help DESE respond to any such questions as your testing partner.    

 

In another state this year we had this happen just by coincidence of their linear transformation and rounding rules and state and their TAC was okay with it. 

 

To avoid it/prevent it, we would have to smooth out several more values in the RSS table.  The issue is the 201 right above the original 200…that would need to change to 202 (b/c the original 200 would change to 201, otherwise you would have another duplicate at 201), and if the SS right above it happened to be 202, it would need to change to 203, and so forth and so on until we found the sweet spot where there was a gap of at least 2 SS points so that we could convert a lower SS without creating another duplication right above it.

 

As a cautionary rule when we must make a tweak to a RSS table, we like to make as few changes as possible (to reduce risk that we might inadvertently mess something else up). This is why I felt it was okay to have two 200s. 

 

Another consideration that came to my mind was whether DESE did any official tracking of SS values over time (like using the State’s mean SS over time to monitor trends or measure outcomes).  I could not recall any and I believe use of the percent of students at and above proficient is the primary outcome that MO/DESE uses.  Perhaps some local districts use mean SSs for tracking for one reason or another, but changing one SS value likely will have little impact (relative to what the SS sem is) and it would be in a favorable  direction b/c the 197 is becoming a little bigger.  (Perhaps the fairest thing to do in this case would be to make another SS value 3 points smaller, but I’ve never known anyone to do this in my 15 years in testing…not to say its never happened, just that that has been my own personal experience).

 

I’m very confident the risk for unintended consequences is small to nonexistent, but I invite Tess or Katie to share their experiences as well.

 

Yours truly,

 

Scott

 

 

 

From: Sireno, Lisa [mailto:Lisa.Sireno@dese.mo.gov]
Sent: Thursday, July 27, 2017 9:15 PM
To: Scott Bishop; Elliot Dunn
Cc: Bates, Shaun; Adam Johnson; Teresa Dawber
Subject: RE: Algebra I and English II Follow-up

 

Scott, are there broader implications this year and for summer 17 for changing the scale score cut? Pros and cons from your perspective?

 

Lisa


From: Scott Bishop [sbishop@questarai.com]
Sent: Thursday, July 27, 2017 9:10 PM
To: Sireno, Lisa; Elliot Dunn
Cc: Bates, Shaun; Adam Johnson; Teresa Dawber
Subject: Re: Algebra I and English II Follow-up

Yes, of course.  I'll be up a while longer too if you had any questions tonight.  

 

 

On Thu, Jul 27, 2017 at 9:06 PM -0500, "Sireno, Lisa" <Lisa.Sireno@dese.mo.gov> wrote:

Not at this time. Are you, Scott, Tess, and/or Katie available for a quick discussion tomorrow morning?

 

Thanks,

Lisa


From: Elliot Dunn [edunn@questarai.com]
Sent: Thursday, July 27, 2017 7:12 PM
To: Sireno, Lisa
Cc: Bates, Shaun; Adam Johnson; Scott Bishop; Teresa Dawber
Subject: FW: Algebra I and English II Follow-up

Hi Lisa,

 

Can you confirm that Janet’s comment is an official DESE request?

 

Thank you,

Elliot

 

From: Duncan, Janet [mailto:Janet.Duncan@dese.mo.gov]
Sent: Thursday, July 27, 2017 3:49 PM
To: Scott Bishop <sbishop@questarai.com>; Katie McClarty <kmcclarty@questarai.com>; Sireno, Lisa <Lisa.Sireno@dese.mo.gov>; Adam Johnson <ajohnson@questarai.com>; Elliot Dunn <edunn@questarai.com>; Teresa Dawber <tdawber@questarai.com>
Cc: Bates, Shaun <shaun.bates@dese.mo.gov>; Riner-Mooney, Angela <Angela.Riner-Mooney@dese.mo.gov>; Strand, Jocelyn <Jocelyn.Strand@dese.mo.gov>; Schepers, Kimberly <Kim.Schepers@dese.mo.gov>
Subject: RE: Algebra I and English II Follow-up

 

My preference is an updated scale score and achievement level.

 

Janet L. Duncan, Director

Accountability Data

Office of Quality Schools

MO Department of Elem. & Sec. Education

Phone:  (573) 751-1362

Fax:  (573) 526-3045

 

From: Scott Bishop [mailto:sbishop@questarai.com]
Sent: Thursday, July 27, 2017 3:47 PM
To: Katie McClarty; Sireno, Lisa; Adam Johnson; Elliot Dunn; Teresa Dawber
Cc: Bates, Shaun; Duncan, Janet; Riner-Mooney, Angela; Strand, Jocelyn; Schepers, Kimberly
Subject: RE: Algebra I and English II Follow-up

 

Thank you Katie,

 

Would DESE like us to change the value of the scale score (SS) that is associated with the new raw score (RS) cut score?  See below.  In Alg I, a RS of 22 is the old cut and it has a SS of 200 in accordance with our business rules.  We will make 21 the new raw score cut per the TAC call.  However, do we change 21’s corresponding SS from 197 to 200 (and then have two RSs with a SS of 200)?  Eng 2 has same situation save for the RS values being 27, 28, and 29.

 

What are your thoughts?  I (Scott) feel having two RS’s with an SS of 200 is okay given this (and summer)) will be last use of the test. 

 

Thank you,

 

Scott

 

rs

SS

stderr

ach_lev

 

ALG I

 

21

197

7

2

 

22

200

7

3

Old RS Cut

23

201

7

3

 

ENG II

 

27

197

6

2

 

28

200

6

3

Old RS Cut

29

201

6

3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

X

 

 

-----Original Message-----
From: Katie McClarty
Sent: Thursday, July 27, 2017 3:34 PM
To: Sireno, Lisa; Adam Johnson; Elliot Dunn; Scott Bishop
Cc: Bates, Shaun; Duncan, Janet; Riner-Mooney, Angela; Strand, Jocelyn; Schepers, Kimberly
Subject: RE: Algebra I and English II Follow-up

 

I'm adding Scott so he can follow up about the specific changes for the cut scores. He and I just got off a phone call about that, and he will be sending our recommendations.

 

Thanks,

Katie

 

Katie McClarty, Ph.D.

Chief Assessment Officer

kmcclarty@questarai.com

 

 

Questar Assessment, Inc.

(952) 997-0399 (Office)

(512) 417-5995 (Mobile)

(952) 997-2770 (Fax)

www.questarai.com

 

Connect with us:

LinkedIn | Facebook | Twitter

 

CONFIDENTIALITY NOTICE: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy, or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you. To reply to our email administrator directly, please send an email to Webmaster@QuestarAI.com.

 

-----Original Message-----

From: Sireno, Lisa [mailto:Lisa.Sireno@dese.mo.gov]

Sent: Thursday, July 27, 2017 3:20 PM

To: Adam Johnson <ajohnson@questarai.com>; Elliot Dunn <edunn@questarai.com>; Katie McClarty <kmcclarty@questarai.com>

Cc: Bates, Shaun <shaun.bates@dese.mo.gov>; Duncan, Janet <Janet.Duncan@dese.mo.gov>; Riner-Mooney, Angela <Angela.Riner-Mooney@dese.mo.gov>; Strand, Jocelyn <Jocelyn.Strand@dese.mo.gov>; Schepers, Kimberly <Kim.Schepers@dese.mo.gov>

Subject: Algebra I and English II Follow-up

 

Hello,

 

As discussed during our TAC meeting this morning, we ask you to proceed with the rescoring of the spring 2017 Algebra I and English II forms.   So that we are on the same page, please return to me a description of the changes to be made in the cut scores.

 

In the meantime, as Adam and I have confirmed this afternoon:

 

Questar will provide a spring 2017 GRF with updated Algebra I and English II scores on August 11.

Questar will provide a plan and timeline for posting updated ISRs and data in ITester admin and the required related communications.

 

Please follow up with questions and clarifications.

 

Thank you,

Lisa

 




AL1 FormH Revised

 ABCDEFGHIJKLMNOPQRSTUVWX
1ORGINAL RSSrsSSstderrach_levREVISED RSSrsSSstderrach_levSpring2017RS_TotalCOUNTPERCENT
2 01004010100401RS Results0100.01651
3 11072211107221180.01321  
4 2124161  2124161  260.00990  Original Revised
5 3134141  3134141  360.00990   Raw ScoreN-CountPercentRaw ScoreN-CountPercent
6 414212141421214200.03301Below Basic0-16 12,39220.450-15 10,153 16.76
7 514811151481115560.09244Basic17-21 11,60119.1516-20 11,557 19.08
8 6153101615310161060.17497Proficient22-34 25,06941.3821-33 25,822 42.62
9 7158101715810172120.34994Advanced35-50 11,52019.0234-50 13,050 21.54
10 81629181629183620.59754Below Basic+Basic  23,99339.60  21,710 35.84
11 91669191669195480.90456Proficient+Advanced  36,58960.40  38,872 64.16
12 10169811016981107921.30732Total 60,582100.00   60,582 100.00
13 111728111172811110591.74804
14 121758112175811213372.20693
15 131788113178811316202.67406
16 141817114181711419803.26830Scale Score CutsBasicProficientAdvanced 
17  15183711518371 1520313.35248Algebra I187200225
18 1618671  16186 18771 2 1622393.69582
19 1718872  17188721723313.84768
20 181907218190721823223.83282
21 191937219193721922983.79321
22 20195722019572 2023673.90710
23 2119772  21197 20072 3 2122833.76845
24 2220073  22200732223153.82127
25 232017323201732321853.60668
26 242037324203732421313.51755
27 252057325205732520803.43336
28 262076326207632620223.33763
29 272096327209632720353.35908
30 282116328211632819533.22373
31 292136329213632918803.10323
32 302156330215633017702.92166
33 312176331217633118343.02730
34 322196332219633217912.95632
35 33221733322173 3315432.54696
36 3422373 34223 22573 4 3415302.52550
37 3522574 35225743513952.30266
38 362277436227743613492.22673
39 372297437229743712782.10954
40 382317438231743811791.94612
41 392347439234743911091.83058
42 402368440236844010201.68367
43 41239844123984419301.53511
44 42242844224284428031.32548
45 43245944324594436661.09934
46 44249944424994445630.92932
47 4525010445250104454600.75930
48 4625011446250114463170.52326
49 4725013447250134472230.36810
50 4825016448250164481290.21293
51 492502244925022449770.12710
52 502503945025039450220.03631


EN2 Form2014H Revised

 ABCDEFGHIJKLMNOPQRSTUVW
1ORIGINAL RSSrsSSstderrach_levREVISED RSSrsSSstderrach_levSpring2017RS_TotalCOUNTPERCENT
2 01052810105281RS Results0110.018
3 11241611124161360.010English II Form 2014H
4 2135111 2135111  470.011  Original Revised
5 314291 314291  5150.024   Raw ScoreN-CountPercentRaw ScoreN-CountPercent
6 4146814146816210.034Below Basic0-18 3,576 5.810-17 2,823 4.58
7 5150715150717350.057Basic19-27 14,098 22.8918-26 12,425 20.17
8 6153716153718720.117Proficient28-37 34,148 55.4427-36 32,962 53.51
9 7156617156619660.107Advanced38-45 9,772 15.8737-50 13,384 21.73
10 815961815961101140.185Below Basic+Basic  17,674 28.69  15,248 24.76
11 916161916161111470.239Proficient+Advanced  43,920 71.31  46,346 75.24
12 10163611016361122240.364Total 61,594 100.00   61,594 100.00
13 11165611116561132730.443
14 12167611216761143460.562
15 13169611316961154110.667
16 14172611417261164990.810Scale Score CutsBasicProficientAdvanced  
17  15174611517461175760.935English II182200225  
18  16176611617661187531.223
19 17178611717861  198691.411
20 1817961 18179 18261 2  2010141.646
21 1918262 19182622111751.908
22 201835220183522212962.104  
23 211855221185522315082.448
24 221875222187522417812.892
25 231895223189522518422.991
26 241915224191522621873.551
27 251936225193622724263.939
28 26195622619562  2826024.224
29 2719762 27197 20062 3  2927894.528
30 2820063 28200633031755.155
31 292016329201633132935.346
32 302036330203633235265.725
33 312056331205633336986.004
34 322086332208633437966.163
35 332106333210633539206.364
36 342126334212633637376.067
37 352156335215633736125.864
38 36218733621873  3831025.036
39 3722173 37221 22573 4  3925244.098
40 3822574 38225744017922.909
41 392278439227844112372.008
42 40232844023284426571.067
43 41236944123694433240.526
44 4224210442242104441140.185
45 432501244325012445220.036
46 4425016444250164
47 4525029445250294


Impact data across admins

 ABCDEFGHIJK
1  
 Algebra I
3 Table 1. Percent of Students at Each Performance Level
4  2014-20152015-20162016-2017 
5  FALL SPRSUMFALLSPR SUMFALLSPRSPR 2017
6   Form GForm HForm GForm HForm GForm GForm GForm HForm H REVISED Cuts
7 Below Basic23.618.121.435.813.020.22420.516.8
8 Basic24.519.023.520.219.627.12519.119.1
9 Proficient39.043.744.530.748.741.73541.442.6
10 Advanced12.919.110.613.418.711.11619.021.5
11 Total 100.0100.0100.0100.0100.0100.0100.0100.0 
12   
13   
14  Table 2. Percent of Students at Proficient and Advanced vs Basic and Below Basic
15  2014-20152015-20162016-2017 
16  FALL SPRSUMFALLSPR SUMFALLSPRSPR 2017
17   Form GForm HForm GForm HForm GForm GForm GForm HForm H REVISED Cuts
18 Below Basic + Basic48.137.144.956.032.647.248.439.635.8
19 Proficient + Advanced51.962.855.144.167.452.851.660.464.2
20 Total 100.0100.0100.0100.0100.0100.0100.0100.0 
21   
22   
23  English II
24 Table 1. Percent of Students at Each Performance Level
25  2014-20152015-20162016-2017 
26   FALL SPRSUMFALLSPR SUMFALLSPRSPR 2017
27  Form GForm HForm GForm HForm GForm GForm GForm HForm H REVISED Cuts
28 Below Basic14.15.021.222.63.220.216.05.84.6
29 Basic30.120.538.629.716.036.428.022.920.2
30 Proficient44.556.137.039.763.040.847.455.453.5
31 Advanced11.318.43.38.017.82.58.615.921.7
32 Total 100.0100.0100.0100.0100.0100.0100.0100.0  
33    
34   
35 Table 2. Percent of Students at Proficient and Advanced vs Basic and Below Basic
36  2014-20152015-20162016-2017 
37   FALL SPRSUMFALLSPR SUMFALLSPRSPR 2017
38  Form GForm HForm GForm HForm GForm GForm GForm HForm H REVISED Cuts
39 Below Basic + Basic44.225.559.852.319.256.744.028.724.8
40 Proficient + Advanced55.874.540.347.780.843.356.071.375.2
41 Total 100.0100.0100.0100.0100.0100.0100.0100.0