Message: RE: Algebra I and English II Follow-up

RE: Algebra I and English II Follow-up
RE: Algebra I and English II Follow-up

From | Scott Bishop | Date Thursday, July 27, 2017 9:44 PM

To | Sireno, Lisa;Elliot Dunn
Cc | Bates, Shaun;Adam Johnson;Teresa Dawber;Katie McClarty
Subject | RE: Algebra I and English II Follow-up

Hi again lisa & shaun:

There are some good friends of mine who I respect a lot that prefer not to have two RSs convert to the same scale score. (Katie is one of them.) However, I believe it is more of a personal preference than a strict rule. There is no psychometric rule against it that I’m aware of and many tests do so very willingly (the ACT assessment being one of them). I believe Katie might say that a concern some have is that users might have questions about why it happens. I’d use the ACT as an example that shows that is okay. Of course, I’d also be willing to help DESE respond to any such questions as your testing partner.

In another state this year we had this happen just by coincidence of their linear transformation and rounding rules and state and their TAC was okay with it.

To avoid it/prevent it, we would have to smooth out several more values in the RSS table. The issue is the 201 right above the original 200…that would need to change to 202 (b/c the original 200 would change to 201, otherwise you would have another duplicate at 201), and if the SS right above it happened to be 202, it would need to change to 203, and so forth and so on until we found the sweet spot where there was a gap of at least 2 SS points so that we could convert a lower SS without creating another duplication right above it.

As a cautionary rule when we must make a tweak to a RSS table, we like to make as few changes as possible (to reduce risk that we might inadvertently mess something else up). This is why I felt it was okay to have two 200s.

Another consideration that came to my mind was whether DESE did any official tracking of SS values over time (like using the State’s mean SS over time to monitor trends or measure outcomes). I could not recall any and I believe use of the percent of students at and above proficient is the primary outcome that MO/DESE uses. Perhaps some local districts use mean SSs for tracking for one reason or another, but changing one SS value likely will have little impact (relative to what the SS sem is) and it would be in a favorable direction b/c the 197 is becoming a little bigger. (Perhaps the fairest thing to do in this case would be to make another SS value 3 points smaller, but I’ve never known anyone to do this in my 15 years in testing…not to say its never happened, just that that has been my own personal experience).

I’m very confident the risk for unintended consequences is small to nonexistent, but I invite Tess or Katie to share their experiences as well.

Yours truly,

Scott

From: Sireno, Lisa [mailto:Lisa.Sireno@dese.mo.gov]
Sent: Thursday, July 27, 2017 9:15 PM
To: Scott Bishop; Elliot Dunn
Cc: Bates, Shaun; Adam Johnson; Teresa Dawber
Subject: RE: Algebra I and English II Follow-up

Scott, are there broader implications this year and for summer 17 for changing the scale score cut? Pros and cons from your perspective?

Lisa

From: Scott Bishop [sbishop@questarai.com]
Sent: Thursday, July 27, 2017 9:10 PM
To: Sireno, Lisa; Elliot Dunn
Cc: Bates, Shaun; Adam Johnson; Teresa Dawber
Subject: Re: Algebra I and English II Follow-up

Yes, of course. I'll be up a while longer too if you had any questions tonight.

Get Outlook for iOS

On Thu, Jul 27, 2017 at 9:06 PM -0500, "Sireno, Lisa" <Lisa.Sireno@dese.mo.gov> wrote:

> Not at this time. Are you, Scott, Tess, and/or Katie available for a quick discussion tomorrow morning?

> Thanks,

> Lisa

> From: Elliot Dunn [edunn@questarai.com]
> Sent: Thursday, July 27, 2017 7:12 PM
> To: Sireno, Lisa
> Cc: Bates, Shaun; Adam Johnson; Scott Bishop; Teresa Dawber
> Subject: FW: Algebra I and English II Follow-up

> Hi Lisa,

> Can you confirm that Janet’s comment is an official DESE request?

> Thank you,

> Elliot

> From: Duncan, Janet [mailto:Janet.Duncan@dese.mo.gov]
> Sent: Thursday, July 27, 2017 3:49 PM
> To: Scott Bishop <sbishop@questarai.com>; Katie McClarty <kmcclarty@questarai.com>; Sireno, Lisa <Lisa.Sireno@dese.mo.gov>; Adam Johnson <ajohnson@questarai.com>; Elliot Dunn <edunn@questarai.com>; Teresa Dawber <tdawber@questarai.com>
> Cc: Bates, Shaun <shaun.bates@dese.mo.gov>; Riner-Mooney, Angela <Angela.Riner-Mooney@dese.mo.gov>; Strand, Jocelyn <Jocelyn.Strand@dese.mo.gov>; Schepers, Kimberly <Kim.Schepers@dese.mo.gov>
> Subject: RE: Algebra I and English II Follow-up

> My preference is an updated scale score and achievement level.

> Janet L. Duncan, Director

> Accountability Data

> Office of Quality Schools

> MO Department of Elem. & Sec. Education

> Phone: (573) 751-1362

> Fax: (573) 526-3045

> From: Scott Bishop [mailto:sbishop@questarai.com]
> Sent: Thursday, July 27, 2017 3:47 PM
> To: Katie McClarty; Sireno, Lisa; Adam Johnson; Elliot Dunn; Teresa Dawber
> Cc: Bates, Shaun; Duncan, Janet; Riner-Mooney, Angela; Strand, Jocelyn; Schepers, Kimberly
> Subject: RE: Algebra I and English II Follow-up

> Thank you Katie,

> Would DESE like us to change the value of the scale score (SS) that is associated with the new raw score (RS) cut score? See below. In Alg I, a RS of 22 is the old cut and it has a SS of 200 in accordance with our business rules. We will make 21 the new raw score cut per the TAC call. However, do we change 21’s corresponding SS from 197 to 200 (and then have two RSs with a SS of 200)? Eng 2 has same situation save for the RS values being 27, 28, and 29.

> What are your thoughts? I (Scott) feel having two RS’s with an SS of 200 is okay given this (and summer)) will be last use of the test.

> Thank you,

> Scott

> rs | SS | stderr | ach_lev |
> ALG I |
> 21 | 197 | 7 | 2 |
> 22 | 200 | 7 | 3 | Old RS Cut
> 23 | 201 | 7 | 3 |
> ENG I I |
> 27 | 197 | 6 | 2 |
> 28 | 200 | 6 | 3 | Old RS Cut
> 29 | 201 | 6 | 3 |

> X

> -----Original Message-----
> From: Katie McClarty
> Sent: Thursday, July 27, 2017 3:34 PM
> To: Sireno, Lisa; Adam Johnson; Elliot Dunn; Scott Bishop
> Cc: Bates, Shaun; Duncan, Janet; Riner-Mooney, Angela; Strand, Jocelyn; Schepers, Kimberly
> Subject: RE: Algebra I and English II Follow-up

> I'm adding Scott so he can follow up about the specific changes for the cut scores. He and I just got off a phone call about that, and he will be sending our recommendations.

> Thanks,

> Katie

> Katie McClarty, Ph.D.

> Chief Assessment Officer

> kmcclarty@questarai.com

> Questar Assessment, Inc.

> (952) 997-0399 (Office)

> (512) 417-5995 (Mobile)

> (952) 997-2770 (Fax)

> www.questarai.com

> Connect with us:

> LinkedIn | Facebook | Twitter

> CONFIDENTIALITY NOTICE: The information contained in this e-mail and any attachments may be confidential. If you are not an intended recipient, you are hereby notified that any dissemination, distribution, or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender and permanently delete the e-mail and any attachments immediately. You should not retain, copy, or use this e-mail or any attachment for any purpose, nor disclose all or any part of the contents to any other person. Thank you. To reply to our email administrator directly, please send an email to Webmaster@QuestarAI.com.

> -----Original Message-----

> From: Sireno, Lisa [mailto:Lisa.Sireno@dese.mo.gov]

> Sent: Thursday, July 27, 2017 3:20 PM

> To: Adam Johnson <ajohnson@questarai.com>; Elliot Dunn <edunn@questarai.com>; Katie McClarty <kmcclarty@questarai.com>

> Cc: Bates, Shaun <shaun.bates@dese.mo.gov>; Duncan, Janet <Janet.Duncan@dese.mo.gov>; Riner-Mooney, Angela <Angela.Riner-Mooney@dese.mo.gov>; Strand, Jocelyn <Jocelyn.Strand@dese.mo.gov>; Schepers, Kimberly <Kim.Schepers@dese.mo.gov>

> Subject: Algebra I and English II Follow-up

> Hello,

> As discussed during our TAC meeting this morning, we ask you to proceed with the rescoring of the spring 2017 Algebra I and English II forms. So that we are on the same page, please return to me a description of the changes to be made in the cut scores.

> In the meantime, as Adam and I have confirmed this afternoon:

> Questar will provide a spring 2017 GRF with updated Algebra I and English II scores on August 11.

> Questar will provide a plan and timeline for posting updated ISRs and data in ITester admin and the required related communications.

> Please follow up with questions and clarifications.

> Thank you,

> Lisa