Alternate Maryland School Assessment
Mathematics and Reading
Grades 3, 5, 8 and 10
2003-2004 Technical Report
|
|
| TABLE OF CONTENTS |
PURPOSE |
|
|
| 1.0 |
HISTORICAL OVERVIEW |
| |
|
|
| |
1.1 |
OVERVIEW OF THE ALTERNATE ASSESSMENT |
|
|
Background |
|
|
|
Purpose of the Assessment |
|
|
Participation in the ALT-MSA |
|
|
|
Organizations and Groups Involved |
|
| |
|
|
| Maryland State Department of Education (MSDE) |
| Pearson Educational Measurement |
| Inclusive Large-Scale Standards and Assessment Group (ILSSA) |
| Advisory Committee | |
| |
|
|
|
|
1.2 |
TEST DESIGN AND BLUEPRINT |
| |
|
Review Standards |
| |
|
Test Design |
| |
|
Test Blueprint |
| |
|
|
|
| 2.0 |
PORTFOLIO ASSESSMENT CONSTRUCTION & ADMINISTRATION |
| |
|
|
|
| |
2.1 |
TIMELINE |
| |
|
|
|
| |
2.2 |
CONTRIBUTORS |
| |
|
Local Accountability Coordinator |
| |
|
Principal |
| |
|
School Testing Coordinator |
| |
|
Test Examiners |
| |
|
Instructional Assistants (Under the supervision of the test examiners) |
| |
|
Student |
| |
|
Parents/Guardians |
| |
|
ALT-MSA Facilitator |
| |
|
|
|
| |
2.3 |
ALT-MSA DEVELOPMENT AND ADMINISTRATION |
| |
|
ALT-MSA Portfolio Planning and Development |
| |
|
Acceptable Evidence of Mastery |
| |
|
Eligible Test Examiners |
| |
|
|
|
| |
2.4 |
PORTFOLIO ORGANIZATION |
| |
|
|
| 3.0 |
SCORING AND REPORTING |
| |
|
|
|
| |
3.1 |
SCORING |
| |
|
Recruitment of Scorers and Scoring Supervisors |
| |
|
Rangefinding |
| |
|
Training |
| |
|
|
The Training Process |
| |
|
|
Introduction |
| |
|
|
Anchor Portfolio Set and Scoring Guide |
| |
|
|
Practice Portfolio Sets |
| |
|
|
Qualifying Portfolio Sets |
| |
|
|
Training of Scoring Supervisors |
| |
|
Distribution of Portfolios to Scoring Teams |
| |
|
Scoring Procedure |
| |
|
Quality Control |
| |
|
|
Backreading |
| |
|
|
Validity Sets |
| |
|
|
Reports Generated to Monitor Scorers and Scoring Accuracy and Control Scorer Drift |
| |
|
|
Security at the Scoring Site |
| |
|
|
| |
3.2 |
STANDARD SETTING |
| |
|
|
| |
3.3 |
REPORTS |
| |
|
Description and Interpretation of Scores |
| |
|
|
Mastery Objective Score |
| |
|
|
Mastery Percentage Score |
| |
|
|
Evidence and Indicators of Positive Practice |
| |
|
Reports |
| |
|
|
Accountability Reports |
| |
|
|
Non-Accountability Reports |
| |
|
|
|
| 4.0 |
RELIABILITY AND VALIDITY |
| |
|
|
| |
4.1 |
RELIABILITY |
| |
|
Reader Agreement |
| |
|
|
| |
4.2 |
VALIDITY |
| |
|
Intrinsic Rational Validity Evidence |
| |
|
Content- and Curricular-Related Validity Evidence |
| |
|
Face Validity |
| |
|
Consequential Validity Evidence |
| |
|
Criterion-Related Validity Evidence |
| |
|
|
|
| REFERENCES |
| |
| APPENDICES |
| |
| |
Appendix A: |
Tables |
| |
|
|
| |
Table 1: |
Participation by Grade, Gender, Ethnicity and SES |
| |
Table 2: |
Scorer Qualification Results |
| |
Table 3: |
Summary of Performance on Validity Sets |
| |
Table 4: |
Percentage of Mastery Objectives Scored "Mastered" by Reading Content Standard |
| |
Table 5: |
Percentage of Mastery Objectives Scored "Mastered" by Mathematics Content Standard |
| |
Table 6: |
Reading Mastery Percentages for All Students Tested |
| |
Table 7: |
Mathematics Mastery Percentages for All Students Tested |
| |
Table 8: |
Average Reading and Mathematics Mastery Percentage Scores |
| |
Table 9: |
Reading Proficiency level Frequencies |
| |
Table 10: |
Reading Proficiency level Frequencies by Free/Reduced Lunch Designation |
| |
Table 11: |
Reading Proficiency Level Frequencies by Ethnicity |
| |
Table 12: |
Mathematics Proficiency Level Frequencies |
| |
Table 13: |
Mathematics Proficiency Level Frequencies by Free/Reduced Lunch Designation |
| |
Table 14: |
Mathematics Proficiency Level Frequencies by Ethnicity |
| |
Table 15: |
Percentage of Portfolios Showing Evidence of Positive Practice |
| |
Table 16: |
Percent Perfect Reader Agreement by Reading Content Standard |
| |
Table 17: |
Percent Perfect Reader Agreement by Mathematics Content Standard |
| |
Table 18: |
Percent Perfect Reader Agreement Over All Students |
| |
Table 19: |
Correlation between Positive Practice Scores and Reading/Mathematics Mastery Scores |
| |
|
|
| |
Appendix B: |
ALT-MSA Timeline for 2003-2004 |
| |
Appendix C: |
ALT-MSA Mastery Objective Review Form |
| |
Appendix D: |
Samples of Required Forms |
| |
Appendix E: |
Performance Scoring Center Staff Roles and Responsibilities |
| |
Appendix F: |
Rangefinding Portfolio Selection Criteria |
| |
Appendix G: |
2003-2004 Scoring Process Document |
| |
Appendix H: |
Steps to Monitor Scoring Accuracy and Remedy Drift |
| |
Appendix I: |
Sample Score Reports | |
| |
| |
| |
| |
|
|
|
|
| Leslie Wilson, Assistant State Superintendent |
Division of Accountability and Assessment
Maryland State Department of Education |
| 200 West Baltimore Street |
|
|
|
|
|
|
| Maryland State Department of Education |
| 200 West Baltimore Street |
|
|
|
|
|
|