SACSCOC Fifth Year Interim Report
Spacer
Assessment : 2012 - 2013 : Educational Programs :
Psychology MA (School Psychology)
Psychology MA (School Psychology)
2 Goals 2 Objectives 2 Indicators 2 Criteria 2 Findings 2 Actions
| GOAL: Foundational Competence In School Psychology |
| Objective |
|
||||
| Students demonstrate competency in the scientific, methodological and theoretical foundations of professional school psychology. |
| Indicator |
|
|||
|
The PRAXIS II School Psychology Exam is a nationally administered examination used to determine an individual’s qualification for licensure to practice within the field. Candidate competency is evaluated with respect to the following test subcategories: 1. Data Based Decision Making (35%), 2. Research-based Academic Practices (12%), 3. Research-based Behavioral and Mental Health Practices (16%), 4. Consultation and Collaboration (12%), 5. Applied Psychological Foundations (13%), and 6. Ethical, Legal, and Professional Foundations (12%). |
| Criterion |
|
|||
|
A minimum score of 165 is required to obtain the credential of Nationally Certified School Psychologist (NCSP), and thus a score of 165 or better has been established by the SSP Program as the criterion for this objective. In addition, candidates are expected to perform at or above the average range provided by the test developers for each of the six subcategories. |
| Finding |
|
|||
| Eleven SSP students took the PRAXIS II examination during the past academic year. Total scores ranged from 168 to 182 with an average score of 174. All students had scores directly reported to our Program which enables an analysis of subcategory performance. Ten of the eleven students (91%) scored at or above the average performance range for the areas of Data-Based Decision Making and Research-Based Academic Practices. Nine of the eleven students (82%) scored at or above the average performance range for the areas of Research-Based Behavioral and Mental Health Practices, Consultation and Collaboration, and Applied Psychological Foundations. Eight of eleven students (73%) scored at or above the average performance range for the area of Ethical, Legal, and Professional Foundations. |
Actions for Objective:
| Action |
|
|||
|
In the past our students have struggled with Research-Based Academic Practices. This past year we put especial interest into those areas and it appears to have paid off: 91% of the students scored above average this year as opposed to 67% last year. We will continue to stress this aspect of our training program. There was some concern about the category Ethical, Legal, and Professional Foundations, as only 73% of our students score at or above the average in that area. This fall, we will revisit that area and determine what steps could and should be taken to have a greater percentage of our students to well in that area. |
| GOAL: Skill Application |
| Objective |
|
||||
|
Candidates in the school psychology program demonstrate knowledge and improving skill application commensurate with their level of training. Specifically, candidates in their final practicum placement and on internship, both held within the public school setting, will demonstrate appropriate application of professional school psychology skills in the areas of assessment, behavioral consultation, academic intervention and counseling. |
| Indicator |
|
|||
|
(1) Satisfactory ratings from Field Supervisors 1(A) Ratings for Practicum II candidates (Year 2 of 3) 1(B) Ratings for candidates on Internship (Year 3 of 3) On-site, or field, supervisors are asked to evaluate each candidate’s performance in order to gauge their professional performance according to the 11 NASP Domains of Competence. These include: 1) Data-Based Decision-Making and Accountability, 2) Consultation and Collaboration, 3) Effective Instruction and Development of Cognitive/Academic Skills, 4) Socialization and Development of Life Skills, 5) Student Diversity in Development and Learning, 6) School and Systems Organization, Policy Development, and Climate, 7) Prevention, Crisis Intervention, and Mental Health, 8) Home/School Community Collaboration, 9) Research and Program Evaluation, 10) School Psychology Practice and Development, and 11) Information Technology. (2) Satisfactory ratings from Program Faculty 2(A) Faculty Rating Forms (FRF) for each of four Portfolio cases submitted 2(B) Procedural Integrity Rubrics (PIR) for each of four Portfolio cases submitted Candidates completing the Internship Portfolio assessment will obtain satisfactory ratings from the Program Faculty on each of four cases submitted. These cases include: 1) an Assessment case, 2) a Behavioral Consultation case, 3) an Academic Intervention case, and 4) a counseling case. Two faculty members will evaluate each case, and the average of these two ratings on both the FRF and the PIR will be reported. Positive Impact Data (3) Quantitative data gathered as part of the case intervention 3(A) Effect Size 3(B) Percent of Non-Overlapping Data Points (PND) Candidates completing the Internship Portfolio assessment will submit quantitative data gathered as part of the case intervention monitoring for three of four cases submitted. These cases include: 1) the Behavioral Consultation case, 2) the Academic Intervention Case, and 3) the Counseling case. Effect size, percent of non-overlapping data points (PND), or other means of quantitatively evaluating candidates positive impact on the student(s) will be calculated. |
| Criterion |
|
|||
|
1A: Candidates are rated by field supervisors according to a three point scale including the following competency rating categories: Improvement Needed (1), Competent (Supervision Needed; 2), and Professionally Competent (No Supervision Needed; 3). Because candidates in their final practicum will be under supervision for two more years, they are expected to maintain an average rating of “2.0” for each of the 11 NASP Domains evaluated. 1B: Candidates are rated by field supervisors according to a three point scale including the following competency rating categories: Improvement Needed (1), Competent (Supervision Needed; 2), and Professionally Competent (No Supervision Needed; 3). Because candidates completing their internship year will continue to be under supervision for one more year, they are expected to maintain an average rating of “2.0” for each of the 11 NASP Domains evaluated. 2A: Candidates completing their internship experience are required to submit four distinct Portfolio cases. Each case will be reviewed by two faculty members and assigned ratings on the Faculty Rating Form (FRF). These ratings will then be averaged across the two faculty raters. The FRF addresses all domains of practice related to the type of case being reviewed. Each item on the FRF includes the following competency rating categories: Pass (score 1), No Pass (score 0), Not Included (score 0), and Not Applicable (removed from the scoring calculation). Candidates are expected to achieve a minimum domain competency average of 85%. In addition, candidates are given a single faculty rating for the overall case completion. This rating ranges from 1 (Very Poor) to 5 (Very Good). Candidates are expected to achieve a minimum average overall rating of 3 across the two faculty raters, which is equivalent to “average” work completed in the field. 2B: Internship portfolio case submissions are also scored by faculty using a Procedural Integrity Rubric, or PIR. Each case PIR includes critical procedures that must be performed as part of completing the case in order for the intern to be judged as following best practices within the field. Each item on the PIR can be scored as follows: 0 = Incomplete, 1 = Needs Improvement (task is completed, with some concerns), 2 = Completed Satisfactorily (Competency Met), and 3 = Exemplary Performance (task is completed at a level above expectations. Each PIR for the four cases submitted has an established cut score equivalent to achievement of at least 85%. Additionally, candidates are expected to obtain no ratings of “0” on any PIR. 3A: Based on the quantitative data included as part of the Behavioral Consultation, Counseling, and Academic Intervention Portfolio case submissions, the candidate’s impact on student behavior and/or learning can be calculated in a variety of ways. Effect size allows for the comparison of the standard mean difference in student performance during baseline and treatment phases of intervention. An effect size of.8 is considered to be of moderate impact. Candidates are expected to demonstrate moderate impact through either effect size or PND calculation for two of the three quantitative cases submitted. 3B: Based on the quantitative data included as part of the Behavioral Consultation, Counseling, and Academic Intervention Portfolio case, the candidate’s impact on student behavior and/or learning can be calculated in a variety of ways. Percent of Non-overlapping Data points, or PND, provides a comparison of the percentage of data points during the treatment phase that do not overlap with the most extreme baseline phase point. A PND calculation of 60% is considered to be of moderate impact. Candidates are expected to demonstrate moderate impact through either effect size or PND calculation for two of the three quantitative cases submitted. |
| Finding |
|
|||
|
Practicum II Field Supervisor Ratings: There were seven candidates who participated in the final Practicum experience during the Spring 2013 semester. Field supervisor rated our candidates, as a whole, very well and solidly within the "Competent" range. All seven candidates (100%) achieved an average supervisor rating equal to or above the target score of 2.0. One candidate did receive below standard rating in two of eleven domains and his progress in these areas will be monitored during the internship year. The cohort average rating within each of the 11 NASP domains met the criterion socre of 2.0. (See Table 1A). Internship Field Supervisor Ratings: There were 11 candidates who participated in the Internship experience during the 2012-13 academic year. Field supervisors rated our candidates, as a whole, very well and solidly within the "Competent" range. All 11 candidates (100%) achieved an average supervisor rating equal to or above the target score of 2.0. The cohort average rating with the 11 NASP Domains met the criterion score of 2.0. (See Table 1B). Faculty Rating Form: Eleven candidates completed their internship portfolios this academic year. Each of four portfolio cases submitted were rated by two faculty members to obtain an average Faculty Rating Form (FRF) rating and an average Overall Case Rating. For the Academic Intervention case, all eight candidates (100%) achieved the criterion of 85% or higher on the average FRF rating and an Overall rating of "3" or higher for the case. For the Assessment case, all eight students (100%) achieved the criterion of 85% or higher on the average FRF rating and an Overall rating of "3" or higher for the case. For the Behavioral Consultation case, all eight candidates (100%) achieved the criterion of 85% or higher on the average FRF rating and an Overall rating of "3" or higher for the case. Finally, for the Counseling case, all eight candidates (100%) achieved the criterion of 85% or higher on the average FRF rating and an Overall rating of "3" or higher for the case. (See Table 2A). PIR Portfolio Reviews: Each Portfolio case completed was evaluated by two faculty raters using the Procedural Integrity Rubric (PIR) in order to obtain an average PIR score. Additionally, candidates were expected to obtain no ratings of "0" on each of the PIR documents. For the Academic Intervention case, all eight candidates (100%) achieved an average PIR score at or above the "cut" score of 24, with no candidates receiving a score of "0" on these case ratings. For the Assessment case, all eight candidates (100%) achieved an average PIR score at or above the "cut" score of 39. With the Behavioral Consultation case, all eight candidates (100%) achieved an average PIR score at or above the "cut" score of 21 with no candidates receiving a score of "0" on these case ratings. Finally, for the Counseling case, all eight candidates (100%) achieved an average PIR score at or above the "cut" score of 21 with no candidates receiving a score of "0" on these case ratings. (See Table 2B). Positive Impact Data for Quantitative Intervention Cases: Candidates' impact on student learning during the Internship experience is evaluated quantitatively through intervention cases submitted as part ot the Portofolio assessment. Three cases, Academic Interevention, Behaivoral Consultation, and Counseling) involve intervention with students and include progress monitoring data. A candidate's positive impact on student functions is evaluated by calculating either an effect size or percentage of nonoverlapping data points. All eight internship candidates (100%) achieved at least a moderate impact on student learning for all three cases submitted. This met and exceeded the expectation of a moderate impact for two of the three cases submitted. (See Table 3AB). |
Actions for Objective:
| Action |
|
|||
|
We are in the process of revising the supervisor rating forms. The revision is two-fold: 1 we are expanding the likert scale from 3 point to 5 point to allow the supervisors more flexibility in rating students; 2. we are now implementing new standards mandated by NASP. These will change supervisor rating and faculty ratings during the next academic year. Hopefully, these changes will allow us to compare supervisor and faculty rating with more specificity. We also are having a supervisor workshop for all practicum and internship supervisors prior to the beginning of the upcoming year. Issues will be put forth, discussed, and any discrepancies in expectations will be addressed. |
| Previous Cycle's "Plan for Continuous Improvement" |
|
We are so pleased with our students' competency achievement. Still there are areas to improve. The domain of research-based academic interventions was a weaker area on the national certification exam. We need to evaluate our curriculum to determine where we might include more emphasis in this area. In the skill applications, we need to address the discrepancy between on-site supervisors who view home/school community collaboration as a weakness and the program faculty who view weaknesses as apparent in information technology, and school and systems organization, policy development, and climate. Perhaps we can explore community engagement activities through the schools to provide instruction and experience in this area. The coursework regarding school and systems organization, policy development, and climate is taught in an interdisciplinary fashion through the Department of Educational Leadership and Counseling. We will need to consult with the faculty teaching this particular area to share our concerns. In the skill application area, our weakest area is in the behavioral consultation case. The data indicate fairly consistently that this is a weaker area although data is within the average to above average range. To strengthen this area, we need to closely examine our expectations and students' understanding of those expectations to clarify any confusion.
|
| Update on Previous Cycle's "Plan for Continuous Improvement" |
|
We are somewhat pleased with the results of the PRAXIS examination. Eleven students took the exam and all performed at an acceptable level. As can be noted, we did have concerns about the "Research-Based Academic Practices" category during the last cycle. The students appear to have responded to our handling of this concern: ten of the 11 students scored within the average performance range on this category; 9 of the 11 fell within the average performance range on "Research-Based Behavioral and Mental Health Practices" category. In all cases in which the student did not perform within the average ranges, they were only one point below what would be acceptable. With these results, we will continue to consult with faculty teaching courses germane to these areas and continue to stress the importance of these areas to the students. |
| Plan for Continuous Improvement |
|
We are happy with the results of this year's students. During the coming academic year, we will increase our communication with supervisors to ensure that effective instruction/supervision is happening both on campus by faculty members and in the field by field supervisors. The plan at this moment is to host a workshop for all field supervisors in August 2013 so that issues, expectations, etc. can be brought to light and discussed to mutually-acceptabe decisions. In addition to that, the coordinator for the School Psychology Program, along with the other core faculty members, will meet with incoming students to ensure that they understand what is expected of them and what is deemed important for their success. We also are changing our rubrics in our evaluation of the students, sites, and the program to reflect the new rubrics currently being put forth by NASP, The National Association of School Psychologists.
|