SACSCOC Fifth Year Interim Report
Spacer
Assessment : 2013 - 2014 : Educational Programs :
Psychology MA (School Psychology)
Psychology MA (School Psychology)
2 Goals 2 Objectives 2 Indicators 2 Criteria 2 Findings 2 Actions
| GOAL: Foundational Competence In School Psychology |
| Objective |
|
||||
| Students demonstrate competency in the scientific, methodological and theoretical foundations of professional school psychology. |
| Indicator |
|
|||
|
The PRAXIS II School Psychology Exam is a nationally administered examination used to determine an individual’s qualification for licensure to practice within the field. Candidate competency is evaluated with respect to the following test subcategories: 1. Data Based Decision Making (35%), 2. Research-based Academic Practices (12%), 3. Research-based Behavioral and Mental Health Practices (16%), 4. Consultation and Collaboration (12%), 5. Applied Psychological Foundations (13%), and 6. Ethical, Legal, and Professional Foundations (12%). |
| Criterion |
|
|||
|
A minimum score of 165 is required to obtain the credential of Nationally Certified School Psychologist (NCSP), and thus a score of 165 or better has been established by the SSP Program as the criterion for this objective. In addition, candidates are expected to perform at or above the average range provided by the test developers for each of the six subcategories. |
| Finding |
|
|||
|
Seven SSP students took the PRAXIS II exam during the past academic year. Total scores ranged from 171 to 185, with an average score of 176. All students had scores directly reported to our Program, which enables an analysis of subcategory performance. All seven students (100%) scored at or above the average performance range for ALL areas assessed, including Data-Based Decision Making, Research-Based Academic Practices, Research-Based Behavioral and Mental Health Practices, Consultation and Collaboration, Applied Psychological Foundations, and Ethical, Legal, and Professional Foundations. |
Actions for Objective:
| Action |
|
|||
|
All members of the cohort scored at or above the acceptable range on the PRAXIS II exam and on each of the subcategories within the exam. We will continue instructing the next cohort of students in the methods that resulted in their success and continually check on how they are progressing toward that goal. In addition, we have opened a search for a new faculty member in the School Psychology Program and will stress during the interviews the importance of maintaining our current level of success. |
| GOAL: Skill Application |
| Objective |
|
||||
|
Candidates in the school psychology program demonstrate knowledge and improving skill application commensurate with their level of training. Specifically, candidates in their final practicum placement and on internship, both held within the public school setting, will demonstrate appropriate application of professional school psychology skills in the areas of assessment, behavioral consultation, academic intervention and counseling. |
| Indicator |
|
|||
|
Ratings Forms (1) Satisfactory ratings from Field Supervisors 1(A) Ratings for Practicum II candidates (Year 2 of 3) 1(B) Ratings for candidates on Internship (Year 3 of 3) On-site, or field, supervisors are asked to evaluate each candidate’s performance in order to gauge their professional performance according to the NASP Standards for the Domains of Competence. These Standards include: II) Data-Based Decision-Making and Accountability, III) Consultation and Collaboration, IV) Direct and Indirect Services at the Student Level {includes 4.1: Interventions and Instructional Support to Develop Academic Skills and 4.2: Interventions and Mental Health Services to Develop Social and Life Skills}, V) Direct and Indirect Services at the Systems Level {includes 5.1: School-Wide Practices to Promote Learning and 5.2: Preventive and Responsive Services}, VI) Direct and Indirect Services to support Family-School Collaboration, VII) Foundations of School Psychologists’ Service Delivery: Diversity, and VIII) Foundations of School Psychologists’ Service Delivery: Research, Program Evaluation, Legal, Ethical and Professional Practice {includes 8.1: Research and Program Evaluation and 8.2: Legal, Ethical, and Professional Practice} (2) Satisfactory ratings from Program Faculty 2(A) Faculty Rating Forms (FRF) for both of two Portfolio case submitted 2(B) Procedural Integrity Rubrics (PIR) for both of two Portfolio cases submitted Candidates completing the Internship Portfolio assessment will obtain satisfactory ratings from the Program Faculty on each of two cases submitted. These cases include the candidate’s choice among: a Behavioral Consultation and Intervention case, an Academic Assessment and Intervention case, and a Counseling case. Two faculty members will evaluate each case, and the average of these two ratings on both the FRF and the PIR will be reported. Positive Impact Data (3) Quantitative data gathered as part of the case intervention 3(A) Effect Size 3(B) Percent of Non-Overlapping Data Points (PND) Candidates completing the Internship Portfolio assessment will submit quantitative data gathered as part of the case intervention monitoring for three of four cases submitted. These cases include: 1) the Behavioral Consultation case, 2) the Academic Intervention Case, and 3) the Counseling case. Effect size, percent of non-overlapping data points (PND), or other means of quantitatively evaluating the candidates positive impact on the student(s) will be calculated. |
| Criterion |
|
|||
|
Skill Application 1A: Candidates are rated by field supervisors according to a five-point scale including the following competency rating categories: Major Area of Concern (Additional, Intensive Supervision Required) {1}, Improvement Needed (Additional Supervision Required) {2}, Meets Expectations for Training Level (Supervision Needed) {3}, Exceeds Expectations for Training Level (Supervision Needed) {4}, Professionally Competent (No Supervision Needed) {5}. Because candidates in their final practicum will be under supervision for two more years, they are expectedto maintain an overall average rating of “3.0” for all of the NASP Domains evaluated. 1B: Candidates are rated by field supervisors according to a five-point scale including the following competency rating categories: Major Area of Concern (Additional, Intensive Supervision Required) {1}, Improvement Needed (Additional Supervision Required) {2}, Meets Expectations for Training Level (Supervision Needed) {3}, Exceeds Expectations for Training Level (Supervision Needed) {4}, Professionally Competent (No Supervision Needed) {5}. Because candidates completing their internship year will continue to be under supervision for one more year, they are expected to maintain an overall average rating of “3.0” for all of the NASP Domains evaluated. 2A: Candidates completing their internship experience are required to submit two distinct Portfolio cases. Each case will be reviewed by two faculty members and assigned ratings on the Faculty Rating Form (FRF).These ratings will then be averaged across the two faculty raters. The FRF addresses all domains of practice related to the type of case being reviewed. Each item on the FRF includes the following competency rating categories: Pass (score 1), No Pass (score 0), Not Included (score 0), and Not Applicable (removed from the scoring calculation). Candidates are expected to achieve a minimum domain competency average of 85%. In addition, candidates are given a single faculty rating for the overall case completion. This rating ranges from 1 (Very Poor) to 5 (Very Good). Candidates are expected to achieve a minimum average overall rating of 3 across the two faculty raters, which is equivalent to “average” work completed in the field. 2B: Internship portfolio case submissions are also scored by faculty using a Procedural Integrity Rubric, or PIR. Each case PIR includes critical procedures that must be performed as part of completing the case in order for the intern to be judged as following best practices within the field. Each item on the PIR can be scored as follows: 0 = Incomplete, 1 = Needs Improvement (task is completed, with some concerns), 2 = Completed Satisfactorily (Competency Met), and 3 = Exemplary Performance (task is completed at a level above expectations. Each PIR for the four cases submitted has an established cut score equivalent to achievement of at least 85%. Additionally, candidates are expected to obtain no ratings of “0” on any PIR. 3A: Based on the quantitative data included as part of the Behavioral Consultation and Intervention, Counseling, and/or Academic Assessment and Intervention Portfolio case submissions, the candidate’s impact on student behavior and/or learning can be calculated in a variety of ways. Effect size allows for the comparison of the standard mean difference in student performance during baseline and treatment phases of intervention. An effect size of .8 is considered to be of moderate impact. Candidates are expected to demonstrate moderate impact through either effect size or PND calculation for both of the cases submitted. 3B: Based on the quantitative data included as part of the Portfolio case submissions, the candidate’s impact on student behavior and/or learning can be calculated in a variety of ways. Percent of Non-overlapping Data points, or PND, provides a comparison of the percentage of data points during the treatment phase that do not overlap with the most extreme baseline phase point. A PND calculation of 60% is considered to be of moderate impact. Candidates are expected to demonstrate moderate impact through either effect size or PND calculation for both of the cases submitted. |
| Finding |
|
|||
|
Skill Application 1A: Table 1A: Practicum II Field Supervisor Ratings There were ten candidates who participated in the final Practicum experience during the Spring 2014 semester. Field supervisors rated our candidates, as a whole, very well and solidly within the “Competent” range. Eight of the ten candidates (80%) achieved an average supervisor rating equal to or above the target score of 3.0. One candidate’s rating document was copied onto PDF missing the even numbered pages, so this candidate’s scores will be added once an accurate copy is received. The other candidate did receive below standard ratings in four of ten Standard areas, with two additional areas rated wholly as “Not Observed.” Her progress in these areas will be monitored closely during the internship year. The cohort average rating within each of the ten Standard areas measured exceeded the criterion score of 3.0. 1B: Table 1B: Internship Field Supervisor Ratings There were seven candidates who participated in the Internship experience during the 2013-2014 academic year. Field supervisors rated our candidates, as a whole, very well and solidly within the “Competent” range. The electronically submitted rating for one candidate could not be read for data entry purposes, and the Program Director is in the process of accessing the hard copy record. The remaining six candidates (100%) achieved an average supervisor rating equal to or above the target score of 3.0. The cohort average rating within each of the ten Standard areasmeasured exceeded the criterion score of 3.0. 2A: Data Tables for FRF Portfolio Reviews Eleven candidates completed their Internship Portfolios this academic year. Two Portfolio cases submitted were rated by two faculty members to obtain an average Faculty Rating Form (FRF) rating and an average overall case rating. For the Academic Intervention case, all ten candidates (100%) graduating in May achieved the criterion of 85% orhigher on the average FRF rating and an overall rating of ‘3’ or higher forthe case. For the Behavioral Consultation case, all eight of ten candidatescurrently rated (100%) achieved the criterion of 85% or higher on theaverage FRF rating and an overall rating of ‘3’ or higher for the case. Thefaculty continue to complete these final portfolio reviews, with onecandidate remaining to be evaluated prior to graduation in August. 2B: Data Tables for PIR Portfolio Reviews Each of four Portfolio cases completed was evaluated by two faculty raters using the Procedural Integrity Rubric (PIR) in order to obtain an average PIR score. Additionally, candidates were expected to obtain no ratings of ‘0’ on each of the PIR documents. For the Academic Intervention case, all ten candidates (100%) achieved an average PIR score at or above the cutscore of 24, with nine of the ten candidates (90%) receiving no scores of ‘0’ on these case ratings. For the Assessment case, all ten candidates (100%) achieved an average PIR score at or above the cut score of 39,with no candidates (100%) receiving any scores of ‘0.’ For the Behavioral Consultation case, all eight candidates reviewed (100%) achieved an average PIR score at or above the cut score of 21, with no candidates (100%) receiving a score of ‘0’ on these case ratings. Finally, for the Counseling case, all eight candidates reviewed (100%) achieved an average PIR score at or above the cut score of 21, with no candidates (100%) receiving a score of ‘0’ on these case ratings. 3A-B: Positive Impact Data for Quantitative Intervention Cases Candidates’ impact on student learning during the Internship experience is evaluated quantitatively through intervention cases submitted as part of the Portfolio assessment. Two cases (i.e., Academic Intervention and Behavioral Consultation) involve intervention with students and include progress monitoring data. A candidate’s positive impact on student functioning is evaluated by calculating either an effect size or percentage of nonoverlapping data points. All ten internship candidates (100%) achieved at least a moderate impact (see definition above) on student learning for one case submitted. All candidates either met or exceeded the expectation of a moderate impact for one of the two cases submitted. |
Actions for Objective:
| Action |
|
|||
| We did not quite achieve the 100% success record on Skill Application for which we had hoped. The program faculty and the chair will meet this coming fall with the external supervisors to welcome them again to the program and to thank them for their past inputs and their past help. During that meeting, we will again stress the needs of the program and the specific needs of the students so that we may get closer to our goal of 100%. We also will more closely monitor the progress of each student whille they are at their external sites. |
| Previous Cycle's "Plan for Continuous Improvement" |
|
We are happy with the results of this year's students. During the coming academic year, we will increase our communication with supervisors to ensure that effective instruction/supervision is happening both on campus by faculty members and in the field by field supervisors. The plan at this moment is to host a workshop for all field supervisors in August 2013 so that issues, expectations, etc. can be brought to light and discussed to mutually-acceptabe decisions. In addition to that, the coordinator for the School Psychology Program, along with the other core faculty members, will meet with incoming students to ensure that they understand what is expected of them and what is deemed important for their success. We also are changing our rubrics in our evaluation of the students, sites, and the program to reflect the new rubrics currently being put forth by NASP, The National Association of School Psychologists.
|
| Update on Previous Cycle's "Plan for Continuous Improvement" |
|
The core faculty of the SSP program and the Chair of the Department met with the external supervisors and went through our expectations and issues that they may have experienced with our students. The meeting was a day-long conference and appeared to have had a positive impact with nearly all of our students meeting criterion. Thus, our "Previous Plan for Continuous Improvement" appears to have had positive results. |
| Plan for Continuous Improvement |
|
The SSP Program works to continuously improve the services and training provided to students in a number of ways. Listed below are the major agenda items for continuous improvement of the Program over the next 12-month period:
1. Program faculty have been working to revise the Program Assessment System following NASP’s adoption of new Standards for Training and Practice. Building upon the work completed last year, this year the faculty will review and update the Faculty Review Form utilized during Internship Portfolio review for behavioral intervention, academic assessment and intervention and counseling cases; 2. Also related to the Program Assessment System, the faculty will be working this year on the development of a Practice Log document in Excel that will allow students to record their hours of practice in the field during various courses throughout their program of study. This log document will be used by the student during all three years they are enrolled in the Program, and will provide cumulative details of hours spent in a variety of professional roles and experiences; 3. The Program and the Department are dedicating both time and resources to expand recruitment efforts. It is hoped that the expanded quality applicant pool will increase the number of qualified admissions to the Program; 4. The Program continues to work to strengthen relationships with local school districts. It is the licensed professionals in the field who provide extremely valuable practical experiences and supervision to our students. Without these individuals, our Program’s quality would decline substantially. A professional development workshop is being planned for this coming January. It is hoped that an event of this type can become an annual occurrence; 5. The Program would like to develop as detailed a database of graduates as possible, dating back to the year of first NASP Program approval. Having contact information of our graduate would support our ongoing contact with them in the form of a semi-annual Program newsletter. This is something we would like to put into place in collaboration with our student organization; 6. The Program would like to develop both an Exit Survey for graduating students as well as a Survey of Program Graduates that would be given randomly to graduates and their employers. These tools would be used to gauge both the impact of the Program based on our graduates as well as trends in the field. |