SACSCOC Fifth Year Interim Report
Spacer
Assessment : 2013 - 2014 : Educational Programs :
Special Education MA/MED (Low Incidence Disabilities And Autism)
Special Education MA/MED (Low Incidence Disabilities And Autism)
2 Goals 4 Objectives 4 Indicators 6 Criteria 6 Findings 4 Actions
| GOAL: Mastery Of Behavior Analysis Certification Board Task List Knowledge And Skills |
| Objective |
|
||||
|
Candidates will demonstrate mastery of knowledge and skills on comprehensive examinations, referenced to the standards set forth by the Behavior Analysis Certification Board. |
| Indicator |
|
|||
|
Faculty-developed rubric scored by two independent faculty members following standards of the Behavior Analysis Certification Board Task List. |
| Criterion |
|
|||
|
90% of candidates will score 80% or better on the rubric (see attached). Particular areas of emphasis include data analysis, experimental design, behavioral support plan development, and application of ethical principles. Because 70% of candidates scored above 80% on their initial attempt in 2012-2013, we will maintain the criterion of 90% of candidates. The domain identified to improve is the area of research design. We have set in motion some interventions for this area. Curriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. In addition, we will be transitioning to written comprehensive exams in order to more fully capture the extent of a candidate's knowledge, and not their nerves at having to perform in person. Thus, this year, we will give candidates a choice of oral or written examinations as we make the transition. Therefore, our scores may be higher than usual for this reason in the following year. |
| Finding |
|
|||
|
Fifty percent of the students scored 80% (40 out of 50) or better upon the initial testing. However, because two of the students taking the exam orally were within 10% of the passing criteria (5 points), the faculty brought the students back into the room and probed for additional information on questions upon which they had performed poorly. Therefore, with additional prompting, those students achieved a passing grade (see the attached score sheet for initial and final scores). Four students completed the exam in a written format. Of those, one passed, one was within the range to qualify for additional questions, and two failed. The one student who earned within a 35 initially passed when given two more questions orally. The two students who failed initially passed when presented with the test orally. Thus, we will be utilizing only oral exams in the future. The average of all initial scores was 38.09 and the average of all final scores was 42.25 out of 50. Interrater reliability was very high across all students with an overall average of 89.75 and a range of 66.67-100 on individual questions. |
Actions for Objective:
| Action |
|
|||
|
Cirriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. Because only 50% of students met the goal on the first testing, this criterion will be maintained. |
| GOAL: Mastery Of Behavior Analysis Certification Board Task List Knowledge And Skills |
| Objective |
|
||||
| Candidates will demonstrate mastery of knowledge and skills on the Formative Evaluation of Behavior Analytic Terms (FEBAT), referenced to the standards set forth by the Behavior Analysis Certification Board. |
| Indicator |
|
|||
|
The FEBAT is a formative evaluation that assesses knowledge on 25 standard behavior analytic terms at six points in the candidate's academic career (see attached- beginning of program, end of first semester, end of second semester, end of summer term, end of first semester of second year, and end of program). Candidates receive course points for completion of the FEBAT, regardless of accuracy on the individual items. Therefore, the only environmental variables that evoke and maintain correct responding are instructional control in the classroom and a history of reinforcement for performing accurately on similar tasks. In addition, the students are urged to do well to provide an accurate assessment of their knowledge with respect to the material as they advance throughout the program. The FEBAT is scored on a faculty-created rubric with a 5-point scale. Scores from 0-4 are administered based on the quality of the definition provided for each term; 4-point answers are those most similar to the definitions provided by Cooper, Heron, and Heward (2007), a textbook based on the Behavior Analysis Certification Board task list. At least 25% of FEBAT administrations are scored by two Board Certified Behavior Analysts to ensure interrater reliability. |
| Criterion |
|
|||
|
80% of candidates will score 80% or better on the FEBAT rubric (see attached) at the final administration. Because this is a new objective for the 2013-2014 school year, we are setting a moderate initial criterion. Curriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. Therefore, achievement of the criterion in the 2013-2014 school year seems plausible. |
| Finding |
|
|||
| No student exceeded 80% on the final administration of the FEBAT. The average score was 50.55 with a range of 35 to 65.5. Due to failure to meet this goal, this criterion will be maintained. See attached for individual scores. |
| Criterion |
|
|||
| Because the FEBAT is administered six times (five times for the 2013-2014 cohort, see attached list) during a candidate's graduate career, it serves as a method of formative evaluation. Therefore, students will improve their performance from the initial administration of the FEBAT to the final administration by 20%. |
| Finding |
|
|||
| Eight out of nine students improved their scores on the FEBAT by at least 20% from the initial administration (average 31.2%). However, given the overall low scores, we will maintain this criterion next year. |
Actions for Objective:
| Action |
|
|||
|
Cirriculum changes are being made to focus more heavily on the task list created by the Behavior Analysis Certification Board. Furthermore, all classes in the future will be taught by Board Certified Behavior Analysts or individuals with a strong background in behavior analysis only. Additionally, review cards featuring terms similar to those in the FEBAT have been incorporated into classes. |
| GOAL: Mastery Of Behavior Analysis Certification Board Task List Knowledge And Skills |
| Objective |
|
||||
|
Candidates will pass the Behavior Analysis Certification Exam at percentages comparable to the national average. |
| Indicator |
|
|||
|
The Special Education MA/MED (Low Incidence Disabilities And Autism) graduate program prepares students to take the Behavior Analysis Certification Exam, the final step in becoming a Board Certified Behavior Analyst. Therefore, the true test of the program's effectiveness is the percentage of students passing the exam every year. The Behavior Analyst Certification Board reports the percentage of students passing the exam and the national average. In 2012, a total of 9 first-time test takers completed the BCBA; 3 passed the examination, resulting in a pass-rate of 33%. The reported national data for 2012 showed a passing rate of 58%; thus, our passing rate was significantly below national average. |
| Criterion |
|
|||
| As a measure of continuous improvement, the criterion for this objective is that the passing rate for each year exceed the passing rate for the previous year as it approaches the national standard. Annual data (starting in 2012) will be graphed and progress will be tracked as the program continues to expand and develop. Therefore, the passing rate for 2013 must exceed 33%. Note that data are available each academic year for the candidates who graduated the previous year. |
| Finding |
|
|||
| The Behavior Analyst Certification Board reports the percentage of students passing the exam and the national average. In 2013, a total of 11 first-time test takers completed the BCBA; 4 passed the examination, resulting in a pass-rate of 36%. The reported national data for 2012 showed a passing rate of 58%; thus, our passing rate was significantly below national average. |
Actions for Objective:
| Action |
|
|||
| As a measure of continuous improvement, the criterion for this objective is that the passing rate for each year exceed the passing rate for the previous year as it approaches the national standard. Annual data (starting in 2012) will be graphed and progress will be tracked as the program continues to expand and develop. Therefore, the passing rate for 2014 must exceed 36%. Note that data are available each academic year for the candidates who graduated the previous year. |
| GOAL: Production Of Scholarly Research Project |
| Objective |
|
||||
|
Candidates will develop a high-quality research proposal in SPED 6314. Candidates will demonstrate mastery of this skill by the end of 6317. |
| Indicator |
|
|||
| The research proposal will be scored by instructors of SPED 6314. |
| Criterion |
|
|||
|
To evaluate the research proposal in SPED 6314, 80% of the candidates will score 80% or better on the attached rubric. Please note that the rubric has changed from last year to more closely reflect the sections, subsections, and writing style of research manuscripts in the field of behavior analysis. Because only 60% of students achieved the 80% criterion last year, it will be maintained this year. Last year, the area identified to improve was written description of procedure on the research proposal. This year, we will maintain that goal in addition to targeting written description of results. |
| Finding |
|
|||
|
Candidates submit a research proposal at the end of the semester. The proposals are scored with the attached rubric that looks at competency domains such as "Introduction", "Method," and "Description of Procedure." Students' performance (n = 10) on the 160 point rubric averaged 82.9%. Range of performance was 61.6-95.7%. The lowest domain of candidate performance was in the area of results. Candidates performed well on providing background information on subject selection, identification of target skill, etc. Findings from this year show a 5.2% average improvement from last year. |
| Criterion |
|
|||
|
We have added a second criterion in 2013-2014 to evaluate continuing improvement on this goal in future years. Candidates complete final drafts of their research projects in both SPED 6314 (Fall) and SPED 6317 (Spring). These are evaluated using the same rubric, the updated rubric used to also assess the first criterion (see above). Therefore, we would like to see a 10% increase in scores on 80% of candidates' final project in SPED 6317, relative to the score they received on the final project they submitted in SPED 6314. The range for improvement will vary based on the candidates' initial performance, but a 10% increase for 80% of candidates is feasible, given the low percentage of candidates that score above 90% on the project in SPED 6314. To achieve this goal, we intend to have the candidates submit multiple drafts throughout the semester to allow us to shape their written behavior. |
| Finding |
|
|||
| To determine improvement in scientific writing ability, scores were compared on the student's research projects from Fall to Spring semesters. All students improved across semesters; only six out of eleven improved more than 10% relative to their Fall performance. The average scores out of 160 points were 130.1 (81.3%) and 151.7 points (94.8%) in Fall and Spring, respectively. Furthermore, the average improvement from Fall to Spring was 18.3% when calulated as percentage of Fall scores and 13.5% when calculated as percentage of the total 160 points. All students made noticeable improvement in their writing across semesters, even if they did not meet the criterion. |
Actions for Objective:
| Action |
|
|||
|
Students will begin working on their research proposals much earlier in their academic careers (Spring semester of their first year). Students will be introduced to a wider variety of research throughout their classes. Students will receive increased emphasis on written description of procedures - clarity and conciseness - in the early portion of the course. Students will be afforded opportunities to submit drafts of the proposal prior to the final submission at the end of the course. The accuracy criterion of 80% of candidates was not obtained - two candidates fell short in their initial drafts in SPED 6314. Thus, 80% will remain the criterion level of accuracy for the research proposal. |
| Previous Cycle's "Plan for Continuous Improvement" |
|
This year, we have already made many changes and plan to continue to do so. We began getting the 2013-2014 students focused on their research projects in Spring of 2013. We have had the students read more research than previous years. We are working to change the course sequence towards both of the aforementioned goals as well. We are having the students conduct small research projects in classes early on to prepare them for their larger research project. We have changed the way that practicum is conducted so that they are receiving maximum experience in the field, if possible. We have changed the curriculum in several classes to focus more strongly on the Behavior Analysis Certification Board Task List in order to prepare them for both comprehensive exams and the BCBA exam. Overall, we are making any change necessary to improve the intensity of the cirricular and research focus for the student.
|
| Update on Previous Cycle's "Plan for Continuous Improvement" |
|
Any failures on the plan for continuous improvement in the last year were due to understaffing. The program accepted too many students into the first year cohort and did not get the new faculty member that was expected. Furthermore, the coordinator of the program was very ill in the last year. We will continue to implement the plan in the coming year with two healthy staff members and a reasonable number of students.
|
| Plan for Continuous Improvement |
|
This year, we have already made many changes and plan to continue to do so. We began getting the 2014-2015 students focused on their research projects in Spring of 2013. We have had the students read more research than previous years. We are working to change the course sequence towards both of the aforementioned goals as well. We are having the students conduct small research projects in classes early on to prepare them for their larger research project. We have changed the way that practicum is conducted so that they are receiving maximum experience in the field, if possible. We have changed the curriculum in several classes to focus more strongly on the Behavior Analysis Certification Board Task List in order to prepare them for both comprehensive exams and the BCBA exam. Overall, we are making any change necessary to improve the intensity of the cirricular and research focus for the student.
|