SACSCOC Fifth Year Interim Report
Spacer
Assessment : 2012 - 2013 : Educational Programs :
Computing Science BS
Computing Science BS
2 Goals 2 Objectives 2 Indicators 2 Criteria 2 Findings 2 Actions
| GOAL: Specialized Competencies |
| Objective |
|
||||
| Students will develop and demonstrate skills and knowledge in their concentration areas. The department offers three concentration areas: Computer Science, Information Systems, and Information Assurance. |
| Indicator |
|
|||
|
During department meetings faculty with expertise in the three concentration areas will discuss students’ performances in their concentration areas. We expect that faculty in each concentration area will evaluate student performance across the 18 hours course sequence for each concentration. |
| Criterion |
|
|||
|
In last year's review the faculty assessed more than 70% of the students were performing at or above expected levels. As a result the faculty set a higher expectation . We expect that the faculty will deem at least 72% or more of students’ performances as acceptable. |
| Finding |
|
|||
|
For the formal assessment and evaluation of the performance of CS students, CS faculty developed tools and rubrics that can address the new ABET-CAC (Computing Accreditation Commission) criteria during the last academic year (2011-2012). Four Program Education Objectives (PEOs)(E01-E04) and eleven Student Outcomes (SOs) ((a)-(k)) were established and then the first self-study was conducted with the two capstone courses, COSC4319 (Software Engineering) and COSC4349 (Professionalism and Ethics) during the Spring 2012 semester. After checking the first assessment on PEOs and SOs, the CS Undergraduate Curriculum Committee (UCC) suggested to obtain more data before drawing conclusions on the first result and/or any changes to the assessment instruments. Accordingly, the adopted assessment instruments have been applied for both the Fall 2012 and the Spring 2013; the assessment results have been discussed; and a number of suggestions/revisions have been made for continuous improvement of the program. The UCC found that the primary assessment rubric for some SOs (except (e), (g), and (h)) was indirect and qualitative, while that for the remaining SOs (i.e., (e), (g) and (h)) was direct and quantitative. To address this, the UCC adopted an additional instrument, TASO (Test for Assessing Student Outcomes), for quantitative measurement of all the eleven SOs. The UCC discussed that PEOs were assessed with the mapping table from SOs to PEOs and thus the performance of PEOs was directly measured in terms of students’ response to SOs. PEOs should be periodically reviewed and revised by the program’s all constituencies to be consistent with their needs, not only with one component; thus the UCC planned to include reviews of both students and Industry Advisory Board. The ABET assessment team, which visited to the CS department in October 2012 for 2012-2013 Accreditation Cycle, pinpointed that Information System concentration received the minimal coverage of Computer Organization and Architecture, and might not receive adequate exposure to a variety of systems through required courses in the degree. |
Actions for Objective:
| Action |
|
|||
|
In order to provide accurate assessment and evaluation for students’ performance, two more assessment results (under the instrument adopted Spring 2012) were collected and compared during the Fall 2012 and the Spring 2013. All the faculty members collaborated to perform the assessment, discussed the collected assessment results, and involved in improving the CS curriculum/program accordingly. In particular, an additional instrument, TASO (Test for Assessing Student Outcomes), was implemented in Spring 2013 and the assessment result will be utilized for the program improvement in the next academic year (i.e., 2013-2014). To address the finding by the ABET assessment team, COSC2329 was modified to include additional architecture and digital logic modules as a temporary solution; thus, the UCC will keep working on developing appropriate solution(s) on this issue, possibly rearranging course contents or/and revising curriculum. The process of periodic review and revision of PEOs and SOs has been initiated with the rubrics and instruments through students’ presentation (at COSC4319), students’ response (at COSC4349), and TASO. |
| GOAL: Ethical Principles, Technical Skills, And Management Skills (core) |
| Objective |
|
||||
| Students will develop and demonstrate knowledge of ethical principles, technical skills, and management skills relevant to the field of computer science. |
| Indicator |
|
|||
| All undergraduate CS students must complete COSC 4319 (Software Engineering) and COSC4349 (Professionalism and Ethics) prior to graduation. In addition, COSC4319, as a capstone course, covers the eight Student Outcomes (except the three outcomes (e), (g) and (h)) out of the eleven Student Outcomes and COSC4349 addresses the remaining three outcomes. Therefore, both the courses are selected for assessment. Additionally, TASO (Test for Assessing Student Outcomes) is implemented to quantitatively measure the eleven SOs. It is required for senior students to take it in senior-level Software Engineering course, prior to graduation regardless of their program concentration. |
| Criterion |
|
|||
|
To perform a quantitative assessment, the following rubrics for the two courses are developed: (1) rubric for Project, (2) rubric for presentation evaluation, (3) rubric for group member evaluation, and (4) rubric for ethics topics. For COSC4319, faculty members observe students’ project presentation and directly evaluated students’ performance based upon the rubrics (1)-(3). One the other hand, for COSC4349, students’ performance is indirectly evaluated through the exam questions that address rubric (4). Each category is rated with the following scale values: (1) 1.0 (below expectations or unacceptable), (2) 2.0 (evolving or developing), (3) proficient (or competent), and (4) outstanding (or exemplary). Student’s performance on Student Outcomes is directly evaluated with each specific rubric, while the performance on objectives is indirectly measured by mapping between Student Outcomes and Program Objectives. Questions on TASO are based on topics from a number of required core courses (including COSC 1436, 1437, 2329, 3318, 3319, 4318, 4319, and 4349). Currently, it contains 26 multiple choice questions and 1 short answer question. Each multiple choice question has five choices, including the last choice of “I don’t know”. The percentage of students who select the correct answer to each given question will be computed and evaluated. |
| Finding |
|
|||
|
The assessment result of Student Outcomes (SOs) ranged (out of the maximum score of 4) as follows: 2.29 (std of 1.11) to 3.15 (std of 0.82), 2.38 (std of 0.9351) to 3.18 (std of 0.85), and 2.18 (std of 1.08) to 3.19 (std of 0.57) for Spring 2012, Fall 2012, and Spring 2013, respectively. To be more specific, the overall performance for Fall 2012 was consistent with that for Spring 2013; however three SOs ((a)-(c)) were relatively lower than the other SOs, which means the students for Spring 2012 and Fall 2012 did not aware well of design and analysis issues. The overall performance for Spring 2013 improved, however the two SOs ((e) and (g)) dropped from the previous two assessment periods. The Program Education Objectives (PEOs) ranges (out of the maximum score of 4) as follows: 2.38 (std of 1.05) to 2.98 (std of 0.90), 2.50 (std of 0.93) to 2.80 (std of 0.91), and 2.82 (std of 0.81) to 3.17 (std of 0.84) for Spring 2012, Fall 2012, and Spring 2013, respectively. The overall performance on PEOs over the three assessment periods is consistent. In particular, students for Spring 2013 performed best with average score > 3 (out of 4). The evaluation of SOs ((e), (g), and (h)) led to the observation that students almost universally resorted to utilitarianism as an ethical perspective when asked to apply a system of ethics to a given scenario. Therefore, it is more desirable to revise COSC4349 to ensure that students to apply other more effective ethical systems. |
Actions for Objective:
| Action |
|
|||
|
To improve confidence and significance of analysis results, number of students, different weights for each outcome, and other factors should be considered. For example, if a class has more than 20 (or 25) students, a random sample of 20 (or 25) students could be selected for assessment. Since COSC4319 and COSC4349 do not have the same number of students, the difference in students needs to be weighted properly. COSC4319 is assessed directly by CS faculty members during students project presentation based upon the prepared assessment rubrics, while COSC4349 is evaluated indirectly (but objectively) through test/exam questions. Therefore, to have the consistent assessment, the objective assessment with TASO was developed and was employed from Spring 2013. |
| Previous Cycle's "Plan for Continuous Improvement" |
|
The first assessment scores on both Program Outcomes and Education Objectives for Spring 2012 is not enough to draw conclusions and/or suggest specific changes to the current curriculum. Therefore, the CS department will obtain more data during Fall 2012 and Spring 2013. Then, based on the analysis result collected at least over two or three consecutive semesters, the current tools and the rubrics will be re-evaluated to customize the curriculum to improve students' performance as well as to meet the ABET-CAC criteria.
In the future, the Program Outcomes and Objectives will be collected with the all the six survey forms and the rubrics for more core CS courses, also including other course activities such as quizzes, assignments, presentations, and exams. Based on the result, the CS undergraduate curriculum committee will plan to share their findings and recommendations to the CS department. Furthermore, CS faculty will discuss those issues with the course instructor(s) for the consistent, repeatable assessment plan and the effective change of the CS curriculum accordingly. |
| Update on Previous Cycle's "Plan for Continuous Improvement" |
|
The extended effort to formally assess students’ learning performance has been performed during this academic year, through the previously adopted rubric and instrument (i.e., faculty members’ evaluation on students’ presentation in COSC4319, exam questions in COSC4349, and the newled established instrument, named TASO. As a result, two more assessment scores on both Program Education Objectives (PEOs) and Student Outcomes (SOs) were obtained during 2012-2013 academic year and were compared with the first one obtained Spring 2012.
Based on the assessment result of , the CS undergraduate curriculum committee (UCC) has shared their findings and recommendations to the CS department. Then, all the CS faculty members have discussed those issues with the course instructor(s) for the consistent, repeatable assessment plan, and initiated the change of the CS curriculum accordingly. For example, COSC2329 now includes additional architecture and digital logic modules and the UCC will possibly rearrange course contents or/and revise curriculum. In addition, the UCC suggested to develop explicit exam questions for COSC4319 to directly measure students' learning performance in the capston Software Engineering course, which will be applied from the next academic year. The six survey forms were developed as initial assessment tools. After further discussion, the CS faculty determined to utilize the instruments of COSC4319 and COSC4349 as the major assessment tools. Based on the findings from the current assessment result, the survey forms will be revised accordingly and utilized from the next academic year, if needed as supplementary tools. Following the original plan to assess PEOs and SOs with more complete set of resources, TASO was developed to cover the topics from the core courses, including COSC1436/1437/2329/3318/3319/4318/43419/4319) and the first test was performed in Spring 2013. The UCC didn't identify any action points from the result of TASO, however the result will be further analyzed and utilized from the next academic year as a process of continuous improvement. |
| Plan for Continuous Improvement |
|
In the current (2012-2013) academic year, the department has focused on developing assessment instruments, which meet the new ABET CAC criteria, and also initiated the application of the developed instruments. However, as the ABET assessment team found, the PEOs (and/or SOs) had not been fully documented (or at least not documented) by all of the program’s constituencies (i.e., the students and the Advisory Board). Therefore, students’ exit survey and the Advisory Board review/survey will be documented and the results will be utilized into the continuous improvement process.
Accordingly, in the next academic year, the UCC will focus on the following items so as to demonstrate the continuous improvement effort with the periodic review and revision: (1) Documentation of the review and revision plan, (2) Documentation of full details of the TASO exam, its implementation, and its utilization as input into the continuous improvement process, (3) Further assessment of student outcomes and analysis of the extent to which they are attained, (4) Details of the utilization of the results and findings for program improvement, and (5) Continuing collection and analysis of the whole process and assessment of the implementation and effectiveness of the process. |