Spacer
Assessment : 2013 - 2014 : Educational Programs :
English MA

2 Goals    3 Objectives    3 Indicators    3 Criteria    3 Findings    3 Actions


GOAL: Academic Writing Skills

Objective  
Demonstrating Critical Thinking, Researching, And Writing Skills: Class Writing
English graduate students will demonstrate their abilities as independent critical thinkers, researchers, and writers capable of employing sophisticated skills in written analysis, synthesis, and evaluation of knowledge and of using a professional idiom in making written arguments. The program's success in achieving this objective will be measured by a holistic assessment of graduate class writing.

Indicator  
Holistic Assessment Of Graduate Writing  
The ability of students to write according to accepted professional standards is a direct indicator of the English MA and MFA programs' success in producing graduates who have acquired appropriate critical thinking, researching, and writing skills and are prepared for future professional endeavors. To that end, a significant amount of student writing is required in English graduate coursework.

To assess the effectiveness of class writing assignments in developing students' ability to make sophisticated arguments about literature, language, and writing disciplines in a critical idiom appropriate to professional standards, the faculty will undertake an annual holistic review of representative graduate student writing produced during the reporting period.
Criterion  
Standards For English Graduate Student Writing  
At least 92% of representative graduate essays evaluated during the holistic assessment will be scored as acceptable or excellent (a combined score of 5 or higher on the scale described below).

A rubric for evaluating graduate student writing is attached.


Assessment Process:

1. To assure that the assessment reviews a representative sampling of writing, graduate professors in both long terms are asked to submit term papers or other significant writing from every third student listed on their class rosters.

2. Two primary readers from among the graduate English faculty independently read and score each essay under review; in the case of an unreliable result, the essay is referred to a secondary reader, who reads the essay independently, without any knowledge of the previous results (see number 5, below)

3. Each primary reader scores each essay on a 4-point scale, with a score of 4 the highest possible. The two primary scores are added to yield a total, with the final scores ranging from 8 (highest possible) to 2 (lowest possible). A combined score of 5 or higher is passing. A score of 7 or 8  indicates an excellent essay; a score of 5 or 6 indicates an acceptable essay; a score of 4 or less indicates an unacceptable essay.

4. Reliability of the two scores is assumed when both scores from the primary readers are congruent, that is, when they are within 1 point of each other. For example, a score of 6 that would be seen as reliable would mean that both readers marked the essay as a 3. A reliable score of 5 would mean that one reader assessed the essay as a 3 while the other reader assessed it as a 2.

5. Should the primary scores for an essay not be reliable—for example, a 4 and a 1, a 3 and a 1, a 4 and a 2—the essay is referred to a secondary reader. If that reader agrees with the higher score, the essay is certified as acceptable or excellent; if the secondary reader agrees with the lower score, the essay is certified as unacceptable.
Finding  
Results Of Holistic Assessment Of English Graduate Student Writing  
On July 23, 2014, a committee of eight English graduate faculty from across a wide range of areas--literature, linguistics, professional writing, and creative writing--undertook the holistic review of graduate student writing for the 2013-2014 academic year. The committee reviewed fifteen essays chosen without prejudice from seven graduate courses in literature, language, and writing disciplines. Two committee members read each essay and rated it on the scale of 1-4 described above; the combined scores are as follow:

Score of 7 (excellent): 2 essays
Score of 6 (acceptable): 3 essays
Score of 5 (acceptable): 7 essays
Score of 4 (unacceptable): 2 essays
Score of 3 (unacceptable): 1 essay

Eighty percent of the essays were deemed excellent or acceptable. This number falls short of the ninety-two percent target.
Actions for Objective:

Action  
Developing Students' Writing Abilities  
Critical writing continues to be an important requirement for graduate students in English, not only because it demonstrates their critical thinking and researching abilities and the breadth of their knowledge but also because it trains them practically for professional researching and writing of their own and for teaching research and writing skills to undergraduates and secondary students. 

Outlining an action for improving student writing is difficult: Professors in graduate English courses often presume that their students come to them with sophisticated critical thinking and writing skills in hand. Although they sometimes provide models of good writing in the field, rarely do they devote time--nor do the restrictions of a busy term allow them to devote time--to teaching writing in their classes.

After the last reporting cycle, the Graduate Director consulted with Dr. Carroll Nardone, Director of Writing in the Disciplines, who revised the assessment rubric for the holistic review. The Graduate Director plans to send this rubric to all graduate professors and students in Fall 2014.

The professor assigned to teach the research and methods course (ENGL 5330) required of all incoming English graduate students in Fall 2014 will also give substantial attention to critical researching and writing, especially by providing creditable models from both peer and professional writing. (A qualification: The catalogue description of the research and methods course allows professors to take different approaches. Some treat it as a scientific approach to gathering information and editing; others treat it as an introductory course in graduate writing and critical approaches. The department decided several years ago that it could not mandate that the professor for this class specifically teach writing.)


GOAL: Demonstration Of Breadth Of Knowledge

Objective  
Demonstrating Critical Thinking And Writing Skills And Breadth Of Knowledge: The Written Comprehensive Examination
English students will demonstrate that they have a graduate-level breadth of knowledge in literature, language, and writing disciplines and that they can express that knowledge in writing. The program's success in achieving this objective can be measured by the pass rate for the written comprehensive examination required of all students who take a graduate English degree at Sam Houston State University.

Indicator  
The Written Comprehensive Examination  
A passing score on the written comprehensive examination is a direct indicator that a student in English has acquired a breadth of knowledge in the subject, has developed critical reading and writing skills appropriate to a graduate-level education in English, and is well-prepared for future professional endeavors. For the examination, students choose three comprehensive areas from among thirteen broad topics in literature, language, and writing disciplines. To demonstrate their mastery of a broad range of materials, they are required to choose at least one British literature area and one American literature area and at least one early (pre-1800) British or American literary area and one later (post-1800) British or American literary area. For each area, students are given a reading list of works selected by faculty area experts.

During the exam itself, the student chooses one of three questions for each area and has two hours to respond to that question. A double-blind grading system is used to evaluate the candidates' proficiency. Three graduate faculty members read and evaluate each essay.
Criterion  
Written Comprehensive Examination Pass Rate  
At least 90% of examination essays will pass (with a grade of pass or high pass).

An examination grading rubric and sample pass, fail, and high pass essays are attached.
Finding  
Results Of Written Comprehensive Examinations  
During the reporting year 2013-2014 (including Summer 2014), MA candidates in English wrote sixty-two comprehensive examination essays; this number includes retakes of essays that had previously failed. The results follow:

Total number of passing essays: 50 (81%)
Total number of failing essays: 7 (11%)
Total number of high passes: 5 (8%)

Eighty-nine percent of the total essays passed (with a grade of pass or high pass).

Conclusions about finding: While this result falls just short of the 90% target, it is substantially better than last year's 82%. It is also probably better representative of our graduate students' abilities and preparation because there was a much larger sample this reporting period (sixty-two essays, as compared with last year's thirty-three).

Because the pass rate has risen steadily over the past three reporting cycles, from 69% for 2011-2012 to 82% for 2012-2013 to this year's 89%, measures implemented to help students prepare for the written comprehensive examination may be yielding results. (An informal survey suggests, for example, that most of the students whose essays passed had attended one or more of the Graduate Director's exam preparation sessions.). However, we need to continue to monitor the pass rate over the next two or three cycles before drawing any firm conclusions.
Actions for Objective:

Action  
Preparing Students For The Written Comprehensive Examination  
Because the pass rate has risen over the past three reporting cycles, we seem to be achieving some success in preparing students for the exam. As much as anything else, this may be a matter of impressing upon the students the importance of sound preparation. We will continue to help students prepare:

1. The Graduate Director will continue to publish an exam prep booklet and to conduct biannual sessions, during which he discusses the exam process, suggests strategies for preparing and for addressing exam questions, and presents exemplary questions and responses.

2. Although the comprehensive examination is expressly separated from graduate coursework, some graduate faculty continue to use typical exam questions for midterm and final examinations, as a way of accustoming students to comprehensive exam expectations and circumstances.

3. Faculty continue to give advice about the exam to students who approach them. Despite some discussion at a recent department meeting, the suggestion that faculty be assigned as mentors to a certain number of students has not yet been implemented.


GOAL: Demonstration Of Breadth Of Knowledge

Objective  
Demonstrating Critical Thinking Skills And Breadth Of Knowledge: Oral Argumentation
English graduate students will demonstrate their knowledge and critical thinking skills through oral arguments. We believe that the ability to make such arguments is necessary for future professional pursuits like teaching and further graduate education. The program's success in achieving this objective can be measured by the pass rate for the oral defense required of all thesis students and the oral comprehensive examination required of all non-thesis students.

Indicator  
The Oral Examination  
A passing grade on the oral examination required of all students who take the English MA or MFA degree at Sam Houston State University is a direct indicator that graduates are able to demonstrate their critical thinking skills and breadth of knowledge in the field. Thesis students sit for a one-hour oral defense of the thesis; having passed the written comprehensive examination, non-thesis students sit for a one-hour oral comprehensive examination covering the same three areas as those on the written exam. A committee of three graduate faculty members examines each student, awarding the candidate a pass, high pass, or fail, according to her or his ability to respond to specific questions. The committee for the oral defense of thesis comprises the members of the student’s reading committee; the oral comprehensive examination committee comprises area experts appointed by the Graduate Director.
Criterion  
Oral Examination Pass Rate  
At least 92% of degree candidates will pass the oral defense of thesis or oral comprehensive exam at the first sitting or upon retaking it.

Thesis defense and oral comprehensive exam grading rubrics are attached.
Finding  
Results Of Oral Examinations  
During the reporting year 2013-2014 (including Summer 2014), twelve students sat for oral comprehensive examinations; six students sat for oral defenses of their theses. Sixteen students earned passes, and two students earned high passes (one each for the oral comprehensive exam and thesis defense).

One hundred percent of the students passed the oral examination during the reporting period. This number exceeds the ninety-two percent target.

Conclusions about findings: In last year's assessment of the oral examination, we suggested that the oral defense of thesis and the oral comprehensive examination are unequal measures of our candidates' abilities to demonstrate critical thinking skills: Thesis students know the subjects of their projects as well as, sometimes even better than, the examining faculty and have a much narrower range of material; the thesis defense sometimes becomes an exercise in congratulations upon a job well-completed. (One other important factor is that supervising faculty do not allow a thesis defense until the candidate is ready to defend her or his project; whille the non-thesis student is required to take the oral comprehensive exam immediately after passing the written exam, then, the thesis student has greater scheduling flexibility.) Non-thesis students, who sit for the oral comprehensive exam, must show mastery of a much wider range of topics in literature, linguistics, and writing disciplines; have less control over the questions asked and the direction of the discussion; and are sometimes examined by faculty experts whom they have not met before the examination.

Faculty continue to express disappointment with the peformances of students in the oral comprehensive examination. During this cycle, however, one student who sat for the comprehensive exam was awarded a high pass (with one student also being awarded a high pass for the thesis defense). Although the number is statistically insignificant, it does suggest that students can meet a standard of excellence in the oral comprehensive exam (see the attached rubric).
Actions for Objective:

Action  
Preparing Students To Make Oral Arguments  
One hundred percent of students who have sat for the oral defense of thesis or oral comprehensive exam in the last five reporting cycles have passed. There are still complaints among examining faculty, however, that students taking the oral comprehensive examination respond with weak arguments and inadequate knowledge.

In our statement of action for the last reporting cycle, we stressed that the oral examination should not be the only measure of a student's ability to express critical thinking skills and breadth of knowledge orally. It is, however, one of the few uniform measures, since not all graduate classes require oral presentations.

In a department meeting specifically about graduate matters, the issue of the oral exam was raised briefly, without much further discussion or consensus, since most attention was focused on the written examination (also one of the few uniform measures of our students' critical thinking abilities and breadth of knowledge).

Although an oral component is not yet required in any graduate courses, the professor assigned to teach the mandatory research and methods class (ENGL 5330) to incoming students in Fall 2014 has agreed to include a formal unit on oral presentations.

Beginning in August 2014, during the comprehensive examination prep sessions, the Graduate Director will also incorporate suggestions for preparing for the oral comprehensive examination. (A qualification: Students are not required to attend the prep sessions.)

Faculty will also continue to encourage students to participate in academic conferences, at which they must not only present their arguments about literature and language orally but also respond to questions and challenges from the professional audience.



Previous Cycle's "Plan for Continuous Improvement"

One measure of our continuing success in producing graduates who have demonstrable critical skills and breadth of knowledge in the field is that our MA students continue to be accepted into respectable PhD and MFA programs. During the 2012-2013 assessment period, graduates were accepted to PhD programs at Arizona State University, the University of Arizona, Carnegie-Mellon University, Drexel University, and the University of North Texas; most of them were awarded full funding. Several other graduates were given teaching positions in two-year colleges. Although these indicators of success cannot be considered measurable because not all of our graduates aspire to such endeavors, as part of our plan for continuous improvement, the faculty will continue to encourage worthy MA graduates to apply for PhD work and teaching positions.

Another measure of the program's success in producing graduates with demonstrable critical researching and writing skills and breadth of knowledge is student participation in professional conferences. Not all students participate in such activities, however, so although faculty will continue to encourage them to present their scholarly and creative work at conferences, participation cannot be considered a measurable indicator of the program's success in achieving its objectives. One suggestion for the future, however, is that participation in at least one scholarly or creative conference or colloquium be a requirement for graduation. In such an event, conference participation could be included a measurable indicator.

Dr. Helena Halmari, English Department Chair, will also continue to pair qualified students with faculty members as research assistants. The requirement for this research assistantship is that the collaboration between student and faculty member lead to a publication and/or conference presentation. Again, because not all students qualify for such assistantships or seek them out, the work undertaken as a research assistant cannot be considered a measurable indicator of the program's success in producing graduates with critical thinking, researching, and writing skills.

In responding specifically to the findings for the three objectives above, we propose the following plan for continuous improvement in the 2013-2014 assessment period:

1. The graduate faculty will undertake a thorough review of comprehensive examination reading lists, to assure that the lists represent both the expectations for breadth of knowledge and current developments in the field.

2. The graduate faculty will undertake a review of comprehensive examination questions, to assure that they are both fair and representative, that they adequately test a student's critical thinking and writing skills and breadth of knowledge, and that they represent current developments in the field.

3. Because faculty who sit on oral comprehensive examination committees still find weaknesses in some students' ability to make critical arguments and demonstrate their breadth of knowledge orally, the department will undertake a pointed discussion about both the nature of and the expectations for this oral exam.

4. To improve its progress toward achieving the objective, the graduate faculty will also consider requiring an oral component in some types of courses or other means by which the program can develop the students' ability to make oral arguments.

5. The graduate faculty will resume the holistic assessment of graduate student writing. To that end, the graduate director has already collected representative writing from all graduate courses taught in Fall 2013.

6. To assure that the rubric for the holistic assessment of writing fairly measures our students' critical thinking, researching, and writing abilities, the graduate director will consult with the University's Director of Writing in the Disciplines, who is a member of the English Department.

7. After any necessary revisions to the rubric have been made, the graduate faculty will discuss the standards for classroom writing and how well that writing develops and/or measures our students' critical abilities. The aim of this discussion will be to reach some sort of departmental consensus on standards for writing in the graduate classroom.

8. To encourage greater faculty mentoring of MA students, the graduate director will propose that each graduate faculty member be assigned four or five students as advisees. While the graduate director will continue with general advising of students, the faculty mentors would meet with their advisees as needed to discuss class researching and writing assignments and to help them prepare for written and oral examinations. Although this advising would be informal, we may require that students meet at least once with their mentors in each long term.
Update on Previous Cycle's "Plan for Continuous Improvement"

The Department of English implemented the following elements of the 2013-2014 plan for continuous improvement:

1. The graduate faculty undertook a thorough review of comprehensive examination reading lists, to assure that they reflect the expectations for breadth of knowledge and currency in the field. This review was completed during the late spring, and updated lists (current 1 August 2014) have now been posted on line.

2. The graduate faculty resumed the holistic review of students' class writing. The results of that evaluation appear above.

Other elements of the plan for continuous improvement from the last reporting cycle were raised in a departmental meeting set aside last spring specifically for graduate program business: (1) a thorough review of comprehensive examination questions, (2) the challenges of incorporating an oral component into classwork, and (3) the possibility of establishing a faculty mentoring system. The graduate faculty decided collectively to postpone these issues until the coming year, especially because much discussion revolved around the nature and the effectiveness of the current written and oral examination system. A committee of five graduate faculty members was formed to review the exam system and make recommendations for future examinations.
Plan for Continuous Improvement

In responding specifically to the findings for the three objectives above, we propose the following plan for continuous improvement in the 2014-2015 assessment period:

1. A committee of five graduate faculty members appointed during Spring 2014 will meet to discuss the written and oral comprehensive examinations, considering specifically how effective they are in measuring the students' critical thinking and writing skills and their breadth of knowledge.

2. The graduate faculty will undertake a thorough review of comprehensive examination questions. This task was postponed from the previous plan for improvement because it depended, in part, on the review of reading lists, which was finally completed in late spring. One persistent concern among graduate faculty is that some students are skirting the requirement that they read all works on an area list and are still managing to pass the exam because too-general questions allow too much flexibility in the responses. In order to assure greater rigor, graduate faculty have suggested more specificity in the questions. The plan for improvement will address this concern.

3. While faculty generally agree upon expectations for the quality of graduate student writing, it is difficult to reach a consensus about specific, measurable standards, in part because of the variety in the kinds of writing expected in various classes: critical term papers, linguistic analyses, papers about pedagogical methods and practices, annotated bibliographies, and expository papers. As one step toward reaching a consensus, the Graduate Director will distribute the recently revised rubric for the holistic assessment of writing to both faculty and students.

4. The graduate faculty will resume the discussion of appointing faculty members as mentors to students. While the Graduate Director will continue the general advisement of students, the faculty mentors would be available to their advisees to discuss class researching and writing assignments and to help them prepare for written and oral examinations.


Sub Content Box

Sam Houston State University
Huntsville, TX 77341
(936) 294-1111
1-866-BEARKAT