Standard 4

[back to CAEP Self-Study Report]

 

Standard 4: The provider demonstrates the impact of its completers on P-12 student learning and development, classroom instruction, and schools, and the satisfaction of its completers with the relevance and effectiveness of their preparation.

NOTE 1: All components must be met for Standard 4

NOTE 2: Standard 4 and the “8 annual reporting measures”

The CAEP January requests for provider annual reports include questions about data on each of the 4.1 through 4.4 components. The request to EPPs defines the minimum expectation each year until reporting across providers can be complete and consistent. Trends in the provider’s cumulative reports since the last accreditation cycle will be included and interpreted as part of the self-study report. Providers are expected to supplement that annual reporting information with other, more detailed, data on the same topics from their own sources. Unconstrained by CAEP’s longer-term goal for consistently defined and commonly reported annual measures, EPPs will have greater flexibility to assemble their best documentation for Standard 4 by employing sources available in their own state, or documentation that they created, if any.

4.1 The provider documents, using multiple measures, that program completers contribute to an expected level of student-learning growth. Multiple measures shall include all available growth measures (including value-added measures, student-growth percentiles, and student learning and development objectives) required by the state for its teachers and available to educator preparation providers, other state-supported P-12 impact measures, and any other measures employed by the provider.

4.2 The provider demonstrates, through structured and validated observation instruments and student surveys, that completers effectively apply the professional knowledge, skills, and dispositions that the preparation experiences were designed to achieve.

4.3 The provider demonstrates, using measures that result in valid and reliable data and including employment milestones such as promotion and retention, that employers are satisfied with the completers’ preparation for their assigned responsibilities in working with P-12 students.

4.4 The provider demonstrates, using measures that result in valid and reliable data, that program completers perceive their preparation as relevant to the responsibilities they confront on the job, and that the preparation was effective.

Candidates in the initial level teacher education program at Marshall University complete a rigorous, challenging, and comprehensive program, as evidenced in CAEP Standards 1, 2, and 3. Preparation provides program completers with the ability to contribute to an expected level of student-learning growth and to effectively instruct their P-12 students. Via multiple measures, the initial level demonstrates this impact as well as the satisfaction of its completers and their employers with the relevance and effectiveness of preparation.

IMPACT ON P-12 STUDENT LEARNING

The initial level of the COEPD utilizes three different measures to indicate completer impact on student learning. These include performance assessment data from the Teacher-in-Residence experience, case study data from 1st year out teachers, and case study data from 3rd year out teachers.  During the TIR experience, candidates currently complete the WVTPA (West Virginia Teacher Performance Assessment) documenting their impact on student learning. The COEPD joined a WV coalition piloting the WVTPA in the fall of 2016 and data are provided for the past two semesters. Prior to this requirement, TIR candidates submitted either the edTPA or PPAT assessment. The TEACHER-IN-RESIDENCE (TIR) is an intensively supervised and mentored residency program for prospective teachers during their senior year that refines their professional practice skills and helps them gain the teaching experience needed to demonstrate competence as a prerequisite to certification to teach in the West Virginia public schools. TIR Programs require authorization of the WVBE pursuant to W.Va Code 18A-3-1(e). TIR participants are full-time teachers and are supported by a local mentor and supervisor. For purposes of addressing Standard 4, TIR participants are considered program completers in their 1st year of teaching.

Beginning with the fall semester of 2014, the initial level program has increased the number of TIR agreements and placements each year, and currently several WV counties are participating in the partnership. The number of participating counties has sharply risen from three in 2014 to the current 16. At this time, TIR agreements are predominantly located in the counties surrounding Huntington, WV and in high need geographical areas, such as the southern coalfields of WV and the increasingly populated eastern panhandle. In addition to tracking where Teacher-in-Residence candidates are placed, the initial level also monitors the content areas of the placements. High school mathematics is the most common, with 56% of the placements, and the second highest area is elementary, with 33%. Following the trends for placements helps the initial level recruit candidates who have an interest in the subject areas most likely to be placed and eventually employed. The initial level continues to work to develop additional TIR agreements and has a goal of partnering with 25 counties by the year 2020.

Analysis of the edTPA data from Fall 2015 and Spring 2016 reveals encouraging numbers. Both mathematics candidates were rated as Proficient or Advanced in all 15 categories. PPAT data for the elementary candidate yielded similar results with scores of “3” or “4” on all 31 categories. Low “N’s” prevented trend analyses, but the initial results seemed positive.

Data from the WVTPA provide more significant insight on the impact TIR candidates are having on P-12 student learning. Of the 9 TIR candidates from the past two semesters, all have scored at least “Emerging” on all 31 WVTPA categories. When specifically examining categories related to impact on student growth, candidate data are trending upward. The “Evidence of Impact” average score (2.33) in fall 2016 increased to 3.00 in spring 2017. Likewise, positive changes were noted in “interpretation of data” mean scores (2.33 increased to 3.33) and “clarity and presentation of evidence” (2.00 increased to 3.00). The initial level will continue to track TIR data and delineate results.

The second and third measures of completer impact on P-12 student learning emerge from a plan developed in the fall of 2016 to capture additional specific data on program completers. Programs began utilizing a CASE STUDIES 1 YEAR OUT INITIAL LEVEL and a CASE STUDIES 3 YEARS OUT INITIAL LEVEL to document completers’ impact on student learning and development and on completers’ teaching effectiveness. Beginning in 2017, both 1st year out and 3rd year out completers were identified and observations conducted. Moving forward, observations will be conducted every spring semester for 1st year out completers and every fall semester for 3rd year out completers. Completers are asked to develop and teach a unit utilizing a pre-posttest design and document student growth. The completers replicate Task 3, Task 6 and Task 7.1 from the WVTPA Performance Assessment. The TPA draws on pedagogical and content knowledge to plan and deliver instruction that builds on each student’s strengths, needs, and prior experience. Through this performance assessment, completers provide evidence of student impact. Pre/post test data are available. Demographic details and +/- change in mean scores are calculated at both the aggregate and individual level. Additionally, completers are observed and evaluated utilizing the WV Evaluation Rubric for Teachers. Principal evaluations are included when possible. Completers are also observed by the Assistant Director of Clinical Experiences, and both quantitative and qualitative data are collected. By the end of three years, all initial level licensure programs will be represented at both junctures and the initial level will be able to identify possible trends by content level, grade level, and experience level.

The first round of observations was completed during the spring and fall 2017 semesters. Six separate case studies were completed on 1st year out teachers in the spring and four separate case studies were completed on 3rd year out teachers in the fall. The first (1st) year out teachers heralded from five different WV counties and taught in both elementary and secondary schools. Aggregate demographic data found the teachers to be employed in schools with an average of 56.23% “percent needy” students, 16.67% “special education” students, 3.7% “racially diverse” students, and 0.08% “English Language Learners”. School size (enrollment) ranged from 339-1971, with an average of 840.83.

The 3rd year out demographics yielded several similar patterns, although none were completed in the same school as the 1st year outs. The four 3rd year out teachers were from three different WV counties, and while all taught at the elementary level, one was a special education teacher, and one was a Wellness (Health and PE) teacher. Aggregate demographic data found the teachers to be employed in schools with an average of 61.76% “percent needy” students, 21.25% “special education” students, 3.95% “racially diverse” students, and 0.75% “English Language Learners”. The only demographic of considerable difference from the 1st year out study was school size (enrollment), with ranges from 89-923 and an average of 433. These differences were attributed to the fact that all of the 3rd year-out teachers were employed at the elementary level. Reviewing school and student demographic data allows the initial level to distinguish if its completers are more successful in certain types of schools.  Identifying these possible trends will assist future program and curriculum development.

The case studies were designed to measure whether or not instruction led to student success in meeting identified goals. Each completer developed and taught a unit utilizing a pre-posttest design and documented student growth with +/- mean percent correct calculated. Results from 1st year out completers clearly validated a positive impact on student learning. All of the six teachers saw increases in pre to post scores, with an average increase at +45.49%. Mean scores on pretests ranged from 0-62.27% correct with a mean of 28.67% while mean scores on post-tests ranged from 44.5-99.5% with a mean of 74.16%.

Results from 3rd year out teachers generated parallel findings. Each of the four teachers saw increases in pre to post scores, with an average increase at +36.72%. Mean scores on pretests ranged from 0-100% with a mean of 28.17% while mean scores on post-tests ranged from 64.89-136.67% with a mean of 64.89%. The one score above 100% was attributed to a Kindergarten testing design using the Dibels program which allows students to score above grade level.

Modified WVTPA scores also provided credible evidence of impact on P-12 student learning. All completers, both 1st year out and 3rd year out, scored either “Emerging” or “Accomplished” on each of the seven measures. Third year out scores were slightly higher on “Interpretation of data” (3.00 vs. 2.67) and “Insights on teaching and learning” (3.00 vs 2.67). Elementary completer scores also trended higher than secondary, with all of the six mean scores exceeding the all completer average for the 1st year out and five of the six mean scores exceeding the all completer average for the 3rd year out.

Initial conclusions are positive on the impact initial level completers are having on students in the P-12 schools. Data from the case studies furnish solid evidence that initial level completers are making a decisive difference in student growth. The case studies will be continued for the next three years to identify more substantial patterns and to examine all content areas and all school levels.

INDICATORS OF TEACHING EFFECTIVENESS

The initial level of the COEPD demonstrates that completers effectively apply the professional knowledge, skills, and dispositions that the preparation experiences were designed to achieve. In order to provide evidence, four different measures of teaching effectiveness are tracked. These include 1) Marshall completers attaining National Board Certification, 2) Performance data from the TIR experience, 3) 1st year out case study data, and 4) 3rd year out case study data.

As the first indicator of teaching effectiveness, the COEPD monitors the number of its completers attaining NBCT (National Board Certified Teachers) in the state of West Virginia. According to the NBCT website, a decade of research shows that students of Board-certified teachers learn more than their peers without Board-certified teachers. Studies have also found that the positive impact of having a board-certified teacher is even greater for minority and low-income students. Board certification allows teachers to hone their practice, showcase their talent, and demonstrate their dedication to their students and profession. Since 2005, 124 Marshall University graduates from 15 different content areas have become nationally board certified. The highest numbers of NBCT completers are in the areas of Reading/Language Arts (33), English (23), Generalist/ECE (13), Mathematics (11), and Exceptional Needs (10). COEPD candidates with NBCT designations have been employed in 28 different West Virginia counties sprinkled throughout the state, predominantly in high need areas. The initial level of the COEPD plans to continue to track NBCT completers.

The second source of teaching effectiveness data stems from the TEACHER-IN-RESIDENCE candidates (previously described). The fall 2015 and spring TIR placements were evaluated with an extended version of the WVPTS evaluation instrument. The fall 2016 and Spring 2017 placements were evaluated using an abbreviated form of the prior instrument. Overall, participants were rated by both their mentors and supervisors as being at least “Accomplished” in terms of their performance on all WVPTS competencies. Twelve of the fifteen measures resulted in mean scores at 3.0 or better and no candidates scored at an “unsatisfactory” level.

The third and fourth measures of completer teaching effectiveness emerged from the CASE STUDIES 1 YEAR OUT INITIAL LEVEL and CASE STUDIES 3 YEARS OUT INITIAL LEVEL detailed previously. Utilizing observation and one-on-one interviews, the case studies were designed to examine six areas of teaching effectiveness. Completers were observed and evaluated by the Assistant Director of Clinical Experiences utilizing the WV Evaluation Rubric for Teachers. Principal evaluations were also shared when available.

Results from both 1st year and 3rd year out completers clearly demonstrate completer competency. Of the 19 measures, scores ranged from 2.33-3.0 for 1st year out and 2.50-3.00 for 3rd year out. No completers were rated as “unsatisfactory” on any variable. Third year out candidates tended to score slightly higher in Standard 2: The Learner and the Learning Environment, specifically on “Unique characteristics of learners” (3.00 vs. 2.67) and “Learner-Centered culture” (3.00 vs. 2.83). Standard 5: Professional Responsibilities for School and Community also naturally demonstrated 3rd year out completers to be more comfortable with understanding how to function at a more advanced school-wide level, with “School-wide collaboration” (2.75 vs 2.33), “Working with Parents and Community” (3.00 vs.2.67), and “Practices and Policies” (3.00 vs. 2.33) scores all trending higher than the 1st year out.

Third year out completers were also more willing/able to share principal evaluations for the case studies. Of the six 1st year out completers, only one (17%) provided his/her evaluation. At the 3rd year out level, 3 of the 4 (75%) granted access. Interestingly, principal evaluations for both 1st and 3rd year out teachers tended to be higher than the Marshall University observer, particularly for 3rd year out teachers, with principal averages higher in all 19 categories.

The first round of case study data produced substantial evidence of the teaching effectiveness of initial level completers in P-12 schools. Data collected document that Marshall University completers are considered to be valuable teachers and are highly effective in their jobs. The case studies will be continued for the next three years to identify more apparent patterns and to examine all content areas and all school levels.

SATISFACTION OF EMPLOYERS

Working within the framework established by the QAS (Quality Assurance System) for collecting evidence of employer satisfaction with completers, initial programs have adopted a multidimensional approach which allows for triangulation of multiple data points. Sources of employer satisfaction data include program-level follow-up studies, Educator Expo Surveys (recruiter surveys), Teacher-in-Residence Program mentor and supervisor surveys, and NBCT performance data. An additional data point (NEXT Survey) will be added in spring 2018 as results from a WVDE/WVHEPC sponsored statewide survey of P-12 supervisors will be available. The NEXT Survey will also allow for benchmarking COEPD data against statewide averages.

Employer follow-up data are collected annually from principals in the COEPD service area. The EMPLOYER SATISFACTION SURVEYS INITIAL LEVEL contains results from a 2014 survey of Cabell County principals and data from a fall 2016 survey of principals in RESA II.  The fall 2016 survey was revised to reflect the WVPTS and WV Teacher Evaluation Form and was focused on both 1st and 3rd year out completers. Data were disaggregated accordingly.

Findings from the 2014 survey of Cabell County principals (N=15) indicate that at least 86% of the principals reported they were “Somewhat Satisfied” or “Very Satisfied” with the performance of COEPD completers on each of the 18 teacher quality standards rated. All 15 principals reported that, based on their assessment of completer preparation and performance, they felt confident in employing other/future COEPD completers.

Data from the revised fall 2016 assessment of RESA II principals (N=12) were disaggregated to provide a focus on both 1st and 3rd year-out COEPD completers. No 1st year-out completers were rated as “Unsatisfactory” on any of the 30 teacher quality standards. The majority of the responses fall in the “Emerging” or “Accomplished” categories, however, at least one responding principal characterized 1st year-out teachers as “Distinguished” on 13 of the 30 teacher quality standards. Principal evaluations of 3rd year-out teachers indicate no ratings of “Unsatisfactory” on any of the 30 standards. All respondents rated 3rd year-out completers as “Accomplished “on 17 of the 30 indicators and 80% of the principals rated them as Accomplished on an additional 10 indicators. When principal ratings of 1st and 3rd year-out teachers are compared, there is a clear trend for principal ratings to be higher for 3rd year-out teachers.

Validity (Face Validity) and reliability (Internal Consistency; Cronbach Alpha==.982) studies were conducted on the 2016 assessment (survey) in summer 2016. A Factor Analysis Study was conducted in summer 2017 and validated that the survey was measuring only one variable. In an effort to further validate the assessment, a content validity study (Lawshe method) was conducted in fall 2017. Study results indicated a need to remove 10 items from the assessment. These items were deleted and a revised assessment was administered in fall 2017.

Employer feedback is routinely solicited as a part of the EPPAC (Educational Personnel Preparation Advisory Committee) meeting held each semester. The EPPAC is the primary COEPD stakeholder group and includes P-12 employers. The data collected include both participant comments and survey feedback. Participant comments are included in EPPAC meeting minutes and have been excerpted and included with survey responses from spring 2017. In the spring 2017 survey, EPPAC members (N=9) were asked to rate COEPD completers in terms of their demonstrated competence on 30 teacher standards aligned with EPP preparation standards. More than 88% of the responses were in the “Emerging” or “Accomplished” categories for 23 of the 30 standards. One third of the respondents rated 3 standards related to Student Management Systems, School/Family/Community, and Ethical Standards as “Unsatisfactory”. Participant comments from the April 2016 and April 2017 meetings were consistent with the survey findings suggesting new teachers are performing satisfactorily and that we should provide additional emphasis on data use, collaboration and teaming skills, and professional learning goal setting.

TEACHER-IN-RESIDENCE (TIR) data discussed earlier provides another measure of employer satisfaction. Mentor and supervisor ratings have proved generally consistent and positive, and three different counties have housed multiple TIR candidates indicating a satisfaction with the partnership.

Data regarding COEPD completers are solicited every other spring semester from P-12 recruiters. Recruiters are asked to complete an EDUCATOR EXPO SURVEYS INITIAL LEVEL during the Job Fair Expo conducted by the Marshall University Career Services Center. Data are provided for the 2013, 2015, and 2017 survey administrations. The actual assessment used in 2013 and 2015 was modified in 2017. Content validity (Lawshe method) and reliability (internal consistency; Cronbach Alpha=.968) were used to assess the quality of the assessment and resulting data. The assessment was determined to have an acceptable level of both validity and reliability. Data from the 2013 (N=4) and the 2015 (N=18) surveys are consistent in their assessment of EPP completers. All but one of the 13 completer qualities assessed were rated as ‘Very Good” or “Excellent”.

The 2017 survey was revised to reflect initial level dispositions and employability questions. Results from the 2017 revised survey (N=18) were consistent with those from the 2013 and 2015 surveys. Ninety-five percent of the recruiters responded either “Excellent” or “Very Good” in terms of COEPD graduates’ knowledge of students and respect for diversity. More than 80% of the recruiters responded similarly for completer ability to identify strategies that impact student learning (84%), utilize instructional resources including technology (84%), and exhibiting a commitment to creating positive learning environments (89%). More than 70% of the recruiters rated completers as “Excellent” or “Very Good” in terms of their knowledge of content and curriculum (78%), their ability to reflect on teaching experience (73%), their commitment to challenging students to think critically (73%), and their commitment to the profession (78%). All 18 of the participating recruiters indicated they would hire a COEPD graduate and 72% indicated they had done so. Those recruiters indicating they had hired COEPD completers gave them overall ratings of “Excellent” (65%), “Very Good” (23%), and “Good” (8%).

SATISFACTION OF COMPLETERS

The initial level of the COEPD values the satisfaction of candidates who have completed its programs. Measuring completers’ satisfaction with their preparation once they are employed provides valuable insight and leads to program improvement, better relationships, and more robust preparation. The initial level demonstrates, using measures that result in valid and reliable data, that program completers perceive their preparation as relevant to the responsibilities they confront on the job, and that the preparation was effective. Four measures are used to determine completer satisfaction. These include 1) Completer Satisfaction Survey from 1st year out (2015-2016 completers), 2) Completer Satisfaction from 3rd year out (2013-2014 completers), 3) NExT Survey, and 4) STUDENT TEACHING SURVEY.

COMPLETER SATISFACTION SURVEYS INITIAL LEVEL yielded several positive results. The survey was administered to both 1st year out (those completers graduating during the 2015-16 academic year and employed in their first year of teaching during the 2016-17 year) and 3rd year out (those completers graduating during the 2013-14 academic year and employed in their third year of teaching during the 2016-17 year) completers who were part of the Teacher Induction Program with Cabell County Schools. Thirty-one (31) 1st year out teachers and thirty-four (34) 3rd year out teachers participated.

Findings from the survey indicate that 1st year out teachers have positive impressions about their preparation. On 27 of the 30 categories, 70% or more of completers rated themselves as either “Very Satisfied” or “Somewhat Satisfied”. Ten of the categories yielded 90% or more of candidates in these two categories. The two lowest areas were in school-wide endeavors and collaborative efforts with the principal. It appears 1st year out teachers felt least prepared to partner with administration and other school colleagues with 39% being “somewhat dissatisfied” or “very dissatisfied” with school mission and 32% “somewhat dissatisfied” or “very dissatisfied” with school, family, and community connections.

Interestingly, third (3rd) year out teachers rated their preparation extremely high. All 30 categories received at least eighty percent (80%) of the completers rating them as “Very satisfied” or “Somewhat satisfied”, and 26 categories were rated at that level by over 90% of the 3rd year out completers. It appears that candidate dissatisfaction with school mission and school connections lessen with time and experience, a seemingly natural improvement. Only 12% of 3rd year out completers classified school mission preparation as “somewhat dissatisfied” and only 9% rated school connections at that level. No 3rd year out candidates were “Very dissatisfied” with any of the categories.

The initial level of the COEPD understands the importance of utilizing survey techniques that are both reliable and valid. The timeline summarized under employer satisfaction details the collection, revision, and quality of both the “Completer Satisfaction Survey” and the data collected from it. Please note the “Completer Satisfaction Survey” mirrors the “Employer Satisfaction Survey” so reliability and validity were determined concurrently. The revised assessment will be administered in spring 2018.

The second measure of completer satisfaction is the NEXT SURVEY (Network for Excellence in Teaching). NExT, founded in 2010, is a partnership of 14 institutions of higher education (IHEs) and the Bush Foundation. NExT collaborated to develop a set of common surveys to support teacher preparation programs in measuring the effectiveness of their programs. NExT shared the instruments with other teacher preparation programs, inviting them to contribute their data to an aggregate data set that will be used in future instrument analyses to strengthen the instruments and ensure their validity and reliability across diverse respondent pools. The surveys include the following: 1) Exit Survey_administered to teacher candidates near the completion of student teaching, 2) Transition to Teaching Survey (TTS)_administered to program completers in the spring following the academic year of graduation, and 3) Supervisor Survey_administered in the spring following the academic year of graduation to employers of program completers who are teaching.

Exit Surveys were administered to Marshall University initial level student teachers during fall 2016 and spring 2017. The Exit Survey collects information on student teachers’ perceptions of and satisfaction with their teacher education programs and student teaching experiences as well as their backgrounds and future plans. Each of the surveys has been found to be highly valid and reliable. The Fall 2016 Exit Survey was administered to candidates who still had classes in session or that were still in contact with their instructors online when the survey became available to WV IHEs in December 2016. The survey link was e-mailed directly to students or posted on the institutional website for candidates to access. The Spring 2017 Exit Survey was administered during class time, when possible, to increase response rates. All eligible candidates were invited to take the survey for both administrations. The 2016-2017 Exit Survey response rate for the institution was 94% (94 out of 117). In comparison, the response rate for the West Virginia aggregate was 71% (419 out of 589).

Data from the NExT survey administration provided strong evidence of initial level completer satisfaction. An overwhelming majority (86.96%) surveyed stated they probably or definitely would recommend Marshall University to other prospective teachers. In the area of “Program Structure/Quality” mean scores ranged from 2.89-3.59 on the eight satisfaction variables. Similar mean scores were found on the nine “Diverse Learners” scale with a range of 2.82-3.39. Even higher scores were found in “Learning Environment” (3.39-3.61), “Instructional Practice” (3.25-3.61), and “Professionalism” (3.26-3.72).

Moving forward, the initial level of the COEPD plans to examine findings from this Exit Survey and compare them to past and future cohorts in order to understand how shifts in coursework and clinical experiences affect candidates’ perceptions of and satisfaction with their teacher education programs. Findings from the Transition to Teaching Survey, administered one year after graduation, may also shed light on whether completers’ perceptions of and satisfaction with their preparedness at graduation align with perceptions of their instructional practice as student teachers.

Summary

In summary and conclusion, the COEPD has developed and implemented a multidimensional approach for compiling, soliciting, and using data regarding initial level program completers. The strategies used are consistent with the overall COEPD framework for collecting completer impact and employer/completer satisfaction data and implement data collection with multiple strategies and data points. Where applicable, assessments (surveys) have been subjected to appropriate validity and reliability assessments and have been determined to meet CAEP criteria for assessments at the “Sufficient” or higher level. Findings from multiple applications of employer and completer follow-up studies, recruiter surveys, EPPAC surveys and participant comments, TIR mentor/supervisor surveys and performance assessments, NBCT performance, and 1st year and 3rd year out case studies clearly indicate initial level completers are effective teachers positively increasing student growth and that both employers and completers are more than satisfied with the preparation and performance of our initial level program completers. COEPD faculty have used data to continually improve and inform the development of the QAS and data collection system, increase the interaction with P-12 stakeholders, and consider a number of changes in the educator preparation program. Future plans include a continued focus on development/refinement of assessments, continuing to identify strategies to increase responses rates, increasing the focus on gathering employer/completer feedback that can be more readily disaggregated by licensure area, and expanding the scope of measuring completer performance in the field via case studies or state assisted measures.

[back to CAEP Self-Study Report]