Standard A.4

[back to CAEP Self-Study Report]

 

Standard A.4 Satisfaction with Preparation The provider documents the satisfaction of its completers from advanced preparation programs and their employers with the relevance and effectiveness of their preparation.

The central focus of CAEP standards is that EPPs use quality evidence for continuous improvement of their programs.  Standard A.4.1 requires that the EPPs “demonstrate that employers are satisfied with completers’ preparation and that completers reach employment milestones such as promotion and retention”.  The EPP must provide evidence that is specific to specialty fields (standards based) and is representative in terms of sample size.

Working within the framework established by the Quality Assurance System (QAS) for collecting evidence of employer satisfaction with completers, the Advanced Level developed a pilot study to use interviews to assess employer satisfaction. It was decided to use an interview model due to the small size of many of the programs. An employer interview survey for School Psychology was developed in spring 2016. The interview was based on the National Association of School Psychologists’ (NASP) standards. The interview protocol was validated by various groups including professors, administrators, state department administrators, and practitioners in the field. Edits were made based on feedback from stakeholders. Interviewers were Graduate Assistants (GAs) in the doctoral program who had experience with research. Training included a discussion of the best method for obtaining data, the use of follow-up questions, and the recommendation for audio recording of the interviews. A random sample of 6 school psychology 2015 graduates were emailed with a request for the contact information of their supervisors. This represented 43% of the cohort. All 6 candidates replied with contact information and agreed to have their supervisors contacted. The employers were then notified by email that they would be called by phone for an interview and the purpose of the interview. The email contained the interview questions and the NASP standards as attachments. The GAs conducted the interviews then a meeting was held with representatives of the AACC to discuss the interviews in order to refine and improve the process for future interviewing.

The AACC compiled the employer interview data and provided it to the School Psychology Program. Overall mean scores were in the proficient to mastery range for all areas indicating employers were satisfied with the program completers’ preparation.  Interviewee responses are summarized for each standard and presented in EMPLOYER EVALUATIONS ADVANCED LEVEL. The lowest mean scores were in communicate assessment results effectively (3.17), effectively implements individual and group therapy (3.2), and assists schools/agencies to evaluate and revise prevention and crisis intervention policies and practices (3.25).  The School Psychology Program reviewed the data along with other sources of data such as Praxis II scores, completer surveys and field supervisor ratings to make changes to the program. Triangulating data from these sources revealed that School Psychology candidates were not feeling confident in conducting therapy in schools or participating in crisis situations. Based on this information, changes were made in coursework and practicums to increase candidate experiences in these areas.

After the successful pilot in spring 2016, the remaining 15 licensure programs at the Advanced Level were asked to develop interview questions based on each program’s standards. They were encouraged to all use the same rating scale (e.g. Mastery, Proficient, Emerging Competence, and Unsatisfactory). This would enable comparisons between programs. Four programs requested to use Qualtrics to collect data rather than interviews. A decision was made to allow two methods of data collection in order to determine the most effective method for obtaining employer satisfaction data. Interview protocols were developed by programs and then reviewed by the Assessment and Accreditation Coordinating Council (AACC). GAs and administrative staff were trained on the interview protocols. These same individuals also assisted with entering the questions into Qualtrics for the programs using a survey method.

The next step was to compile the names of all the candidates who had graduated in spring 2016. There were challenges finding employers of graduates because tracking information is not provided by the state. Contact information was obtained from student files, social media such as LinkedIn, professors, and colleagues of candidates. One hundred and seventeen candidates were emailed and requested to provide information for their immediate employer. Of these, only 30 replied with their supervisor’s information.

The data from two programs are available in the EMPLOYER EVALUATIONS ADVANCED LEVEL. One utilized the interview format while one elected to use Qualtrics, There are data from the interviews of employers for the School Psychology candidates from spring 2016 and 2017 and Qualtrics employers surveys for the Principalship candidates for spring 2017. The Principalship questions were aligned with the ELCC standards.

The Survey was sent to 20 identified employers of One-Year Out Graduates of the Principalship Certificate, and results from this revised survey (N=8) indicated that Ninety-five percent of the employers responded either “Proficient” or “Mastery” in terms of COEPD graduates’ preparation to meet ELCC Standards.  100% of the employers rated completers as ‘Proficient” or “Mastery” in terms of their knowledge of Vision, Collaboration, and Context, and 87.5% “Proficient” or “Mastery” with regards to School Culture and Instructional Programs, Integrity, fairness, and ethics. None rated any of the completers as Unsatisfactory in any of the categories. The results of this survey indicated a positive satisfaction with the preparation of these completers by their employers.

While other employer data has been obtained from other programs, the small sample size makes generalizability difficult. Due to the obstacles encountered in tracking candidates, COEPD has developed a Google Doc which all programs will utilize when candidates exit the programs. The Google Doc will collect candidate information such as cell phone number, personal email, and employer information. Candidates will also be informed that they will be contacted at one year and three years to obtain employer and completer information. Utilizing the Google Doc along with an improved data base that was established in April 2017, COEPD will be better able to track candidates after graduation.

Another method COEPD utilized to obtain employer satisfaction, was to survey personnel directors. Each year COEPD invites personal directors from LEAs in southern and central WV to Marshall University. At this meeting, personnel directors share their needs and concerns. Program Directors from COEPD Advanced Level attend to present current information about programs. This information sharing helps to improve the quality of the COEPD programs and permits us to determine the needs of the counties.

Attending personnel directors were asked to complete a three-part survey (Summary of 2016-2017 PERSONNEL DIRECTORS SURVEY) about advanced program completers and COEPD responsiveness to LEAs.  Parts A and D included questions in which the personnel directors responded to six statements about COEPD advanced program completers who were initially employed as teachers in their schools during 2016-17 or who were first year employees in new non-teaching positions such as principal, counselor, etc.  Parts B and C included questions about the responsiveness of the Huntington and South Charleston campuses to LEA requests/needs.  Each statement utilized a Strongly Disagree (SD), Disagree (D), Agree (A), Strongly Agree (SA) scale for responses.  Eleven directors responded to the survey.  Survey results are provided in PERSONNEL DIRECTORS..

All of the responding directors either Agreed or Strongly Agreed that COEPD advanced program completers demonstrated a high level of commitment to students and schools, exhibited a high degree of preparation, demonstrated a commitment to ethical standards of professional conduct, demonstrated a knowledge of technology, and exhibited dependability and responsibility.  Overall, 60% of the respondents Strongly Agreed and 40% Agreed that, based on their assessment of the preparation, performance, and professionalism of COEPD advanced program completers employed in their systems, they were generally satisfied and felt confident in employing other COEPD advanced program completers.

COEPD Responsiveness:  Huntington Campus (Part B)

All responding Personnel Directors either Agreed or Strongly Agreed that they are generally satisfied with the quality of records received related to COEPD advanced program completers seeking employment in their school systems, and the timeliness with which these records were received.,  Similarly, all respondents either Agreed or Strongly Agreed (90%) they were generally satisfied with the timeliness and quality of responses to their inquiries regarding completers and completer records related to perspective and current employees.

COEPD Responsiveness:  South Charleston Campus (Part C)

All responding Personnel Directors either Agreed or Strongly Agreed that they are generally satisfied with the quality of records received related to COEPD advanced program completers seeking employment in their school systems, and the timeliness with which these records were received.,  Similarly, all respondents either Agreed or Strongly Agreed (90%) they were generally satisfied with the timeliness and quality of responses to their inquiries regarding advanced program completers and completer records related to future and current employees.

Open ended questions at the end provided areas to consider when training candidates.

• Marshall University has always been great partners.  The students are generally well prepared and high-quality.  Marshall listens to our concerns and the University tries to work with us to provide appropriate programming.
• It appears many graduate do not have the “social” skills – appear awkward – have trouble communicating in interviews – cannot make eye contact.  I also feel that it should be a requirement that the supervising teacher comment under each area in the student teacher evaluation.  It appears we get evaluations that do not truly reflect the students’ ability.  When it gets to this point we need complete honesty and transparency on these evaluations.  We rely on them when hiring.

CAEP Standard A.4.2 requires that Educator Preparation Providers (EPPs) “demonstrates that advanced program completers perceive their preparation as relevant to the responsibilities they confront on the job and that the preparation was effective.”

Using guidelines provided by the QAS, programs were asked to develop completer surveys aligned with their standards. Programs made the completer survey parallel to the employer survey.  Utilizing the same candidate data base as the employer surveys, Advanced Level graduates from spring 2016 were sent separate emails with links to the Qualtrics completer surveys. GAs and administrative staff entered the surveys into Qualtrics. Completer data from the Principalship and School Psychology program for spring 2016 and 2017 are available in COMPLETER SATISFACTION ADVANCED LEVEL.

The Survey was sent to 15 2016 completers in the School Psychology Program, and results from this revised survey (N=11) indicated that 88.5 percent of the completers responded either “Proficient” or “Mastery” in terms of COEPD graduates’ preparation to meet NASP Standards.  The highest Program Categories of Data Based Decision Making/Accountability, Consultation and Collaboration, Research and Program Evaluation and Legal, Ethical and Professional Practice, fairness and ethics was rated at 98-100% as ‘Proficient” or “Mastery” and the lowest two were Interventions/Mental health Services to Develop Social/Life Skills and Preventive and Responsive Services at 73-75%.  Overall less than 3% of the completers rated their satisfactions as Unsatisfactory. The results of this survey indicated a positive satisfaction with the preparation of these completers.

Principalship: The 2017 survey was revised to reflect the ELCC standards and completer satisfaction with preparation questions. The Survey was sent to 80 identified One-Year Out Graduates of the Principalship Certificate, and results from this revised survey (N=22) indicated that 87.9 percent of the completers responded either “Proficient” or “Mastery” in terms of COEPD graduates’ preparation to meet ELCC Standards.  The highest Program Category of Integrity, fairness and ethics was rated at 95.5% as ‘Proficient” or “Mastery” and the lowest was Context at 77.5%.  Overall less than 8% of the completers rated their satisfactions as Unsatisfactory. The results of this survey indicated a positive satisfaction with the preparation of these completers.

The AACC has provided the completer data to the programs. The programs will review the data along with other program data to make program improvements.

Completer surveys from other Advanced Level programs were not reported due to low response rates. As stated above, the development of a better tracking system will enable COEPD to increase the response rate. Additionally, informing students at exit that they will be contacted at one year and three years should increase participation. A rotation system has been developed to survey all candidates within a three-year cycle starting spring 2016. The Quality of Evidence Work Group (QEWG) has plans to evaluate the validity and reliability of the completer surveys.

Summary/Conclusions

The strategies used are consistent with the overall COEPD framework for collecting employer satisfaction data and include data collection using multiple strategies and data points.  Employers rated candidates as either mastery or proficient in the assessed areas. Overall, Personnel Directors indicate COEPD advanced program completers who were first time employees in their systems in 2016-17 were committed to schools, well prepared, knowledgeable about technology, dependable and responsible, and demonstrate ethical standards and conduct.  Personnel Directors also report they feel confident in employing other COEPD completers in their school systems.  Directors also reported a general satisfaction with the quality and timeliness of the completer records and the responsiveness to inquiries from both the Huntington and South Charleston campuses. Findings from multiple applications of employer follow-up studies clearly indicate employers are more than satisfied with the preparation and performance of our initial program completers.  COEPD faculty have used employer and completer data to continually improve and inform the development of the QAS and data collection system, increase the interaction with P-12 stakeholders, and consider a number of changes in the educator preparation program.  Future plans include a continued focus on development/refinement of assessments, continuing to identify strategies to increase responses rates, and increasing the focus on gathering employer and completer feedback that can be more readily disaggregated by licensure area.

[back to CAEP Self-Study Report]