{"id":18,"date":"2020-10-05T09:47:51","date_gmt":"2020-10-05T14:47:51","guid":{"rendered":"http:\/\/www.marshall.edu\/coepd-assessment\/?page_id=18"},"modified":"2026-02-23T16:37:43","modified_gmt":"2026-02-23T21:37:43","slug":"content-validity-procedure","status":"publish","type":"page","link":"https:\/\/www.marshall.edu\/coepd-assessment\/content-validity\/content-validity-procedure\/","title":{"rendered":"Content Validity & Reliability Procedure"},"content":{"rendered":"

CAEP Standard Alignment<\/strong>: R5\/RA5: Quality Assurance System and Continuous Improvement
\nCAEP Standard Component Alignment<\/strong>: R5.2\/RA5.2: Data Quality<\/p>\n

\n\t\n
\n
\n\n\t\t\t
\n\t\n\t\t\t\t
<\/div>\n\n\t\t
\n\t\t\t\n\t\t\t
\n\t\t\t\tCAEP Criteria for Evaluation of EPP-Created Assessments and Surveys (2022)\n\t\t\t<\/div>\n\n\t\t\t
<\/div>\n\t\t<\/div>\n\n\t\t\n\t\t
\n\t\t\t\t\t\t\t
Before you proceed with the content validity and reliability procedures, please view the CAEP Criteria for Evaluation of EPP-Created Assessments and Surveys.<\/div>\n\t\t\t\t\t<\/div>\n\n\t\t\t\t\t
\n\t\t\t\t\t\n\t\tView Criteria<\/span>\n\t\t<\/span><\/span>\n\t<\/a>\n\n\t\t\t<\/div>\n\t\t\n\t\t\n\t\t\t<\/div>\n\t\n\n<\/div>\n<\/div>\n\n
\n
\n\n\t\t\t
\n\t\n\t\t\t\t
<\/div>\n\n\t\t
\n\t\t\t\n\t\t\t
\n\t\t\t\tCAEP Criteria for Evaluation of EPP-Created Surveys (2022)\n\t\t\t<\/div>\n\n\t\t\t
<\/div>\n\t\t<\/div>\n\n\t\t\n\t\t
\n\t\t\t\t\t\t\t
Before you proceed with the content validity and reliability procedures, please view the CAEP Criteria for Evaluation of EPP-Created Surveys.<\/div>\n\t\t\t\t\t<\/div>\n\n\t\t\t\t\t
\n\t\t\t\t\t\n\t\tView Criteria<\/span>\n\t\t<\/span><\/span>\n\t<\/a>\n\n\t\t\t<\/div>\n\t\t\n\t\t\n\t\t\t<\/div>\n\t\n\n<\/div>\n<\/div>\n\n\n<\/div>\n\n

Introduction<\/h3>\n

The College of Education and Professional Development (COEPD) at Marshall University has established a content validity procedure for all Education Preparation Provider (EPP) created assessments and surveys, including key assessments, performance tasks, clinical evaluations, and national board-certified exams.\u00a0 The EPP adopted the procedure to evaluate its assessments in Spring 2022. The content validity and reliability procedures are used by both initial- and advanced-level programs.\u00a0 Procedures follow the guidelines outlined in the CAEP Evaluation Framework document for EPP-Created Assessments to design, pilot, and judge the adequacy of the assessments created by the EPP.<\/p>\n

The purpose of the content validity procedure is to provide guidance for collecting evidence and to document the adequate technical quality of assessment instruments and rubrics used to evaluate candidates in the COEPD.<\/p>\n

CAEP Defined Assessments<\/h3>\n

CAEP uses the term \u201cassessments\u201d to cover content tests, observations, projects or assignments, and surveys \u2013 all of which are used with candidates.\u00a0 Surveys are often used to gather evidence on candidate preparation and candidate perceptions about their readiness to teach.\u00a0 Surveys are also helpful to measure the satisfaction of graduates or employers with preparation and the perceptions of clinical faculty about the preparedness of EPP completers.<\/p>\n

Assessments and rubrics are used by faculty to evaluate candidates and provide them with feedback on their performance.\u00a0 Assessments and rubrics should address relevant and meaningful candidate knowledge, performance, and dispositions, aligned with CAEP standards.\u00a0 An EPP will use assessments that comprise evidence offered in accreditation self-study reports to examine candidates at various points from admission through completion consistently.\u00a0 These are assessments that all candidates are expected to complete as they pass from one stage of preparation to the next or that are used to monitor the progress of candidates\u2019 developing proficiencies during one or more stages of preparation.<\/p>\n

EPP-Defined Assessment<\/h3>\n

The definition of assessment adopted by the EPP includes three significant processes: data collection from a comprehensive and integrated set of assessments, analysis of data for forming judgments, and use of analysis in making decisions.\u00a0 Based on these three processes, assessment is operationally defined as a process in which data\/information is collected, summarized, and analyzed as a basis for forming judgments.\u00a0 Judgments then form the basis for making decisions regarding continuous improvement in our programs.<\/p>\n

EPP Five-Year Review Cycle<\/h3>\n

The EPP established a consistent process to review all EPP-created assessments\/rubrics on a five-year cycle when possible.<\/p>\n

Purpose<\/h3>\n

This policy establishes formal procedures for ensuring the content validity of all Educator Preparation Provider (EPP)-created assessments within the College of Education and Professional Development (COEPD) at Marshall University. Content validity ensures assessments accurately measure intended constructs, align to professional standards, reflect P\u201312 expectations, and support continuous improvement.<\/p>\n

Scope<\/h3>\n

These procedures apply to:<\/p>\n

    \n
  • Key assessments (initial- and advanced-level programs)<\/li>\n
  • Disposition instruments<\/li>\n
  • Clinical evaluation tools (when modified by the EPP)<\/li>\n
  • Survey instruments (completer, employer, stakeholder)<\/li>\n
  • Substantively revised rubrics<\/li>\n<\/ul>\n

    Governing Framework<\/h3>\n

    All assessments must demonstrate alignment to applicable professional standards, including CAEP, InTASC, WVPTS, or Specialized Professional Association (SPA) standards.<\/p>\n

    Conditions Requiring Content Validation<\/h3>\n
      \n
    • Development of a new key assessment<\/li>\n
    • Substantive rubric revision<\/li>\n
    • Updates to applicable standards<\/li>\n
    • Data trends suggesting construct misalignment<\/li>\n
    • Stakeholder feedback indicating measurement concerns<\/li>\n
    • Routine 3\u20135 year revalidation cycle<\/li>\n<\/ul>\n

      Content Validity Procedures<\/h3>\n

      Step One: Define the Construct<\/h4>\n

      Program faculty define the construct, identify transition point alignment, and document operational definitions.<\/p>\n

      Step Two: Standards Alignment Matrix<\/h4>\n

      Develop a detailed alignment matrix mapping each rubric element to applicable professional standards.<\/p>\n

      Step Three: Expert Panel Selection<\/h4>\n

      The program coordinator or responsible individual coordinates a validation panel including:<\/p>\n

        \n
      • Program faculty (2\u20133)<\/li>\n
      • Clinical partner representative<\/li>\n
      • External stakeholder or employer<\/li>\n
      • Current program candidate<\/li>\n
      • Program completer<\/li>\n
      • Content specialist (advanced programs, if applicable)<\/li>\n<\/ul>\n

        The program coordinator or designated responsible individual should actively engage and utilize all members of the advisory board, when applicable, to ensure broad representation, shared expertise, and comprehensive program input.<\/p>\n

        Step Four: Rating Process<\/h4>\n

        Panelists independently rate each rubric element using a 3-point relevance scale:<\/p>\n

          \n
        • 3 = Essential<\/li>\n
        • 2 = Useful but Not Essential<\/li>\n
        • 1 = Not Important<\/li>\n<\/ul>\n

          Panelists also provide qualitative feedback regarding clarity, alignment, and representativeness.<\/p>\n

          Step Five: Content Validity Index (CVI) Calculation<\/h4>\n

          The Content Validity Index (CVI) is calculated using the proportion of panelists rating an item as ‘Essential (3).’<\/p>\n

          Item-Level CVI (I-CVI) = Number of ratings of 3 \u00f7 Total number of raters<\/p>\n

          Scale-Level CVI (S-CVI) = Average of all I-CVI values across rubric elements<\/p>\n

          COEPD Benchmarks:<\/h5>\n
            \n
          • I-CVI \u2265 .78 required for retention<\/li>\n
          • S-CVI \u2265 .80 required for instrument approval<\/li>\n
          • Items below threshold must be revised and re-evaluated<\/li>\n<\/ul>\n

            Step Six: Revision and Recalibration<\/h4>\n

            Items not meeting benchmark thresholds are revised for clarity, alignment, and measurability. All revisions are documented in a formal revision log.<\/p>\n

            Step Seven: Governance Review<\/h4>\n

            Validation findings are presented to EPPAC or content specific advisory board for affirmation of professional relevance and workforce alignment.<\/p>\n

            Step Eight: Implementation and Rater Training<\/h4>\n

            Prior to implementation, calibration sessions are conducted. High-stakes assessments require interrater reliability evidence (\u226580% agreement or ICC \u2265.70).<\/p>\n

            Documentation and Archiving<\/h4>\n

            The program coordinator or designated responsible individual must clearly document each phase of the content validity process, including rater selection, materials provided, rating results, calculations, and any resulting revisions. Upon completion, all documentation must be submitted to the COEPD Director of Assessment for review and recordkeeping.<\/p>\n

            Continuous Monitoring<\/h4>\n

            Annual data reviews examine distribution patterns, ceiling effects, construct overlap, and stakeholder feedback. Identified concerns trigger revalidation procedures.<\/p>\n

            Roles and Responsibilities<\/p>\n

              \n
            • Facilitate expert panel processes<\/li>\n
            • Design rating instruments<\/li>\n
            • Calculate CVI metrics<\/li>\n
            • Ensure documentation integrity<\/li>\n
            • Prepare accreditation evidence<\/li>\n
            • Report validation outcomes in Annual Assessment Reports<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"

              CAEP Standard Alignment: R5\/RA5: Quality Assurance System and Continuous Improvement CAEP Standard Component Alignment: R5.2\/RA5.2: Data Quality Introduction The College of Education and Professional Development (COEPD) at Marshall University has established a content validity procedure for all Education Preparation Provider (EPP) created assessments and surveys, including key assessments, performance tasks, clinical evaluations, and national board-certified<\/p>\n","protected":false},"author":553,"featured_media":0,"parent":6,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"page-full-width.php","meta":{"_acf_changed":false,"footnotes":""},"class_list":["post-18","page","type-page","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/pages\/18","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/users\/553"}],"replies":[{"embeddable":true,"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/comments?post=18"}],"version-history":[{"count":76,"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/pages\/18\/revisions"}],"predecessor-version":[{"id":591,"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/pages\/18\/revisions\/591"}],"up":[{"embeddable":true,"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/pages\/6"}],"wp:attachment":[{"href":"https:\/\/www.marshall.edu\/coepd-assessment\/wp-json\/wp\/v2\/media?parent=18"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}