Date of Award

Fall 12-2022

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

Educational Foundations & Leadership

Program/Concentration

Educational Psychology and Program Evaluation

Committee Director

Joanna Garner

Committee Member

Shanan Chappell Moots

Committee Member

Linda Bol

Abstract

Validity in psychometrics refers to the degree to which evidence and theory supports the interpretations drawn from a test, and Messick’s Contemporary Validity Theory (1994) includes several facets with well-established evidence collection methods. However, there is a lack of consensus on appropriate methods of evaluating the facet of consequential validity, which is the degree to which interpretation of scores could have consequences for test-takers. The primary objective of this study was to illustrate a method of identifying potential consequences of survey in the stage of manual development. This method was placed in the context of the Ambassador Questionnaire (AQ) used in Engineering Ambassador (EA) programs that aim to broaden participation in the discipline with outreach. Because this program focuses on equity and inclusion, a second objective was to determine if the AQ could include a subscale sensitive to equity perceptions. A mixed method was employed to address these objectives. Qualitative interviews informed both the capacity for interviews to solicit potential consequences of use and the utility of an equity perception subscale to EA chapter advisors. Quantitative analyses included factor analytic techniques. Existing AQ data was proportionately stratified for separate exploratory and confirmatory factor analyses to determine if controlling for demographic differences would reinforce the AQ’s existing factor structure that omits pilot items sensitive to perceptions of equity differences.

Ten advisors were interviewed and a total of eleven hours of interview data was collected. Several themes related to consequences of use emerged from these interviews, including use of the AQ to track student growth, potential influences on students as a result of taking the AQ, programmatic changes that could be made with AQ data, and sharing of AQ data with interested parties. These findings suggest that interviewing stakeholders during the development of an instrument’s manual can provide developers with foreseeable consequences. Integration of advisor feedback on items sensitive to perceptions of equity differences with quantitative findings that reinforce the AQ’s existing factor structure also suggests that a scale sensitive to said perceptions would be best served by an additional subscale with added items or an independent equity perceptions scale.

Rights

In Copyright. URI: http://rightsstatements.org/vocab/InC/1.0/ This Item is protected by copyright and/or related rights. You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s).

Copyright, 2022, by Melissa G. Kuhn, All Rights Reserved.

DOI

10.25777/rafn-mf64

ISBN

9798371976734

ORCID

0000-0002-3197-7442

Download

Share

COinS