Date of Award
Doctor of Philosophy (PhD)
Educational Foundations & Leadership
Educational Psychology and Program Evaluation
Shanan Chappell Moots
Validity in psychometrics refers to the degree to which evidence and theory supports the interpretations drawn from a test, and Messick’s Contemporary Validity Theory (1994) includes several facets with well-established evidence collection methods. However, there is a lack of consensus on appropriate methods of evaluating the facet of consequential validity, which is the degree to which interpretation of scores could have consequences for test-takers. The primary objective of this study was to illustrate a method of identifying potential consequences of survey in the stage of manual development. This method was placed in the context of the Ambassador Questionnaire (AQ) used in Engineering Ambassador (EA) programs that aim to broaden participation in the discipline with outreach. Because this program focuses on equity and inclusion, a second objective was to determine if the AQ could include a subscale sensitive to equity perceptions. A mixed method was employed to address these objectives. Qualitative interviews informed both the capacity for interviews to solicit potential consequences of use and the utility of an equity perception subscale to EA chapter advisors. Quantitative analyses included factor analytic techniques. Existing AQ data was proportionately stratified for separate exploratory and confirmatory factor analyses to determine if controlling for demographic differences would reinforce the AQ’s existing factor structure that omits pilot items sensitive to perceptions of equity differences.
Ten advisors were interviewed and a total of eleven hours of interview data was collected. Several themes related to consequences of use emerged from these interviews, including use of the AQ to track student growth, potential influences on students as a result of taking the AQ, programmatic changes that could be made with AQ data, and sharing of AQ data with interested parties. These findings suggest that interviewing stakeholders during the development of an instrument’s manual can provide developers with foreseeable consequences. Integration of advisor feedback on items sensitive to perceptions of equity differences with quantitative findings that reinforce the AQ’s existing factor structure also suggests that a scale sensitive to said perceptions would be best served by an additional subscale with added items or an independent equity perceptions scale.
In Copyright. URI: http://rightsstatements.org/vocab/InC/1.0/ This Item is protected by copyright and/or related rights. You are free to use this Item in any way that is permitted by the copyright and related rights legislation that applies to your use. For other uses you need to obtain permission from the rights-holder(s).
Copyright, 2022, by Melissa G. Kuhn, All Rights Reserved.
Kuhn, Melissa G..
"Investigating the Process of Consequential Validity with the Ambassador Questionnaire"
(2022). Doctor of Philosophy (PhD), Dissertation, Educational Foundations & Leadership, Old Dominion University, DOI: 10.25777/rafn-mf64