Estimating the magnitude of method bias on account of text similarity using a natural language processing-based technique

A number of potential biases have been identified that may contribute a spurious component to the observed correlation between variables. One such potential bias is the manner in which items are worded. This paper presents a technique and a program of research to estimate the magnitude of method bias arising from item wording. We hypothesize that the greater the level of text similarity between items employed to capture predictor and criterion variables, the greater the magnitude of the observed effect size between them. Two samples will be employed; one investigating the perceived usefulness-use correlations reported in the TAM literature and the other investigating the attitude behavior (physical activity) correlation reported literature drawing upon Theory of Planned Behavior (TPB). An NLP-based technique is developed to rate predictor-criterion pairs on similarity. The hypothesis will be tested by meta-regressing the predictor-criterion correlations against their respective similarity scores. Implications for research and practice are discussed.