Assessing Q-Assessor: A Validation Study of Computer-Based Q Sorts versus Paper Sorts

Abstract

Q methodology studies traditionally have relied on face-Io-face administration to lead subjects correctly through the steps involved in the Q sort. The cost and time commitments of one-on-one supervision limit Q methodology's potential applicability to geographically scattered samples. Unsupervised paper-based techniques are less costly to administer, but can compromise study conclusions by introducing unmeasured methodological variability. Computer-based systems using the Internet can ensure accurate performance of the Q sort, administer studies to subjects anywhere, and collect results immediately and cost-effectively. Two validation studies are described for a prototype Internet-based system, Q-Assessor. In the pilot study, 6 subjects performed Q sorts via Q-Assessor. They also completed a traditionally administered paper-based sort. Q-Assessor compared favorably in the time required for participants to complete the study and for the investigator to process the results into a database as well as the subject satisfaction with and preference for the study methodology. This led to a larger-scale validation study of the computer-mediated method. Thirty subjects were given the option of completing a Q sort either online or on paper. More than half (17) selected the computer-based alternative as the more convenient method. Because 13 elected to do paper-based sorts, comparisons were allowed which showed 110 apparent difference in the reliability or validity between the methods.