< Back to previous page


The Influence of Methodological Choices on Data Quality and Quantity in Experience Sampling Studies

Book - Dissertation

Experience sampling is becoming an increasingly popular tool to investigate individuals' daily life. In studies using Experience Sampling Methodology (ESM), participants are typically asked to fill in several questionnaires per day, over several days, while following their normal daily routines. Despite the increased popularity of ESM, methodological developments in this field have been lagging behind. Setting up ESM studies involves many choices and currently we do not have a thorough understanding of how these choices influence the data that is being collected. This lack of methodological research makes it difficult to evaluate and compare results from different ESM studies. The current doctoral project investigated the effects of several central design choices on the characteristics of data collected via ESM. Part of the work relies on an experimental study, in which participants were randomly assigned to receive either 30 or 60 items, 3, 6, or 9 times per day over 14 days. Based on these data, we investigated the effects of the sampling frequency (i.e., how many assessments per day) and the questionnaire length on measures of data quality and quantity and other characteristics of the data. We observed that a longer ESM questionnaire is associated with more perceived burden, lower compliance, and more reported careless responding than a shorter version. In contrast, no such changes were observed for different sampling frequencies. Moreover, the questionnaire length, but not the sampling frequency, was associated with changes in response behavior over days in the study. We also detected several reactive effects that may affect ESM data. Specifically, we observed systematic changes in completion times, the variance of ESM measures, and the mean levels of ESM measures of emotional awareness and affect over time. In addition, participants described various reactive effects in a follow-up interview. Subsequently, we review the literature on questionnaire development for ESM studies and formulate recommendations for applied researchers. We then focus on the measurement of affect in ESM studies. While we did not find clear signs of the sampling protocol influencing the structure of affect, our results suggest that a conceptualization in terms of positive and negative affect may brush over important nuances in the affective structure. Finally, we turn to a largely ignored design choice in ESM studies—the response delay. In a large, pooled dataset of nine ESM studies, we observed that reported affect changes as a function of the response delay. Although we cannot identify the underlying mechanism with certainty, we highlight several possibilities and discuss how future research could help us understand the response delays better. Taken together, the research compiled in this thesis contributes to our understanding of the effects of design choices on the data collected in ESM studies. We offer some tentative recommendations for ESM study design and discuss related considerations in detail at the end of this thesis. Further, we discuss some remaining open questions that need to be answered to bring ESM to its full potential.
Publication year:2021