Skip to Content
[CAIDA - Center for Applied Internet Data Analysis logo]
Center for Applied Internet Data Analysis
www.caida.org > publications : papers : 2020 : improving_efficiency_qoe_crowdtesting
Improving the Efficiency of QoE Crowdtesting
R. Mok, G. Kawaguti, and J. Okamoto, "Improving the Efficiency of QoE Crowdtesting", in ACM Quality of Experience in Visual Multimedia Applications (QOEVMA), Oct 2020.
|   View full paper:    PDF    |  Citation:    BibTeX   |

Improving the Efficiency of QoE Crowdtesting

Ricky K. P. Mok1
Ginga Kawaguti2
Jun Okamoto2
1

CAIDA, San Diego Supercomputer Center, University of California San Diego

2

NTT, Japan

Crowdsourced testing is an increasingly popular way to study the quality of experience (QoE) of applications, such as video streaming and web. The diverse nature of the crowd provides a more realistic assessment environment than laboratory-based assessments allow. Because of the short life-span of crowdsourcing tasks, each subject spends a significant fraction of the experiment time just learning how it works. We propose a novel experiment design to conduct a longitudinal crowdsourcing study aimed at improving the efficiency of crowdsourced QoE assessments. On Amazon Mechanical Turk, we found that our design was 20% more cost-effective than crowdsourcing multiple one-off short experiments. Our results showed that subjects had a high level of revisit intent and continuously participated in our experiments. We replicated the video streaming QoE assessments in a traditional laboratory setting. Our study showed similar trends in the relationship between video bitrate and QoE, which confirm findings in prior research.

Keywords: QoE, routing, topology
  Last Modified: Wed Oct-14-2020 17:14:09 UTC
  Page URL: https://www.caida.org/publications/papers/2020/improving_efficiency_qoe_crowdtesting/index.xml