REFORM: Reputation Based Fair and Temporal Reward Framework for Crowdsourcing

Crowdsourcing is an effective method to collect data by employing distributed human population. Researchers introduce Peer-Based Mechanisms (PBMs) in crowdsourcing settings to incentivize agents to report accurately. We observe that with PBMs, crowdsourcing systems may not be fair. Unfair rewards for the agents may discourage participation. This work aims to build a general framework that assures fairness for PBMs in a temporal setting, i.e., where reports are time-sensitive. Towards this, we introduce two notions of fairness for PBMs, namely 𝛾-fairness and qualitative fairness. To satisfy these notions, our framework provides trustworthy agents with additional chances of pairing. We introduce Temporal Reputation Model (TERM) to quantify agents' trustworthiness across tasks. Having TERM, we present our iterative framework, REFORM, that can adopt the reward scheme of any existing PBM. We demonstrate REFORM's significance by deploying the framework with RPTSC's reward scheme and prove that REFORM with RPTSC considerably improves fairness; while incentivizing truthful and early reports.