RumourEval: bringing veracity checking to the community
While Pheme has been about advancing our own technology and understanding around veracity and fake news, there’s a much broader opportunity to make progress if these advances are shared. One way we can do this is by presenting problems from Pheme to the research community, using our data and the ways we found of framing the rumour detection problem.
SemEval is an event that runs roughly every year. In it, there are many tasks, and teams compete on each task to try and get the best performance. In 2017, we ran a task on Pheme problems, namely stance detection and veracity assessment. We called it RumourEval. Teams had about four months to develop their systems ahead of a final evaluation, where some new, previously-unseen data was given out for processing.
We could base the demo data on Pheme’s earlier work: the stance dataset from WP3. To make this a challenge, the data used had to be something new and hidden. At our Athens meeting, we chose two rumours that developed during 2016: the kidnapping of a London YouTuber, and the poor health of Hillary Clinton. Both were falsifiable, which was important – we needed to know the real truth of these in order to know how to assess systems.
In the end, we had over a dozen submissions, with teams from Asia, Europe and America entering. Some got great results, and we ended up with a new dataset and many new papers from a wide range of institutions addressing Pheme’s problems – some of which had interesting insights. The data is now openly available, and, as is typical for SemEval tasks, is a benchmark for future work in the area.
The findings will be reported at the ACL conference in Vancouver, this August. For further details, you can read the full RumourEval task report paper.
Work on automatic detection of online claims far from over Do they think it’s true? Automatic Rumour Stance Classification.
Comments are currently closed.