Abstract

This paper presents several methods of automatic context enrichment of sentences that need to be evaluated, tagged or fact-checked by human judges. We have created a corpus of medical Web articles. Sentences from this corpus have been fact-checked by medical experts in two modes: contextually (reading the entire article and evaluating sentence by sentence) and without context (evaluating sentences from all articles in random order). It is known that non-contextual evaluation is faster, but some sentences are impossible to evaluate without context. We have designed and evaluated several methods of summarizing context that we hypothesized were suitable for supporting evaluation of sentences without reading the entire text. Then, we collected new assessments from medical experts for the sentences with enriched context. The context enrichment methods have been evaluated using two measures: conversion, which calculates how frequently a method allows experts to evaluate sentences that were impossible to evaluate without context, and agreement, which depends on how frequently the new expert evaluations match with evaluations from experts who had read the whole text before rating a sentence. Our results show that the best method achieves a high conversion rate, while providing experts with a condensed context summary. Moreover, the method significantly reduces the time needed to evaluate one sentence, compared to the baseline method (which provides the expert with the entire paragraph surrounding the target sentence). The problem of automatically enhancing the context of a sentence for fast fact-checking or tagging has not appeared in other studies before. We present preliminary results of the research in this area and a framework for testing potential new methods.

Citations

Authors (3)