An Empirical Study on Utilizing Neural Network for Event Information Retrieval

Event information retrieval (EIR) is the task of retrieving news articles in response to the event-oriented query, which tailors for dealing with a large scale of news articles on breaking news (e.g., natural disaster). However, the challenges caused by the dynamic and complex natures of the event make EIR less studied than traditional information retrieval (IR). The existing studies usually employed either traditional IR models or machine learning approaches over hand-crafted features to retrieve events, which fails to capture more-complex structures of events with event evolution well. To address this issue, we exploit neural models incorporating the characteristics of events to EIR task. Experimental results demonstrate the effectiveness of our method over two datasets, which can regard as the simple but strong baseline for further research in EIR.


Introduction
With the explosion of massive news articles generated from various news agencies, it is difficult for people to deal with such a great many news articles. EIR [1] is the task of retrieving news articles in response to an event query, which tailors for dealing with this information overload. Accurately retrieving news articles about a specific event will benefit many downstream applications, such as event monitoring, news recommendation, event tracking, etc.
However, the existing challenges make EIR task less studies than traditional IR. To see its challenge, we analyse the collection of news articles and conclude that the challenge is primarily caused by the dynamic and complex nature of events. Take the Malaysia Airlines Flight 370 disaster ("MH370") as an example (see figure 1). Given the query "MH370 debris", whose goal is to find news articles related to the query. By our analysis of the retrieval results, two of the most important ones are pointed out:  event dynamics: the event consists of several subevents (e.g., "MH370 disappearance", "Find MH370 debris") with event evolution.
 structure complexity: a subevent usually associates to multiple new articles and a news article may correspond to multiple subevents in different periods.
A few of existing studies [2][3][4] [3] presented the novel event mining and feature generation methods for ranking news events. These solutions employ either traditional information retrieval models or machine learning approaches over hand-crafted features to retrieval events, which fails to capture more-complex structures of events with event evolution well.
Fortunately, neural models such as neural ranking models (NRMs) [5, 6] and pre-trained neural language models (e.g., BERT [7,8]) have achieved promising results in ad-hoc retrieval task. However, there is not yet an end-to-end approach for EIR task. Motivated by this, we investigate whether neural models can be exploited to improve the performance of EIR. To our best knowledge, our work is the first contribution to employ neural models to EIR by incorporating the characteristics of events. Experimental results demonstrate the better performance using our method over two realworld datasets, which can regard as the simple but strong baseline for further research.

Neural Models
Neural ranking model is the heart of the retrieval system, which determines the quality of the retrieval system definitively. With the advance of deep learning, several studies have leveraged shallow or deep neural networks to the ranking problem effectively in IR, such as [8,9]. Depending on the different application scenarios, neural models can be categorized into two groups, including models (i.e., DSSM [10], ArcI/II [11], MatchPyramid [12]) for text matching and models (i.e., DRMM [13], DRMMTKS [14], DUET [15], KNRM [16], Conv-KNRM [5], BERT [8]) for ad-hoc retrieval. However, these models above-mentioned do not consider IR specificities, such as query term importance, exact matching signals for text matching. As such, we only employ models developed for ad-hoc retrieval to EIR task.

EIR by Neural Models
The input text is usually short and does not involve the complex structure in traditional tasks, such as ad-hoc retrieval, which can be input to neural models directly. By contrast, news articles are usually long texts and involve more-complex structures. If we truncate a fix-length text from the beginning of the document as input, it will fail to capture the sufficient semantics of events. An alternative is employing query expansion and news summarization to deal with event dynamics and structure complexity, respectively.
Previous studies have shown that external resources are effective for query expansion. Therefore, we use events collected from Wikipedia worldwide current events as external resources to expand the original query. Concretely, this process involves three key steps: (1) crawl the relevant Wikipedia pages for each event with current events portal, (2) obtain candidate terms of each event from these Wikipedia pages with RAKE [17], and (3) select the top three candidates to expand the original query. In addition, we assume that an event can be viewed as an abstract representation of long news articles. So, we use the abstractive news summarization approach [18] to extract the event. Finally, we feed the extensions of query and the refined news article into neural models for EIR.

Experiments and Results
We perform the following experiments aim to explore (1) whether neural models are more suitable for EIR task than traditional approaches, and (2) whether our approach using query expansion and news summarization to neural models achieves a better overall ranking quality than without using our approach.

Setup
We use two publicly available datasets for evaluation in EIR task, including JustEvents [19] and Event-Dataset [20]. The JustEvents dataset consists of 250 events spanned from 18th January 2011 to 7th February 2011, which is collected from more than 1.8 million Wikipedia pages. The Event-Dataset contains 35 past and future events occurred during the years of 1997 to 2022. These events are well reported all over the world, such as "MH370 Disappearance", "Kashmir Earthquake". The detailed information of datasets is displayed on table 1. We evaluate the effectiveness of the approach based on two types of approaches. One is traditional models, including the Okapi BM25 [21] and BM25P. Another is neural models that we have mentioned in Section 2.1, including DRMM, DRMMTKS, DUET, KNRM, Conv-KNRM and BERT.
Considering small datasets, we conduct 10-fold cross validation to minimize the problem of overfitting. We use MatchZoo toolkit to train these NRMs. We use the cross entropy [22] loss function to rank. In order to evaluate the performance of the proposed approach for EIR in this paper, we use three standard metrics for comparison: MAP [23], nDCG@5 [24], and nDCG@10. In addition, statistical differences between models are computed using the Fisher randomization test [25] ( 0.05

Results Analysis
A summary of performance results of traditional models (i.e., BM25, BM25P) and neural models over two datasets is displayed in table 2. As we can see from the JustEvents dataset, all the neural models perform significantly better than traditional models. For the Event-Dataset dataset, we find that all neural models cannot compete with traditional models. A possible reason is that the Event-Dataset dataset is too small (only 35 queries) to take effect on neural models for improving performance. Considering this issue, we only use the JustEvents dataset to evaluate our approach. The experiment result is reported in table 3. Our approach employing query expansion and news summarization achieves significant performances in all neural models. According to this experiment, we investigate

Conclusions
In this paper, we employ neural models incorporating the dynamics and complex nature of events to EIR task. First, we analyse the challenge of EIR and conclude that the challenge is primarily caused by the dynamic and complex nature of events. Then we investigate whether neural models can be exploited to EIR by incorporating these characteristics of events. Finally, the experimental results with our approach outperform other the state-of-the-art approaches over real-world datasets in terms of effectiveness, which can regard as the simple but strong baseline for further research in EIR. As future work we plan to design a neural model tailored for EIR, which can avoid the pipeline way of processing query and news articles.