An overview of methods for treating selectivity in big data sources
Maciej Beręsewicz , Risto Lehtonen , Fernando Reis , Loredana Di Consiglio , Martin Karlberg
AbstractOfficial statistics is now considering seriously big data as a significant data source for producing statistics. It holds the potential for providing faster, cheaper, more detailed and completely new types of statistics. However, the use of big data brings also several challenges. One of them is the nonprobabilistic character of most sources of big data, as very often they were not designed to produce statistics. The resulting selectivity bias is therefore a major concern when using big data. This paper presents a statistical approach to big data, searching for a definition meaningful from the statistical point of view and identifying its main statistical characteristics. It then argues that big data sources share many characteristics with Internet opt-in panel surveys and proposes this as a reference to address selectivity and coverage problems in big data. Coverage and the self-selection process are briefly discussed in mobile network data, Twitter, Google Trends and Wikipedia page views data. An overview of methods which can be used to address selectivity and eliminate, or mitigate, bias is then presented, covering both methods applied at individual level, i.e. at the level of the statistical unit, and at domain level, i.e. at the level of the produced statistics. Finally, the applicability of the methods to the several big data sources is briefly discussed and a framework for adjusting selectivity in big data is proposed.
|Publisher name (outside publisher list)||Publications Office of the European Union|
|Publishing place (Publisher address)||Luxembourg|
|Book series /Journal (in case of Journal special issue)||Statistical Working Papers - EUROSTAT, ISSN 2315-0807 , (0 pkt)|
|Publication size in sheets||5.7|
|Keywords in English||big data, selectivity|
|Score||= 20.0, 21-04-2020, MonograhOrBookMainLanguagesAuthor|
|Citation count*||13 (2020-09-23)|
* presented citation count is obtained through Internet information analysis and it is close to the number calculated by the Publish or Perish system.