Fake News Detection Project Report
Fake News Detection Project Report
Abstract
The most common algorithms used by fake news detection
In our modern era where internet is ubiquitous, systems include machine learning algorithms such as Sup-
everyone relies on various online resources for
port Vector Machines, Random Forests, Decision trees, Sto-
news. Along with the increase in use of social me-
chastic Gradient Descent, Logistic Regression and so on. In
dia platforms like Facebook, Twitter etc. news this project we have attempted to implement two out of
spread rapidly among millions of users within a
these algorithms to train and test our results. We have used a
very short span of time. The spread of fake news
combination of both off the shelf datasets as well as ex-
has far reaching consequences like creation of bi- panded it by crawling content on the web. The main chal-
ased opinions to swaying election outcomes for the
lenge throughout the project has been to build a set of uni-
benefit of certain candidates. Moreover, spammers
form clean data and to tune parameters of our algorithms to
use appealing news headlines to generate revenue attain the maximum accuracy.
using advertisements via click-baits. In this project,
we aim to perform a binary classification of vari-
We observed that the Random Forests algorithm with a
ous news articles available online with the help of simple term frequency-inverse document frequency vector
concepts pertaining to Artificial Intelligence, Natu-
performed the best out of the four algorithms we tested. In
ral Language Processing and Machine Learning.
section 2 we describe the data collection, building of the
dataset and the text preprocessing techniques used. In sec-
1 Introduction tion 3, the three different models and their algorithms are
discussed. Section 4 discusses evaluation of results and the
With the growing popularity of mobile technology and so- scope for future enhancements.
cial media, information is accessible at one’s fingertips.
Mobile applications and social media platforms have over-
thrown traditional print media in the dissemination of news 2 Data Collection
and information. It is only natural that with the convenience
and speed that digital media offers, people express prefer-
The dataset for this project was built with a mix of both real
ence towards using it for their daily information needs. Not
and fake news. Most of the data was manually crawled and
only has it empowered consumers with faster access to di- extracted, whereas some were used off the shelf. The entire
verse data, it has also provided profit seeking parties with a
dataset amounted to 125,600 news articles out of which
strong platform to capture a wider audience.
15,600 were fake news and 1,10,000 were real news.
With the outburst of information, it is seemingly tedious for
To efficiently collect such huge data, we created a multi-
a layman to distinguish whether the news he consumes is
threaded web crawler. We ran the crawler using up to 100
real or fake. Fake news is typically published with an intent threads at a time and download the raw HTML body content
to mislead or create bias to acquire political or financial
of the crawled pages.
gains. Hence it may tend to have luring headlines or inter-
esting content to increase viewership. The sources of real news include Yahoo News, AOL, Reu-
ters, Bloomberg and The Guardian among many. Sources
In the recent elections of United States, there has been much
for fake news include TheOnion. UsaNewsFlash, Truth-Out,
debate regarding the authenticity of various news reports The Controversial Files and so on.
favoring certain candidates and the political motives behind
them. Amidst such growing concerns, the detection of fake
To extract important content from the crawled pages we
news gains utmost importance to prevent its negative im- used two strategies. First was to reduce noise by removing
pacts on individuals and society.
insignificant and irrelevant information like images, tables,
headers, footers, special symbols, navigation bars etc. The
second strategy was to extract HTML div tags from the re- 4. Feature Generation
maining content having the id property as ‘content’ or some
variations of ‘content’. With this we noticed we were able to For generation of features from the given data, we first per-
extract most of the important information across many formed tokenization on the raw text of articles. We then
webpages. Since each website has its own style of layout generated tf-idf feature vectors as described below.
and parameters, a one size fit all strategy would have failed,
and hence we leveraged a generic approach.
4.1 Term Frequency - Inverse Document Fre-
The collected data was processed using various text prepro- quency
cessing measures, as explained later and stored in CSV files.
The real and fake data were then merged and shuffled to get
a CSV file containing a consolidated randomized dataset. The tf-idf is a statistical measure that reflects the importance
of a particular word with respect to a document in a corpus.
From the consolidated randomized dataset we picked 20845 It is often used in information retrieval and text mining as
records at random which contained approximate 50% real one of the components for scoring documents and perform-
news and 50% fake news articles. From these records, 80% ing searches. It is a weighted measure of how often a word
was used for training the detection model and 20% was re- occurs in a document relative to how often it occurs across
served for testing the model. all documents in the corpus. Term frequency is the number
of times a term occurs in a document. Inverse document
2.1 Real News frequency is the inverse function of the number of docu-
ments in which it occurs.
The News Aggregator Dataset from the UCI Machine
Learning Repository was used to extract real news. This
dataset consists of links to the originally published news
articles in their websites. We extracted these URLS and
crawled them to download the news content using Beauti-
fulSoup.
2.2 Fake News Hence a term like “the” that is common across a collection
will have lesser tf-idf values, as its weight is diminished by
For fake news we used Kaggle’s ‘Getting Real about Fake the idf component. Hence the weight computed by tf-idf
News’ dataset. The CSV file with data was available off the represents the importance of a term inside a document.
shelf for use, and we had to perform minimal text pro-
cessing on this data. The tokenized data was used to generate a sparse matrix of
tf-idf features for representation. This represented our fea-
ture vector and was used in subsequent prediction algo-
2.3 Text Preprocessing rithms.
Since most of the data was crawled and extracted manually, 3. Prediction Algorithms
we had to first go through the data to understand organiza-
tion and formatting of text. The data was made uniform and We implemented two different algorithms from scratch for
comparable by converting it into a uniform UTF-8 encod- the prediction model which were: Logistic Regression mod-
ing. There were some cases where we encountered weird el and the Naïve Bayes classifier model. The algorithms and
symbols and letters incompatible with the character set the details of implementation have been explained in the
which had to be removed. We noticed that the data from sections below. In addition to these we also trained and test-
news articles were often organized into paragraphs. So, we ed our dataset on two other models: Random Forests model
performed trimming to get rid of extra spaces and empty and Support Vector Machine model. Given the short time
lines in text. frame of the project, the last two algorithms were prudently
implemented with the help of scikit-learn libraries.
3.1 Logistic Regression
Logistic Regression is a Machine Learning technique used 3.2 Naïve Bayes Classifier
to estimate relationships among variables using statistical
methods. This algorithm is great for binary classification This is a simple yet powerful classification model that
problems as it deals with predicting probabilities of classes, works remarkably well. It uses probabilities of the elements
and hence our decision to choose this algorithm as our base- belonging to each class to form a prediction. The underlying
line run. It relies on fitting the probability of true scenarios assumption in the Naïve Bayes model is that the probabili-
to the proportion of actual true scenarios observed. Also, ties of an attribute belonging to a class is independent of the
this algorithm does not require large sample sizes to start other attributes of that class. Hence the name ‘Naive’.
giving fairly good results.
4 Evaluation Metrics
We used the following three metrics for the evaluation of
our results. The use of more than one matrix helped us eval-
uate the performance of the models from different perspec-
tives.
1. False Positive: Predicted as fake news but are actual- In the implementation of Logistic Regression, we collected
ly true news. the loss in each iteration in an array. Then we plotted the
2. False Negative: Predicted as true news but are actu- curve of the loss in each iteration vs the number of itera-
ally fake news. tions. The graph is given below:
3. True Positive: Predicted as fake news and are actual-
ly fake news.
4. True Negative: Predicted as true news and are actu-
ally true news.
MODELS
METRICS Naïve Logistic SVM Random
Bayes Regression Forest
Accuracy 57% 98 % 97% 97%
Precision 0.59 0.98 0.98 0.98
Recall 0.70 0.98 0.97 0.96
The confusion matrix data for each model run on data set
size 20,000 split into 80% Training and 20% Test data are
as follows:
MODELS
METRICS Naïve Logistic SVM Random
Bayes Regression Forest
True Positive 1538 2155 2125 2114
True Negative 756 1750 1754 1756 Figure 10. Scatter plot for Logistic Regression model
False Positive 1048 54 45 48
False Negative 658 41 76 82
Total Test Data 4000 4000 4000 4000
Support Vector Machine:
Scatter Plots for various Models on a small set of data i.e
1000 are shown below. In each figure the first subplot
shows the prediction value plotted in red, whereas the sec-
ond subplot shows the actual values plotted in green.
Naïve Bayes:
References