TextMining PDF
TextMining PDF
Text Mining
[email protected]
10th January 2016
Visit http://HandsOnDataScience.com/ for more Chapters.
Text Mining (or Text Analytics) applies analytic tools to learn from collections of text data,
like social media, books, newspapers, emails, etc. The goal can be considered to be similar to
humans learning by reading such material. However, using automated algorithms we can learn
from massive amounts of text, very much more than a human can. The material could consist of
millions of newspaper articles to perhaps summarise the main themes and to identify those that
are of most interest to particular people. Or we might be monitoring twitter feeds to identify
emerging topics that we might need to act upon, as it emerges.
The required packages for this chapter include:
library(tm)
#
library(qdap)
#
library(qdapDictionaries)
library(dplyr)
#
library(RColorBrewer)
#
library(ggplot2)
#
library(scales)
#
library(Rgraphviz)
#
As we work through this chapter, new R commands will be introduced. Be sure to review the
commands documentation and understand what the command does. You can ask for help using
the ? command as in:
?read.csv
We can obtain documentation on a particular package using the help= option of library():
library(help=rattle)
This chapter is intended to be hands on. To learn effectively, you are encouraged to have R
running (e.g., RStudio) and to run all the commands as they appear here. Check that you get
the same output, and you understand the output. Try some variations. Explore.
Copyright 2013-2015 Graham Williams. You can freely copy, distribute,
or adapt this material, as long as the attribution is retained and derivative
work is provided under the same license.
Draft Only
Data Science with R
Hands-On
Text Mining
The primary package for text mining, tm (Feinerer and Hornik, 2015), provides a framework
within which we perform our text mining. A collection of other standard R packages add value
to the data processing and visualizations for text mining.
The basic concept is that of a corpus. This is a collection of texts, usually stored electronically,
and from which we perform our analysis. A corpus might be a collection of news articles from
Reuters or the published works of Shakespeare. Within each corpus we will have separate documents, which might be articles, stories, or book volumes. Each document is treated as a separate
entity or record.
Documents which we wish to analyse come in many different formats. Quite a few formats are
supported by tm (Feinerer and Hornik, 2015), the package we will illustrate text mining with in
this module. The supported formats include text, PDF, Microsoft Word, and XML.
A number of open source tools are also available to convert most document formats to text files.
For our corpus used initially in this module, a collection of PDF documents were converted to text
using pdftotext from the xpdf application which is available for GNU/Linux and MS/Windows
and others. On GNU/Linux we can convert a folder of PDF documents to text with:
system("for f in *.pdf; do pdftotext -enc ASCII7 -nopgbrk $f; done")
The -enc ASCII7 ensures the text is converted to ASCII since otherwise we may end up with
binary characters in our text documents.
We can also convert Word documents to text using anitword, which is another application
available for GNU/Linux.
system("for f in *.doc; do antiword $f; done")
Module: TextMiningO
Page: 1 of 46
Draft Only
Data Science with R
1.1
Hands-On
Text Mining
There are a variety of sources supported by tm. We can use getSources() to list them.
getSources()
## [1] "DataframeSource" "DirSource"
## [5] "XMLSource"
"ZipSource"
"URISource"
"VectorSource"
In addition to different kinds of sources of documents, our documents for text analysis will come
in many different formats. A variety are supported by tm:
getReaders()
##
##
##
##
##
[1]
[3]
[5]
[7]
[9]
"readDOC"
"readPlain"
"readRCV1asPlain"
"readReut21578XMLasPlain"
"readTagged"
"readPDF"
"readRCV1"
"readReut21578XML"
"readTabular"
"readXML"
Module: TextMiningO
Page: 2 of 46
Draft Only
Data Science with R
1.2
Hands-On
Text Mining
Text Documents
We load a sample corpus of text documents. Our corpus consists of a collection of research
papers all stored in the folder we identify below. To work along with us in this module, you
can create your own folder called corpus/txt and place into that folder a collection of text
documents. It does not need to be as many as we use here but a reasonable number makes it
more interesting.
cname <- file.path(".", "corpus", "txt")
cname
## [1] "./corpus/txt"
We can list some of the file names.
length(dir(cname))
## [1] 46
dir(cname)
## [1]
## [2]
## [3]
## [4]
## [5]
## [6]
....
"acnn96.txt"
"adm02.txt"
"ai02.txt"
"ai03.txt"
"ai97.txt"
"atobmars.txt"
Length Class
Mode
2
PlainTextDocument list
2
PlainTextDocument list
Module: TextMiningO
Page: 3 of 46
Draft Only
Data Science with R
## ai02.txt
## ai03.txt
## ai97.txt
....
Hands-On
2
2
2
Text Mining
PlainTextDocument list
PlainTextDocument list
PlainTextDocument list
Module: TextMiningO
Page: 4 of 46
Draft Only
Data Science with R
1.3
Hands-On
Text Mining
PDF Documents
If instead of text documents we have a corpus of PDF documents then we can use the readPDF()
reader function to convert PDF into text and have that loaded as out Corpus.
docs <- Corpus(DirSource(cname), readerControl=list(reader=readPDF))
This will use, by default, the pdftotext command from xpdf to convert the PDF into text
format. The xpdf application needs to be installed for readPDF() to work.
Module: TextMiningO
Page: 5 of 46
Draft Only
Data Science with R
1.4
Hands-On
Text Mining
Word Documents
A simple open source tool to convert Microsoft Word documents into text is antiword. The
separate antiword application needs to be installed, but once it is available it is used by tm to
convert Word documents into text for loading into R.
To load a corpus of Word documents we use the readDOC() reader function:
docs <- Corpus(DirSource(cname), readerControl=list(reader=readDOC))
Once we have loaded our corpus the remainder of the processing of the corpus within R is then
as follows.
The antiword program takes some useful command line arguments. We can pass these through
to the program from readDOC() by specifying them as the character string argument:
docs <- Corpus(DirSource(cname), readerControl=list(reader=readDOC("-r -s")))
Here, -r requests that removed text be included in the output, and -s requests that text hidden
by Word be included.
Module: TextMiningO
Page: 6 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
We can (and should) inspect the documents using inspect(). This will assure us that data has
been loaded properly and as we expect.
inspect(docs[16])
##
##
##
##
##
##
##
##
<<VCorpus>>
Metadata: corpus specific: 0, document level (indexed): 0
Content: documents: 1
[[1]]
<<PlainTextDocument>>
Metadata: 7
Content: chars: 44776
Module: TextMiningO
Page: 7 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
We generally need to perform some pre-processing of the text data to prepare for the text analysis. Example transformations include converting the text to lower case, removing numbers and
punctuation, removing stop words, stemming and identifying synonyms. The basic transforms
are all available within tm.
getTransformations()
## [1] "removeNumbers"
## [4] "stemDocument"
"removePunctuation" "removeWords"
"stripWhitespace"
The function tm map() is used to apply one of these transformations across all documents within
a corpus. Other transformations can be implemented using R functions and wrapped within
content transformer() to create a function that can be passed through to tm map(). We will
see an example of that in the next section.
In the following sections we will apply each of the transformations, one-by-one, to remove unwanted characters from the text.
Module: TextMiningO
Page: 8 of 46
Draft Only
Data Science with R
3.1
Hands-On
Text Mining
Simple Transforms
We start with some manual special transforms we may want to do. For example, we might want
to replace /, used sometimes to separate alternative words, with a space. This will avoid the
two words being run into one string of characters through the transformations. We might also
replace @ and | with a space, for the same reason.
To create a custom transformation we make use of content transformer() to create a function
to achieve the transformation, and then apply it to the corpus using tm map().
toSpace
docs <docs <docs <-
<<VCorpus>>
Metadata: corpus specific: 0, document level (indexed): 0
Content: documents: 1
[[1]]
<<PlainTextDocument>>
Metadata: 7
Content: chars: 44776
Module: TextMiningO
Page: 9 of 46
Draft Only
Data Science with R
3.2
Hands-On
Text Mining
<<VCorpus>>
Metadata: corpus specific: 0, document level (indexed): 0
Content: documents: 1
[[1]]
<<PlainTextDocument>>
Metadata: 7
Content: chars: 44776
General character processing functions in R can be used to transform our corpus. A common
requirement is to map the documents to lower case, using tolower(). As above, we need to
wrap such functions with a content transformer():
Module: TextMiningO
Page: 10 of 46
Draft Only
Data Science with R
3.3
Hands-On
Text Mining
Remove Numbers
Module: TextMiningO
Page: 11 of 46
Draft Only
Data Science with R
3.4
Hands-On
Text Mining
Remove Punctuation
Module: TextMiningO
Page: 12 of 46
Draft Only
Data Science with R
3.5
Hands-On
Text Mining
<<VCorpus>>
Metadata: corpus specific: 0, document level (indexed): 0
Content: documents: 1
[[1]]
<<PlainTextDocument>>
Metadata: 7
Content: chars: 32234
Stop words are common words found in a language. Words like for, very, and, of, are, etc, are
common stop words. Notice they have been removed from the above text.
We can list the stop words:
length(stopwords("english"))
## [1] 174
stopwords("english")
##
[1]
##
[6]
## [11]
## [16]
## [21]
## [26]
....
"i"
"our"
"yours"
"his"
"herself"
"them"
"me"
"ours"
"yourself"
"himself"
"it"
"their"
"my"
"ourselves"
"yourselves"
"she"
"its"
"theirs"
"myself"
"you"
"he"
"her"
"itself"
"themselves"
"we"
"your"
"him"
"hers"
"they"
"what"
Module: TextMiningO
Page: 13 of 46
Draft Only
Data Science with R
3.6
Hands-On
Text Mining
Module: TextMiningO
Page: 14 of 46
Draft Only
Data Science with R
3.7
Hands-On
Text Mining
Strip Whitespace
Module: TextMiningO
Page: 15 of 46
Draft Only
Data Science with R
3.8
Hands-On
Text Mining
Specific Transformations
We might also have some specific transformations we would like to perform. The examples here
may or may not be useful, depending on how we want to analyse the documents. This is really
for illustration using the part of the document we are looking at here, rather than suggesting
this specific transform adds value.
toString <- content_transformer(function(x, from, to) gsub(from, to, x))
docs <- tm_map(docs, toString, "harbin institute technology", "HIT")
docs <- tm_map(docs, toString, "shenzhen institutes advanced technology", "SIAT")
docs <- tm_map(docs, toString, "chinese academy sciences", "CAS")
inspect(docs[16])
##
##
##
##
##
##
##
##
<<VCorpus>>
Metadata: corpus specific: 0, document level (indexed): 0
Content: documents: 1
[[1]]
<<PlainTextDocument>>
Metadata: 7
Content: chars: 30117
Module: TextMiningO
Page: 16 of 46
Draft Only
Data Science with R
3.9
Hands-On
Text Mining
Stemming
Module: TextMiningO
Page: 17 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
A document term matrix is simply a matrix with documents as the rows and terms as the columns
and a count of the frequency of words as the cells of the matrix. We use DocumentTermMatrix()
to create the matrix:
dtm <- DocumentTermMatrix(docs)
dtm
##
##
##
##
##
<<DocumentTermMatrix
Non-/sparse entries:
Sparsity
:
Maximal term length:
Weighting
:
We can inspect the document term matrix using inspect(). Here, to avoid too much output,
we select a subset of inspect.
inspect(dtm[1:5, 1000:1005])
## <<DocumentTermMatrix
## Non-/sparse entries:
## Sparsity
:
## Maximal term length:
## Weighting
:
##
....
The document term matrix is in fact quite sparse (that is, mostly empty) and so it is actually
stored in a much more compact representation internally. We can still get the row and column
counts.
class(dtm)
## [1] "DocumentTermMatrix"
"simple_triplet_matrix"
dim(dtm)
## [1]
46 6508
<<TermDocumentMatrix
Non-/sparse entries:
Sparsity
:
Maximal term length:
Weighting
:
We will use the document term matrix for the remainder of the chapter.
Module: TextMiningO
Page: 18 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
We can obtain the term frequencies as a vector by converting the document term matrix into a
matrix and summing the column counts:
freq <- colSums(as.matrix(dtm))
length(freq)
## [1] 6508
By ordering the frequencies we can list the most frequent terms and the least frequent terms:
ord <- order(freq)
# Least frequent terms.
freq[head(ord)]
## aaaaaaeaceeaeeieaeaeeiiaiaciaiicaiaeaeaoeneiacaeaaeooooo
##
1
##
aab
##
1
##
aadrbltn
##
1
##
aadrhtmliv
##
1
##
aai
##
1
....
Notice these terms appear just once and are probably not really terms that are of interest to us.
Indeed they are likely to be spurious terms introduced through the translation of the original
document from PDF to text.
# Most frequent terms.
freq[tail(ord)]
##
##
use
1366
mine
1446
data
3101
These terms are much more likely to be of interest to us. Not surprising, given the choice
of documents in the corpus, the most frequent terms are: data, mine, use, pattern, dataset,
can.
Module: TextMiningO
Page: 19 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
# Frequency of frequencies.
head(table(freq), 15)
## freq
##
1
2
## 2381 1030
3
503
4
311
5
210
6
188
7
134
8
130
9
82
10
83
11
65
578
1
609
1
611
1
616
1
703
1
709
1
776
1
12
61
13
54
14
52
15
51
tail(table(freq), 15)
## freq
## 483
##
1
544
1
547
1
555
1
So we can see here that there are 2381 terms that occur just once.
Module: TextMiningO
Page: 20 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
We can convert the document term matrix to a simple matrix for writing to a CSV file, for
example, for loading the data into other software if we need to do so. To write to CSV we first
convert the data structure into a simple matrix:
m <- as.matrix(dtm)
dim(m)
## [1]
46 6508
For very large corpus the size of the matrix can exceed Rs calculation limits. This will manifest
itself as a integer overflow error with a message like:
## Error in vector(typeof(x$v), nr * nc) : vector size cannot be NA
## In addition: Warning message:
## In nr * nc : NAs produced by integer overflow
If this occurs, then consider removing sparse terms from the document term matrix, as we discuss
shortly.
Once converted into a standard matrix the usual write.csv() can be used to write the data to
file.
write.csv(m, file="dtm.csv")
Module: TextMiningO
Page: 21 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
We are often not interested in infrequent terms in our documents. Such sparse terms can be
removed from the document term matrix quite easily using removeSparseTerms():
dim(dtm)
## [1]
46 6508
data
3101
graham
108
inform
467
time
483
use william
1366
236
table(freq)
## freq
## 108
##
1
236
1
467
1
Module: TextMiningO
Page: 22 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
One thing we often to first do is to get an idea of the most frequent terms in the corpus. We use
findFreqTerms() to do this. Here we limit the output to those terms that occur at least 1,000
times:
findFreqTerms(dtm, lowfreq=1000)
## [1] "data" "mine" "use"
So that only lists a few. We can get more of them by reducing the threshold:
findFreqTerms(dtm, lowfreq=100)
##
[1]
##
[6]
## [11]
## [16]
## [21]
## [26]
....
"accuraci"
"algorithm"
"appli"
"attribut"
"base"
"case"
"acsi"
"allow"
"applic"
"australia"
"build"
"chang"
"adr"
"also"
"approach"
"australian"
"call"
"claim"
"advers"
"analysi"
"area"
"avail"
"can"
"class"
"age"
"angioedema"
"associ"
"averag"
"care"
"classif"
mine
induct
0.90
0.72
need statistician
0.63
0.63
major
mani
challeng
0.70
foundat
0.62
come
know
0.65
general
0.62
answer
0.64
boost
0.61
If two words always appear together then the correlation would be 1.0 and if they never appear
together the correlation would be 0.0. Thus the correlation is a measure of how closely associated
the words are in the corpus.
Module: TextMiningO
Page: 23 of 46
Draft Only
Data Science with R
10
Hands-On
Text Mining
Correlations Plots
current
contain
case
decis
comput
australian
collect
condit
call
base
appli
can
applic
also
dataset
cost
data
compar
claim
avail
allow
consist
attribut
cluster
chang
area
analysi
angioedema
confer
age
algorithm
accuraci
consid
common
associ
classif
class
combin
australia
classi
build
databas
advers
acsi
adr
day
approach
averag
care
csiro
plot(dtm,
terms=findFreqTerms(dtm, lowfreq=100)[1:50],
corThreshold=0.5)
Rgraphviz (Hansen et al., 2016) from the BioConductor repository for R (bioconductor.org) is
used to plot the network graph that displays the correlation between chosen words in the corpus.
Here we choose 50 of the more frequent words as the nodes and include links between words
when they have at least a correlation of 0.5.
By default (without providing terms and a correlation threshold) the plot function chooses a
random 20 terms with a threshold of 0.7.
Module: TextMiningO
Page: 24 of 46
Draft Only
Data Science with R
11
Hands-On
Text Mining
Correlations PlotOptions
current
contain
case
decis
comput
australian
collect
condit
call
base
appli
can
applic
also
dataset
cost
data
compar
claim
avail
allow
consist
attribut
cluster
chang
area
analysi
angioedema
confer
age
algorithm
accuraci
consid
common
associ
classif
class
combin
australia
classi
build
databas
advers
acsi
adr
day
approach
averag
care
csiro
plot(dtm,
terms=findFreqTerms(dtm, lowfreq=100)[1:50],
corThreshold=0.5)
Module: TextMiningO
Page: 25 of 46
Draft Only
Data Science with R
12
Hands-On
Text Mining
data
mine
3101
1446
cluster algorithm
616
611
use
1366
rule
609
pattern
887
featur
578
dataset
776
set
555
can
709
tree
547
model
703
method
544
wf
<- data.frame(word=names(freq), freq=freq)
head(wf)
##
word
## data
data
## mine
mine
## use
use
## pattern pattern
## dataset dataset
....
freq
3101
1446
1366
887
776
We can then plot the frequency of those words that occur at least 500 times in the corpus:
library(ggplot2)
subset(wf, freq>500)
ggplot(aes(word, freq))
geom_bar(stat="identity")
theme(axis.text.x=element_text(angle=45, hjust=1))
%>%
+
+
3000
freq
2000
1000
e
us
tre
e
se
t
le
ru
n
tte
r
pa
el
od
m
e
in
m
et
ho
r
tu
fe
a
et
ta
s
da
ta
da
us
te
r
cl
n
ca
al
go
rit
hm
word
Module: TextMiningO
Page: 26 of 46
Draft Only
Data Science with R
13
Hands-On
Text Mining
Word Clouds
gain
preadmiss commonwealth
correspond give
function
piatetskyshapiro appropri
inhibitor simpli suggest
tune select independ
debian
construct
residualleverag
propos
comparison
experiment
domain leverag depend expect
estim
accuraci possibl
patholog
major
cart
total
servic
reaction rst actual
speci automat
messag
better australian order
relat
particular evolutionari see minimum dier
structur framework
togeth
gap
valid transact
activ
reason decis popul
even
stage therefor
applicinteract
tabl context
nd
studi
general rnn
averag statist export best
section second
hadi
advantag
understand
mutarc oc
indic clear
risk also sourc
igi intellig calcul
unit accord
drug
ensembl
denit one produc correl
artici alendron within
conddistribut sequenc mani determin descript standard
test
techniqu supp probablrefer
employ
predict
help
age
appear
input
miss ratio similar
clinic four publish altern
purpos breiman
might
project now acsisplit
issu
divers
top
singl
nal
larger tradit hidden less machin
make
deliv
prune
induct busi high
need consist
node weight
group insur
cca
start real
log introduct rather
som
classi
benet
hot
period
much
assess
year
observ
grow open
investig
intern
link methodolog transform databas current
train
item upon
mml
quit
level
increas
engin evalu
choos
new
review
market var premium press
sinc
public
address
rare
tar complex
conclus
episod deploy densiti
receiv entiti
emerg
ace
gis
varieti element output
month
area
chosen
occur
match
utar
subset
compon
end esophag version
interv thus locat
success list
natur
fraud
event score expert
architectur care line adr run
instal exist illustr
equat
must user specif
consider due
smyth situat experi spatial
diseas
scienc
le
consid
build
next
parallel drgcallsegment eect
text ieee
gnu
william main combin
without
obtain
remain
forward addit
govern
begin
describ categor
visual monitorthree
dicult allowmedic built avail
creat
environ caa den term advers
improv
kdd
abl introduc scheme
nugget program becom multipl
base
advanc
t
strength worker
includ
region common regress sever
usag
move
act
well
global
howev type various
analysi optim recent condit
basic simplwhole miner
interfac forest chang
threshold direct
reduc smaller australia
present extract practic higher rankcount known frequent mean individu
rattl seri http
outcommathemat
csiro anoth
usual
hepat prepar nation
world huang rang view exclus tmip ann behaviour
claim angioedema normal operfemal cost
dimension work
contain
stream
report demonstr author
requir
suppa
summari confer discov distinct
industri
donohostahel access factor linux deriv complet
prototyp
technolog
layer
custom
univers take
rate will sampl
classif connect
respect
control
size polici
step
g day interpret form
approach paramet geograph limit
vol
support graham softwar
occurr exposur visualis subsequ
key
canberra
variabl typic
queensland prior detect
interesting
preprocess
pbs
goal
consequ pmml appli
signic
way mbs
made intrus initi
interest
class
tempor
certain
postdischarg
hybrid
frequenc
organis
laboratori
follow
medicar concept
problem
analys
mine
number
literatur
process least captur
packag
small
comput
detail
guroften
featur
space
larg
sequenti
yes
origin
task
associ
among administr
abstract
involv
idea
good
may
search
found discuss
portfolio
code
pattern
like
futur
focusrecord
admiss proceed
plot
neural
design
patient
given
two
knowledg
develop copyright
cover
partit
unexpect
still
right
time
result
tree
exampl
discoveri
highlight
theori
outlier
can
show
case
set
spot
respons
candid
vector
degre
million
target
map
inform
page
perform
valu
part
dataset
local
method
aim
model
measur
identi
data
algorithm
taxat
templat
use
proport
random
system
We can generate a word cloud as an effective alternative to providing a quick visual overview of
the frequency of words in a corpus.
The wordcloud (?) package provides the required function.
library(wordcloud)
set.seed(123)
wordcloud(names(freq), freq, min.freq=40)
Notice the use of set.seed() only so that we can obtain the same layout each timeotherwise
a random layout is chosen, which is not usually an issue.
Module: TextMiningO
Page: 27 of 46
Draft Only
Data Science with R
Text Mining
featur
model
will section
set method
support present
techniqu
group
inform relat insur
mani
train
associ
drug
outlier
approach
13.1
Hands-On
studi includ
structur number
describ
cluster
data
pattern
measur
one
variabl
william
map
detect
two
use sequenc
exampl
rst
mine
problem
system knowledg
kdd risk
larg discoveri
new
tree
develop select rule
dataset
provid
timeprocess case
health can patient
high
decis
statist
random
research
sourc
period
valu
work test
generat perform databas paper
dier
comput
gur
popul
also
identirecord
learn
servic
class
general
function may similar
tablevent
analysi
classif
interest base
algorithm applic
tempor
result
To increase or reduce the number of words displayed we can tune the value of max.words=. Here
we have limited the display to the 100 most frequent words.
set.seed(142)
wordcloud(names(freq), freq, max.words=100)
Module: TextMiningO
Page: 28 of 46
Draft Only
Data Science with R
13.2
Hands-On
Text Mining
openparticular
high
use system
approach
expert
algorithm dataset
predict
train page increas cluster
discoveri
confer
may usualprocess featur kdd
howev
group weight insur acsi build paper number classif http
angioedema
tempor
refer
test
set
period
research
australian nd singl machin
comput proceed follow window
techniqu popul
detect
oper
larg
small
understand
statist
cost
data
general
allow given
patient
map
generat
studi
can
care sourc
time
develop
domain
often
tree
method
rattl
pattern
inform
support
class exist
rule
three packag
model
classi
mine
associ
knowledg
consid
interesting
A more common approach to increase or reduce the number of words displayed is by tuning the
value of min.freq=. Here we have limited the display to those words that occur at least 100
times.
set.seed(142)
wordcloud(names(freq), freq, min.freq=100)
Module: TextMiningO
Page: 29 of 46
Draft Only
Data Science with R
13.3
Hands-On
Text Mining
openparticular
high
use system
approach
expert
algorithm dataset
predict
train page increas cluster
discoveri
confer
may usualprocess featur kdd
howev
group weight insur acsi build paper number classif http
angioedema
tempor
refer
test
set
period
research
australian nd singl machin
comput proceed follow window
techniqu popul
detect
oper
larg
small
understand
statist
cost
data
general
allow given
patient
map
generat
studi
can
care sourc
time
develop
domain
often
tree
method
rattl
pattern
inform
support
class exist
rule
three packag
model
classi
mine
associ
knowledg
consid
interesting
We can also add some colour to the display. Here we make use of brewer.pal() from RColorBrewer (Neuwirth, 2014) to generate a palette of colours to use.
set.seed(142)
wordcloud(names(freq), freq, min.freq=100, colors=brewer.pal(6, "Dark2"))
Module: TextMiningO
Page: 30 of 46
Draft Only
Varying the Scaling
result
small
consisttarget
unexpect
tempor
studi
identi
discov
requir
dataset
import
like individu
reaction
pattern
predict
allow
refer den
weight
visual
network
machin
generat
step
usual need
neural
state
point
analysi
number
sequenc paper
patient
size
care
rule
may
eect
model
journal
singl
graham condit
record
claim
featur
process
given
technolog
learn
kdd chang
describ vector entiti
interest one
rst sourc
can
cluster discoveripopul
algorithm tree forest
relat
packag
use
structur
window contain
rnn angioedema
experi
scienc
consid
classi
map
object
work
current
confer
medic
distribut dier
event
inform
random
search
train
within
expect
case adr
mean
base
select
show
new
occur
pmml
unit
accuraci transact compar
cost
csiro
lead
discuss
time
avail
open
formerror
includ
hot
subset
drugpolici level
attribut
implement
exist
techniqu
method
evalu
follow
three
appli rank
averag
total combin section
call well
make
rattl
two
common
advers
tabl
period
type
area
subspac
risk
sampl
general approach
user
univers william
larg
insur acsi
statist
valu
increas
report
support
eci
hospit propos function
collect
research particular
tool link
australia
age
interv node
repres year
problem intern observ databas explor
utar
will
interesting
often
detect
system
day similar
domain
health
decis howev
oper
regress
exampl
expert
also
australian
mine
understand
develop
Text Mining
data
group
estim
class
13.4
Hands-On
set
proceed
spot
test
present
knowledg
nugget
associ
episod
comput
We can change the range of font sizes used in the plot using the scale= option. By default the
most frequent words have a scale of 4 and the least have a scale of 0.5. Here we illustrate the
effect of increasing the scale range.
set.seed(142)
wordcloud(names(freq), freq, min.freq=100, scale=c(5, .1), colors=brewer.pal(6, "Dark2"))
Module: TextMiningO
Page: 31 of 46
Draft Only
Data Science with R
Text Mining
Rotating Words
scienc follow
confer approach
rule tree
hybrid dataset
use cluster
pattern
repres search
expert
outlier
error
g variabl
refer
state rattl
statist
support
total intellig
order
discuss
studi
usual
mean
condit
singl wellcase
understand
class cost
discov detect
often
set
learn
network
page rst current node
high
kdd
visual
given transact task
indic need insur featur
acsi paper hot process system
chang day distribut
multipl
may
predict
medic
avail
spot
discoveri collect
train
machin
map
algorithm
13.5
Hands-On
signic
model
interest
period
estim
test
eci
level
associ
can
forest
oper
time
data
describ
larg
form
individu
den
small
domain
method
import
build william
exist
drug
http
lead univers
area combin
proceed
result
user
includ
tool subset
exampl
gur
weight
group journal accuraci explor step random reaction ratio
occur
subspac
like rank eect record two regress
also observ
consist
target
call
section base mani
pmml
interv report
inform
year angioedema will
evalu
intern graham
age
valu
analysi event
averag make
appli new sampl
open
classif
utar
tabl
servic
work
health
structur patient perform link select
propos
object
common
neural identi
classi attribut show databas
nugget
vector
experi
popul size allow
measur distanc unexpect
entiti present
csiro
expect
rnn
sourc
stage
similar
interesting
implement general adr australian
australia howev unit three risk
problem
hospit sequenc dier
technolog
generat one
techniqu
advers claim within applic compar requir
type provid
consid
care window nd
point
develop particular
relat
mine
packag polici
episod knowledg
comput
contain
research decis
We can change the proportion of words that are rotated by 90 degrees from the default 10% to,
say, 20% using rot.per=0.2.
set.seed(142)
dark2 <- brewer.pal(6, "Dark2")
wordcloud(names(freq), freq, min.freq=100, rot.per=0.2, colors=dark2)
Module: TextMiningO
Page: 32 of 46
Draft Only
Data Science with R
14
Hands-On
Text Mining
The qdap (Rinker, 2015) package provides an extensive suite of functions to support the quantitative analysis of text.
We can obtain simple summaries of a list of words, and to do so we will illustrate with the
terms from our Term Document Matrix tdm. We first extract the shorter terms from each of our
documents into one long word list. To do so we convert tdm into a matrix, extract the column
names (the terms) and retain those shorter than 20 characters.
words <- dtm
as.matrix
colnames
(function(x) x[nchar(x) < 20])
%>%
%>%
%>%
We can then summarise the word list. Notice, in particular, the use of dist tab() from qdap to
generate frequencies and percentages.
length(words)
## [1] 6456
head(words, 15)
## [1] "aaai"
"aab"
## [6] "aadrhtmliv" "aai"
## [11] "abbrevi"
"abc"
"aad"
"aam"
"abcd"
"aadrbhtm"
"aba"
"abdul"
"aadrbltn"
"abbrev"
"abel"
summary(nchar(words))
##
##
Median
6.000
Max.
19.000
8
651
11
200
table(nchar(words))
##
##
##
##
##
3
579
18
21
4
5
6
867 1044 1114
19
16
7
935
9
397
10
268
12
138
13
79
14
63
15
34
16
28
17
22
dist_tab(nchar(words))
##
interval freq cum.freq percent cum.percent
## 1
3 579
579
8.97
8.97
## 2
4 867
1446
13.43
22.40
## 3
5 1044
2490
16.17
38.57
## 4
6 1114
3604
17.26
55.82
## 5
7 935
4539
14.48
70.31
## 6
8 651
5190
10.08
80.39
## 7
9 397
5587
6.15
86.54
## 8
10 268
5855
4.15
90.69
## 9
11 200
6055
3.10
93.79
## 10
12 138
6193
2.14
95.93
....
Module: TextMiningO
Page: 33 of 46
Draft Only
Data Science with R
14.1
Hands-On
Text Mining
Number of Words
900
600
300
0
5
10
Number of Letters
15
20
A simple plot is then effective in showing the distribution of the word lengths. Here we create a
single column data frame that is passed on to ggplot() to generate a histogram, with a vertical
line to show the mean length of words.
data.frame(nletters=nchar(words))
ggplot(aes(x=nletters))
geom_histogram(binwidth=1)
geom_vline(xintercept=mean(nchar(words)),
colour="green", size=1, alpha=.5)
labs(x="Number of Letters", y="Number of Words")
Module: TextMiningO
%>%
+
+
+
Page: 34 of 46
Draft Only
Data Science with R
Letter
14.2
Hands-On
Text Mining
Letter Frequency
E
I
A
R
T
O
N
S
L
U
C
M
D
P
H
G
B
F
V
K
Y
W
X
Q
J
Z
0%
2%
4%
6%
Proportion
8%
10%
12%
Next we want to review the frequency of letters across all of the words in the discourse. Some
data preparation will transform the vector of words into a list of letters, which we then construct
a frequency count for, and pass this on to be plotted.
We again use a pipeline to string together the operations on the data. Starting from the vector of words stored in word we split the words into characters using str split() from stringr
(Wickham, 2015), removing the first string (an empty string) from each of the results (using
sapply()). Reducing the result into a simple vector, using unlist(), we then generate a data
frame recording the letter frequencies, using dist tab() from qdap. We can then plot the letter
proportions.
library(dplyr)
library(stringr)
words
%>%
str_split("")
sapply(function(x) x[-1])
unlist
dist_tab
mutate(Letter=factor(toupper(interval),
levels=toupper(interval[order(freq)])))
ggplot(aes(Letter, weight=percent))
geom_bar()
coord_flip()
labs(y="Proportion")
scale_y_continuous(breaks=seq(0, 12, 2),
label=function(x) paste0(x, "%"),
Module: TextMiningO
%>%
%>%
%>%
%>%
%>%
+
+
+
+
Page: 35 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
expand=c(0,0), limits=c(0,12))
Module: TextMiningO
Page: 36 of 46
Draft Only
Data Science with R
Letter
14.3
Hands-On
Text Mining
.010
.019
.013
.010
.010
.007
.005
.003
.002
.002
.001
.001
.001
.000
.000
.000
.000
.000
.000
.006
.001
.004
.002
.002
.002
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.013
.003
.007
.006
.004
.004
.003
.002
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.008
.002
.005
.005
.004
.003
.002
.001
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.006
.021
.010
.016
.014
.008
.005
.003
.002
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.005
.001
.003
.002
.001
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.004
.001
.004
.004
.002
.002
.002
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.005
.005
.002
.004
.003
.002
.001
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.007
.015
.009
.011
.012
.009
.007
.005
.003
.002
.001
.001
.001
.000
.000
.000
.000
.000
.000
.002
.000
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.002
.000
.001
.003
.001
.000
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.005
.005
.008
.008
.006
.004
.004
.002
.001
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
0.000
.009
.003
.007
.005
.003
.003
.002
.002
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
0.005
.005
.010
.012
.008
.007
.009
.005
.004
.003
.002
.001
.001
.000
.000
.000
.000
.000
.000
.000
0.010
.005
.021
.009
.008
.009
.005
.005
.003
.002
.002
.001
.001
.000
.000
.000
.000
.000
.000
.000
0.015
.011
.003
.006
.005
.002
.002
.001
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.001
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.009
.012
.013
.009
.010
.009
.006
.004
.002
.002
.001
.001
.000
.000
.000
.000
.000
.000
.000
.015
.004
.011
.008
.007
.006
.005
.003
.002
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.008
.005
.012
.013
.009
.008
.007
.005
.003
.002
.001
.001
.001
.000
.000
.000
.000
.000
.000
.004
.010
.005
.005
.004
.003
.002
.002
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.003
.001
.003
.002
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.005
.002
.002
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.001
.002
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.001
.001
.002
.001
.001
.001
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.001
.000
.000
.001
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
.000
10
Position
11
12
13
14
15
16
17
18
Proportion
0.020
19
The qheat() function from qdap provides an effective visualisation of tabular data. Here we
transform the list of words into a position count of each letter, and constructing a table of the
proportions that is passed on to qheat() to do the plotting.
words
lapply(function(x) sapply(letters, gregexpr, x, fixed=TRUE))
unlist
(function(x) x[x!=-1])
(function(x) setNames(x, gsub("\\d", "", names(x))))
(function(x) apply(table(data.frame(letter=toupper(names(x)),
position=unname(x))),
1, function(y) y/length(x)))
qheat(high="green", low="yellow", by.column=NULL,
values=TRUE, digits=3, plot=FALSE)
labs(y="Letter", x="Position") +
theme(axis.text.x=element_text(angle=0))
guides(fill=guide_legend(title="Proportion"))
Module: TextMiningO
%>%
%>%
%>%
%>%
%>%
%>%
+
+
Page: 37 of 46
Draft Only
Data Science with R
14.4
Hands-On
Text Mining
Miscellaneous Functions
We can generate gender from a name list, using the genderdata (?) package
devtools::install_github("lmullen/gender-data-pkg")
name2sex(qcv(graham, frank, leslie, james, jacqui, jack, kerry, kerrie))
## The genderdata package needs to be installed.
## Error in install genderdata package(): Failed to install the genderdata package.
## Please try installing the package for yourself using the following command:
##
install.packages("genderdata", repos = "http://packages.ropensci.org", type
= "source")
Module: TextMiningO
Page: 38 of 46
Draft Only
Data Science with R
15
Hands-On
Text Mining
Word Distances
Continuous bag of words (CBOW). Word2Vec associates each word in a vocabulary with a unique
vector of real numbers of length d. Words that have a similar syntactic context appear closer
together within the vector space. The syntactic context is based on a set of words within a
specific window size.
install.packages("tmcn.word2vec", repos="http://R-Forge.R-project.org")
## Installing package into /home/gjw/R/x86 64-pc-linux-gnu-library/3.2
## (as lib is unspecified)
##
## The downloaded source packages are in
## '/tmp/Rtmpt1u3GR/downloaded_packages'
library(tmcn.word2vec)
model <- word2vec(system.file("examples", "rfaq.txt", package = "tmcn.word2vec"))
## The model was generated in '/home/gjw/R/x86_64-pc-linux-gnu-library/3.2/tm...
distance(model$model_file, "the")
##
## 1
## 2
## 3
## 4
## 5
....
Word
a
is
and
an
please
CosDist
0.8694174
0.8063422
0.7908007
0.7738196
0.7595193
Module: TextMiningO
Page: 39 of 46
Draft Only
Data Science with R
16
Hands-On
Text Mining
Here in one sequence is collected the code to perform a text mining project. Notice that we would
not necessarily do all of these steps so pick and choose as is appropriate to your situation.
# Required packages
library(tm)
library(wordcloud)
# Locate and load the Corpus.
cname <- file.path(".", "corpus", "txt")
docs <- Corpus(DirSource(cname))
docs
summary(docs)
inspect(docs[1])
# Transforms
toSpace <- content_transformer(function(x, pattern) gsub(pattern, " ", x))
docs <- tm_map(docs, toSpace, "/|@|\\|")
docs
docs
docs
docs
docs
docs
<<<<<<-
tm_map(docs,
tm_map(docs,
tm_map(docs,
tm_map(docs,
tm_map(docs,
tm_map(docs,
content_transformer(tolower))
removeNumbers)
removePunctuation)
removeWords, stopwords("english"))
removeWords, c("own", "stop", "words"))
stripWhitespace)
Module: TextMiningO
Page: 40 of 46
Draft Only
Data Science with R
17
Hands-On
Text Mining
Module: TextMiningO
Page: 41 of 46
Draft Only
Data Science with R
18
Hands-On
Text Mining
LDA
Topic Models such as Latent Dirichlet Allocation has been popular for text mining in last 15
years. Applied with varying degrees of success. Text is fed into LDA to extract the topics
underlying the text document. Examples are the AP corpus and the Science Corpus 1880-2002
(Blei and Lafferty 2009). PERHAPS USEFUL IN BOOK?
When is LDA applicable - it will fail on some data and need to choose number of topics to
find and how many documents are needed. HOw do we know the topics learned are correct
topics.
Two fundemental papers - independelty discovered: Blei, Ng, Jordan - NIPS 2001 with 11k
citations. Other paper is Pritchard, Stephens, and Donnelly in Genetics June 200 14K citations
- models are exactly the same except for minor differences: except topics versus population
structures.
No theoretic analysis as such. How to guarantee correct topics and how efficient is the learning
procedure?
Observations:
LDA wont work on many short tweets or very few long documents.
We should not liberally over-fit the LDA with too many redundant topics...
Limiting factors:
We should use as many documents as we can and short documents less than 10 words wont
work even if there are many of them. Need sufficiently long documents.
Small Dirichlet paramenter helps especially if we overfit. See Long Nguens keynote at PAKDD
2015 in Vietnam.
number of documents the most important factor
document length plays a useful role too
avoid overfitting as you get too many topics and dont really learn anything as the humn needs
to cull the topics.
New work detects new topics as they emerge.
library(lda)
## Error in library(lda):
# From demo(lda)
library("ggplot2")
library("reshape2")
data(cora.documents)
## Warning in data(cora.documents):
data(cora.vocab)
## Warning in data(cora.vocab):
Module: TextMiningO
Page: 42 of 46
Draft Only
Data Science with R
Hands-On
Text Mining
theme_set(theme_bw())
set.seed(8675309)
K <- 10 ## Num clusters
result <- lda.collapsed.gibbs.sampler(cora.documents,
K, ## Num clusters
cora.vocab,
25, ## Num iterations
0.1,
0.1,
compute.log.likelihood=TRUE)
## Error in eval(expr, envir, enclos):
topic.proportions[is.na(topic.proportions)] <-
1 / K
Module: TextMiningO
Page: 43 of 46
Draft Only
Data Science with R
19
Hands-On
Text Mining
Thanks also to Tony Nolan for suggestions of some of the examples used in this chapter.
Some of the qdap examples were motivated by http://trinkerrstuff.wordpress.com/2014/
10/31/exploration-of-letter-make-up-of-english-words/.
Module: TextMiningO
Page: 44 of 46
Draft Only
Data Science with R
20
Hands-On
Text Mining
References
Bilisoly R (2008). Practical Text Mining with Perl. Wiley Series on Methods and Applications
in Data Mining. Wiley. ISBN 9780470382851. URL http://books.google.com.au/books?id=
YkMFVbsrdzkC.
Feinerer I, Hornik K (2015). tm: Text Mining Package. R package version 0.6-2, URL https:
//CRAN.R-project.org/package=tm.
Hansen KD, Gentry J, Long L, Gentleman R, Falcon S, Hahne F, Sarkar D (2016). Rgraphviz:
Provides plotting capabilities for R graph objects. R package version 2.12.0.
Neuwirth E (2014). RColorBrewer: ColorBrewer Palettes. R package version 1.1-2, URL
https://CRAN.R-project.org/package=RColorBrewer.
R Core Team (2015). R: A Language and Environment for Statistical Computing. R Foundation
for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
Rinker T (2015). qdap: Bridging the Gap Between Qualitative Data and Quantitative Analysis.
R package version 2.2.4, URL https://CRAN.R-project.org/package=qdap.
Wickham H (2015). stringr: Simple, Consistent Wrappers for Common String Operations. R
package version 1.0.0, URL https://CRAN.R-project.org/package=stringr.
Williams GJ (2009). Rattle: A Data Mining GUI for R. The R Journal, 1(2), 4555. URL
http://journal.r-project.org/archive/2009-2/RJournal_2009-2_Williams.pdf.
Williams GJ (2011). Data Mining with Rattle and R: The art of excavating data for knowledge
discovery. Use R! Springer, New York.
This document, sourced from TextMiningO.Rnw bitbucket revision 76, was processed by KnitR
version 1.12 of 2016-01-06 and took 41.3 seconds to process. It was generated by gjw on nyx
running Ubuntu 14.04.3 LTS with Intel(R) Xeon(R) CPU W3520 @ 2.67GHz having 8 cores and
12.3GB of RAM. It completed the processing 2016-01-10 09:58:57.
Copyright 2013-2015 [email protected]
Module: TextMiningO
Page: 45 of 46
Draft Only