Sentiment Analysis on Twitter

How to do a Twitter Sentiment Analysis?

Or: What´s the mood on Twitter?

Hello there!

Today I want to show you how to do a so-called Sentiment Analysis. It is about analyzing the mood on Twitter about a certain Keyword. You get a number of tweets which contain a keyword you can define, filter out the text of these tweets and then see if there are more positive or negative words. Of course you can´t just do it by hand; you need a tool doing the work for you.

Our tool:

Our main tool is called R. (yes just R, it´s not a typo)

It is a free “software environment for statistical computing and graphics” and is available for Unix platforms, Windows and MacOS.

It´s available here: http://www.r-project.org/

It has a comfortable installer, so this step shouldn´t be a problem.

After installing you can open the GUI and get the following screen:

Ok now we can download our other tool: twitteR

It´s a script written for R.

You don´t have to download it from a website, you can do it directly from within R.

You can to it with:

Install.packages(‘twitteR’, dependencies=T)

You then have to select a CRAN mirror, from where you want to download it and click ok. (you can show what ever mirror you want)

R will now download the package and install it.

Then we have to activate it for our current session with:

library(twitteR)

library(plyr)

Your screen should look like this now:

Ok now we come to a tricky part:

The Twitter Authentification

 

Since Twitter released the Version 1.1 of their API a OAuth handshake is necessary for every request you do. So we have to verify our app.

First we need to create an app at Twitter.

Got to https://dev.twitter.com/ and log in with your Twitter Account.

Now you can see your Profile picture in the upper right corner and a drop-down menu. In this menu you can find “My Applications”.

Click on it and then on “Create new application”.

You can name your Application whatever you want and also set Description on whatever you want. Twitter requires a valid URL for the website, you can just type in http://test.de/ ; you won´t need it anymore.

And just leave the Callback URL blank.

Click on Create you´ll get redirected to a screen with all the OAuth setting of your new App. Just leave this window in the background; we´ll need it later

Continue to R and type in the following lines (on separate lines):

reqURL <- "https://api.twitter.com/oauth/request_token"

accessURL <- "http://api.twitter.com/oauth/access_token"

authURL <- "http://api.twitter.com/oauth/authorize"

consumerKey <- "yourconsumerkey"

consumerSecret <- "yourconsumersecret"

twitCred <- OAuthFactory$new(consumerKey=consumerKey,consumerSecret=consumerSecret,requestURL=reqURL,accessURL=accessURL,authURL=authURL)

download.file(url="http://curl.haxx.se/ca/cacert.pem", destfile="cacert.pem")

twitCred$handshake(cainfo="cacert.pem")

registerTwitterOAuth(twitCred)

You have to replace yourconsumerkey and yourconsumersecret with the data provided on your app page on Twitter, still opened in your webbrowser.

The command twitCred$handshake(cainfo=”cacert.pem”) will ask you to go a certain URL and entert he PIN you receive on this page.

The Data Mining:

Ok we passed the authentication and can now go on with getting the tweets we want from Twitter.

Type in:

tweets = searchTwitter("#apple", n=200, cainfo="cacert.pem")

This makes twitteR get 200 Tweets with the keyword #apple in it (you can change the keyword of course).

After waiting a few seconds you can use length(tweets) to see how many tweets were actually saved; maybe for some keywords the number existing is actual smaller than our sample size n.

Now we have our Tweets.

The Analyzing:

To be able to analyze our tweets, we have to extract their text and save it into the variable tweets.text by typing:

Tweets.text = laply(tweets,function(t)t$getText())

What we also need are our lists with the positive and the negative words.

We can find them here:

https://github.com/mjhea0/twitter-sentiment-analysis/tree/master/wordbanks

After downloading the ZIP you can put them in a folder on your Computer; you should just keep the absolute path in mind.

We now have to load the words in variables to use them by typing:

pos = scan('/Users/julian/Documents/positive-words.txt', what='character', comment.char=';')

neg = scan('/Users/julian/Documents/negative-words.txt', what='character', comment.char=';')

Of course you have to change the path, but we have our two lists: pos and neg

Now we have to insert a small algorhytm written by Jeffrey Breen analyzing our words.

Just copy-paste the following lines and hit enter:

score.sentiment = function(sentences, pos.words, neg.words, .progress='none')

{

require(plyr)

require(stringr)

# we got a vector of sentences. plyr will handle a list

# or a vector as an "l" for us

# we want a simple array ("a") of scores back, so we use

# "l" + "a" + "ply" = "laply":

scores = laply(sentences, function(sentence, pos.words, neg.words) {

# clean up sentences with R's regex-driven global substitute, gsub():

sentence = gsub('[[:punct:]]', '', sentence)

sentence = gsub('[[:cntrl:]]', '', sentence)

sentence = gsub('\\d+', '', sentence)

# and convert to lower case:

sentence = tolower(sentence)

# split into words. str_split is in the stringr package

word.list = str_split(sentence, '\\s+')

# sometimes a list() is one level of hierarchy too much

words = unlist(word.list)

# compare our words to the dictionaries of positive & negative terms

pos.matches = match(words, pos.words)

neg.matches = match(words, neg.words)

# match() returns the position of the matched term or NA

# we just want a TRUE/FALSE:

pos.matches = !is.na(pos.matches)

neg.matches = !is.na(neg.matches)

# and conveniently enough, TRUE/FALSE will be treated as 1/0 by sum():

score = sum(pos.matches) - sum(neg.matches)

return(score)

}, pos.words, neg.words, .progress=.progress )

scores.df = data.frame(score=scores, text=sentences)

return(scores.df)

}

The final steps:

Type in:

analysis = score.sentiment(Tweets.text, pos, neg)

Congrats, your first sentiment Analysis was now saved.

You can get a table by typing:

table(analysis$score)

Or the mean by typing:

mean(analysis$score)

Or get a histogram with:

hist(analysis$score)

The positive values stand for positive tweets and the negative values for negative tweets. The mean tells you about the overall mood of your sample.

 

 

 

 

 

Note:

Sometimes it doesn´t work because there are some tweets with invalid characters in it. Then you have to do the data mining again or change the keyword. As soon an update is available I will update this article.