This article is just a small summary about the authentication process with Twitter. Its about creating the Twitter app and doing the handshake cause you have to do it every time you want to get data from Twitter with R.
Since Twitter released the Version 1.1 of their API a OAuth handshake is necessary for every request you do. So we have to verify our app.
First we need to create an app at Twitter.
Got to https://dev.twitter.com/ and log in with your Twitter Account.
Now you can see your Profile picture in the upper right corner and a drop-down menu. In this menu you can find “My Applications”.
Click on it and then on “Create new application”.
You can name your Application whatever you want and also set Description on whatever you want. Twitter requires a valid URL for the website, you can just type in http://test.de/ ; you won´t need it anymore.
And just leave the Callback URL blank.
Click on Create you´ll get redirected to a screen with all the OAuth setting of your new App. Just leave this window in the background; we´ll need it later
Before we go on, make sure you have installed the newest version of the twitteR package from github.
Therefore you can use the following code:
install.packages(c("devtools", "rjson", "bit64", "httr")) #RESTART R session! library(devtools) install_github("twitteR", username="geoffjentry") library(twitteR)
Now the twitteR package is up-to-date and we can use the new and very easy setup_twitter_oauth() function which uses the httr package.
The oAuth process:
First you have to get your api_key and your api_secret as well as your access_token and access_token_secret from your app settings on Twitter. Just click on the “API key” tab to see them.
api_key <- "YOUR API KEY" api_secret <- "YOUR API SECRET" access_token <- "YOUR ACCESS TOKEN" access_token_secret <- "YOUR ACCESS TOKEN SECRET" setup_twitter_oauth(api_key,api_secret,access_token,access_token_secret)
And that´s it.
If you want to test your authentication just try to get some tweets with:
searchTwitter("iphone")
Now you can continue your Twitter research with one of my Twitter examples:
Or follow me on Twitter to stay up-to-date about everything connected to R and Data Science
Hi Julian,
I have downloaded twitter data using R in .csv format. These are the fields :
text favorited favoriteCount replyToSN created truncated replyToSID id replyToUID statusSource screenName retweetCount isRetweet retweeted longitude latitude
For a given tweet in .csv file, the value of retweetCount>0 though the value for retweeted=0. I am not able to exactly discern the meaning for these three different fields (retweetCount, isRetweet, retweeted).
Could you please help me understand the same?
TIA
Hey Nikhil,
So
retweetCount – How often it was retweeted
isRetweet – Is this tweet a retweet or an original tweet?
retweeted – Was the tweet retweeted? (true/false)
So there seems to be a problem with the API. I don´t really know why it returns these values but I found a lot of similar questions in the Twitter forum but no real answer.
I will take a look at this topic again.
Regards
A small clarification with respect to the above post
Value for retweeted is FALSE even though retweetCount>0.
Hi julian
I am calculating sentiment from twitter tweets.I have a question can we fetch conversations from a single tweet that is fetched in search twitter function.
Hey Onkar,
I think that this is not possible on the direct way. You can just search all the tweets you downloaded for connections. To have access to the whole Twitter data you would need something like a Firehose access.
Regards
I get the following error by executing : twitCred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”)):
Error in function (type, msg, asError = TRUE) :
Could not resolve host: api.twitter.com; Host not found, try again
how to resolve this issue?
Hey phdcs,
sorry for the late answer.
It seems like there was something wrong with the Twitter API. It should work again now, does it?
Regards
Pingback: Analysis of devices used to post on Twitter – R code | kapoorabhishek
Well after I make the authentication and all the necessary, I get this as a result!
twitterMap(“socialmedia2day”, fileName=”Map_socialmedia2day.pdf”, nMax=3000)
Getting data from Twitter, this may take a moment.
Error in twInterfaceObj$doAPICall(paste(“users”, “show”, sep = “/”), params = params, :
OAuth authentication is required with Twitter’s API v1.1
Problem on twitter????
Hey Tasos,
Did you receive a “true” after the authentication process?
Regards
after I pasted the PIN in R then I had just the next line to give a command.
no true no false!
thx.
Hey Tasos,
it seems like you forgot the last row in the code.
The authentication is not done after you inserted the PIN. You have to execute the line:
registerTwitterOAuth(twitCred)
This line will then, if everything works well, return a “TRUE”.
Then you are ready to go.
Regards
Hey Julian!
Everything works until the point:
> twitCred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”))
Error: Unauthorized
>
> registerTwitterOAuth(twitCred)
Error in registerTwitterOAuth(twitCred) :
oauth has not completed its handshake
Any idea?
Hey Barns,
You have to execute the code line by line. Did you do so? Because otherwise the code skips the line where you have to enter your PIN.
Regards
Yep, did it line by line, but after running line 14, console says “Error: Unauthorized”
The keys are definetely correct (Used Mozilla and my personal Account)
Could you please copy-and-paste your R history here? It could help me finding the error.
Regards
Thanks again for your quick response! Somehow i got it yesterday until that point. I think the problem was a double-twitter-login with two different accounts…
same problem, but I use just one twitter account
R logs
> library(twitteR,igraph)
Le chargement a nécessité le package : ROAuth
Le chargement a nécessité le package : RCurl
Le chargement a nécessité le package : bitops
Le chargement a nécessité le package : digest
Le chargement a nécessité le package : rjson
> library(RCurl)
> options(RCurlOptions = list(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”)))
> require(twitteR)
> reqURL accessURL authURL apiKey apiSecret twitCred twitCred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”))
Erreur : Unauthorized
> twitCred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”))
Erreur : Unauthorized
can you help me please ?
Hey,
I just updated the tutorial.
Please refresh the page and see if it works with the new code.
Regards
library(twitteR)
library(stringr)
library(RCurl)
reqURL <-"https://api.twitter.com/oauth/request_token"
accessURL <- "http://api.twitter.com/oauth/access_token"
authURL <- "http://api.twitter.com/oauth/authorize"
consumerKey="xxxxxxxxxxx" % i have this key
comsumerSecret="xxxxxxxxxx" % i have this key
download.file(url="http://curl.haxx.se/ca/cacert.pem",destfile="cacert.pem") % for what??
twitCred <- OAuthFactory$new(consumerKey=consumerKey,
consumerSecret=consumerSecret,
requestURL=reqURL,
accessURL=accessURL,
authURL=authURL)
twitCred$handshake(cainfo="cacert.pem") # curl is a command line tool for transfering
registerTwitterOAuth(twitCred)
Hai julian,
I wanna ask you several question.
1. I try to run this code but still error because "unauthorized". i am working with mac os.
How to solve it?
2.This code for what?? should i download this file?
download.file(url="http://curl.haxx.se/ca/cacert.pem",destfile="cacert.pem")
3.it there have different setting for the cacert.pem???
Hey, using the most recent version of the code should solve all your problems. I updated the tutorial a few weeks ago.
Regards
Pingback: Create Twitter Wordcloud with Sentiments | julianhi's Blog
Please help!! I am having “unauthorized” when started searching. This problem has been several days. So I decided to create another application today. Could that be my application setting? Below is my R console output:
> Cred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”))
To enable the connection, please direct your web browser to:
https://api.twitter.com/oauth/authorize?oauth_token=YZGu3jRN2ZxG9IwvhvEJMcR9oXTw2GcV8geGp49ME
When complete, record the PIN given to you and provide it here: 8357550
> registerTwitterOAuth(Cred)
[1] TRUE
> save(Cred, file=”twitter authentication rkresnadi1.Rdata”)
> twitlst <- searchTwitter("sustainable development", n=1000, cainfo="cacert.pem")
[1] "Unauthorized"
Error in twInterfaceObj$doAPICall(cmd, params, "GET", …) :
Error: Unauthorized
Hey Stu,
your code seems to be ok. I saw that you changes the name of the Credentials Object to Cred, but that shouldn´t be a problem if you did this in your whole code. But you receive a “TRUE” which normally tells you that the authentication was successful. Could you also please post the first part of your code?
Regards
please why am i getting error: forbidden
> library(twitteR)
Loading required package: ROAuth
Loading required package: RCurl
Loading required package: bitops
Loading required package: digest
Loading required package: rjson
Warning messages:
1: package ‘twitteR’ was built under R version 3.0.2
2: package ‘ROAuth’ was built under R version 3.0.2
3: package ‘RCurl’ was built under R version 3.0.2
4: package ‘digest’ was built under R version 3.0.2
> library(plyr)
Attaching package: ‘plyr’
The following object is masked from ‘package:twitteR’:
id
Warning message:
package ‘plyr’ was built under R version 3.0.2
> library(stringr)
Warning message:
package ‘stringr’ was built under R version 3.0.2
> library(ROAuth)
> download.file(url=”http://curl.haxx.se/ca/cacert.pem”, destfile=”cacert.pem”)
trying URL ‘http://curl.haxx.se/ca/cacert.pem’
Content type ” length 250283 bytes (244 Kb)
opened URL
downloaded 244 Kb
> requestURL accessURL = “http://api.twitter.com/oauth/access_token”
> authURL consumerKey = “myKey”
> consumerSecret = “mySecret”
> Cred Cred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”) )
Error: Forbidden
Hey nolly,
Please use the updates code. You use old code.
Then everything should work.
Regards
where is updated code in your blog please give me your link
Hey satya
The updated code is the code above. But the code used in the question before was an Old version.
So just use the code from the Tutorial and everything should work.
Regards
Hi julianhi,
I’m having the same problem retrieving tweets.
> tweets = searchTwitter(“sustainable development”, n=200, cainfo=”cacert.pem”)
[1] “Unauthorized”
Error in twInterfaceObj$doAPICall(cmd, params, “GET”, …) :
Error: Unauthorized
Here is my code, I’m putting it in line by line. Using RStudio Version 0.98.501 on Mac.
> library(RCurl)
> options(RCurlOptions = list(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”)))
> require(twitteR)
> reqURL accessURL authURL apiKey apiSecret twitCred twitCred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”))
To enable the connection, please direct your web browser to:
https://api.twitter.com/oauth/authorize?oauth_token=KiysFMXkw1ggdHGviVv5U8Yef4ooZxw7AZ89L1LSmFM
When complete, record the PIN given to you and provide it here: 1234567
> registerTwitterOAuth(twitCred)
[1] TRUE
> tweets = searchTwitter(“sustainable development”, n=200, cainfo=”cacert.pem”)
[1] “Unauthorized”
Error in twInterfaceObj$doAPICall(cmd, params, “GET”, …) :
Error: Unauthorized
Please help!
CMG
Hi J,
here’s the complete code, I messed up the last post. Like I said, I’m using RSudio on Mac
> library(RCurl)
Loading required package: bitops
> options(RCurlOptions = list(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”)))
> require(twitteR)
Loading required package: twitteR
Loading required package: ROAuth
Loading required package: digest
Loading required package: rjson
> reqURL accessURL authURL apiKey apiSecret twitCred twitCred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”))
To enable the connection, please direct your web browser to:
https://api.twitter.com/oauth/authorize?oauth_token=epAv0cZJWd7sRayMvKwDPVjQtPiVnmX1dK1AKEZjVg
When complete, record the PIN given to you and provide it here: 1234567
> registerTwitterOAuth(twitCred)
[1] TRUE
> tweets = searchTwitter(“MMR”, n=200, cainfo=”cacert.pem”)
[1] “Unauthorized”
Error in twInterfaceObj$doAPICall(cmd, params, “GET”, …) :
Error: Unauthorized
Ok, there isn’t enough room to put up the whole code, can I email it to you?
Hey CMG,
sorry for the late answer but I was very busy.
I can´t find a bug in your code at the moment.
But did you try do execute it in the normal R console?
Sometimes the authentication does not really complete in RStudio.
Regards
Pingback: Mining some Twitter data | DH + Data Viz
Hello, I am having also the error forbidden during the handshake:
To enable the connection, please direct your web browser to:
http://api.twitter.com/oauth/authorize?oauth_token=FWJevpPdDnZrbuqmmkyrQzcUZzRNR3we83LlJdAXnY
When complete, record the PIN given to you and provide it here: 1817865
Error: Forbidden
And I tried many times…..
here is my code:
library(twitteR)
library(RCurl)
# Set SSL certs globally
options(RCurlOptions = list(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”)))
require(twitteR)
reqURL <- "https://api.twitter.com/oauth/request_token"
accessURL <- "http://api.twitter.com/oauth/access_token"
authURL <- "http://api.twitter.com/oauth/authorize"
consumerKey <- "xpArd5lrnnjxxxxxxxxxxxxxxxxxx"
consumerSecret <- "jFF4fG3wrhx7Ya8P8xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
twitCred <- OAuthFactory$new(consumerKey=consumerKey,
consumerSecret=consumerSecret,
requestURL=reqURL,
accessURL=accessURL,
authURL=authURL)
twitCred$handshake(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"))
registerTwitterOAuth(twitCred)
What did I miss ? Do you think SSL is not working properly ?
Hey,
I just updated the tutorial.
Please refresh the page and see if it works with the new code.
Regards
Good afternoon, I’ve read through the comments and while I saw the same issue (Error: Unauthorized) I didn’t see a way to resolve it.
> twitCred$handshake(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”))
To enable the connection, please direct your web browser to:
https://api.twitter.com/oauth/authorize?oauth_token=xyz…
When complete, record the PIN given to you and provide it here:
You mention to use the most updated code but I think I am. Here’s what I’m using:
library(RCurl)
# Set SSL certs globally
options(RCurlOptions = list(cainfo = system.file(“CurlSSL”, “cacert.pem”, package = “RCurl”)))
require(twitteR)
reqURL <- "https://api.twitter.com/oauth/request_token"
accessURL <- "https://api.twitter.com/oauth/access_token"
authURL <- "https://api.twitter.com/oauth/authorize"
apiKey <- "removed"
apiSecret <- "removed"
twitCred <- OAuthFactory$new(consumerKey=apiKey,consumerSecret=apiSecret,requestURL=reqURL,accessURL=accessURL,authURL=authURL)
twitCred$handshake(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"))
registerTwitterOAuth(twitCred)
Hey,
I just updated the tutorial.
Please refresh the page and see if it works with the new code.
Regards
Pingback: Five minutes of Twitter Data Tutorial | Austin+Wehrwein
Pingback: Cluster your Twitter Data with R and k-means - ThinkToStart
Pingback: Sentiment Analysis on Twitter with Datumbox API - ThinkToStart
Hey, I am having this problem while I am trying to extract 1499 tweets with #flipkart….
In doRppAPICall(“search/tweets”, n, params = params, retryOnRateLimit = retryOnRateLimit, :
1499 tweets were requested but the API can only return 100
Hey abhi,
the Twitter API can just return a certain amount of tweets for a given keyword. This is a restriction by twitter. So for you there is no way of getting more than 100 tweets at the moment for your given searchterm. If you want to get more over time you can use the streaming API and save the tweets directly when they are posted on twitter. But this is real time and it needs some time to collect tweets.
Regards
May be you can add
retryOnRateLimit = TRUE
to your searchTwitter call.
But I´m not sure if this still works in the latest twitteR package.
Regards
And julianhi, one more request….
Is there any way to extract tweets between certain date range…please help #stuck
This would be a sample function call with date limits:
searchTwitter”iphone”, since=’2011-03-01′, until=’2011-03-02′)
Thanks Alot julianhi u just save my life……
Saving lifes is my job 😉
Regards
Hi julianhi! I have a problem with this script, i wish that you can help me
install.packages(“ROAuth”)
install.packages(“twitteR”)#comandi riguardo a twitter
install.packages(“wordcloud”)#per analizzare graficamente i testi
install.packages(“tm”)#permette di lavorare su dati di testo
install.packages(“stringr”)
install.packages(“RCurl”)
library(“ROAuth”)
library(“twitteR”)
library(“wordcloud”)
library(“tm”)
library(“stringr”)
library(“RCurl”)
require(twitteR)
require(ROAuth)
require(stringr)
download.file(url=”http://curl.haxx.se/ca/cacert.pem”, destfile=”cacert.pem”)
cred <- OAuthFactory$new(consumerKey='xxxx',
consumerSecret='xxx',
requestURL='https://api.twitter.com/oauth/request_token',
accessURL='https://api.twitter.com/oauth/access_token',
authURL='https://api.twitter.com/oauth/authorize'😉
cred$handshake(cainfo = system.file("CurlSSL", "cacert.pem", package = "RCurl"))
save(cred, file="twitter authentication.Rdata")
registerTwitterOAuth(cred)
r_stats<- searchTwitter("#Rstats", n=1500, cainfo="cacert.pem")
Finally I have this error:
[1] "Unauthorized"
Errore in twInterfaceObj$doAPICall(cmd, params, "GET", …) :
Error: Unauthorized
but when I do registerTwitterOAuth(cred) my output is TRUE then I think my autentication is ok. Can you help me?
Thanks a lot
Angela
Hey Angela,
I updated the code a few days ago. Please refresh the site and you should see it 😉
The auth process got a lot easier in the newest twitteR version.
Does this work for you?
Regards
Perfect! thanks a lot! I see this blog and I found some errors.. but with your code is all ok.. Can I ask you some question? In my prior code there is a destfile called cancert.pem, (sorry for the question, I’m not very well in R and in english too) but what is its rule in the code? thanks 🙂
The cacert.pem file is a collection of trusted root certification authorities. But it is enough to know that it was necessary for the steps to create the oAuth connection. The old code had a lot of manual steps and in the new version of the twitteR package these steps were replace with the httr package. And so you just need the one line code from my tutorial for the authentication. All these steps you had to do before like downloading the cacert.pem file happen automatically.
Regards
perfect! thanks Julianhi 🙂
Sorry, I have another problem 😦 in this code:
#save text
r_stats_text <- sapply(r_stats, function(x) x$getText())
r_stats_text
#create corpus
r_stats_text_corpus <- Corpus(VectorSource(r_stats_text))
#clean up
r_stats_text_corpus <- tm_map(r_stats_text_corpus, tolower)
r_stats_text_corpus <- tm_map(r_stats_text_corpus, removeNumbers)
r_stats_text_corpus <- tm_map(r_stats_text_corpus, removePunctuation)
r_stats_text_corpus <- tm_map(r_stats_text_corpus, removeWords, stopwords('english'))
r_stats_text_corpus wordcloud(r_stats_text_corpus)
Error: inherits(doc, “TextDocument”) is not TRUE
can you help me?
thanks a lot!
Hi Julian,
As for the searchTwitter command, i notice that you said due to the restriction, it can only return the recent ones. But what is ‘recent’? like 1 week? 2 week? 1 month? 3 months?
and in the searchTwitter command, i can only search for one keyword at a time. is there any other method that I can get the results that contain several keywords? for example, i want to get the tweets contain keywords of ‘A’ or ‘B’ or ‘C’ (not AND).
Best Regards
Hey,
sorry for the late answer.
Recent is not defined by twitter. It always depends on the amount of tweets which contain for example the hashtag you are searching for. You have to try it out yourself for the keyword you are searching.
I´m not sure how you could solve your second problem. But why don´t you just do 3 separate searches and save them into a dataframe. Wouldn´t that be a solution?
Regards
Ok, thank you for your reply! i’ll try =]
Could you please help me on how I can solve the following problem in R:
> twitCred$handshake()
To enable the connection, please direct your web browser to:
http://api.twitter.com/oauth/authorize?oauth_token=OdB28pxghH4qGjGKTq5pMaEyrJk7ankT9xWfZs7x4
When complete, record the PIN given to you and provide it here: 5345731
Error: Forbidden
Hey,
seems like you used some old code. I updated the tutorial some time ago. Please refresh the page and delete your cache and try using the new code.
Regards
Hey Julianhi,
Can you suggest me how to extract tweets from certain locations like India or any country…….