Spark Streaming with Twitter, you can get public tweets by using Twitter API.

Spark Streaming with Twitter, you can get public tweets by using Twitter API.

To start, register a developer account at
To have consumer key, consumer secret key, access token and access secret, you will need to create an app and you will need to provide a web site URL, it does not have to be working.
Spark streaming use case with scala with Twitter
Get twitter streaming tweets that contains that contains β€œhappy” or β€œmoney” case insensitive
1
package com.jentekco.spark
2
​
3
import org.apache.spark._
4
import org.apache.spark.SparkContext._
5
import org.apache.spark.streaming._
6
import org.apache.spark.streaming.twitter._
7
import org.apache.spark.streaming.StreamingContext._
8
​
9
import org.apache.log4j.{Level, Logger}
10
​
11
/** Listens to a stream of Tweets only contains "happy" or "money" case insensitive
12
you can change it to whatever keyword you want to limit to
13
*/
14
object SearchKeywordTweets {
15
16
/**
17
I have included working build.sbt for you to run
18
sbt assembly to create jar file, check sbt sub folder for the build.sbt
19
*/
20
21
def main(args: Array[String]) {
22
23
24
Logger.getLogger("org").setLevel(Level.ERROR)
25
val ssc = new StreamingContext("local[*]", "SearchTwitterTweets", Seconds(1))
26
System.setProperty("twitter4j.oauth.consumerKey","<your Twitter consumer key>")
27
System.setProperty("twitter4j.oauth.consumerSecret", "<your twitter consumer secret>")
28
System.setProperty("twitter4j.oauth.accessToken", "<your twitter access token>")
29
System.setProperty("twitter4j.oauth.accessTokenSecret", "<your twitter token secret>")
30
31
32
val tweets = TwitterUtils.createStream(ssc, None)
33
34
// Extract the text
35
val tweets_collection = tweets.map(each_tweet => each_tweet.getText())
36
37
//Set your search criteria to only retain these meet your search condition
38
val focus_tweets_collection=tweets_collection.filter(text=>text.toLowerCase.contains("happy") | text.toLowerCase.contains("money"))
39
​
40
//Display your result
41
focus_tweets_collection.print()
42
​
43
//If you intent to run on Non windows machine, use proper path instead of windows drive letter!
44
ssc.checkpoint("d:/checkpoint/")
45
ssc.start()
46
ssc.awaitTermination()
47
}
48
}
Copied!
​
Last modified 1yr ago
Copy link