This tutorial is out of date and no longer maintained.
This tutorial will show how to use the Twitter Streaming APIs to process tweets in real-time from a Laravel application. There are a variety of use cases for this: perhaps you want to auto-respond to mentions of your company, run a contest via Twitter, or create support tickets when users complain about your product. For this tutorial, we’ll build a “featured tweets” widget to display approved tweets on our app’s home page.
You might first look at the Twitter REST APIs, which many developers are familiar with. These APIs allow you to do a lot of cool things: read and write tweets on behalf of app users, manage followers and lists, and more. However, if you want to process tweets as they are sent, the REST APIs are very difficult to manage. Pagination and rate limiting become a logistical nightmare quickly.
Whereas the REST APIs force you to request information, the Streaming APIs feed information to you. You can read more about how it works in the documentation, but it’s basically a pipe from Twitter to your application which delivers a continual flow of tweets.
By the end of the tutorial, you’ll have a constantly updating collection of tweets in your app that you can approve or disapprove to show up in your home page. Here’s a preview of what you’ll end up with.
Here are the major pieces of the app that we’ll be creating:
TwitterStream
which will extend Phirehose and place tweets onto the queue as they arriveConnectToStreamingAPI
, which will open a connection to the Streaming API with an instance of TwitterStream
ProcessTweet
, which will contain the handler to process tweets when they are pulled off the queueTweet
Eloquent modelWhile reading through the tutorial, you can follow along with this GitHub repo. It has separate commits for each step:
https://github.com/dabernathy89/Laravel-Twitter-Streaming-API-Demo
Using the Laravel installer, create a new Laravel instance:
- laravel new twitter-stream-test
Once that’s done, add the app to your Homestead configuration file. Don’t forget to edit your hosts
file with whatever custom domain you added to the configuration. Boot up Homestead:
- homestead up --provision
SSH into Homestead:
- homestead ssh
Finally, cd
into your app’s folder.
Next, you need to pull in the Phirehose library. Run composer require fennb/phirehose
to grab the latest version. Phirehose doesn’t support PSR-0 or PSR-4 autoloading, so you’ll need to use Composer’s classmap
autoloading option. Add a line just after the database
entry:
"classmap": [
"database",
"vendor/fennb/phirehose/lib"
],
ProcessTweet
JobI mentioned in the overview that we will be putting all of the tweets from the Streaming API onto a queue. In Laravel 5.2, there are special Job classes that are meant for processing items in a queue. Your app will have a job called ProcessTweet
which will be responsible for pulling tweets off the queue and doing something with them. You can create the job with a simple Artisan command:
- php artisan make:job ProcessTweet
This will generate a Job class in your app/Jobs
folder. You only need to make two adjustments to it for now.
handle()
method. For now you can just var_dump()
some basic information from the tweet.<?php
namespace App\Jobs;
use App\Jobs\Job;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Queue\ShouldQueue;
class ProcessTweet extends Job implements ShouldQueue
{
use InteractsWithQueue, SerializesModels;
protected $tweet;
/**
* Create a new job instance.
*
* @return void
*/
public function __construct($tweet)
{
$this->tweet = $tweet;
}
/**
* Execute the job.
*
* @return void
*/
public function handle()
{
$tweet = json_decode($this->tweet,true);
var_dump($tweet['text']) . PHP_EOL;
var_dump($tweet['id_str']) . PHP_EOL;
}
}
TwitterStream
ClassTo use Phirehose, you need to create a simple class that extends the base Phirehose class. This can go wherever you like, but I’ve just placed it directly in the app
folder. Here’s the full class:
<?php
namespace App;
use OauthPhirehose;
use App\Jobs\ProcessTweet;
use Illuminate\Foundation\Bus\DispatchesJobs;
class TwitterStream extends OauthPhirehose
{
use DispatchesJobs;
/**
* Enqueue each status
*
* @param string $status
*/
public function enqueueStatus($status)
{
$this->dispatch(new ProcessTweet($status));
}
}
A couple of things to note:
DispatchesJobs
trait to make it easy to push the tweet onto the queue.enqueueStatus
, which will be called by Phirehose for each tweet.TwitterStream
ClassYou need to register the TwitterStream
class with the Laravel container so that it can pull in its dependencies properly. In the register
method of your AppServiceProvider
class, add the following:
$this->app->bind('App\TwitterStream', function ($app) {
$twitter_access_token = env('TWITTER_ACCESS_TOKEN', null);
$twitter_access_token_secret = env('TWITTER_ACCESS_TOKEN_SECRET', null);
return new TwitterStream($twitter_access_token, $twitter_access_token_secret, Phirehose::METHOD_FILTER);
});
You will also need to add use Phirehose;
and use App\TwitterStream;
at the top of your service provider. Don’t worry about the environment variables for now - you’ll create them shortly.
Generate an Artisan command so that you can initiate the Streaming API connection over the command line:
php artisan make:console ConnectToStreamingAPI
This will generate the boilerplate console command class. Then you need to:
TwitterStream
class through the constructorhandle()
method, finish configuring the Phirehose object, including your search terms, and open the connectionHere’s what the completed command looks like. It will pull in tweets that contain the keyword scotch_io
:
<?php
namespace App\Console\Commands;
use Illuminate\Console\Command;
use App\TwitterStream;
class ConnectToStreamingAPI extends Command
{
/**
* The name and signature of the console command.
*
* @var string
*/
protected $signature = 'connect_to_streaming_api';
/**
* The console command description.
*
* @var string
*/
protected $description = 'Connect to the Twitter Streaming API';
protected $twitterStream;
/**
* Create a new command instance.
*
* @return void
*/
public function __construct(TwitterStream $twitterStream)
{
$this->twitterStream = $twitterStream;
parent::__construct();
}
/**
* Execute the console command.
*
* @return mixed
*/
public function handle()
{
$twitter_consumer_key = env('TWITTER_CONSUMER_KEY', '');
$twitter_consumer_secret = env('TWITTER_CONSUMER_SECRET', '');
$this->twitterStream->consumerKey = $twitter_consumer_key;
$this->twitterStream->consumerSecret = $twitter_consumer_secret;
$this->twitterStream->setTrack(array('scotch_io'));
$this->twitterStream->consume();
}
}
You also need to register the command. Simply update the $commands
property in the App\Console\Kernel
class:
protected $commands = [
Commands\ConnectToStreamingAPI::class
];
Generate an app with Twitter, and grab the keys and access tokens. Add them to your .env
file:
TWITTER_CONSUMER_KEY=KEY
TWITTER_CONSUMER_SECRET=SECRET
TWITTER_ACCESS_TOKEN=TOKEN
TWITTER_ACCESS_TOKEN_SECRET=SECRET
You can set up the queue however you want, but for demonstration purposes, the database driver will work fine. To set that up, update your .env
file:
QUEUE_DRIVER=database
By default, your app will look for a database with the name homestead
. Don’t forget to update this in your .env
file if you want a separate database for your app:
DB_DATABASE=twitterstreamtest
You’ll also need to run the following commands to prepare the database queue:
- php artisan queue:table
- php artisan migrate
Now you should be ready to start processing some tweets. Run your Artisan command:
php artisan connect_to_streaming_api
You’ll see some information about the connection in your console from Phirehose. It won’t tell you every time a tweet is processed, but it will give you an update every once in a while on the status of the connection.
Before too long, you should have some tweets about Scotch.io sitting in a queue. To check if tweets are coming in, head over to your database and look in the jobs
table:
Now let’s try processing the queue. Open up a new shell window, navigate to your app in Homestead, and pull the first item off the queue:
php artisan queue:work
If everything went right, you should see the text and id of the tweet in the console, from the handle()
method in the ProcessTweet
job.
Now that you have successfully connected to the Streaming API, it’s time to start building your featured tweets widget. Start by generating a Tweet
model and a corresponding database migration:
- php artisan make:model Tweet --migration
We’re going to add some properties to our model, so go ahead and set those now. Below is what your completed model and migration should look like - note that I’m using a string for the model ID, because Twitter IDs are massive.
<?php
namespace App;
use Illuminate\Database\Eloquent\Model;
class Tweet extends Model
{
protected $fillable = ['id','json','tweet_text','user_id','user_screen_name','user_avatar_url','public','approved'];
}
<?php
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
class CreateTweetsTable extends Migration
{
/**
* Run the migrations.
*
* @return void
*/
public function up()
{
Schema::create('tweets', function (Blueprint $table) {
$table->string('id');
$table->text('json');
$table->string('tweet_text')->nullable();
$table->string('user_id')->nullable();
$table->string('user_screen_name')->nullable();
$table->string('user_avatar_url')->nullable();
$table->boolean('approved');
$table->timestamps();
});
}
/**
* Reverse the migrations.
*
* @return void
*/
public function down()
{
Schema::drop('tweets');
}
}
Finally, migrate your database:
- php artisan migrate
Right now the ProcessTweet
job is just displaying some information about the tweet in the console. Update the handle
method so that it saves the tweets to the database:
public function handle()
{
$tweet = json_decode($this->tweet,true);
$tweet_text = isset($tweet['text']) ? $tweet['text'] : null;
$user_id = isset($tweet['user']['id_str']) ? $tweet['user']['id_str'] : null;
$user_screen_name = isset($tweet['user']['screen_name']) ? $tweet['user']['screen_name'] : null;
$user_avatar_url = isset($tweet['user']['profile_image_url_https']) ? $tweet['user']['profile_image_url_https'] : null;
if (isset($tweet['id'])) {
Tweet::create([
'id' => $tweet['id_str'],
'json' => $this->tweet,
'tweet_text' => $tweet_text,
'user_id' => $user_id,
'user_screen_name' => $user_screen_name,
'user_avatar_url' => $user_avatar_url,
'approved' => 0
]);
}
}
When that’s done, you can run the queue listener to empty out the queue and import the tweets into your database:
php artisan queue:listen
You should now be able to see some tweets in your database:
Thanks to Laravel’s auth scaffolding, this is probably the easiest step. Just run:
php artisan make:auth
For now your app will only display the featured tweets widget on the main landing page. Authenticated users are going to be allowed to approve or disapprove tweets, so they will receive all tweets (paginated). Visitors will only be able to see the five most recent approved tweets.
Pass the tweets into the existing welcome
view by updating the main route in your routes.php
file:
Route::get('/', function () {
if (Auth::check()) {
$tweets = App\Tweet::orderBy('created_at','desc')->paginate(5);
} else {
$tweets = App\Tweet::where('approved',1)->orderBy('created_at','desc')->take(5)->get();
}
return view('welcome', ['tweets' => $tweets]);
});
You’ll need to create three blade templates for the featured tweets widget, all stored in the resources/views/tweets
directory:
list.blade.php
will be the public-facing list of recent tweetslist-admin.blade.php
will be the admin-facing list of all tweetstweet.blade.php
will be a small partial common to both of the tweet listsThe list-admin
view is the most complex. The list is wrapped in a form and includes some radio inputs so that registered users can easily approve tweets.
Here are the three templates, in order:
// list.blade.php
@foreach($tweets as $tweet)
<div class="tweet">
@include('tweets.tweet')
</div>
@endforeach
// list-admin.blade.php
<form action="/approve-tweets" method="post">
{{ csrf_field() }}
@foreach($tweets as $tweet)
<div class="tweet row">
<div class="col-xs-8">
@include('tweets.tweet')
</div>
<div class="col-xs-4 approval">
<label class="radio-inline">
<input
type="radio"
name="approval-status-{{ $tweet->id }}"
value="1"
@if($tweet->approved)
checked="checked"
@endif
>
Approved
</label>
<label class="radio-inline">
<input
type="radio"
name="approval-status-{{ $tweet->id }}"
value="0"
@unless($tweet->approved)
checked="checked"
@endif
>
Unapproved
</label>
</div>
</div>
@endforeach
<div class="row">
<div class="col-sm-12">
<input type="submit" class="btn btn-primary" value="Approve Tweets">
</div>
</div>
</form>
{!! $tweets->links() !!}
// tweet.blade.php
<div class="media">
<div class="media-left">
<img class="img-thumbnail media-object" src="{{ $tweet->user_avatar_url }}" alt="Avatar">
</div>
<div class="media-body">
<h4 class="media-heading">{{ '@' . $tweet->user_screen_name }}</h4>
<p>{{ $tweet->tweet_text }}</p>
<p><a target="_blank" href="https://twitter.com/{{ $tweet->user_screen_name }}/status/{{ $tweet->id }}">
View on Twitter
</a></p>
</div>
</div>
With your blade templates ready, you can now bring them into your welcome
view:
@extends('layouts.app')
@section('content')
<div class="container">
<div class="row">
<div class="col-md-8 col-md-offset-2">
<div class="tweet-list">
@if(Auth::check())
@include('tweets.list-admin')
@else
@include('tweets.list')
@endif
</div>
</div>
</div>
</div>
@endsection
To get some very basic styling in your list, add the following CSS to your app.blade.php
file:
.tweet {
padding: 10px;
border: 1px solid #ccc;
border-radius: 5px;
margin-bottom: 20px;
}
With that done, you can now view the widget! Head to the browser and visit your app’s home page. If you’re not seeing any tweets, make sure to run the queue listener (php artisan queue:listen
) to process anything that might still be in it. An authorized user should see something like this (with a few more tweets… and less blurry):
The final step is to make the admin list from Step 14 functional. The form in the list-admin
template currently points to a nonexistent route. You need to add it to your routes.php
file. Inside that route we’ll do some basic logic to approve or disapprove tweets. Here’s how it should look:
Route::post('approve-tweets', ['middleware' => 'auth', function (Illuminate\Http\Request $request) {
foreach ($request->all() as $input_key => $input_val) {
if ( strpos($input_key, 'approval-status-') === 0 ) {
$tweet_id = substr_replace($input_key, '', 0, strlen('approval-status-'));
$tweet = App\Tweet::where('id',$tweet_id)->first();
if ($tweet) {
$tweet->approved = (int)$input_val;
$tweet->save();
}
}
}
return redirect()->back();
}]);
With that route in place, you should now be able to mark tweets as approved or disapproved. Try approving some, and then visit the page as an unauthenticated user. It should look something like this:
That’s it! You’ve connected your Laravel app to the Twitter Streaming API. However, it’s not quite production ready. There are a few other things you should consider:
Thanks for learning with the DigitalOcean Community. Check out our offerings for compute, storage, networking, and managed databases.
This textbox defaults to using Markdown to format your answer.
You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!
Working on improving health and education, reducing inequality, and spurring economic growth? We'd like to help.
Get paid to write technical tutorials and select a tech-focused charity to receive a matching donation.