How To Process Tweets in Real-Time with Laravel

This tutorial will show how to use the Twitter Streaming APIs to process tweets in real-time from a Laravel application. There are a variety of use cases for this: perhaps you want to auto-respond to mentions of your company, run a contest via Twitter, or create support tickets when users complain about your product. For this tutorial, we'll build a "featured tweets" widget to display approved tweets on our app's home page.

You might first look at the Twitter REST APIs, which many developers are familiar with. These APIs allow you to do a lot of cool things: read and write tweets on behalf of app users, manage followers and lists, and more. However, if you want to process tweets as they are sent, the REST APIs are very difficult to manage. Pagination and rate limiting become a logistical nightmare quickly.

Whereas the REST APIs force you to request information, the Streaming APIs feed information to you. You can read more about how it works in the documentation, but it's basically a pipe from Twitter to your application which delivers a continual flow of tweets.

Table of Contents

    By the end of the tutorial, you'll have a constantly updating collection of tweets in your app that you can approve or disapprove to show up in your home page. Here's a preview of what you'll end up with.

    Overview

    The Tools

    • Public Streaming API: There are three Streaming APIs, but the one we care about is the Public API - and more specifically, the 'filter' endpoint. This endpoint will deliver a stream of all public tweets, filtered by the keywords that you define.
    • Phirehose: The Phirehose library takes care of authentication and other implementation details for the Streaming API.
    • Homestead: Although you can build your app with your environment of choice, this tutorial will assume you are using Homestead.

    Notes

    • The Streaming APIs are treated specially by Twitter. You can only have one connection open from a single Twitter app.
    • Because of the special nature of the connection to the Streaming API, it's important that it is separate from the web-facing part of your app. In this tutorial, we'll be creating the connection with an Artisan console command.
    • Because the Streaming APIs are so finnicky, it's important not to do anything intensive with the tweets in the same process that is collecting them. In this tutorial you'll simply be placing the tweets onto a queue.
    • Be careful with the terms you are tracking. If you try to track all mentions of Taylor Swift, your app will probably melt.

    App Structure

    Here are the major pieces of the app that we'll be creating:

    • A class called TwitterStream which will extend Phirehose and place tweets onto the queue as they arrive
    • An Artisan command, ConnectToStreamingAPI, which will open a connection to the Streaming API with an instance of TwitterStream
    • A job class, ProcessTweet, which will contain the handler to process tweets when they are pulled off the queue
    • A Tweet Eloquent model
    • A few blade templates to display the tweet widget

    While reading through the tutorial, you can follow along with this Github repo. It has separate commits for each step:

    https://github.com/dabernathy89/Laravel-Twitter-Streaming-API-Demo

    Step 1: Create a New Laravel App

    Using the Laravel installer, create a new Laravel instance:

    laravel new twitter-stream-test

    Once that's done, add the app to your Homestead configuration file. Don't forget to edit your hosts file with whatever custom domain you added to the configuration. Boot up Homestead:

    homestead up --provision

    SSH into Homestead:

    homestead ssh

    Finally, cd into your app's folder.

    Step 2: Install Phirehose

    Next, you need to pull in the Phirehose library. Run composer require fennb/phirehose to grab the latest version. Phirehose doesn't support PSR-0 or PSR-4 autoloading, so you'll need to use Composer's classmap autoloading option. Add a line just after the database entry:

            "classmap": [
                "database",
                "vendor/fennb/phirehose/lib"
            ],

    Step 3: Create the ProcessTweet Job

    I mentioned in the overview that we will be putting all of the tweets from the Streaming API onto a queue. In Laravel 5.2, there are special Job classes that are meant for processing items in a queue. Your app will have a job called ProcessTweet which will be responsible for pulling tweets off the queue and doing something with them. You can create the job with a simple Artisan command:

    php artisan make:job ProcessTweet

    This will generate a Job class in your app/Jobs folder. You only need to make two adjustments to it for now.

    1. You know that this class will be processing a tweet (even though we haven't set that up yet). Pass a $tweet variable into the constructor and set it as a property.
    2. Do something with the tweet in the handle() method. For now you can just var_dump() some basic information from the tweet.
    <?php
    
    namespace App\Jobs;
    
    use App\Jobs\Job;
    use Illuminate\Queue\SerializesModels;
    use Illuminate\Queue\InteractsWithQueue;
    use Illuminate\Contracts\Queue\ShouldQueue;
    
    class ProcessTweet extends Job implements ShouldQueue
    {
        use InteractsWithQueue, SerializesModels;
    
        protected $tweet;
    
        /**
         * Create a new job instance.
         *
         * @return void
         */
        public function __construct($tweet)
        {
            $this->tweet = $tweet;
        }
    
        /**
         * Execute the job.
         *
         * @return void
         */
        public function handle()
        {
            $tweet = json_decode($this->tweet,true);
            var_dump($tweet['text']) . PHP_EOL;
            var_dump($tweet['id_str']) . PHP_EOL;
        }
    }
    

    Step 4: Create The TwitterStream Class

    To use Phirehose, you need to create a simple class that extends the base Phirehose class. This can go wherever you like, but I've just placed it directly in the app folder. Here's the full class:

    <?php
    
    namespace App;
    
    use OauthPhirehose;
    use App\Jobs\ProcessTweet;
    use Illuminate\Foundation\Bus\DispatchesJobs;
    
    class TwitterStream extends OauthPhirehose
    {
        use DispatchesJobs;
    
        /**
        * Enqueue each status
        *
        * @param string $status
        */
        public function enqueueStatus($status)
        {
            $this->dispatch(new ProcessTweet($status));
        }
    }

    A couple of things to note:

    • The class pulls in the DispatchesJobs trait to make it easy to push the tweet onto the queue.
    • There is only a single method, enqueueStatus, which will be called by Phirehose for each tweet.

    Step 5: Register the TwitterStream Class

    You need to register the TwitterStream class with the Laravel container so that it can pull in its dependencies properly. In the register method of your AppServiceProvider class, add the following:

            $this->app->bind('App\TwitterStream', function ($app) {
                $twitter_access_token = env('TWITTER_ACCESS_TOKEN', null);
                $twitter_access_token_secret = env('TWITTER_ACCESS_TOKEN_SECRET', null);
                return new TwitterStream($twitter_access_token, $twitter_access_token_secret, Phirehose::METHOD_FILTER);
            });

    You will also need to add use Phirehose; and use App\TwitterStream; at the top of your service provider. Don't worry about the environment variables for now - you'll create them shortly.

    Step 6: Create the Artisan Command

    Generate an Artisan command so that you can initiate the Streaming API connection over the command line:

    php artisan make:console ConnectToStreamingAPI

    This will generate the boilerplate console command class. Then you need to:

    • Update the command's signature and description
    • Pull in an instance of the TwitterStream class through the constructor
    • In the command's handle() method, finish configuring the Phirehose object, including your search terms, and open the connection

    Here's what the completed command looks like. It will pull in tweets that contain the keyword scotch_io:

    <?php
    
    namespace App\Console\Commands;
    
    use Illuminate\Console\Command;
    use App\TwitterStream;
    
    class ConnectToStreamingAPI extends Command
    {
        /**
         * The name and signature of the console command.
         *
         * @var string
         */
        protected $signature = 'connect_to_streaming_api';
    
        /**
         * The console command description.
         *
         * @var string
         */
        protected $description = 'Connect to the Twitter Streaming API';
    
        protected $twitterStream;
    
        /**
         * Create a new command instance.
         *
         * @return void
         */
        public function __construct(TwitterStream $twitterStream)
        {
            $this->twitterStream = $twitterStream;
    
            parent::__construct();
        }
    
        /**
         * Execute the console command.
         *
         * @return mixed
         */
        public function handle()
        {
            $twitter_consumer_key = env('TWITTER_CONSUMER_KEY', '');
            $twitter_consumer_secret = env('TWITTER_CONSUMER_SECRET', '');
    
            $this->twitterStream->consumerKey = $twitter_consumer_key;
            $this->twitterStream->consumerSecret = $twitter_consumer_secret;
            $this->twitterStream->setTrack(array('scotch_io'));
            $this->twitterStream->consume();
        }
    }
    

    You also need to register the command. Simply update the $commands property in the App\Console\Kernel class:

        protected $commands = [
            Commands\ConnectToStreamingAPI::class
        ];

    Step 7: Create a Twitter App

    Generate an app with Twitter, and grab the keys and access tokens. Add them to your .env file:

    TWITTER_CONSUMER_KEY=123
    TWITTER_CONSUMER_SECRET=123
    TWITTER_ACCESS_TOKEN=123
    TWITTER_ACCESS_TOKEN_SECRET=123

    Step 8: Configure Your Queue Driver

    You can set up the queue however you want, but for demonstration purposes, the database driver will work fine. To set that up, update your .env file:

    QUEUE_DRIVER=database

    By default, your app will look for a database with the name homestead. Don't forget to update this in your .env file if you want a separate database for your app:

    DB_DATABASE=twitterstreamtest

    You'll also need to run the following commands to prepare the database queue:

    php artisan queue:table

    php artisan migrate

    Step 9: Read some tweets!

    Now you should be ready to start processing some tweets. Run your Artisan commmand:

    php artisan connect_to_streaming_api

    You'll see some information about the connection in your console from Phirehose. It won't tell you every time a tweet is processed, but it will give you an update every once in a while on the status of the connection.

    Before too long, you should have some tweets about Scotch.io sitting in a queue. To check if tweets are coming in, head over to your database and look in the jobs table:

    Now let's try processing the queue. Open up a new shell window, navigate to your app in Homestead, and pull the first item off the queue:

    php artisan queue:work

    If you don't have anything in the queue yet, you can add your own by sending this tweet (be sure to open it in a new tab).

    If everything went right, you should see the text and id of the tweet in the console, from the handle() method in the ProcessTweet job.

    Step 10: Set Up the Tweet Model

    Now that you have successfully connected to the Streaming API, it's time to start building your featured tweets widget. Start by generating a Tweet model and a corresponding database migration:

    php artisan make:model Tweet --migration

    We're going to add some properties to our model, so go ahead and set those now. Below is what your completed model and migration should look like - note that I'm using a string for the model ID, because Twitter IDs are massive.

    <?php
    
    namespace App;
    
    use Illuminate\Database\Eloquent\Model;
    
    class Tweet extends Model
    {
        protected $fillable = ['id','json','tweet_text','user_id','user_screen_name','user_avatar_url','public','approved'];
    }
    <?php
    
    use Illuminate\Database\Schema\Blueprint;
    use Illuminate\Database\Migrations\Migration;
    
    class CreateTweetsTable extends Migration
    {
        /**
         * Run the migrations.
         *
         * @return void
         */
        public function up()
        {
            Schema::create('tweets', function (Blueprint $table) {
                $table->string('id');
                $table->text('json');
                $table->string('tweet_text')->nullable();
                $table->string('user_id')->nullable();
                $table->string('user_screen_name')->nullable();
                $table->string('user_avatar_url')->nullable();
                $table->boolean('approved');
                $table->timestamps();
            });
        }
    
        /**
         * Reverse the migrations.
         *
         * @return void
         */
        public function down()
        {
            Schema::drop('tweets');
        }
    }

    Finally, migrate your database:

    php artisan migrate

    Step 11: Make the ProcessTweet Job Useful

    Right now the ProcessTweet job is just displaying some information about the tweet in the console. Update the handle method so that it saves the tweets to the database:

        public function handle()
        {
            $tweet = json_decode($this->tweet,true);
            $tweet_text = isset($tweet['text']) ? $tweet['text'] : null;
            $user_id = isset($tweet['user']['id_str']) ? $tweet['user']['id_str'] : null;
            $user_screen_name = isset($tweet['user']['screen_name']) ? $tweet['user']['screen_name'] : null;
            $user_avatar_url = isset($tweet['user']['profile_image_url_https']) ? $tweet['user']['profile_image_url_https'] : null;
    
            if (isset($tweet['id'])) {
                Tweet::create([
                    'id' => $tweet['id_str'],
                    'json' => $this->tweet,
                    'tweet_text' => $tweet_text,
                    'user_id' => $user_id,
                    'user_screen_name' => $user_screen_name,
                    'user_avatar_url' => $user_avatar_url,
                    'approved' => 0
                ]);
            }
        }

    When that's done, you can run the queue listener to empty out the queue and import the tweets into your database:

    php artisan queue:listen

    You should now be able to see some tweets in your database:

    Step 12: Set Up Authentication

    Thanks to Laravel's auth scaffolding, this is probably the easiest step. Just run:

    php artisan make:auth

    Step 13: Pass Tweets Into the Welcome View

    For now your app will only display the featured tweets widget on the main landing page. Authenticated users are going to be allowed to approve or disapprove tweets, so they will receive all tweets (paginated). Visitors will only be able to see the five most recent approved tweets.

    Pass the tweets into the existing welcome view by updating the main route in your routes.php file:

    Route::get('/', function () {
        if (Auth::check()) {
            $tweets = App\Tweet::orderBy('created_at','desc')->paginate(5);
        } else {
            $tweets = App\Tweet::where('approved',1)->orderBy('created_at','desc')->take(5)->get();
        }
    
        return view('welcome', ['tweets' => $tweets]);
    });

    Step 14: Create The Blade Templates

    You'll need to create three blade templates for the featured tweets widget, all stored in the resources/views/tweets directory:

    • list.blade.php will be the public-facing list of recent tweets
    • list-admin.blade.php will be the admin-facing list of all tweets
    • tweet.blade.php will be a small partial common to both of the tweet lists

    The list-admin view is the most complex. The list is wrapped in a form and includes some radio inputs so that registered users can easily approve tweets.

    Here are the three templates, in order:

    // list.blade.php
    @foreach($tweets as $tweet)
        <div class="tweet">
            @include('tweets.tweet')
        </div>
    @endforeach
    // list-admin.blade.php
    <form action="/approve-tweets" method="post">
    {{ csrf_field() }}
    
    @foreach($tweets as $tweet)
        <div class="tweet row">
            <div class="col-xs-8">
                @include('tweets.tweet')
            </div>
            <div class="col-xs-4 approval">
                <label class="radio-inline">
                    <input
                        type="radio"
                        name="approval-status-{{ $tweet->id }}"
                        value="1"
                        @if($tweet->approved)
                        checked="checked"
                        @endif
                        >
                    Approved
                </label>
                <label class="radio-inline">
                    <input
                        type="radio"
                        name="approval-status-{{ $tweet->id }}"
                        value="0"
                        @unless($tweet->approved)
                        checked="checked"
                        @endif
                        >
                    Unapproved
                </label>
            </div>
        </div>
    @endforeach
    
    <div class="row">
        <div class="col-sm-12">
            <input type="submit" class="btn btn-primary" value="Approve Tweets">
        </div>
    </div>
    
    </form>
    
    {!! $tweets->links() !!}
    // tweet.blade.php
    <div class="media">
        <div class="media-left">
            <img class="img-thumbnail media-object" src="{{ $tweet->user_avatar_url }}" alt="Avatar">
        </div>
        <div class="media-body">
            <h4 class="media-heading">{{ '@' . $tweet->user_screen_name }}</h4>
            <p>{{ $tweet->tweet_text }}</p>
            <p><a target="_blank" href="https://twitter.com/{{ $tweet->user_screen_name }}/status/{{ $tweet->id }}">
                View on Twitter
            </a></p>
        </div>
    </div>

    Step 15: Show the Widget in Your Welcome View

    With your blade templates ready, you can now bring them into your welcome view:

    @extends('layouts.app')
    
    @section('content')
    <div class="container">
        <div class="row">
            <div class="col-md-8 col-md-offset-2">
    
                <div class="tweet-list">
                    @if(Auth::check())
                        @include('tweets.list-admin')
                    @else
                        @include('tweets.list')
                    @endif
                </div>
    
            </div>
        </div>
    </div>
    @endsection
    

    To get some very basic styling in your list, add the following CSS to your app.blade.php file:

            .tweet {
                padding: 10px;
                border: 1px solid #ccc;
                border-radius: 5px;
                margin-bottom: 20px;
            }

    With that done, you can now view the widget! Head to the browser and visit your app's home page. If you're not seeing any tweets, make sure to run the queue listener (php artisan queue:listen) to process anything that might still be in it. An authorized user should see something like this (with a few more tweets... and less blurry):

    Step 16: Add the Route for Approving Tweets

    The final step is to make the admin list from Step 14 functional. The form in the list-admin template currently points to a nonexistent route. You need to add it to your routes.php file. Inside that route we'll do some basic logic to approve or disapprove tweets. Here's how it should look:

    Route::post('approve-tweets', ['middleware' => 'auth', function (Illuminate\Http\Request $request) {
        foreach ($request->all() as $input_key => $input_val) {
            if ( strpos($input_key, 'approval-status-') === 0 ) {
                $tweet_id = substr_replace($input_key, '', 0, strlen('approval-status-'));
                $tweet = App\Tweet::where('id',$tweet_id)->first();
                if ($tweet) {
                    $tweet->approved = (int)$input_val;
                    $tweet->save();
                }
            }
        }
        return redirect()->back();
    }]);

    With that route in place, you should now be able to mark tweets as approved or disapproved. Try approving some, and then visit the page as an unauthenticated user. It should look something like this:

    Conclusion

    That's it! You've connected your Laravel app to the Twitter Streaming API. However, it's not quite production ready. There are a few other things you should consider:

    • You will likely want to configure Supervisor, which Laravel uses to monitor queue listeners, to monitor your Artisan command and restart it if it fails.
    • You'll want to handle the call to Supervisor into your deployment process.
    • Your search terms might change, and Phirehose provides a method for updating them without interrupting your script.

    Thanks to @mand0z, who's implemented the Phirehose library in Laravel, for answering a couple of questions for me.

    Daniel Abernathy

    1 post

    I'm a freelance web developer working primarily with WordPress. I live in Austin, TX with my amazing wife Emily and our chubby cat Banana.

    In addition to web development, I also enjoy hiking (I'm hitting the Appalachian Trail for the 3rd year in a row this summer), traveling, and Texas barbecue.