Build a Twitter Like Live Search Feed with Appbase.io and JavaScript

How to build a realtime feed with appbase.io and Javascript - a detailed tutorial post

Ever tried Twitter Live Search and wondered how it works? On the surface, you search for a #hashtag or a keyword and Twitter shows you a continuous stream of results, including new tweets that appear after you searched!

Twitter Live Search for 'Trump'

Image: We search for 'Trump' in the Twitter Live Mode. Notice the popup showing 20 new updates while we are on the screen.

How is Twitter able to return a dynamic result feed for any searched keyword? One way to implement a Twitter Search style live feed is to return the original search results from a database (SQL, ElasticSearch, Mongo etc.) and then have a separate feed component using a realtime channel (think socket.io).

Live feed implementation

Image: Architecture for a naive realtime feed

We can't know for sure how Twitter internally implements the realtime feed, but the approach described above, while easy to get started, requires scaling two different components and suffers from data consistency issues while offering an okay-ish O(MxN) time complexity (where M=data insertions, N=open keyword searches).

In this post, we describe a scalable approach to building realtime feeds using Meetup's RSVPs as an example with appbase.io, a hosted data streams API. On the backend, we use a Node.JS worker to fetch Meetup's RSVP API and insert them into an appbase.io app. On the frontend, we use jQuery (an alternate post with React.JS instead coming soon) to create the feed UI and query appbase.io in realtime with different keyword filters.

Enter meetupblast!

Meetupblast shows a live feed of meetup RSVPs searchable by cities and topics. Like Twitter Live Search, it returns a dynamic feed of RSVP data by the selected cities and topics. It's a great way to find cool meetups other people are attending in your city.

(http://appbaseio.github.io/meetupblast/)

Image: Screenshot of meetupblast, tap on the image for redirection to the live demo.

Or continue reading to see how we built this.

Key Ingredients

The meetupblast recipe can be broken into two key pieces: the backend worker and the user interface.

  1. Backend Worker:
    • We use the Meetup's long-polling RSVP stream endpoint to get realtime RSVPs.
      Note: This demo wouldn't have been possible without meetup supporting an API endpoint to extract the RSVP information. So, cheers to them!
    • We then store this data in appbase.io, which provides us a convenient way to query both historical data and realtime feeds - we call it the streaming DB ;)
  2. User Interface:
    • Querying appbase.io for Meetup RSVP feeds by cities and topics.
    • The UI / UX logic maintained with a jQuery frontend. And we use typeahead for displaying the search interface.

What is meetupblast?

Meetupblast shows live RSVPs from people across the globe. It has some neat filters that allow you to select cities and the topics of interest, to see what meetups are happening in your city.

showing a filtered view

Image: Let's check for RSVPs in the all the singles tagged meetups in San Francisco.

Now when a person RSVPs for a singles meetup, you would see it show up on the meetupblast feed. Now don't get any ideas for stalking unsuspecting people. It's just a cool way to find the most happening meetups!

A fun fact: While building this app, we noticed that there are about 120 RSVPs on meetup.com every minute, that's 2 every second. Or in 30 days, we would see over 5 million RSVPs - that's a good data set size to query in realtime.

Deep Dive

Now that we know what meetupblast does, let's get into the meat of how the app works.

Backend Worker

Our backend worker is a simple Node.JS code that keeps running forever on a DigitalOcean droplet.

Let's see how our worker consumes meetup RSVP data and then stores it into appbase.io.

http.get(meetup_url, function(res) {
    res.on('data', function(chunk) { // 'data' event handler is called everytime there is a new meetup RSVP
        var data = JSON.parse(chunk);
        meetup_data.push(data);  // capture the RSVP responses in an array
    });
});
appbaseRef.index({
    type: DATA_TABLE,            // collection to store the data into
    body: meetup_data[counter]
}).on('data', function(res) {
    console.log("successfully indexed one meetup RSVP");
})

Meetup provides us a JSON object for every RSVP. We then write this data into appbase.io as soon as it arrives. appbase.io is built as a streaming layer on ElasticSearch, and provides a continuous query interface for streaming JSON results.

An RSVP JSON data looks like this:

"visibility": "public",
"response": "yes",
"guests": 0,
"member": {
    "member_id": 185034988,
    "photo": "http://photos1.meetupstatic.com/photos/member/d/0/0/0/thumb_248333248.jpeg",
    "member_name": "Wilma"
},
"rsvp_id": 1566804180,
"mtime": 1440266230993,
"event": {
    "event_name": "Wednesday Badminton @ Flying Dragon",
    "event_id": "224809211",
    "time": 1440630000000,
    "event_url": "http://www.meetup.com/Richmond-Hill-Friends/events/224809211/"
},
"group": {
    "group_topics": [
    {
        "urlkey": "social",
        "topic_name": "Social"
    },
    {
        "urlkey": "board-games",
        "topic_name": "Board Games"
    },
    {
        "urlkey": "movie-nights",
        "topic_name": "Movie Nights"
    }
    ],
"group_city": "Richmond Hill",
"group_country": "ca",
"group_id": 1676043,
"group_name": "Richmond Hill Friends",
"group_lon": -79.4,
"group_urlname": "Richmond-Hill-Friends",
"group_state": "ON",
"group_lat": 43.84
}

User Interface

The User Interface is a small frontend that queries appbase.io for realtime meetups based on specific cities and tags and displays them in a neat list.

Image: Showing meetups by city and topic tags - try it live here.

The app directory structure looks like this:

meetupblast/
|_ _ _ _ app/
|        |_ _ _ _ app.js
|        |_ _ _ _ request.js
|        |_ _ _ _ variable.js
|_ _ _ _ index.html

The codebase can be accessed at the meetupblast github repo.

app.js

This is the entry point of the codebase and brokers responsibility to variable.js and request.js.

request.js

Creates a meetup_request prototype which is responsible for three different queries.

  1. Fetching the list of the 1000 most popular cities and topic where meetups happen,
  2. A realtime query fetching live meetup data from the selected cities and topics,
  3. Pagination showing older RSVPs when a user scrolls to the bottom of the meetup feed.

variable.js

Instantiates a meetup prototype object which is responsible for the tag selection logic when a user ticks a city / topic and displaying a new meetup RSVP inside the feed.

A sneakpeak into continuous queries

Here's a sneakpeak into how the continuous query for fetching live meetup data works.

We use the appbase-js library (available via bower and npm) for making continuous queries over the meetup feed data.

appbaseRef = new Appbase({
    url: "https://qz4ZD8xq1:a0edfc7f-5611-46f6-8fe1-d4db234631f3@scalr.api.appbase.io",
    appname: "meetup2"
})
/* A query object for meetups in the cities 'San Francisco' and 'Boston' */
var $composeQuery = {
  "filtered": {
    "filter" : {
      "terms" : { 
        "group_city_simple" : ["San Francisco", "Boston"]
      }
    }
  }
}
/* A query request to fetch initial results */
appbaseRef.search({
  type: "meetup",
  size: 100,
  body: {
    query: $composeQuery
  }
}).on('data', function(res) {
  // initial matches are returned in the array 'res.hits.hits'
  console.log("initial search results: ", res.hits.hits);
});

/* Same query request for streaming realtime results */
appbaseRef.searchStream({
  type: "meetup",
  body: {
    query: $composeQuery
  }
}).on('data', function(res) {
    // realtime updates with one new RSVP
    console.log("new data feed: ", res);
})

search() and searchStream() are alter-egos, the former returns the initial query matches and the latter streams RSVP updates in realtime.

Summary

We created meetupblast as a way to find happening meetups by looking up what other people are RSVPing to.

  1. We use the Meetup's long-polling API to get the live data of meetups.
  2. We use appbase.io to transform this into queryable live data.
  3. We put together a small frontend that allows neat filtering based on cities and topics.

Without further ado, here are the important links: meetup worker code, meetupblast and the demo.

For the curious, try building a date filter which only shows RSVP feeds between a specific date range.

Siddharth Kothari

Siddharth is the co-founder and CEO at appbase.io. He has been a Javascript aficionado for four years now.

In his spare time, he likes endurance sports and contributing to open-source projects: currently docbase and dejaVu.