Introduction

Have you ever tried to search a site and got disappointed at the page having to reload? Or better still, you have to wait for an awfully long time while looking at a preloading spinning? I can bet that sucks.

Do you know you as a developer can make near real-time search engines on your sites?

Elasticsearch is a distributed, RESTful search and analytics engine capable of solving a growing number of use cases. Elasticsearch is built on top of Apache Lucene, which is a high-performance text search engine library.

In today's lesson, you will learn how to build a real-time search engine using Node.js, Elasticsearch, and Vue.js. As a result, a basic comprehension of Vue.js and Node.js (Express) is needed to follow this tutorial.

Getting Started

Let's get started with setting up the environment for this lesson. Since you will use Node.js, the easiest way to get started is to create a new folder and run npm init. Create a new folder called elastic-node, change directory into the new folder and then run npm init:

//create a new directory called elastic-node
mkdir elastic-node
//change directory to the new folder created
cd elastic-node
//run npm init to create a package.json file
npm init

The above commands take you through the process of creating a package.json file, which is required to run any Node.js library. Next, you need to install libraries which will be needed for the real-time search engine. The libraries needed are:

  • Express: This library will run our server.
  • Body-parser: This library works with Express to parse body requests.
  • Elasticsearch: This is the official Node.js library for Elasticsearch, which is the engine on which the real-time search will be built.

To install these libraries, run:

npm install express body-parser elasticsearch

Now the first part of your environment is set up. However, Elasticsearch itself is missing from your setup. You will need to install Elasticsearch itself. There are different ways to install Elasticsearch. If you are using a Debian Linux operating system, you could just download the .deb file and install using dpkg.

//download the deb package
curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.4.deb
//install the deb package using dpkg
sudo dpkg -i elasticsearch-5.6.4.deb

For other distributions/operating systems, you can find a guide on how to install Elasticsearch here.

Elasticsearch is not started automatically after installation. Elasticsearch can be started and stopped using the service command:

// start the Elasticsearch service
sudo -i service elasticsearch start
// stop the Elasticsearch service
sudo -i service elasticsearch stop

To configure Elasticsearch to start automatically when the system boots up, run:

//reload the systemctl daemon
sudo /bin/systemctl daemon-reload
// enable elastic search so it can be called as a service
sudo /bin/systemctl enable elasticsearch.service

After running the command above, you can start and stop Elasticsearch by running:

// start the Elasticsearch service
sudo systemctl start elasticsearch.service
// stop the Elasticsearch service
sudo systemctl stop elasticsearch.service

To check the status of Elasticsearch:

// check status of Elasticsearch
sudo service elasticsearch status

Note: A great tool while working with Elasticsearch is the Google Chrome Elastic toolbox. It helps you to take a quick look at your indexes and documents.

Indexing Data In Elasticsearch

Create a data.js file in your root folder and add:

//data.js
//require the Elasticsearch librray
const elasticsearch = require('elasticsearch');
// instantiate an Elasticsearch client
const client = new elasticsearch.Client({
   hosts: [ 'http://localhost:9200']
});
// ping the client to be sure Elasticsearch is up
client.ping({
     requestTimeout: 30000,
 }, function(error) {
 // at this point, eastic search is down, please check your Elasticsearch service
     if (error) {
         console.error('Elasticsearch cluster is down!');
     } else {
         console.log('Everything is ok');
     }
 });

Let me explain what you have done in the code-block above: first, you required the Elasticsearch library and set up a new Elasticsearch client passing in an array of one host. If you notice, the host is http://localhost:9200. This is because, by default, Elasticsearch listens on port 9200. Next, you ping the Elasticsearch client to be sure the server is up. If you run node data.js you should get a message that says Everything is ok.

Understanding Indexes

Unlike normal databases, an Elasticsearch index is a place to store related documents. For example, you will create an index called scotch.io-tutorial to store data of type cities_list. This is how it's done in Elasticsearch:

//data.js
// create a new index called scotch.io-tutorial. If the index has already been created, this function fails safely
client.indices.create({
      index: 'scotch.io-tutorial'
  }, function(error, response, status) {
      if (error) {
          console.log(error);
      } else {
          console.log("created a new index", response);
      }
});

Add this piece of code after the ping function you had written before. Now, run node data.js again, you should get two messages:

  • Everything is okay
  • Created a new index (with the response from Elasticsearch)

Adding Documents To Indexes

The Elasticsearch API makes it easy for documents to be added to already created indexes. It is as simple as:

// add a data to the index that has already been created
client.index({
     index: 'scotch.io-tutorial',
     id: '1',
     type: 'cities_list',
     body: {
         "Key1": "Content for key one",
         "Key2": "Content for key two",
         "key3": "Content for key three",
     }
 }, function(err, resp, status) {
     console.log(resp);
 });

The code block above is explanatory. The body refers to the document you want to add to the scotch.io-tutorial index, while the type is more of a category. However, note that if the id key is omitted, Elasticsearch will auto-generate one.

However, in this lesson, your document will be a list of all the cities in the world. If you are to add each city one by one, It will take days, if not weeks to completely index all. Luckily, Elasticsearch has a bulk function used to process bulk data.

First, grab the JSON file containing all cities in the world here and save into your root folder as cities.json

It's time to use the bulk API to import the large chunk of data we have:

//data.js
// require the array of cities that was downloaded
const cities = require('./cities.json');
// declare an empty array called bulk
var bulk = [];
//loop through each city and create and push two objects into the array in each loop
//first object sends the index and type you will be saving the data as
//second object is the data you want to index
cities.forEach(city =>{
   bulk.push({index:{ 
                 _index:"scotch.io-tutorial", 
                 _type:"cities_list",
             }          
         })
  bulk.push(city)
})
//perform bulk indexing of the data passed
client.bulk({body:bulk}, function( err, response  ){ 
         if( err ){ 
             console.log("Failed Bulk operation".red, err) 
         } else { 
             console.log("Successfully imported %s".green, cities.length); 
         } 
}); 

Here, you have looped through all the cities in your JSON file, and at each loop, you append an object with the index and type of the document you will be indexing. Notice that there are two pushes to the array during the loop? This is because the bulk API expects an object containing the index definition first, then the document you want to index. For more information on that, you can check here

Next, you called the client.bulk function passing in the new bulk array as the body. This indexes all your data into Elasticsearch with the index of scotch.io-tutorial and type cities_list.

Express To The Mix

Your Elasticsearch instance is up and running, and you can connect with it using Node.js. It's time to use Express to serve a landing page and use the setup you have running so far.

Create a file called index.js and add:

//index.js
//require the Elasticsearch librray
const elasticsearch = require('elasticsearch');
// instantiate an elasticsearch client
const client = new elasticsearch.Client({
   hosts: [ 'http://localhost:9200']
});
//require Express
const express = require( 'express' );
// instanciate an instance of express and hold the value in a constant called app
const app     = express();
//require the body-parser library. will be used for parsing body requests
const bodyParser = require('body-parser')
//require the path library
const path    = require( 'path' );

// ping the client to be sure Elasticsearch is up
client.ping({
     requestTimeout: 30000,
 }, function(error) {
 // at this point, eastic search is down, please check your Elasticsearch service
     if (error) {
         console.error('elasticsearch cluster is down!');
     } else {
         console.log('Everything is ok');
     }
 });


// use the bodyparser as a middleware  
app.use(bodyParser.json())
// set port for the app to listen on
app.set( 'port', process.env.PORT || 3001 );
// set path to serve static files
app.use( express.static( path.join( __dirname, 'public' )));
// enable CORS 
app.use(function(req, res, next) {
  res.header("Access-Control-Allow-Origin", "*");
  res.header('Access-Control-Allow-Methods', 'PUT, GET, POST, DELETE, OPTIONS');
  res.header("Access-Control-Allow-Headers", "Origin, X-Requested-With, Content-Type, Accept");
  next();
});

// defined the base route and return with an HTML file called tempate.html
app.get('/', function(req, res){
  res.sendFile('template.html', {
     root: path.join( __dirname, 'views' )
   });
})

// define the /search route that should return elastic search results 
app.get('/search', function (req, res){
  // declare the query object to search elastic search and return only 200 results from the first result found. 
  // also match any data where the name is like the query string sent in
  let body = {
    size: 200,
    from: 0, 
    query: {
      match: {
          name: req.query['q']
      }
    }
  }
  // perform the actual search passing in the index, the search query and the type
  client.search({index:'scotch.io-tutorial',  body:body, type:'cities_list'})
  .then(results => {
    res.send(results.hits.hits);
  })
  .catch(err=>{
    console.log(err)
    res.send([]);
  });

})
// listen on the specified port
app .listen( app.get( 'port' ), function(){
  console.log( 'Express server listening on port ' + app.get( 'port' ));
} );

Looking at the code above, notice you have:

  • Required the Express, body-parser and path libraries.

  • Set a new instance of Express to the constant called app.

  • Set the app to use the bodyParser middleware.

  • Set the static part of the app to a folder called public (I'm yet to create this folder).

  • Defined a middleware which adds CORS header to the app.

  • Defined a GET route for the root URL of the app represented by / and in this route, I returned a file called template.html which is in the views folder (I'm also yet to create this folder and the file template.html)

  • Defined a GET route for the /search URL of the app which uses a query object to search for the match of the data passed to it via the query string. The main search query is included within the query object. You can add different search queries to this object. For this query, you add a key with the query and return an object telling it that the name of the document you are looking for should match req.query['q'].

    Besides the query object, the search body can contain other optional properties, including size and from. The size property determines the number of documents to be included in the response. If this value is not present, by default ten documents are returned. The from property determines the starting index of the returned documents. This is useful for pagination.

Understanding The Search API Response

If you were to log out the response from the search API, it includes a lot of information.

{ took: 88,
timed_out: false,
_shards: { total: 5, successful: 5, failed: 0 },
hits:
{ total: 59,
 max_score: 5.9437823,
 hits:
  [ {"_index":"scotch.io-tutorial",
  "_type":"cities_list",
  "_id":"AV-xjywQx9urn0C4pSPv",
  "_score":5.9437823,"
  _source":{"country":"ES","name":"A Coruña","lat":"43.37135","lng":"-8.396"}},
    [Object],
...
    [Object] ] } }

The response includes a took property for the number of milliseconds it took to find the results, timed_out, which is only true if no results were found in the maximum allowed time, _shards for information about the status of the different nodes (if deployed as a cluster of nodes), and hits, which includes the search results.

Within the hits property, we have an object the following properties:

total shows the total number of matched items.

max_score is the maximum score of the found items.

hits is an array that includes the found items.

The explanation above is the reason in the search route, you returned response.hits.hits, which houses the documents found.

Creating The HTML Template

First, create two new folders in your root folder named views and public which you referenced in the section above. Next, create a file called template.html in the views folder and paste:

<!-- template.html -->
<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/vue"></script>
<div class="container" id="app">
    <div class="row">
        <div class="col-md-6 col-md-offset-3">
            <h1>Search Cities around the world</h1>
        </div>
    </div>
    <div class="row">
        <div class="col-md-4 col-md-offset-3">
            <form action="" class="search-form">
                <div class="form-group has-feedback">
                    <label for="search" class="sr-only">Search</label>
                    <input type="text" class="form-control" name="search" id="search" placeholder="search" v-model="query" >
                    <span class="glyphicon glyphicon-search form-control-feedback"></span>
                </div>
            </form>
        </div>
    </div>
    <div class="row">
        <div class="col-md-3" v-for="result in results">
            <div class="panel panel-default">
                <div class="panel-heading">
                <!-- display the city name and country  -->
                    {{ result._source.name }}, {{ result._source.country }} 
                </div>
                <div class="panel-body">
                <!-- display the latitude and longitude of the city  -->
                    <p>lat:{{ result._source.lat }}, long: {{ result._source.lng }}.</p>
                </div>
            </div>
        </div>
    </div>
</div>
<!--- some styling for the page -->
<style>
    .search-form .form-group {
        float: right !important;
        transition: all 0.35s, border-radius 0s;
        width: 32px;
        height: 32px;
        background-color: #fff;
        box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset;
        border-radius: 25px;
        border: 1px solid #ccc;
    }

    .search-form .form-group input.form-control {
        padding-right: 20px;
        border: 0 none;
        background: transparent;
        box-shadow: none;
        display: block;
    }

    .search-form .form-group input.form-control::-webkit-input-placeholder {
        display: none;
    }

    .search-form .form-group input.form-control:-moz-placeholder {
        /* Firefox 18- */
        display: none;
    }

    .search-form .form-group input.form-control::-moz-placeholder {
        /* Firefox 19+ */
        display: none;
    }

    .search-form .form-group input.form-control:-ms-input-placeholder {
        display: none;
    }

    .search-form .form-group:hover,
    .search-form .form-group.hover {
        width: 100%;
        border-radius: 4px 25px 25px 4px;
    }

    .search-form .form-group span.form-control-feedback {
        position: absolute;
        top: -1px;
        right: -2px;
        z-index: 2;
        display: block;
        width: 34px;
        height: 34px;
        line-height: 34px;
        text-align: center;
        color: #3596e0;
        left: initial;
        font-size: 14px;
    }
</style>

In the code snippet above, there are two main sections which are:

  • HTML code: In this section, you started by requiring three different libraries which are 1.) Bootstrap CSS, for styling the page. 2.) Axios js, for making HTTP requests to our server and 3.) Vue.js, a minimalistic framework which you will use for our view.

  • CSS code: Here, you have applied styling to make the search input hide and reveal itself once you hover over the search icon.

    Next, there is an input for the search box which you assigned its v-model to query (this will be used by Vue.js). After this, you looped through all our results (this loop and the result variable will be made available by Vue.js). Note that while looping here, you had to access the_source property of your data. This should look familiar based on the response that elastic search returns.

Run node index.js command, browse to http://localhost:3001/, you will see:

Next, add a script tag in your template.html file, add:

//template.html
// create a new Vue instance
var app = new Vue({
    el: '#app',
    // declare the data for the component (An array that houses the results and a query that holds the current search string)
    data: {
        results: [],
        query: ''
    },
    // declare methods in this Vue component. here only one method which performs the search is defined
    methods: {
        // make an axios request to the server with the current search query
        search: function() {
            axios.get("http://127.0.0.1:3001/search?q=" + this.query)
                .then(response => {
                    this.results = response.data;

                })
        }
    },
    // declare Vue watchers
    watch: {
        // watch for change in the query string and recall the search method
        query: function() {
            this.search();
        }
    }

})

Vue.js code: In this section, you declared a new instance of Vue, mounting it on the element with the id of app. You declared data properties which include 1.) query which you had attached to the search input, and 2.) results which is an array of all results found.

In the methods quota, you have just one function called search, which triggers a GET request to the search route passing along the current input in the search box. This in turns returns a response that is then looped in the HTML code block.

Finally, you use what is called watchers in Vue.js, which performs an action anytime data being watched for changes. Here, you are watching for a change in the query data, and once it changes, the search method is fired.

If you re-run the node index.js command now and browse to http://localhost:3001/, it should work as seen:

Searching From The Client Side

What if I do not want to send requests to my server every time a search occurs? Can I search the Elasticsearch engine directly from the client side? YES.

While the above approach works, some developers might not be comfortable with always hitting their servers for every search term, while some feel it's more secure to search from the server side.

However, it is possible to search from the client side. Elasticsearch offers a browser build which can make searches. Let me take you through a quick sample.

First, add a new route to your Express file and restart your server:

//index.js
// decare a new route. This route serves a static HTML template called template2.html
app.get('/v2', function(req, res){
  res.sendFile('template2.html', {
     root: path.join( __dirname, 'views' )
   });
})

In this code block above, you have created a new route for the URL at /v2 and all you did in this route is to return a static HTML file called template2.html which will be created soon.

Next, you need to download the client library for Elasticsearch here. After downloading, extract and copy elasticsearch.min.js to the public folder in your application root.

Note: It is important to know if you try to connect with the Elasticsearch engine from the client side, you might experience CORS issue. To solve this issue, locate your Elasticsearch config file (For Debian/Ubuntu, it can be found at /etc/elasticsearch/elasticsearch.yml. For other operating systems, find out where it is located here and add the following to the bottom of the file:

#/etc/elasticsearch/elasticsearch.yml

http.cors.enabled : true
http.cors.allow-origin : "*"

After that is done, restart your Elasticsearch instance

// restart the Elasticsearch service
sudo service elasticsearch restart

Next, create a file called template2.html in your views folder and add:

<!-- template2.html -->
<link rel="stylesheet" type="text/css" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css">
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script src="https://cdn.jsdelivr.net/npm/vue"></script>
<div class="container" id="app">
    <div class="row">
        <div class="col-md-6 col-md-offset-3">
            <h1>Search Cities around the world</h1>
        </div>
    </div>
    <div class="row">
        <div class="col-md-4 col-md-offset-3">
            <form action="" class="search-form">
                <div class="form-group has-feedback">
                    <label for="search" class="sr-only">Search</label>
                    <input type="text" class="form-control" name="search" id="search" placeholder="search" v-model="query" >
                    <span class="glyphicon glyphicon-search form-control-feedback"></span>
                </div>
            </form>
        </div>
    </div>
    <div class="row">
        <div class="col-md-3" v-for="result in results">
            <div class="panel panel-default">
                <div class="panel-heading">
                <!-- display the city name and country  -->
                    {{ result._source.name }}, {{ result._source.country }} 
                </div>
                <div class="panel-body">
                <!-- display the latitude and longitude of the city  -->
                    <p>lat:{{ result._source.lat }}, long: {{ result._source.lng }}.</p>
                </div>
            </div>
        </div>
    </div>
</div>
<script src="/elasticsearch.min.js"></script>
<style>
    .search-form .form-group {
        float: right !important;
        transition: all 0.35s, border-radius 0s;
        width: 32px;
        height: 32px;
        background-color: #fff;
        box-shadow: 0 1px 1px rgba(0, 0, 0, 0.075) inset;
        border-radius: 25px;
        border: 1px solid #ccc;
    }

    .search-form .form-group input.form-control {
        padding-right: 20px;
        border: 0 none;
        background: transparent;
        box-shadow: none;
        display: block;
    }

    .search-form .form-group input.form-control::-webkit-input-placeholder {
        display: none;
    }

    .search-form .form-group input.form-control:-moz-placeholder {
        /* Firefox 18- */
        display: none;
    }

    .search-form .form-group input.form-control::-moz-placeholder {
        /* Firefox 19+ */
        display: none;
    }

    .search-form .form-group input.form-control:-ms-input-placeholder {
        display: none;
    }

    .search-form .form-group:hover,
    .search-form .form-group.hover {
        width: 100%;
        border-radius: 4px 25px 25px 4px;
    }

    .search-form .form-group span.form-control-feedback {
        position: absolute;
        top: -1px;
        right: -2px;
        z-index: 2;
        display: block;
        width: 34px;
        height: 34px;
        line-height: 34px;
        text-align: center;
        color: #3596e0;
        left: initial;
        font-size: 14px;
    }
</style>

Next, add a script tag in your template2.html file and add:

//template2.html
// instantiate a new Elasticsearch client like you did on the client
var client = new elasticsearch.Client({
    hosts: ['http://127.0.0.1:9200']
});
// create a new Vue instance
var app = new Vue({
    el: '#app',
    // declare the data for the component (An array that houses the results and a query that holds the current search string)
    data: {
        results: [],
        query: ''
    },
    // declare methods in this Vue component. here only one method which performs the search is defined
    methods: {
        // function that calls the elastic search. here the query object is set just as that of the server.
        //Here the query string is passed directly from Vue
        search: function() {
            var body = {
                    size: 200,
                    from: 0,
                    query: {
                        match: {
                            name: this.query
                        }
                    }
                }
                // search the Elasticsearch passing in the index, query object and type
            client.search({ index: 'scotch.io-tutorial', body: body, type: 'cities_list' })
                .then(results => {
                    console.log(`found ${results.hits.total} items in ${results.took}ms`);
                    // set the results to the result array we have 
                    this.results = results.hits.hits;
                })
                .catch(err => {
                    console.log(err)

                });


        }
    },
    // declare Vue watchers
    watch: {
        // watch for change in the query string and recall the search method
        query: function() {
            this.search();
        }
    }

})

The HTML and JavaScript snippet above is very identical to the one in the section above. The only differences are:

  • You did not require Axios, you required elasticsearch.js instead.
  • At the top of the script tag, you initiated the Elasticsearch client as it was done on the server-side.
  • The search method does not perform an HTTP request, but rather searches the Elasticsearch engine as done in the search route on the server side.

If you browse to http://localhost:3001/v2, It should work as shown:

Conclusion

In this lesson, you have gotten a grasp of how to index data in Elasticsearch and searching data in Elastic search. Also, you have learned how to implement a real-time search using the client library for Elasticsearch. What are your thoughts about Elasticsearch? Let's discuss them in the comment section.