Building a Slack Clone in Meteor.js: Real-Time Data

Free Course

Build Your First Node.js Website

Node is a powerful tool to get JavaScript on the server. Use Node to build a great website.

This is the second of a five-part series on building a Slack clone using Meteor. The aim of these tutorials are not just for you to blindly follow instructions, but it’s our hope that you’ll understand the thought process and reasoning behind the architecture.

If you followed along with our previous article, you’d understand the killer features of Meteor, installed it on your machine and created the layout and styles of our apps using templates. We have also explained what Meteor does behind the scenes.

In this lesson, we will code the beef of the application and produce a fully-functional, live-updating chat room.
to

Data Storage

First, we need to store our messages so they are persistent – so we can access them later. Doing so will also allow other clients connecting to our application to receive the updates in real time. It wouldn’t be much fun if you just talk to yourself, would it?

To store our data, we need a database. The officially supported data storage system used in Meteor is MongoDB, although there are many contributed packages for MySQL (numtel:meteor-mysql), Postgres (austinrivas:meteor-postgresql, meteor-stream:meteor-postgres) and many more. You can also suggest additional database support on the Meteor Roadmap, there’re ones for SQL and Redis specifically.

MongoDB fits well with what we are doing, and here are some reasons why.

MongoDB_Logo

Mongo vs SQL

Mongo is a NoSQL document database. A Mongo database consists of collections, which are similar to tables in SQL. A collection is literally a collection of documents, similar to a SQL table being a collection of rows.

But SQL has a flat key-value structure, Mongo allows much more flexibility because any valid JSON file* will make up a valid document. This means you can store nested structures – arrays inside objects, which are inside another array inside another object etc. – directly into the collection as a document (although heavy nesting is not necessarily a good idea!).

Because we model the messages, the users, the chatrooms all as objects (which may be nested), storing it in JSON format is a breeze. If you were to do this in SQL, you might have to create many tables and configure many relationships; there may also be performance issues as “joins are expensive”.

*Mongo actually stores your document as BSON, or Binary JSON, which can be viewed, in part, as an extension to JSON that allows for more data types, such as Date.

Object-Relational Mapping

Usually, when you work with a relational database in JavaScript, you’d interact with the database through an ORM (Object-relational mapping), and not with the database directly. This is because, as I said, the table structure of relational databases are flat, but for object-orientated JavaScript, we need to work with objects, which can have heavily-nested properties. The ORM will provide methods for us to query the database indirectly, saving us having to do complicated joins and also improve performance if used correctly.

If you’ve worked on a project involving Node.js or io.js with PostgreSQL, MySQL, MariaDB, SQLite or MSSQL, you might have come across the Sequelize ORM. If you’ve worked on the Laravel framework in PHP, you’d be greatly appreciative of their powerful Eloquent ORM.

Sequelize logo

However, because we are working with a document database, all we have to do is simply make the object, and store the object, as-is, into our collection. Getting information is also just as easy, take the object and parse it without any transformation. Nested objects are fine without extra configurations, so there are no need for an ORM. This greatly reduces the complexity and the amount of code you need to write.

Schema-less

Mongo is also schema-less. So if you’d like, you can insert objects with different properties into the collection and it will store it for you. For example, if you want to add an extra field to you messages; traditionally, you’d have to create a new column in your SQL table and fill the other entries with an empty string or NULL. Using MongoDB means you can add a property to one document without it affecting the other documents in your collection.

This does not mean Mongo cannot have a schema – a very common misconception! In fact, the Simple Schema Meteor package is one of the most popular packages, and provides a schema for Meteor collections. So Mongo is schema-less because a schema is not required or imposed onto the collections, not because it cannot have one.

Right Tools for the Right Job

MongoDB is by no means the answer for everything. Some people jump on the bandwagon and use MongoDB for everything – this is not the way to go!

The reason why I made you read all that text is because we must remember – there’s never a data storage system that can cater for everything, and you should pick the right tools for the right job. If your data structure is highly relational (meaning each field has many relations with other fields), then maybe SQL will be a better fit.

Collections

In MongoDB, our documents are stored inside collections. Defining a new collection is easy:


Messages = new Mongo.Collection("messages");

One line is all it takes to create a new collection. But where do we put this line? Remember that one of the seven principles of Meteor is database everywhere, this means our collection should be available on both the client and the server.

Jog your memory back to the last article – if we want a file to be available on both client and server, we can actually store that file anywhere apart from inside the special directories (client/, server/, public/, private/, test/). It’s my personal preference, or dare I say convention, to store it inside a collections/ directory on the application root directory.

collections/messages.js


Messages = new Mongo.Collection("messages");

Now, we can open up another terminal and navigate to the root of the application. There, we will open up a Mongo shell


$ meteor mongo
MongoDB shell version: 2.6.7
connecting to: 127.0.0.1:3001/meteor

And we can see that currently, the database is empty.


meteor:PRIMARY> show dbs
admin   (empty)
local   0.063GB
meteor  (empty)

That’s because Meteor doesn’t create the collection unless we are actually using it. But to test that it works, why don’t you have a go at inserting some dummy data into it? Just change collection/messages.js to the following:


Messages = new Mongo.Collection("messages");
Messages.insert({greeting: "hello"}, function() {});

If you query the database again, the collection is no longer empty, and you can see a new messages collection. Heck, you can even use db.collection.find() to get the message you just inserted!


meteor:PRIMARY> show databases
admin   (empty)
local   0.063GB
meteor  0.031GB
meteor:PRIMARY> show collections
messages
system.indexes
meteor:PRIMARY> db.messages.find()
{ "_id" : "L84orvrSeZDCNEGWY", "greeting" : "hello" }

You can also use show dbs instead of show databases, and show tables instead of show collections

Data Seeding

So our Meteor collections work. Now, let’s seed our messages collection with some dummy data.

You can do so very simply by doing this:


Meteor.startup(function () {
    if (Messages.find().count() === 0) {
        for( var i = 0; i < 10; i++ ) {
            Messages.insert({text: "A dummy message"});
        }
    }
});

Meteor.startup() is part of the Core API and allows you to define function(s) to run as soon as the server has been started. So here, we are simply checking whether the Messages collection is populated, and if not, insert 10 documents into it.

But using this method, all the messages are the same! You can of course specify an array of messages, but then that’s hard work! Luckily for us, there are plenty of packages out there to help us fake some data!

Faking it

You might have previously seeded your database with Faker, a name that’s associated with tools that generate fake data such as words, names, phone number, emails and pretty much anything you can think of. It has a version for Node, one of PHP, one for Perl and one for Ruby; this is of course not an exhaustive list.

But Meteor also comes with its own package – anti:fake, which works well alongside dburles:factory. They are not feature-rich as the Faker packages, but it is enough for our use, and it will also give you an opportunity to learn how to install Meteor packages.

Meteor has its own packages repository – Atmosphere. Head over there now and search for fake. And select the anti:fake package.

atmosphere-fake

To install anti:fake, just run the meteor add command.


meteor add anti:fake

The first part of the package name is, by convention, the package author, and the latter part is the name of the package. You’ll see the package being added successfully.


anti:fake  added, version 0.4.1
anti:fake: Random text and data generator

If you now look inside .meteor/packages and .meteor/versions, you’ll see an entry for the anti:fake package. You should check these files into the repository, so people who clone your application can download the same packages (and version) as you.

Next, do the same with the dburles:factory package.

Now we have the tools, let’s create a seeder.js file inside our server/ directory, where we will insert dummy data into our collection when the server starts. For now, anyone can post a message (which means, in effect, everyone posts as the same user).


Meteor.startup(function() {
  Factory.define('message', Messages, {
    text: function() {
        return Fake.sentence();
    }
  });

  // Add this if you want to remove all messages before seeding
  Messages.remove({});

  if (Messages.find({}).count() === 0) {
    _(10).times(function(n) {
      Factory.create('message');
    });
  }
});

And now, query the messages collection again using meteor mongo, and find it filled with dummy data!


meteor:PRIMARY> db.messages.find()
{ "_id" : "f7obDSzes2sz7KFHM", "text" : "Is remo esy ingment warddifulercom comsurco caldecondie addis eddi tyentheac eris." }
{ "_id" : "gjAdHexFzNWEvqt8L", "text" : "Inble minly erthebe conna inger." }
{ "_id" : "wmGfZnDkfGYdDvAE5", "text" : "Eslecin deelofthe suren therey dia cy in inomander moo." }
{ "_id" : "Ft5Xa4eHmy7Cpm5Lh", "text" : "Canthees se ter iningra lybeper torterlyextion dis noenan la esgenthetytion fimentreerde." }
{ "_id" : "g7RkhEkGZnz6MP69m", "text" : "E ament tytaly comastiono terthe." }
{ "_id" : "FagjoGcmsBL5ovpFz", "text" : "Conmentcallightal atra in ytureci talex tainestmis thethe." }
{ "_id" : "jYZxTev3HkcG5LkMR", "text" : "Titheaf ingex et ied pitiono." }
{ "_id" : "LFZ6eXawjsoGf3y29", "text" : "Tionre reees lylydicomters ersit tyre tinlyar." }
{ "_id" : "QT6KHmkWcfMtBT4mH", "text" : "Confac calde tuyingoras tirire re lyertionar." }
{ "_id" : "u7EA6vCD7QicRyM3C", "text" : "Uedin secalcon sii anaterpar dencom yernun ertheno." }

If you want to delete your entire database and start from scratch, quit your application and run meteor reset. Be careful as this will remove all the data in the database.

Next, let’s use this dummy data instead of hard-coding it into our template(s) or helper(s). But before we do, Christmas came early for all of us!

RoboMongo

meteor mongo is great, but wouldn’t it be better to have a GUI though? Well, here’s a little gem called Robomongo!

Robomongo is the MongoDB management tool. It has a great GUI, with different view modes. It also embeds the same JavaScript engine that powers MongoDB’s shell – this means you can run shell commands from within Robomongo, but now with the help of autocomplete and syntax highlighting.

If you want to connect to a remote database, you can do so; and if you don’t want to open up port 27017, you can even connect through SSH!

Note, however, the current version of Robomongo does not fully-support MongoDB 3.x. You can keep track of the issue on GitHub.

Go ahead and download it for your platform now!

Now, let’s use Robomongo to see our new collection.


$ robomongo

The GUI of Robomongo will pop up asking you to connect to a database. By default, Meteor will run on port 3000 and the MongoDB that comes with Meteor will run on port 3001.

So on the popup window that appears, click on ‘Create’ to configure a connection to MongoDB.

Robomongo asking listing out all configured Mongo Connections

Next, give your connection a name so you can recognize it later, I usually choose the name of the application as the connection name.

create-connection

Press ‘Save’. Select your connection and click ‘Connect’

Selecting the Slack connection and connecting to it

And now, navigate down the tree and double-click on the messages collection, where all the documents will appear on your right!

Viewing the messages collection using Robomongo

If you already have MongoDB running on your machine and you want to use that instead, just add the environment variable MONGO_URL=mongodb://localhost:27017/meteor, assuming your Mongo installation is running on the default Mongo port of 27017 and the database name is meteor.

Using Collections

OK, back to our application. Remember from our previous article, we had an array defining all our messages in client/app.js.


Template.messages.helpers({
  messages: [
    { text: "All these messages" },
    { text: "Uses the same template" },
    { text: "But have a different data context" },
    { text: "It's why these messages are all different!" },
    { text: "Hey!" },
    { text: "What's up man!" },
    { text: "Hello" }
  ]
});

But now, we can use our Messages collection instead! There are many methods available to interact with collections, and you can find them in the official documentation. For now, we only need to use two – collection.find() to retrieve documents, and collection.insert() to add a document.

Now, replace the entire client/app.js with the following


Template.messages.helpers({
  messages: Messages.find({})
});

When I passed in an empty object (or pass nothing in at all) it will retrieve all the records in the collection. So now when your application refreshes, you should see the list of messages you inserted into the collection! Try it!

blank-screen-slack

Well that didn’t work…let’s open up the console and find out why!


Uncaught ReferenceError: Messages is not defined

But we did define it in collections/messages.js, what’s the deal? The deal is a little thing called load order.

Previous documentation on file load order was a little vague, and led to much confusion. So I took the liberty and re-written the Meteor documentation on file load order, in the hope that it makes more sense.

These are the five rules, which should be applied in order:

  1. HTML template files are always loaded before everything else
  2. Files beginning with main. are loaded last
  3. Files inside any lib/ directory are loaded next
  4. Files with deeper paths are loaded next
  5. Files are then loaded in alphabetical order of the entire path

Going back to our malfunctioning application, because the entire path of our client/app.js is before the path for collections/messages.js alphabetically, the client/app.js file is being loaded before collections/messages.js, hence the error we get.

Let’s experiment a little and rename our collections directory to aaaaa. You’ll find the messages coming back again, because aaaaa... comes before client... alphabetically!

All messages are showing again after tinkering with load order

But that’s a really stupid name, so let’s utilize rule #3 and change collections/messages.js to lib/collections/messages.js instead. Great! Everything works!

Adding to Collection

Adding to the collection is just as easy. Now, instead of manually appending a block of HTML onto the page, we just add the message to the Messages collection. (We’re also checking that the input box is not empty before inserting)


Template.footer.events({
  'keypress input': function(e) {
    var inputVal = $('.input-box_text').val();
    if(!!inputVal) {
      var charCode = (typeof e.which == "number") ? e.which : e.keyCode;
      if (charCode == 13) {
        e.stopPropagation();
        Messages.insert({text: $('.input-box_text').val()});
        $('.input-box_text').val("");
        return false;
      }    
    }
  }
});

Let’s try another little experiment. We will restart our application so we get our seeded messages again. We will now submit a new message. After this, we will quit our meteor process on the server (e.g. pressing Ctrl + C)

If you look in the console now, you’ll see several messages like this:


GET http://localhost:3000/sockjs/info?cb=w8_t8rtb3s net::ERR_CONNECTION_REFUSED

Now, let’s type in a second message into the box and hit Enter. Then, restart meteor. You’ll see two things:

  1. The second message appeared on your screen immediately after you pressed the carriage return key , even when the server is down
  2. You’re able to recover the second message but the first one is gone (rightly so because in our seeder we erase all previous messages)

This is possible because of Meteor’s minimongo.

Minimongo

Minimongo can be seen as a simple version of MongoDB that sits on the client. When we inserted into our Messages collection, we are inserting into this local version of Mongo, as well as the remote version.

Reactivity

In Meteor, reactivity works by it detecting a change in a data source, and then re-running some code to recompute the new values. Here’re a list of reactive data sources:

  • Session variables
  • Database queries on Collections
  • Meteor.status
  • The ready() method on a subscription handle
  • Meteor.user
  • Meteor.userId
  • Meteor.loggingIn

Don’t worry about most of them, but just notice that database queries are a reactive data source. This means as soon as the local database updated, minimongo will detect this change and triggers a recomputation that leads to our helper getting the updated values and reactively adding the message onto the screen. This is why as soon as you hit Enter, the message appeared on your screen immediately.

This reactivity is achieved through Blaze, which in turns uses Tracker. But which is too much to get into that right now. Just understand that Tracker provides reactivity, and Blazes uses the updated information to update the view (or user interface), all done without polling for changes periodically.

Because Meteor is modular, you’re not tied down to Blaze. You can use Meteor with React and Angular, for example. If you want to learn how to do that, this video on React + Meteor, and this talk on Angular + Meteor, may interests you.

React JS

Latency Compensation

Somewhat related to reactivity, Meteor implements latency compensation. This means Meteor will update the UI even without confirmation from the server, just like our message. But then when the ‘real’ data comes back from the server, the UI is updated if, and only if, it needs to. This is extremely useful to providing good user experience when you are very sure of the expected outcome, as it stops the user having to appear to wait for confirmation.

Here, we know, by the design of our application, that messages by anyone will be inserted into the database, so that’s a good use case. Later on, when we deal with users and authentication, maybe the insert will not be successful, but then in those case, we can update the UI in a logical fashion, such as making the text red to indicate failure, instead of just removing it.

How useful latency compensation is depends on the use case and your implementation. But it’s there as a feature should you need it.

Offline Use

Meteor may also be useful for brief offline use. As you saw, the message we sent when the server was down didn’t go through immediately, but when it booted up again, the client will try to communicate with it and pass up any changes to the data it received til then. That’s why the second message showed up.

But this is also the case when the client loses connection with the server. As soon as the client gets back this connection, those messages will still be sent to the server.

Publish & Subscribe

By default, Meteor comes with the autopublish package, which automatically publishes all collections in the entire database. This means the client have access to (is subscribed to) all the collections on the server.

So if we open up two different browsers (or the same browser but one is in ‘private’ mode). When we insert a message from one client, the change is seen on the other. This is because every client is automatically subscribed to the collections, and as soon as the collection updates, all the clients update also.

Updates across different clients

Obviously, this have huge security implications in a real chatroom because we don’t want people to read our private chats! So in the next article, we will explore the idea of adding users to our application, and using publication and subscription so users can only see certain rooms.

Daniel Li

I am a full-stack web developer in Hong Kong. Do check out my profile, blog and projects, many of which are open-source! I get a real buzz from hearing words like Meteor and Node, if you do too, throw me a tweet!