Amazon AD Banner

Have you ever wonder how URL shorteners like bit.ly or goo.gl work? Well, we're going to actually build a simple shortener so you can learn how it's done and get accustomed to technologies like Hapi.js (API Framework made by Walmart) and MongoLab (MongoDB database as a service) because I'll deploy it to Heroku but you can use your own MongoDB database installed on your local machine or wire a relational database (I recommend SQLite) or any other that you know of or suits your needs instead (for this project, using Redis may also be a good idea because of efficiency and scaling).

I will use Axios as the request library and Bluebird for the Promise implementation in older browsers. But, you can use jQuery's $.ajax() instead or another request library called "fetch" which uses the native implementation of fetch(), this also needs a polyfill for fetch and Promise.

One more thing! The backend code will be written in ES6 (ES2015, EcmaScript 6), which means, you will see a lot of arrow functions, const declarations, and template literals. I skipped the new module importing and exporting syntax in favor of legacy but you can use that if you want; for the ES6 syntax to work you need to either deploy it to a server (Heroku, HyperDev) that has NodeJS 4.4 or higher (that's the version I battle-tested it) or if you'll follow this tutorial on your local machine; make sure you have the latest stable version of NodeJS.

What we will be building

The following is front-end of the API, I enabled CORS so anyone can make use of it. In hapi.js, it's super easy to enable CORS but more on that when we get to the good stuff. The main goal is to let the user write or paste a long URL (can be short too) in the input textbox and when he clicks on the "SHORTEN!" button, a DIV will appear on the page with the link shortened that he can copy and start using right away.

Demo Github Repository

I know the design isn't the greatest but this isn't a web design tutorial (and I don't have great design skills), there are things that are lacking from the app and I leave them up to you if you want to keep learning and improving :).

Requirements to follow

There are some things that you have to make sure are installed on your machine or server before doing all of this. If you don't, you need to go ahead and download them, install the tools and now you can start following this tutorial.

If you aren't sure how on earth to connect MongoLab (MLab) to NodeJS and Mongoose, this article may be of use.

That's all, a code editor and a lot of patience is a must. We won't be using Gulp or any other build tool since this is a very small project.

Step 1: The Backend

It's time to start getting things done, I'm going to set up the backend and boilerplate code first, then the API (hapi + mongo) to make a POST request and save new shortened URLs, enable static file serving and lastly, the front-end or page design.

Create a folder somewhere in your system and inside that folder, create the following subfolders and files:

/ Root folder of the project
  - [Folder] Public
    - 404.css
    - scripts.js
    - styles.css
  - [Folder] Views
    - 404.html
    - index.html
  - createhash.js
  - package.json
  - routes.js
  - shortener.js

Doesn't matter if the files are empty, we will be filling them step by step; the first file that we need to fill is package.json, this file contains the list of packages (dependencies) that our app depends on. It also contains information about the project and the author, info that you can replace (author, tags, license, description, name, version):

{
  "name": "url-shortener",
  "version": "1.0.0",
  "description": "A simple url shortener with hapi.js",
  "main": "shortener.js",
  "scripts": {
    "start": "node shortener.js"
  },
  "keywords": [
    "url",
    "shortener",
    "app",
    "hapi"
  ],
  "author": "codetuts",
  "license": "ISC",
  "dependencies": {
    "hapi": "^14.1.0",
    "inert": "^4.0.1",
    "joi": "^9.0.4",
    "mongoose": "^4.5.8"
  }
}

The dependencies that are listed there are needed for our project to run flawlessly. Once you are ready to run the app, it's imperative that you install those dependencies (actually, let's do it now, as in NOW!) by running the following command in your terminal (once you have navigated to your project folder):

npm install

Simple, huh? This way, NPM sees all the dependencies in your package.json file and installs them, creating a new folder called node_modules in the process.

NOTE: When deploying to Github and then to Heroku, you SHOULD NOT include the node_modules folder, Heroku automatically runs the package installer command in the shadows and lets you know everything is ready. To avoid this, you can create a .gitignore file in your project folder and add node_modules/ to it.

The main entry point (shortener.js)

This is the file that NodeJS will run to start the server, the MongoDB connection and start listening to connections.

const Hapi     = require('hapi');  
const server   = new Hapi.Server();  
const routes   = require('./routes');  
const mongoose = require('mongoose');  
const mongoUri = process.env.MONGOURI || 'mongodb://localhost/shortio';  
// If you're testing this locally, change mongoUri to:
// 'mongodb://localhost:27017/shortio'

/* MONGOOSE AND MONGOLAB
 * ----------------------------------------------------------------------------
 * Mongoose by default sets the auto_reconnect option to true.
 * We recommend setting socket options at both the server and replica set level.
 * We recommend a 30 second connection timeout because it allows for 
 * plenty of time in most operating environments.
 =============================================================================*/

const options = {  
  server: {
    socketOptions: { keepAlive: 300000, connectTimeoutMS: 30000 }
  }, 
  replset: {
    socketOptions: { keepAlive: 300000, connectTimeoutMS : 30000 }
  }
};

mongoose.connect(mongoUri, options);

const db = mongoose.connection;

The first part is the set of const declarations; we're declaring constants that will hold the modules installed by NPM. The last constant, mongoUri has a curious value, doesn't it? process.env.MONGOURI means that I don't want to expose my MLab credentials (the MONGOURI looks like this: 'mongodb://username:password@host:port/db') to the world and I'd rather set it as an environment variable (Heroku has this as "config variables" and HyperDev has a .env file). If you are following this tutorial on your local machine, replace process.env.MONGOURI with 'mongodb://localhost/shortio' or 'mongodb://127.0.0.1:27017/shortio' if it fails.

Now, the options object literal contains some settings you can provide to the Mongoose driver, these options can be omitted (and not pass the object to mongoose.connect) but MLab recommends those settings for it to work properly. As you may know, Mongoose is one of the most popular MongoDB clients/drivers for NodeJS, it has Schemas and a nice Promise implementation.

Next, it's time to actually connecting to the database and start the server with Hapi, add the following lines below the aforementioned code:

/* SERVER INITIALIZATION
 * -----------------------------------------------------------------------
 * We initialize the server once the connection to the database was set
 * with no errors; we also need to set CORS to true if we want this
 * API to be accessible in other domains. In order to serve static files
 * I used the Hapi plugin called 'inert', hence the call to 'require'.
 =======================================================================*/

server.connection({  
  port: process.env.PORT || 3000,
  routes: { cors: true }
});

server.register(require('inert'), (err) => {  
  db.on('error', console.error.bind(console, 'connection error:'))
    .once('open', () => {
      server.route(routes);

      server.start(err => {
        if (err) throw err;

        console.log(`Server running at port ${server.info.port}`);
      });
    });
});

The code may seem complicated but it's not. The first piece is the configuration of the connection that Hapi will create to start the server; the server will run on port 3000 if process.env.PORT is not set and all the routes will have CORS enabled, you can disable this by setting cors: true to false if you're not comfortable with others using the API in other domains (although they could use crossorigin.me); you could also implement JSONP but that's beyond the scope of this tutorial.

Next, as the official Hapi.js documentation states, if you want to serve static files (it also supports templating engines like Handlebars or Jade/Pug) for rendering views, scripts, CSS and images, you need a plugin for Hapi called inert; we already installed it with the package.json file so don't worry. I wrapped the database and server initialization inside the server.register() statement in order to let Hapi know I'm using the inert plugin, otherwise it'd throw an error saying there's no such thing as a "file handler."

Lastly, we start the connection with MongoDB and pass a bound callback (console.error) to db.on('error', callback) for it to handle connection errors and using the Promise syntax .once, we tell Mongoose what to do once the connection was opened. In this scope, we're telling Hapi to use our routes.js module (that we haven't even filled yet, patience, patience) and then start the server watching for errors with the if (err) throw err; expression. Hapi provides an object called server.info, here I can found the port that Hapi is currently using and console.log it.

The Routes and Schema module (routes.js)

Open up the routes.js file and in here, we'll be working with the Mongoose Schema and the routes, the code seems long but it's because there's a lot of object literals that take up lines.

const Joi        = require('joi');  
const mongoose   = require('mongoose');  
const Schema     = mongoose.Schema;  
const createHash = require('./createhash');  
const hashLen    = 8; /* 8 chars long */  
// Local machine? Set baseUrl to 'http://localhost:3000'
// It's important that you don't add the slash at the end
// or else, it will conflict with one of the routes
const baseUrl    = process.env.BASE_URL || 'http://your-domain.com';

/* CREATING MONGOOSE SCHEMAS
 ================================================*/

const redirSchema = new Schema({  
  shortUrl: String,
  url: String,
  createdAt: Date
});

const Redir = mongoose.model('Redir', redirSchema);

Again, we require mongoose because we need it to set the Schema, a schema the structure that our MongoDB documents (objects, entries, records) will follow and we're inserting "redirections" into our database, that's why I named my schema "Redir". The Joi plugin was also installed with package.json and it's useful to validate input data that is sent to us by the user (in this case, to validate that the provided POST request is valid).

We're also requiring another module that we're yet to create, createHash, which is there to generate a "n" characters long hash or randomized string for our shortened URLs and attach it to your base URL (your-domain.com for example), it will generate something like http://your-domain.com/xH83dUp1 and if we then go to http://your-domain.com/xH83dUp1, it will redirect you to the original URL sent with the POST request. Also, we store how long we want those hashes in a constant, I liked it 8 characters long but you can make it larger or shorter. Lastly, it's important to set the base URL, heroku subdomain or your own domain, if you're working on localhost, set it to localhost with the corresponding port and the HTTP at the beginning (it's important that you don't add a slash at the end).

The schema consists of three keys: the shortened URL that is going to be generated by a simple algorithm once we receive a POST request, the original URL passed to us and the date when it was created (optional); I'm never going to use the Date again in this example but I like to place it there just in case (it's highly, highly optional). From now on, when we want to perform searches and insertions on that model or collection, we use a Schema method Redir.someMethod(...) like .findOne or .save.

If you feel comfortable with more modularization, move all the Schema logic to a separate file and require it from routes.js.

The final part of the routes.js file is of course, the routes themselves! This is the array that we are going to export for it to be used in the entry point file. Add the following code below the schema declarations:

/* EXPORTING THE ROUTES
 ======================================================*/

module.exports = [  
  // The array of routes goes here
];

Nope, that's not the whole file, I will add route by route and ask you to insert them one by one right inside the array, I do this because the expanded array looks rather large and I want to exaplain each route one by one.

The first route that we need to add to the array is the root directory /, what will happen when the visitor enters the page, we'd normally serve an index.html file right? This is going to be the frontend view where users are going to land. Add the following code inside the array:

{
  method: 'GET',
  path: '/',
  handler(request, reply) {
    reply.file('views/index.html');
  }
},

That right there is how Hapi.js declares routes, first, the GET method is decided because we're requesting data not sending it; the path is the forward slash that represents the root path. A handler is needed for each route, this is the method that will execute when a user enters the path or sends the request; it takes two parameters: request and reply. Similar to req and res in Express, request and reply work similarly, with request we can view information about the request, in this tutorial we'll only cover request.params and request.payload (for POST requests). What I put inside that handler was simple a .file() method (added by the Inert plugin) which sends HTML to the requester, in this case, the browser renders the file.

The next route is for serving images, css and javascript files to our frontend html files in the views folder; it follows the same logic as the previous route:

{
  method: 'GET',
  path: '/public/{file}',
  handler(request, reply) {
    reply.file(`public/${request.params.file}`);
  }
}

The only difference is that we're serving whatever filename is requested in the public path, that's what {file} is there for, that's a parameter extractor (more about parameters here) and it's telling the handler that the first and only parameter is called "file." We then reply with the concatenation of the public folder and the filename requested as a parameter. So, when someone asks for http://your-domain.com/public/myfile.ext, it will serve it back to the requester.

Pay attention to the following route, it's the most important and the core of our little experiment:

{
  method: 'POST',
  path: '/new',
  handler(request, reply) {
    const uniqueID = createHash(hashLen);
    const newRedir = new Redir({
      shortUrl: `${baseUrl}/${uniqueID}`,
      url: request.payload.url,
      createdAt: new Date()
    });

    newRedir.save((err, redir) => {
      if (err) { reply(err); } else { reply(redir); }
    });
  },
  config: {
    validate: {
      payload: {
        url: Joi.string()
                .regex(/^https?:\/\/([\da-z\.-]+)\.([a-z\.]{2,6})([\/\w \.-]*)*\/?$/)
                .required()
      }
    }
  }
},

We will make use of a super simple HTML form (using AJAX) to send a POST request to our server, that's why I set up a POST request handler routed to the /new path (it can be whatever you want, /shorten for example). First, we need to add an extra property to the route object, the config object.

This configuration object will have another object as a property, the validate object will, in turn, have a payload property with an object literal holding the validation for each parameter; we're only receiving one (the original URL) and it uses Joi to check if it's a string, if it matches the given pattern (this one is a piss poor URL validation, there are better ones) and says that it's required, if not provided, give an error back.

Now, in the handler, we declare a new Redir instance with our keys (properties), the shortUrl is going to be generated by the function imported from the module in the first lines of the file, the original URL is subtracted from the payload (data sent via POST request) and the date is generated by JavaScript's native Date constructor. Once the instance is saved, it's time to save it to the database; we need to pass it a callback to handle the error if something happened with the database or return the newly created object (the Redir instance) that's going to be sent back to the requester as JSON if everything went well. Said JSON object will be used by the front-end (the app) to tell the user the new shortened URL or let him know of any error that happened.

The last route is, you guessed, the redirection route, we're going to catch a parameter in the root path and find that parameter (should be a hash/ID) in the database and if it finds it, redirect the user to the original URL (not the shortened one):

{
  method: 'GET',
  path:'/{hash}',
  handler(request, reply) {
    const query = {
      'shortUrl': `${baseUrl}/${request.params.hash}`
    };

    Redir.findOne(query, (err, redir) => {
      if (err) { return reply(err); }
      else if (redir) { reply().redirect(redir.url); }
      else { reply.file('views/404.html').code(404); }
    });
  }
}

This one is rather simple, we caught any parameter introduced in the root path (examples: my-domain.com/cheese, my-domain.com/12345678, my-domain.com/p4r4m3t3r) and it's gonna find it in the Redirs collection (database), it's only going to retrieve the first match. We pass the query to Mongoose's findOne method saying "find redirections that in the shortened URL, have a hash the same as the one requested in the URL", then, if it found one, redirect to the original URL before being shortened, if it didn't find one, send the 404.html view with the status code of 404 (overriding the default 200 which means success). Of course, not forgetting to handle a possible database error first.

In the end, the routes.js file should look something like this.

Generating the hash in createhash.js

We're almost done in the backend, just open the createhash.js file and fill it with the following code:

/* FUNCTIONS TO GENERATE THE RANDOM HASH
 * ---------------------------------------------------------------------
 * I'm only using numbers, letters from A-Z both lowercase & uppercase,
 * however, you can add other characters to increase the randomness.
 =====================================================================*/

function randomChar() {  
  var n = Math.floor(Math.random() * 62);
  if(n < 10) return n; // 0-9
  if(n < 36) return String.fromCharCode(n + 55); // A-Z
  return String.fromCharCode(n+61); // a-z
}

function createHash(len) {  
  var str = '';
  while(str.length < len) str += randomChar();
  return str;
}

module.exports = createHash;

There's not much to explain here, the first function generates a random character string which can include numbers or letters (both lowercase and uppercase). The second function assembles a string consisting of len random characters long. The module.exports expression makes it available for routes.js to use.

IMPORTANT: This will create a random ID but not a unique one, you can hard code a recursive workaround by checking if the hash exists in the database but that'd require extra DB queries, highly inefficient. You can use this NPM package to generate unique hashes recommended by a user on Reddit. If you guessed correctly, implementing this code at a larger scale may cause some shortened URLs to be overridden, I had this in mind but I didn't think it was that big of a deal for a small sample project.

Front what? The design, the client-side JavaScript, and the CSS/HTML files to be displayed. We have our API to POST into the database new shortened URLs but how do we send them? You can start testing it by using the Postman app and sending a form or www-urlencoded POST request with the "url" parameter and a valid URL as its value. If it returns an object containing the Schema data (shortenedUrl, createdAt and URL), you will know the backend is usable.

Start with the index.html file located in the views folder, I won't paste the whole thing here because it's a bit long but you can copy the contents of this file into the one you created.

In this file, I'm invoking the Roboto font, Miligram.css (a small CSS framework), font-awesome for the icons and then your own CSS file for that particular view. At the bottom go the JavaScript files, I invoked Bluebird first, Bluebird allows older browsers (and not so modern ones) to have Promises, Axios requires Promises to work so, if I didn't include this library it wouldn't work in IE and older versions of popular browsers. Axios is required before our scripts.js file and it's basically an HTTP request library (like jQuery's $.ajax) to send the POST request to the backend.

IMPORTANT: You should not do this in production, always minify and bundle your scripts and CSS files into a single file. Axios and Bluebird make up for more Kilobytes than jQuery alone so you may want to consider using jQuery's Ajax implementation instead.

For the main CSS, copy the contents of this CSS file into your styles.css file located in the public folder (this file is also kinda long).

To finish with the front page, we need scripts.js and I'm going to explain step by step what it does. Open the file (located in the public folder) and fill it with:

// Cache DOM elements in memory
var form   = document.getElementById('shorten-form');  
var urlBox = form.elements[0];  
var link   = document.getElementById('link');  
var shrBox = document.getElementById('shortened');

// Callback function passed to Axios' .post().then()
function displayShortenedUrl(response) {  
  link.textContent = response.data.shortUrl;
  link.setAttribute(
    'href', response.data.shortUrl
  ); // Set the link's href attribute
  shrBox.style.opacity = '1';
  urlBox.value = ''; // Reset input
} // End of function to update the view

// Callback function passed to Axios' error handler
function alertError(error) {  
  // Handle server or validation errors
  alert('Are you sure the URL is correct? Make sure it has http:// at the beginning.');
} // End of function to display errors on the page

form.addEventListener('submit', function(event) {  
  event.preventDefault();

  // Send the POST request to the backend  
  axios.post('/new', { url: urlBox.value })
       .then(displayShortenedUrl)
       .catch(alertError);
});

The first thing we have to do is cache some HTML elements for later access, we need the input box where we paste the original URL, the link that gets displayed with the shortened URL, the div that contains the shortened URL (hidden by default), and the form that contains the input box to send the original URL and the "Shorten!" button.

We need the form to attach an event listener to it, the 'submit' event. This means that we can execute a function when we press ENTER after pasting or writing the URL or when we click the submit button. The event.preventDefault(); prevents the browser from reloading when we submit the URL, then we have the POST request with Axios, we send the request to the /new path as stated in the backend's routes.js file. With promises, we can use the .then and .catch methods; if the insertion and creating of the shortened URL in the database was successful, we pass a callback function that receives the response from the server and if the server returned an error, we do a not recommended JavaScript alert telling the user what went wrong.

The way we implement the callback function is by accepting a response object from Axios as its only parameter and then use the link HTML element updating its href attribute (with the shortened URL) and the textContent (also with the shortenedUrl) to let the user know about the now shortened URL. After we do this, we show the hidden DIV that contains this information by setting its opacity to 0 (you can change the opacity transition to a display: block; with no transition and changing that element's CSS to display: none; and remove the opacity: 0; declaration and transition). And before finishing the function, clear the contents of the input box.

The 404 View

404 Template

We also need the view that we sent in the backend when the shortened URL doesn't exist, here's both the HTMl and the CSS for their corresponding views/404.html and public/404.css files.

<!DOCTYPE html>  
<html>  
  <head>
    <meta charset="utf-8">
    <title>404 - This is awkward...</title>
    <link href="https://fonts.googleapis.com/css?family=Montserrat" rel="stylesheet">
    <link rel="stylesheet" href="public/404.css">
  </head>
  <body>
    <div class="overlay"></div>
    <div class="on-top">
      <h1>That shortened URL doesn't exist...</h1>
      <p>
        It doesn't exist for now, double check what you got the correct URL, remember that these shortened URLs are case sensitve. Sorry for the inconvenience.
      </p>
      <a href="/" class="back">Back to Shortio</a>
    </div>
  </body>
</html>

That's the HTML for the view, as you see, there is a message and a button to go back to the app.

As for the CSS, since it's a long file and I don't think it's a good idea to put it here; you can go ahead and copy the contents of this CSS file into your public/404.css file. Please don't use the Travolta image in production, I'm not sure how well it fares with copyright issues and it's silly to use it professionally.

Conclusion and a challenge

That's all, to run the app, if you're working on your local machine, go to the terminal, navigate to the project folder and run the following command:

npm start

This will invoke the entry point with NodeJS and start the connections that we established in shortener.js. If it gives you any error, let me know in the comment section and I'll see what happened. Since I deployed the project to Heroku, all I had to do was move every file (except the node_modules folder) to a Github repository (which I added below the screenshot, besides the demo link) and deploy it from Heroku, no procfile needed, no hassle, just a manual deploy within the web app.

Try using it and finding bugs, improve it, make some extra functionality happen like revealing shortened URLs using an endpoint like /reveal/{hash}, or you can double check if the hash already exists before inserting the new one and avoid duplicated shortened URLs; I'm thinking maybe you can instead of storing the full shortened URL, store the pure hash and assemble the shortened URL in the frontend with the scripts.js file. Make any improvement that would make you learn more.

I also want to thank the Reddit community for giving me feedback, no offense taken, we're here to learn and improve and everything I can possibly add or remove from this tutorial is well received. Cheers!