Tutorial Create and Read From DB

This continues from part 2 of the tutorial track

Part III - Create our DB and read stuff from it

Step 1 – Install MongoDB

Note: This should already have been done in part 0 and part1 of the tutorial track

Step 2 – Run mongod and mongo

Create C:/data/db (or D:/data/db, if your project folder is on a different drive)

Open a new command prompt or shell into that directory, and run mongodb

C:/data/db

mongod

You'll see the Mongo server start up. This is going to take a while if it's the first time, because it has to do some preallocating of space and a few other housekeeping tasks. Once it says "[initandlisten] waiting for connections on port 27017", you're good. There's nothing more to do here; the server is running. Now you need to go back to your original command prompt and type:

C:\node\nodetest1

mongo

You'll see something like the following:

Mongo Console

MongoDB shell version v4.4.4
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 4.4.4
Server has startup warnings:
[a bunch of warnings that you can ignore for this tutorial]
>

Additionally, if you're paying attention to your mongod instance, you'll see it mention that a connection has been established. All right, you've got MongoDB up and running, and you've connected to it with the client. We'll use this client to manually work on our database, for a bit, but it's not necessary for running the website. Only the server daemon (mongod) is needed for that.

Step 3 – Create a Database

The first thing we're going to do is create a database in which to store stuff. In your Mongo console, type the following:

Mongo Console

use nodetest1

Now we're using the database "nodetest1," which is great except nothing actually exists yet. To make the database exist, we have to add some data. We're going to start off by doing that right inside of the Mongo client.

Step 4 – Add some Data

My favorite thing about MongoDB is that it uses JSON for its structure, which means it was instantly familiar for me. If you're not familiar with JSON, you'll need to do some reading, as I'm afraid that's outside the scope of this tutorial.

Let's add a record to our collection. For the purposes of this tutorial, we're just going to have a simple database of usernames and email addresses. Our data format will thus look like this:

Mongo Console

{
    "_id" : 1234,
    "username" : "cwbuecheler",
    "email" : "cwbuecheler@nospam.com"
}

You can create your own _id assignment if you really want, but I find it's best to let Mongo just do its thing. It will provide a unique identifier for every single top-level collection entry. Let's add one and see how it works. In your Mongo client, type this:

Mongo Console

db.users.insert({ "username" : "testuser1", "email" : "testuser1@testdomain.com" })

Something important to note here: that db stands for our database, which as mentioned above we've defined as nodetest1. The users part is our collection. Note that there wasn't a step where we created the users collection. That's because the first time we add to it, it's going to be auto-created. Handy. OK, Hit enter. Assuming everything went right, you should see WriteResult({ "nInserted" : 1 }), which is Mongo's odd way of telling you everything went according to plan. That's not very exciting, though, so type this:

Mongo Console

db.users.find().pretty()

In case you're curious, the .pretty() method gives us linebreaks. It will return:

Mongo Console

{
    "_id" : ObjectId("5202b481d2184d390cbf6eca"),
    "username" : "testuser1",
    "email" : "testuser1@testdomain.com"
}

Except, of course, your ObjectId will be different, since as mentioned, Mongo is automatically generating those. That's all there is to writing to MongoDB from the client app, and if you've ever worked with JSON services before, you are probably going "oh, wow, that's going to be easy to implement on the web." ... you're right!

A quick note on DB structure: obviously in the long run you're unlikely to be storing everything at the top level. There are a ton of resources on the internet for schema design in MongoDB. For example, Chris has a Five Minute React series of tutorials which dives into it.

Now that we've got one record, let's add a a couple more. In your Mongo console, type the following:

Mongo Console

newstuff = [{ "username" : "testuser2", "email" : "testuser2@testdomain.com" }, { "username" : "testuser3", "email" : "testuser3@testdomain.com" }]
db.users.insert(newstuff);

Note that, yes, we can pass an array with multiple objects to our collection. Handy! Another use of db.usercollection.find().pretty() will show all three records:

Mongo Console

{
        "_id" : ObjectId("5202b481d2184d390cbf6eca"),
        "username" : "testuser1",
        "email" : "testuser1@testdomain.com"
}
{
        "_id" : ObjectId("5202b49ad2184d390cbf6ecb"),
        "username" : "testuser2",
        "email" : "testuser2@testdomain.com"
}
{
        "_id" : ObjectId("5202b49ad2184d390cbf6ecc"),
        "username" : "testuser3",
        "email" : "testuser3@testdomain.com"
}

Now we're going to start actually interacting with the web server and site that we set up earlier. You can quit out of the MongoDB console with ctrl-c or by typing "exit" and hitting enter. We're done here.

Step 5 – Hook Mongo up to Node

This is where the rubber meets the road. Let's start by building a page that just spits out our DB entries in a mildly pretty form. Here's the HTML we're shooting to generate:

<ul>
    <li><a href="mailto:testuser1@testdomain.com">testuser1</a></li>
    <li><a href="mailto:testuser2@testdomain.com">testuser2</a></li>
    <li><a href="mailto:testuser3@testdomain.com">testuser3</a></li>
</ul>

I know this isn't rocket science, but that's the point. We're just doing a simple DB read-and-write in this tutorial, not trying to build a whole website. First things first, we need to add a few lines to our main app.js file—the heart and soul of our app—in order to actually connect to our MongoDB instance. Open C:\node\nodetest1\app.js and at the top you'll see:

C:\node\nodetest1\app.js

var createError = require('http-errors');
var express = require('express');
var path = require('path');
var cookieParser = require('cookie-parser');
var logger = require('morgan');

Now add these three lines:

C:\node\nodetest1\app.js

var express = require('express');
var path = require('path');
var favicon = require('serve-favicon');
var logger = require('morgan');
var cookieParser = require('cookie-parser');
var bodyParser = require('body-parser');

// New Code
var mongo = require('mongodb');
var monk = require('monk');
var db = monk('localhost:27017/nodetest1');

These lines tell our app we want to talk to MongoDB, we're going to use Monk to do it, and our database is located at localhost:27017/nodetest1. Note that 27017 is the default port your MongoDB instance should be running on. If for some reason you've changed it, obviously use that port instead. Now look further down in the file, where you have this:

C:\node\nodetest1\app.js

app.use('/', indexRouter);
app.use('/users', usersRouter);

We need to do some work here. Those app.use statements (along with the others you'll find in app.js) are establishing middleware for Express. The short, simple explanation is: they're providing custom functions that the rest of your app can make use of. It's pretty straightforward, but due to chaining it needs to come before our route definitions, so that they can make use of it.

Above the two lines just mentioned, add the following:

C:\node\nodetest1\app.js

// Make our db accessible to our router
app.use(function(req,res,next){
    req.db = db;
    next();
});

NOTE: If you don't put this above the routing stuff mentioned above (app.use('/', indexRouter);), your app WILL NOT WORK.

We already defined "db" when we added Mongo and Monk to app.js. It's our Monk connection object. By adding this function to app.use, we're adding that object to every HTTP request (ie: "req") our app makes. This is fine and will not cause performance issues. Unlike MySQL, MongoDB connections don't need to be manually opened and closed, so the db object will just be there, not using any resources, until we need it.

So, again, that code needs to go above our routing code. Your entire app.js should look like this, now:

C:\node\nodetest1\app.js

var createError = require('http-errors');
var express = require('express');
var path = require('path');
var cookieParser = require('cookie-parser');
var logger = require('morgan');

// New Code
var mongo = require('mongodb');
var monk = require('monk');
var db = monk('localhost:27017/nodetest1');

var indexRouter = require('./routes/index');
var usersRouter = require('./routes/users');

var app = express();

// view engine setup
app.set('views', path.join(__dirname, 'views'));
app.set('view engine', 'jade');

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

// Make our db accessible to our router
app.use(function(req,res,next){
    req.db = db;
    next();
});

app.use('/', indexRouter);
app.use('/users', usersRouter);

// catch 404 and forward to error handler
app.use(function(req, res, next) {
  next(createError(404));
});

// error handler
app.use(function(err, req, res, next) {
  // set locals, only providing error in development
  res.locals.message = err.message;
  res.locals.error = req.app.get('env') === 'development' ? err : {};

  // render the error page
  res.status(err.status || 500);
  res.render('error');
});

module.exports = app;

Next thing we need to do is modify our route so that we can actually show data that's held in our database, using our db object.

Step 6 – Pull your data from Mongo and display it

Open up C:\node\nodetest1\routes\index.js in your editor. It's still got the index route, and the goofy /helloworld route. Let's add a third:

C:\node\nodetest1\routes\index.js

/* GET Userlist page. */
router.get('/userlist', function(req, res) {
    var db = req.db;
    var collection = db.get('users');
    collection.find({}, {}, function(e, users){
        res.render('./userlist', {
            "userlist" : users
        });
    });
});

OK ... that's getting fairly complicated. All it's really doing, though, is extracting the "db" object we passed to our http request, and then using that db connection to fill our users variable with database documents, ie: user data. Then we do a page render just like the other two GETs in this route file.

Basically, we tell our app which collection we want to use (users) and do a find, then return the results as the variable users. Once we have those documents, we then do a render of userlist (which will need a corresponding Pug template), giving it the userlist variable to work with, and passing our database documents to that variable.

NOTE: The above is the basic super-bare-bones method of creating a callback to respond to a process which the system might have to wait for. (TODO Give them the run-down on callback hell and async/await)

Scoutradioz uses a custom package called scoutradioz-utilities to simplify database access. Add this line to the top of the page:

C:\node\nodetest1\routes\index.js

const utilities = require('@firstteam102/scoutradioz-utilities');

Note that the "scoutradioz-utilities" package is mostly a shorthand. Instead of doing "var db = req.db; var col = db.get(colName)" every time, "await utilities.find(colName, query, options)" is a shorthand to reduce the amount of code needed to be written.

Add another block below the '/userlist' method:

C:\node\nodetest1\routes\index.js

router.get('/userlist2', async function(req, res) {
    
    var users = await utilities.find('users', {}, {});
    
    res.render('./userlist', {
        userlist: users
    });
});

Next let's set up our Pug template. Navigate to C:\node\nodetest1\views\ and open index.pug. Once again, immediately save it as userlist.pug. Then edit the HTML so it looks like this:

C:\node\nodetest1\view\userlist.pug

extends layout

block content
  h1.
    User List
  ul
    each user, i in userlist
      li
        a(href=`mailto:${user.email}`)= user.username

This is saying that we're going to pull in the set of documents we just called userlist over in the route file, and then for each entry (named 'user' during the loop), get the email and username values from the object and put them into our html. We've also got the count — i — handy in case we need it, though in this instance we don't.

We're all set. Save that file, and let's restart our node server. Remember how to do that? Go to your command prompt, head for C:\node\nodetest1\ and ctrl-c to kill your server if it's still running from way back before. Then type:

C:\node\nodetest1

C:\node\nodetest1>npm start

Now open your browser and head to http://localhost:3000/userlist and marvel at the results.

(TODO INSERT SCREENSHOT OF PAGE WITH BULLET LIST OF THREE USERS)

You're now pulling data from the DB and spitting it out onto a web page. Nice!