CRUD Operations and File Upload using Node.js and MongoDB

Last Updated : 07 May, 2024

Within the computer programming world, CRUD is an elision for the 4 operations Create, Read, Update, and Delete. Create, Read, Update, and Delete (CRUD) are the four basic functions that models should be able to do, at most. In RESTful applications, CRUD often corresponds to the HTTP methods like POST, PUT, DELETE, and GET respectively. Each database requires some functions to be called so that users can perform some operations. These operations are used by the users to perform certain types of queries on a particular data within the database. The basic of the operation is CRUD operations. CRUD operations and the File Upload will be done with the help of the Node js framework and its packages along with MongoDB and Postman.

Before deep-diving into how the code works let us know the prerequisites of the applications. The local machine should have Node js, Postman and MongoDB installed as the prerequisites.

Node js (npm): If one doesn't have Node.js and npm install go to the official link to install Node.js and run the following command on the terminal for downloading npm.

npm install npm@latest -g

Postman: Postman is used to manage the requests. It helps in testing the HTTP requests, with the help of GUI without copying and pasting each HTTP request everywhere. Go to the official link to download PostMan for free.
MongoDB: MongoDB is the database that is used in CRUD operations. If you don't have a MongoDB account, don't worry the article will be giving step-by-step instructions for it too.

This project is required to be built in node js with MongoDB as the backend. It provides the following capability through the API interface:

CSV File Upload from Multipart Form
Scan the uploaded CSV and push its content into MongoDB collection
API to perform CRUD operations on the above collection

The above API are using a passport package for basic authentication to manage access

Framework and Runtime Environment Used: Here are the major frameworks and run time environments that are used in the project.

Node js
NPM package Manager
Mongo DB
Postman

Step 1: Make the Node.js project. To do that write down the following command in the terminal.

npm init

After writing the code into the terminal you need to fill out the information. To make it default press ENTER at each section. At the end, you will get a filed named package.json in your directory.

Some of the packages as dependencies are used in this project namely - express, express-session, bcryptjs, fast-csv, csvtojson, dotenv, multer, passport, passport-local, and mongoose. Run the command to install these packages.

Step 2: Set up a server. To set up the server for the application, make 2 files named app.js and .env. app.js will act as the server file and .env will help in loading the environment variables.

Inside the .env file write port number for your localhost PORT = "Port Number" and in app.js write down the following code to run up the server.

JavaScript

const express = require('express');
const app = express();
const dotenv = require('dotenv');

dotenv.config();

app.use(express.json({limit: '20mb'}))
app.use(express.urlencoded({ extended: false, limit: '20mb' }))

/* Setting up server */ 
app.listen(process.env.PORT, function(){
    console.log("This server port is up and running ");
})

To access the routes folder where all the requests are given write the following line -

JavaScript

/* Initializing the path for routes */
app.use("/", require("./routes"));

The final code for now in app.js will look like this -

JavaScript

const express = require('express');
const app = express();
const dotenv = require('dotenv');

dotenv.config();

app.use(express.json({limit: '20mb'}))
app.use(express.urlencoded({ extended: false, limit: '20mb' }))

/* Initializing the path for routes */
app.use("/", require("./routes"));

/* Setting up server */ 
app.listen(process.env.PORT, function(){
    console.log("This server port is up and running ");
})

Step to run the application: Open the terminal and writing node app.js will start the server.

Step 3: Connect to MongoDB collection. If one doesn't know how to connect to MongoDB go through the following steps, otherwise skip the following steps -

Go to MongoDB Atlas
Set up an account
Build a new project
Build a database (it might take a couple of minutes)
Go to database access and fill out the username, password, and IP address section (You can five your current IP address as default)
Click connect and choose to connect your application and it will give you an address. Copy the address and paste it inside .env file

The final .env file will look something like this -

DB_CONNECT = "YOUR PATH"
PORT = "Port Number"

After making a collection connect the Node.js application with MongoDB by writing the following code in app.js.

JavaScript

global.dbconn = "";

/* Connected the app with mongoose */ 
mongoose.connect(
    process.env.DB_CONNECT,
    { useNewUrlParser: true, useUnifiedTopology: true },
    (client, err) =>{
        try{
            console.log("Connected to db: " + client)
        }catch(err){
            console.log(err);
        }
    }
);

Step 4:This step describes the project flow structure. The project is divided into the various section. There is a middleware Folder along with the controller, routes, and config. In the _others_ folder there is a dummy CSV file for upload. Upon using the routes and uploading it, the CSV file is uploaded in uploads folder. Make the following files in each section.

NOTE: The csv_controller.js and csv.js are just programs for CRUD Operations.

NOTE: No need to make csv copy.js as it is just a copy csv.js.

Schema: To put it simply, the schema is a skeleton structure of the whole database. It allows us to define various fields for users to enter.

Step 5: Make a schema for the user and for crud operation. Since the data needs to be dynamic we will be defining the schema for the database.

Under models: User.js

JavaScript

const mongoose = require('mongoose');

/* Creating the schema with name, email, password and date */
const UserSchema = new mongoose.Schema({
    name: {
      type: String,
      required: true
    },
    email: {
      type: String,
      required: true
    },
    password: {
      type: String,
      required: true
    },
    date: {
      type: Date,
      default: Date.now
    }
  });
  
  /* Exporting schema with collection as CrudOperations */
  const User = mongoose.model('User', UserSchema);
  
  module.exports = User;

A schema with the name User is created that accepts name, email, password, and date as its fields.

crud.js

JavaScript

const mongoose = require('mongoose');

/**
 * Creating the schema with username, identifier, 
   firstName, lastName and updated
 */
const crud = new mongoose.Schema({
    username:{
        type: String,
        required: true,
    },
    identifier: {
        type: Number,
        required: true,
    },
    firstName:{
        type:String,
        required: true,
    },
    lastName:{
        type:String,
        required: true,
    },
    updated: {
        type: Date,
        required:true,
        default: Date.now,
    }
});

/**
 * Exporting schema with collection as CrudOperations
 */
module.exports = mongoose.model('CrudOperations', crud);

Within crud.js a schema with accepting fields username, identifier, firstName, lastName, updated is created.

Step 6: Having a proper authentication and authorization is very important in every organizations. For that each organizations put a layer of security. The files auth.js and passport.js are doing just that. But since the article helps in performing CRUD operations and file upload we will not be deep diving into it.

FILENAME: auth.js

JavaScript

module.exports = {
    /**
     * Ensuring authentication
     */
    ensureAuthenticated: function(req, res, next) {
      if (req.isAuthenticated()) {
        return next();
      }
      req.flash('error_msg', 'Please log in to view that resource');
      res.redirect('/users/login');
    },
    forwardAuthenticated: function(req, res, next) {
      if (!req.isAuthenticated()) {
        return next();
      }
      res.redirect('/dashboard');      
    }
  };

The auth.js file see whether the user has logged into the account. If yes, the user is authorized to view the page otherwise it is redirected to the home page

FILENAME: passport.js

JavaScript

const LocalStrategy = require('passport-local').Strategy;
const bcrypt = require('bcryptjs');

// Load User model
const User = require('../models/User');

module.exports = function(passport) {
  passport.use(
    new LocalStrategy({ usernameField: 'email' }, (email, password, done) => {
      // Match user
      User.findOne({
        email: email
      }).then(user => {
        if (!user) {
          return done(null, false, { message: 'That email is not registered' });
        }

        // Match password
        bcrypt.compare(password, user.password, (err, isMatch) => {
          if (err) throw err;
          if (isMatch) {
            return done(null, user);
          } else {
            return done(null, false, { message: 'Password incorrect' });
          }
        });
      });
    })
  );

  passport.serializeUser(function(user, done) {
    done(null, user.id);
  });

  passport.deserializeUser(function(id, done) {
    User.findById(id, function(err, user) {
      done(err, user);
    });
  });
};

Along with these 2 files, there is a need for session variable and authorization to include passport into this project. To do so, app.js is opened and passport middleware and session is initialized.

FILENAME: app.js

JavaScript

// Passport Config
require('./config/passport')(passport);

app.use(express.json({limit: '20mb'}))
app.use(express.urlencoded({ extended: false, limit: '20mb' }))

// Express session
app.use(
    session({
      secret: 'secret',
      resave: true,
      saveUninitialized: true
    })
);
  
  // Passport middleware
  app.use(passport.initialize());
  app.use(passport.session());

Step 7: A middleware is made using with the help of multer package. To use the multer package write the following command in the terminal.

npm install --save multer

The package multer helps in handling multipart/form-data, which is primarily used for uploading file. Multer adds a body object and a file or files object to the request object. The body object contains the values of the text fields of the form, the file or files object contains the files uploaded via the form.

FILENAME: uploader.js

JavaScript

const path = require('path')
const multer = require('multer')

var storage = multer.diskStorage({
    destination: function(req, file, cb){
        cb(null, 'uploads/')
    },
    filename: function(req, file, cb){
        let ext = path.extname(file.originalname)
        cb(null, Date.now() + ext);
    }
})

var upload = multer({
    storage: storage,
    fileFilter: function(req, file, callback){
        if(
            file.minetype == "text/csv"
        ){
            callback(null, true)
        } else{
            console.log("Error in uploading")
            callback(null, false)
        }
    },
    limits: {
        fileSize: 1024 * 1024 * 2
    }
})

module.exports = upload

Step 8: After the authentication is done with passport and multer as middleware is added into the project, the routes and the controller for the file upload and CRUD operations are needed to be completed.

Before jumping into the operations first the routes are needed to be initialized. To do so, index.js is used.

FILENAME: index.js

JavaScript

const express = require("express");
var router = express.Router();

/* Initializing other routes */
router.use("/", require("./csv"));
router.use('/', require('./users.js'));

module.exports = router;

To use the passport authentication the routes and controller for the user are made in users.js and user_controller.js.

FILENAME: users.js

JavaScript

const express = require('express');
const router = express.Router();
const userController = require("../controller/user_controller")
/* Register router with passport package */
router.post('/register', userController.register);
  
/* Login router */ 
router.post('/login', userController.login);

/* Logout router  */ 
router.get('/logout', userController.logout);

module.exports = router;

It makes the route for the register and login as a post method and a logout system as a get method. Inside the user_controller.

FILENAME: user_controller.js

JavaScript

const express = require('express');
const router = express.Router();
const bcrypt = require('bcryptjs');
const passport = require('passport');
// Load User model
const User = require('../models/User');
const { forwardAuthenticated } = require('../config/auth');

exports.register = (req, res) => {
    console.log("Request: " + JSON.stringify(req.body))
    const { name, email, password, password2 } = req.body;
    let errors = [];
  
    /* If condition to check whether all credentials are filled */
    if (!name || !email || !password || !password2) {
      errors.push({ msg: 'Please enter all fields' });
    }
    
    /* If condition to check whether password 
    and password2 matches or not */
    if (password != password2) {
      errors.push({ msg: 'Passwords do not match' });
    }
  
    /* If condition to check in case password 
    length is greater than 3 or not */
    if (password.length < 3) {
      errors.push({ msg: 'Password must be at least 3 characters' });
    }
  
    if (errors.length > 0) {
      res.send('register error')
    } else {
    
    /* Checking if user exists */
      User.findOne({ email: email }).then(user => {
        if (user) {
          errors.push({ msg: 'Email already exists' });
          res.send('register user exists');
        } 
        
       /* Creating the user */ 
       else {
          const newUser = new User({
            name,
            email,
            password
          });
          
          /* Bcrypt hashing the password for user privacy */
          bcrypt.genSalt(10, (err, salt) => {
            bcrypt.hash(newUser.password, salt, (err, hash) => {
              if (err) throw err;
              newUser.password = hash;
              newUser
                .save()
                .then(user => {
                  res.send("Register Successful");
                })
                .catch(err => console.log(err));
            });
          });
        }
      });
    }
  }

exports.login = (req, res, next) => {

    /* Authenticating if login was successful or 
    not with the help of passport */
passport.authenticate('local', {
    successRedirect: res.send("Login Successful"),
    failureRedirect: res.send("Error in Login"),
    failureFlash: false
})(req, res, next);
}

exports.logout = (req, res) => {
    req.logout();
   /* Logging out */
    res.send("User Logout");
}

The register exporter first checks whether the credentials given are correct or not. If they are correct, then the password is encrypted with the help of bcrypt . And after that a new user is saved inside the schema. To check whether the function works or not a request is sent through with the help of postman for register, login and logout.

Register: Sending a post request http://localhost:3000/register with the help of postman.

After the registration is successful, the same user can be seen in MongoDB atlas.

Login: Sending a post request http://localhost:3000/login

Logout: Logout send out a get request.

Crud Operations and File Upload: Using the files csv.js and csv_controller.js, all the CRUD operations and the file upload will be performed. Various API's are called which help in achieving various functions. Like http://localhost:3000/fileUpload helps in uploading the CSV file and pushing the data into Mongoose. Similarly, http://localhost:3000/CRUDcreate, http://localhost:3000/CRUDread, http://localhost:3000/CRUDupdate, http://localhost:3000/CRUDdelete helps in other CRUD operations CREATE, READ, UPDATE, and DELETE respectively.

Create: We will be sending out the post request. As the name signifies, they create functions all the users to enter a new record in the database. For a relational database, a create function is called upon known as INSERT. In non-relational databases like MongoDB provides two different options to insert documents into the collection - db.collection.insertOne()
db.collection.insertMany().

FILENAME: csv.js

JavaScript

/* Creating a schema for crud operation C: Create */
router.post("/CRUDcreate",  forwardAuthenticated, csvController.create)

FILENAME: csv_controller.js

JavaScript

exports.create = async function(req, res){
    /* Initializing the schema and putting in CRUDcreate */
    const CRUDcreate = new CRUDoperations ({
        username: req.body.username,
        identifier:req.body.identifier,
        firstName: req.body.firstName,
        lastName: req.body.lastName
    });

    /* Try Catch */
    try{
        /* Saving the data in mongoose */
        const savedCRUD = await CRUDcreate.save();
        /* Sending the response back */
        res.status(200);
        res.send(savedCRUD);
    }catch(err){
        /* Sending the error back */
        res.status(400).send(err);
    }   
}

Once the operation is done after adding the necessary packages in the files, .save() is used to save the information to the database.

Read: The read function in the CRUD operation is similar to a search function. What it does it helps the user to search and retrieve particular information of a particular data. In MongoDB we use db.collection.find() or db.collection.findOne() to retrieve the information.

FILENAME: csv.js

JavaScript

/* Router for CRUD operation R: read */
router.get("/CRUDread", csvController.read)

FILENAME: csv_controller.js

JavaScript

exports.read = function(req, res){
    /* Using find() for reading all the data */
    CRUDoperations.find({}, function(err, fetch){
        /* In case of error */
        if(err) res.status(400).send(err);

        /* Sending back the response */
        res.status(200).send(fetch);        
    });
}

Update: The update function is used to modify the data present in the collection of the MongoDB database. TO change a record users may have to modify multiple fields. This requires carefully updating the documents as the changes done while updating the collection is permanent and irreversible. 3 different methods can be used in MongoDB for updating - db.collection.updateOne(), db.collection.updateMany(), db.collection.replaceOne().

FILENAME: csv.js

JavaScript

/* CRUD operation 'update' router */
router.post("/CRUDupdate", csvController.update)

FILENAME: csv_controller.js

JavaScript

exports.update = async function(req, res){
    /* Taking the id */
    let id = req.body._id;

    try{
        /* Using findByIdAndUpdate */
        const CRUDupdate = await CRUDoperations.findByIdAndUpdate({_id: id},
        
        /* Setting the value of identifier as 1967 of corresponding id */
        {$set:{
                identifier: 1969
              }
        },
        {
            useFindAndModify: false 
        });

        /* Sending the response back to the server */
        res.status(200).send(CRUDupdate);
    }catch(err){
        /* Sending error back to the server */
        res.status(400).send(err);
    } 
}

Delete: The last CRUD operation is Delete. As the name suggests it help in deleting a single collection from a document.

db.collection.deleteOne()
db.collection.deleteMany()

FILENAME: csv.js

JavaScript

/* Router to perform delete of CRUD operations */
router.post("/CRUDdelete", csvController.delete)

FILENAME: csv_controller.js

JavaScript

exports.delete =  async function(req, res){
    /* Taking the id of the collection */
    let id = req.body._id;

    /* Using Try and catch for deletion */
    try{
        /* Using findbyIdAndRemove operation to remove 
        the data with corresponding id */
        const CRUDdel = await CRUDoperations.findByIdAndRemove(
        id, function(err, res){
            if (err){
                /* Sending error back to the server */
                res.status(400).send(err);
                console.log(err)
            }
            else{
                
                /* Sending the response back to the server */
                console.log("Removed User : ", res);
            }
        },
        {
            useFindAndModify: false 
        })
    }catch(err){
        res.status(400).send(err);
    }
}

File Upload: As discussed before file upload is done with the help of package multer.

FILENAME: csv.js

JavaScript

const fileStorageEngine = multer.diskStorage({
    destination: (req, file, cb) => {
        cb(null, './uploads')
    },
    filename: (req, file, cb) => {
        cb(null, Date.now() + "--" + file.originalname);
    },
});

const upload = multer({ storage: fileStorageEngine });

router.post('/single', upload.single('fileUpload'), (req, res) => {
    console.log('single route')
    console.log('file:'+JSON.stringify(req.file));
    res.send("single file upload success");
}); 


router.post("/fileUpload",upload.single('fileCSV'), 
    csvController.fileupload)

FILENAME: csv_controller.js

JavaScript

exports.fileupload = function(req, res){
    console.log("Inside file Upload!!")
    console.log('single route')
    console.log('file:'+JSON.stringify(req.file.path));


    let stream = fs.createReadStream(req.file.path)     
        
    let csvData = [];

    let csvStream = fastcsv
        .parse()
        .on('error', error => console.error(error))
        .on("data", function(data) {
            //console.log("Data Parse: " + JSON.stringify(data))
            dt = data[0].split(',')
            //console.log("DT: " + dt[0])
            csvData.push({
                username: dt[0],
                identifier: dt[1],
                firstName: dt[2],
                lastName: dt[3]
            });
            //console.log((csvData));
        })
        .on("end", async function() {
            // remove the first line: header
            csvData.shift();

            // save to the MongoDB database collection
            try{
                console.log("client:" + CRUDoperations);
                let CRUDinsert = await CRUDoperations.insertMany(
                    csvData
                )
                console.log("CRUD Insert Many" + CRUDinsert)
                res.status(200).send(CRUDinsert);

            } catch(err){
                console.log("db error:"+err);
                res.status(400).send(err);
            }
            console.log(JSON.stringify(csvData));
        });

    stream.pipe(csvStream);
}

Node.js CRUD Operations Using Mongoose and MongoDB Atlas

arnavmahajan01

Improve

Article Tags :

CRUD Operations and File Upload using Node.js and MongoDB

Similar Reads

Thank You!

What kind of Experience do you want to share?