MongoDB Data Migration to Atlas

5 min readMar 29, 2021

Two years ago I had to build an application as part of the final project as a student at the university. I called it “Glaciar” and the idea was visualise medioambiental data available on open datasets.

To store the data, I code some ETL and use an instance of Mongo in MLAB. It was a great “database as a service” very easy to use, and it gave you 512 Mb free and it was OK.

In 2018, the company MongoDB, Inc. acquierd mLab and in this post I would like to share the steps to migrate the dataset from my own Mongo databases, (wherever mlab, running on local pc, or inside docker) to Mongo Atlas.

Let’s start

Preconditions

I already have:

The Raw source Data in CSV files (link)
A docker running with a MongoDB inside

Step 00: CSV data to local MongoDB (Optional)

This is an optional step, but if want to popularte the database from the CSV files, I already have the scripts in Github, but the result is a MongoDB database with data:

Local Monogo DB initial (Runing on a Docker listening on port 27018)

Step 01: Creating the Cluster on MongoDB Atlas

The steps to create the cluster are very straightfoward after creating the MongoDB Atlas account, so I will show the resulting cluster created:

On a new organization “Glaciar.org” a proyect “Glaciar Live” and The Empty Cluster

Step 02: Data Dump from tocal and Restore in Atlas

To export the data I am runing a pretty simple command with datadump:

$ mongodump --port 27018 --db viedma-r0

The reason to indicate only the port 27018, is that the host and users are the default ones: the database is running on a docker localost and the local database is not usign user and password on this very simple version on local. The export result is:

From the file system Where I had the binary database.

$ mongorestore --uri mongodb+srv://<db-user>:<db-pass>@cluster0.cccll.mongodb.net

Step 03: Very it from Compass

And the data is visible on the Compass Client

And, in the collection view

Step 04: Using Other GUI like Robo 3T

If you are using Robo 3T, you can also connect to your cluster, but it is a little tricky to connect to your replica set, compared with other clients. There is a good post about how to archive it, and the config that I use is it:

With the connection:

Configuration Connection from Robo 3T to mongo DB in Atlas

Step 05: Connect the Node JS Backend

Config the connection string for node.js was a little tricky at begining, but, after upgrading the Mongoose Driver to the last version (very important!) I could successfully connect to MongoDB using the string connection provided by MpngoDB Atlas.

So, upgrading Mongoose to “mongoose”: “5.12.2” and I was using the script

mongodb+srv://<user>:<pass>@MyCluster0.nnnnn.mongodb.net/myDatabase?retryWrites=true&w=majority

And, last, but not least, in Mongo DB Atlas you can restrict the IP Address from where you are goingo to accept connections, for example, if you have a local environment running on your local pc you should add your own local IP, but, when you are deploying in live, for example Heroku, you should open the IP address range (to 0.0.0.0) or, may be, much better, config a VPC peering between your Heroku account and MongoDB Atlas, but it is a little out of the scope of this post … but I left a link to another one

The Result in live environment:

Source Code:

Let’s see the highlights of the source code available on GitHub:

Source Code Backend config From GitHub :

/*URI like (from .env variables):mongodb+srv://<user>:<pass>@MyCluster0.nnnnn.mongodb.net/myDatabase?retryWrites=true&w=majority*/const atlas_uri = process.env.GlaciaR_Viedma_backend__MONGODB_ATLAS_URI;console.log(‘getMongoConfig(Atlas MongoDB)= ‘, atlas_uri); if (atlas_uri) { 
   return atlas_uri 
}

And the source portion where I am using the driver:

var mongoose = require('mongoose').set('debug', true)
mongoose.connect(Global.getMongoConfig(), function(err, res) {
    if(err) {
        console.log('Error connecting to the database. ' + err);
    } else {
        console.log('Connected to Database ... ');
    }
});

Backend:

Let’s check the backend data, available on Heroku

http://glaciar-org-backend.herokuapp.com/

Front End: glaciar.org

Final Words

In this post we describe a common but necessary task, because after the MLAB acquisition from Mongo Atlas, the apps needs to migrate the data and adjust the connection string to be able to keep it running, like my own “glaciar” app.

With Compass we have also the option to use a MongoDB up to 512 Mb for free in a stable environment, which it is pretty enough to labs and testing

Let see in another one!

Regards

Pablo