Migrating from MongoDB (back) to PostgreSQL
We helped a customer move from MongoDB back to PostgreSQL. It’s an interesting story about some of the challenges of scaling a database platform and ways that PostgreSQL provides very reliable, robust, scalable data services.
The customer is a software company that provides mobile enterprise security and threat protection. Their product provides security for mobile devices by identifying and eliminating apps that perform malicious actions. The product is used as a gatekeeper to install apps on enterprise managed devices, providing white lists, black lists, approved apps, private app stores, data security, digital rights management, etc. The product is in use at over 50 large (F500) enterprises and runs on a few million handsets.
For this MDM product, the data model supports typical e-commerce or app store functionality including ratings, classification and categorization, comments & reviews, and version information as well as additional enterprise functionality such as delegated authority and combining both isolated per-customer data and shared common data for ratings and compatibility comparisons.
Fortunately, this customer has had success in the market and a growing user base — along with data and query volumes. Additionally, as their customers deployed the MDM solution to more countries, they wanted a to operate in widespread data centers and wanted availability across data centers.
At the time of conversion, the SaaS backend supports a moderate workload, with 1,000 queries per second with 100 concurrent connections from 7 application servers primarily serving API requests from installed mobile apps. The database was growing to 50GB in size.
Problems with Mongo
About two years earlier, this customer had moved from PostgreSQL to MongoDB because they believed a NoSQL solution would be easier to scale and offer application benefits for the fast changing data model. They were excited by the high availability, no single point of failure, and semi-structured data model capabilities.
However, in their actual experience, MongoDB did not scale. They encountered problems with the single master node failure model and trying to add a new node to the Mongo cluster to freeze or become busy re-shuffling data. They were unable to find a partition scheme that worked for their data that Mongo could actually support in different regions and growing a single cluster across data centers did not work well enough for them.
Moving Back to PostgreSQL
We recommended AWS RDS for Postgres and migrated the customer’s data platform from MongoDB back to PostgreSQL. We architected a multi datacenter deployment using a single write node and multiple read replicas across data centers using streaming replication. We leveraged the JSON capabilities in Postgres to support a dynamic and flexible data models essentially providing NoSQL inside a relational database.
The customer has been very happy with both the reliability and overall high availability of the deployment and with their ability to scale the data layer to support more users and higher query volumes. Originally we planned to introduce a caching tier to maintain high performance with growing queries, but the customer has been able to scale by adding read replicas and has chosen to keep the architecture simpler by not including an additional technology layer.