The paradigm shift from conventional on-premise database servers to cloud-native database services may very well be among the most disruptive shifts in software engineering. For a long time, the process of creating a database involved configuration of physical servers, setting up expensive storage area networks (SANs), implementing manual pipeline replication, and hiring dedicated database administrators (DBAs).
Cloud databases have enabled software developers to abstract away the hardware infrastructure necessary to run the database servers. However, not only is the shift changing the perception of database infrastructure among developers, but it is also revolutionizing the software engineering industry as a whole with the help of utility-based API.
Bare Metal Infrastructure vs. Serverless Data Layers
The reason why such a dramatic paradigm shift occurs is the infrastructure abstraction itself. Working with a bare-metal infrastructure implies that an engineer will be limited by the actual capabilities of the hardware setup. Scaling up the performance of the database will include purchases of additional memory, disk space, downtime for migration, etc.
On the other hand, cloud databases offer a great level of elasticity to scale up the data layer infrastructure based on application needs. Starting from infrastructure-as-a-service (bare-metal infrastructure), continuing with platform-as-a-service (managed database platform), and ending up with serverless database services, cloud database management technologies have evolved tremendously.
A serverless database eliminates the notions of “servers,” “provisioned IOPS,” and “instanced storage.” The engineer doesn’t have to worry about the underlying infrastructure and memory/disk space needed for their application to run efficiently. Storage and computing power are separated, with compute power spinning up/down according to the needs to handle queries, and storage scaling out across distributed clusters.
In addition to that, the engineer will be charged precisely according to the number of reads/writes performed on the database and the size of storage gigabytes consumed. Such pricing enables small startups to create scalable applications capable of handling enterprise-level traffic without any upfront investments in setting up the infrastructure.
Accelerating Software Development Lifecycle
The second innovation introduced by cloud databases is the ability to accelerate the software development lifecycle. Configuring local or staging environments replicating production setups used to be a time-consuming and error-prone process. Configuration drift was almost inevitable.
However, with the help of modern cloud database platforms, one can create a database instance instantly using API. With the help of Infrastructure as Code (IaC) tools like Terraform or Pulumi, an engineer can provision a database instance for testing purposes, run automated integration tests, and destroy the instance in minutes.
Not only that, but features such as automatic backups, point-in-time recovery (PITR), and zero-downtime schema migrations are taken care of by the cloud provider. If an engineer accidentally messed up their production deployment and corrupted the database, it can be easily restored to its previous state via the cloud console or CLI.
All those features provide a great level of safety net to experiment without the fear of data loss. It enables developers to ship features much faster.
Multi-Model & Distributed Global Database Architecture
Previously, the RDBMS database was the default database system for any application. However, when web applications started being deployed across the globe and storing vast amounts of semi-structured data, conventional relational database systems proved to be incapable of handling the load.
That’s why the NoSQL movement emerged, trading consistency for scalability and flexibility. However, modern cloud database platforms have torn down those limitations with two innovations:
Multi-Model Databases: The latest generation of cloud database platforms is able to store and query relational tables, JSON documents, key-value stores, graph networks, and vector embeddings within the same database engine. The “architecture sprawl” concept, when an application uses 5+ databases, becomes outdated.
Distributed Database with Strong Consistency: Thanks to distributed computing technologies, cloud databases have overcome the limitations imposed by the CAP theorem. Modern cloud databases are capable of synchronizing data across multiple geographical locations.
Users in London and San Francisco will be able to interact with the application data layer in real-time with sub-second latency, and the cloud database will take care of data integrity and prevention of write conflicts.
Enabling AI & Real-Time Analytics Out-of-the-Box
Previously, databases were divided into operational databases (OLTP) and analytical databases (OLAP). Daily extraction of data from the production database with the help of ETL pipelines and loading it into a data warehouse for data scientists to run complex analytics queries without affecting production performance was a common practice.
However, cloud databases are revolutionizing that architecture with hybrid transactional/analytical processing (HTAP). With the help of cloud, it becomes possible to separate storage from computing resources. Therefore, it is possible to conduct analytics queries on live data without affecting production performance.
In addition to that, thanks to the emergence of AI and Large Language Models (LLM), cloud databases are introducing vector search support. It is possible to store vector embeddings together with regular data rows within the database. Thus, one can build semantic search engines, recommendation systems, and even AI-powered chatbots in their primary data store.
The Role of the Backend Engineer & Security Compliance
Securing the database infrastructure used to be a time-consuming process of constantly monitoring and tuning firewalls, operating system security patches, and user access controls. A small mistake in the process could potentially result in a catastrophic data breach.
However, thanks to cloud databases, the task of securing database infrastructure at the basic level has become the responsibility of the cloud provider. Such features as data encryption, automatic security patches, and IAM-based access control are provided out-of-the-box. In addition, cloud database compliance certifications such as SOC 2, HIPAA, GDPR, and others are inherited from the cloud platform.
All those innovations have fundamentally changed the role of a backend engineer:
The Role of the Modern Backend Engineer: Now that cloud is handling database infrastructure, the backend engineer can focus on other things. In particular, he/she will need to design efficient data models, optimize queries, create APIs, and develop data access patterns.
FAQ
Is there a risk of vendor lock-in when using cloud databases?
Yes. Usage of proprietary features of the cloud database might make migration to another vendor impossible. However, most modern cloud databases rely on open-source database engines such as PostgreSQL, MySQL, or Redis. Thus, sticking to standard SQL and keeping database dialect compatibility will reduce migration issues.
How does cloud databases compare to local databases in terms of network latency?
Since all requests to cloud databases require network hops through public/private cloud networks, there will always be network latency for each request compared to querying the database running locally. Those problems are solved by using connection pools, deploying application servers to the same availability zone as the database, and caching with the help of Redis.
Are cloud databases more expensive than hosting a database on a dedicated server?
For bursty or small-scale workloads, cloud databases are substantially cheaper, since you pay only for what you use. However, for large-scale or 24/7 databases, cloud infrastructure can be more expensive than leasing physical hardware. However, considering the cost of maintaining the infrastructure and backing up the data, cloud databases prove to be cheaper.
What is the difference between unmanaged and managed cloud databases?
Unmanaged cloud databases (Infrastructure-as-a-Service or IaaS) still imply configuring, managing, and securing a bare-metal server in the cloud, installing database software, performing backups, upgrades, and auto-scaling. Managed cloud databases (Platform-as-a-Service or Serverless) take care of all these aspects and provide database access via an endpoint.
