Why NoSQL databases?

Figure: Google Trends - The interest on NoSQL is clearly rising

As you probably know, adoption of NoSQL databases is on the rise; and we see more and more NoSQL data stores being developed. So, what is the secret? Why is there a rise in the adoption? Here are some of the key point I think of:

- Some applications do not require all the complicated features offered by RDBMS, instead they want to scale. For example, ranking pages based on search keywords. NoSQL databases scales out (horizontal scaling) better than RDBMS.
- RDBMS is poor at handling unstructured data such as documents, user reviews, comments, etc. Certain NoSQL databases, such as MongoDB, and couchDB, are built specifically for such purposes.
- For rapid development, there is an increased desire to avoid data/schema pre-design. For example, in MongoDB the developer specify the schema in the code itself.
- Simpler APIs to access the data. For example, key-value stores such as Redis and BerkelyDB have a very simple store and fetch methods.
- Most of the NoSQL databases are open-source and works/scales better on commodity servers compared to RDBMS.
- Law latency compared to RDBMS for certain operations.
- Some applications do not require strict consistency guarantees. They can live with weaker eventual consistency (eventually everyone sees the same data). For example, twitter/facebook feeds do not need to be super consistent across all users. NoSQL takes advantage of this fact to scale better.

It's all good with NoSQL and let's switch from RDBMS to NoSQL, or is it not? NoSQL is not for all applications. In fact, not for 90% of the current business use cases. So, when you decide to use NoSQL think twice about the requirements before you jump into it. Here are some pointers to consider before you dive into long term adoption of NoSQL:

Some dangers of NoSQL:
- No schema - NoSQL databases such as MongoDB do not use fixed schemas and the schemas are not defined inside the database; schemas are defined in the code itself. While it looks great to have such flexibility, in the long run, it can create management issues if these schemas are not document properly as there is no fixed schema to look at.
- Denormalized data - maintain relationship in the code; this can create data quality issues
- Inconsistent APIs by different NoSQL providers (compared to SQL) - switching between products is not easy. So, fully understand the capabilities and limitations of the NoSQL database you are going to use before you commit to it.

NoSQL is not for you when you want:
- Strong consistency guarantees
- To execute complex queries (for example, MongoDB does not support transactions, nor joins)
- To maintain normalized data at the database to support integrity of the data.
- Strong compliance and security features - currently NoSQL databases lacks rigorous security controls
- To use transactions

While the adoption of NoSQL databases continue to rise, it is important to understand that one specific NoSQL database or a RDBMS may not meet all your organizations requirements. Depending the on the type of data you are working with (e.g. structural, documents, key-values, etc.) and the type of workload (e.g. OLTP, OLAP, batch processing, search, stream processing, etc.), you may have to utilize multiple databases in your organization. The industry has identified this need and that is why we increasingly hear about polyglot persistence. It basically means that multiple databases are utilized to build applications. We will look into polyglot persistence in another blog post.

[Aug 2014] What kind of DBMS are currently being used/built?

What kind of databases are trending? (becoming popular more)

As you can see, there is an increasing popularity towards non relational database systems. I was surprised to see the graph DBMS curve. But then again, the rise of sensors and connected devices, which produce so much graph data, should be no surprise to see the above trend.

In the chart above, it may give you the wrong impression that organizations are moving away from relational RDBMSes. However, that is not the case. All DBMSes except RDBMS together makes only a small market (around 10%). Therefore, small change in popularity for non relational DBMS is very visible. To confirm this fact, look at the chart below. Still RDBMS rules the world! IMHO, it will remain this way for the foreseeable future. 

What kind of databases are people building these days? As you can see, most software companies invest on relational systems. However, at the same time, they are investing more on other types of database systems mainly due to the following facts:
1. Amount of data is getting bigger and bigger (e.g. "every two days we create as much data as up to 2003", Google CEO)
2. Data is more connected than ever before (e.g.: rise of social media, sensor networks, connected devices; Cisco estimates 50 billion connected devices by 2020.)
3. Data is unstructured or semi-structured (e.g.: Amazon review comments, tweets, blogs)

I encourage you to visit db-engines.com to see more drilled down stats.

(Source: http://db-engines.com/en/ranking_categories)

Is relational database (RDBMS) doomed in a sea of NoSQL?

Short Answer: No. Relational databases are created out of the need to support certain business needs which NoSQL databases cannot replace. NoSQL databases addresses business use cases where relational databases fall short. 

Long Answer: Now we are going to get in to the details:

RDBMS provides strong consistency guarantees: specifically, it guarantees ACID (Atomicity - all or nothing of the transaction is committed, Consistency - transactions leave the database in a consistent state - data stays valid in terms of integrity constraints, Isolation - each transaction is independent and Durability - once committed, the data stays ) properties. Due to the strong consistency guarantee, all users/apps see the same data at the same time. Downside of such strong guarantees is that they are more difficult to scale.

On the other hand, NoSQL systems support BASE (Basically Available - available as in CAP theorem, Soft state - state changes over time, Eventually consistent - with time all users will see the same output) properties which are less strict than ACID properties. Due to the loose consistency guarantee, NoSQL systems scale out easier than RDBMS. Further, unlike RDBMS systems, NoSQL systems do not have fixed schemas to work with. Developers decide the schema in their code which may lead to faster development. 

As you can see, RDBMS and NoSQL systems demonstrate different set of properties. When you want to have fast data at the expense of providing inaccurate data at times (e.g. Twitter/FB feeds) and/or scale based on changing demand, NoSQL is a better solution. However, when you want to have super valid/consistent data and support complex and dynamic queries, RDBMS is a better solution. In fact, there are already attacks on NoSQL based systems due to lack of consistency. The recent attack on flexcoin is one such example. This teaches us to use the right tool for the job at hand. Hence, it is less like to see the demise of RDBMS technology in the near future.