Why NoSQL databases?

Figure: Google Trends - The interest on NoSQL is clearly rising

As you probably know, adoption of NoSQL databases is on the rise; and we see more and more NoSQL data stores being developed. So, what is the secret? Why is there a rise in the adoption? Here are some of the key point I think of:

- Some applications do not require all the complicated features offered by RDBMS, instead they want to scale. For example, ranking pages based on search keywords. NoSQL databases scales out (horizontal scaling) better than RDBMS.
- RDBMS is poor at handling unstructured data such as documents, user reviews, comments, etc. Certain NoSQL databases, such as MongoDB, and couchDB, are built specifically for such purposes.
- For rapid development, there is an increased desire to avoid data/schema pre-design. For example, in MongoDB the developer specify the schema in the code itself.
- Simpler APIs to access the data. For example, key-value stores such as Redis and BerkelyDB have a very simple store and fetch methods.
- Most of the NoSQL databases are open-source and works/scales better on commodity servers compared to RDBMS.
- Law latency compared to RDBMS for certain operations.
- Some applications do not require strict consistency guarantees. They can live with weaker eventual consistency (eventually everyone sees the same data). For example, twitter/facebook feeds do not need to be super consistent across all users. NoSQL takes advantage of this fact to scale better.

It's all good with NoSQL and let's switch from RDBMS to NoSQL, or is it not? NoSQL is not for all applications. In fact, not for 90% of the current business use cases. So, when you decide to use NoSQL think twice about the requirements before you jump into it. Here are some pointers to consider before you dive into long term adoption of NoSQL:

Some dangers of NoSQL:
- No schema - NoSQL databases such as MongoDB do not use fixed schemas and the schemas are not defined inside the database; schemas are defined in the code itself. While it looks great to have such flexibility, in the long run, it can create management issues if these schemas are not document properly as there is no fixed schema to look at.
- Denormalized data - maintain relationship in the code; this can create data quality issues
- Inconsistent APIs by different NoSQL providers (compared to SQL) - switching between products is not easy. So, fully understand the capabilities and limitations of the NoSQL database you are going to use before you commit to it.

NoSQL is not for you when you want:
- Strong consistency guarantees
- To execute complex queries (for example, MongoDB does not support transactions, nor joins)
- To maintain normalized data at the database to support integrity of the data.
- Strong compliance and security features - currently NoSQL databases lacks rigorous security controls
- To use transactions

While the adoption of NoSQL databases continue to rise, it is important to understand that one specific NoSQL database or a RDBMS may not meet all your organizations requirements. Depending the on the type of data you are working with (e.g. structural, documents, key-values, etc.) and the type of workload (e.g. OLTP, OLAP, batch processing, search, stream processing, etc.), you may have to utilize multiple databases in your organization. The industry has identified this need and that is why we increasingly hear about polyglot persistence. It basically means that multiple databases are utilized to build applications. We will look into polyglot persistence in another blog post.

Introduction to Oracle 12c Multitenant Database - Part 1

Up until 12c, Oracle database had one instance can work with only one database (barring RAC - Real Application Clusters; multiple instances share one single physical database). Before we look at what 12c is, let's try to understand why one instance - one database approach is not optimal with the current advances in hardware and software. Usually in an organization, for different applications, one or more databases are used. Often, these databases run on multiple physical servers, possibly in different operating systems. Usually one database instance per one server. The servers are grossly underutilized. What can we do about it?

Figure: Oracle legacy database (11g and before) - one-to-one mapping

One possible solution is to run multiple instances-databases in the same server. It works to some extent to support a few database instances per machine, but the approach still is not so efficient in terms of resource utilization. The reason is that multiple instances do not share background processes, system/process memory or Oracle meta-data. Further, it is difficult to management many database instances given the fact that some organizations deploy thousands of them. This naturally leads to the question whether we can create a shared resource architecture to support multiple instances.

Figure: Oracle legacy database - each application uses multiple database instances (source: http://goo.gl/HznYKe)

Another possible solution is to isolate database files for different applications using a virtual private database (VPD); that way we do not have duplicate background processes, system/process memory or Oracle meta-data. However, it is difficult manage security, isolation, and transport data in VPD.

Having seen the pros and cons of the above two approaches, it is logical think if we can combine the best of both and create a better database system. Yes, you guessed right, that's exactly what Oracle 12c (c stands for cloud) has done! Oracle 12c is called CDB (Container Database) or Pluggable Database.

CDB consists of two levels. A root container called CDB$ROOT and one or more pluggable databases PDBs. A PDB is very similar to a legacy database except that it shares background processes, memory and meta-data with other PDBs. Unlike VPD based solution, at the same time, it provides stronger security and isolation controls, and further easier provisioning via cloning and portability via pluggable feature.

Figure: The same legacy deployment with new CDB - consolidated to one machine (source: http://goo.gl/HznYKe)

Think like a USB device to be a PDB and computers with USB ports as CDB container. Once the CDB$ROOT is up and running, you can plug in any number of PDBs to the CDB. There is a seed PDB called PDB$CDB which you can use to clone new PDBs. You can move (plug/unplug) PDBs between CDBs.  

Figure: CDB - Multitentant architecture

We will look further into the CDB concepts in the next blog post.

[Aug 2014] What kind of DBMS are currently being used/built?

What kind of databases are trending? (becoming popular more)

As you can see, there is an increasing popularity towards non relational database systems. I was surprised to see the graph DBMS curve. But then again, the rise of sensors and connected devices, which produce so much graph data, should be no surprise to see the above trend.

In the chart above, it may give you the wrong impression that organizations are moving away from relational RDBMSes. However, that is not the case. All DBMSes except RDBMS together makes only a small market (around 10%). Therefore, small change in popularity for non relational DBMS is very visible. To confirm this fact, look at the chart below. Still RDBMS rules the world! IMHO, it will remain this way for the foreseeable future. 

What kind of databases are people building these days? As you can see, most software companies invest on relational systems. However, at the same time, they are investing more on other types of database systems mainly due to the following facts:
1. Amount of data is getting bigger and bigger (e.g. "every two days we create as much data as up to 2003", Google CEO)
2. Data is more connected than ever before (e.g.: rise of social media, sensor networks, connected devices; Cisco estimates 50 billion connected devices by 2020.)
3. Data is unstructured or semi-structured (e.g.: Amazon review comments, tweets, blogs)

I encourage you to visit db-engines.com to see more drilled down stats.

(Source: http://db-engines.com/en/ranking_categories)

Is relational database (RDBMS) doomed in a sea of NoSQL?

Short Answer: No. Relational databases are created out of the need to support certain business needs which NoSQL databases cannot replace. NoSQL databases addresses business use cases where relational databases fall short. 

Long Answer: Now we are going to get in to the details:

RDBMS provides strong consistency guarantees: specifically, it guarantees ACID (Atomicity - all or nothing of the transaction is committed, Consistency - transactions leave the database in a consistent state - data stays valid in terms of integrity constraints, Isolation - each transaction is independent and Durability - once committed, the data stays ) properties. Due to the strong consistency guarantee, all users/apps see the same data at the same time. Downside of such strong guarantees is that they are more difficult to scale.

On the other hand, NoSQL systems support BASE (Basically Available - available as in CAP theorem, Soft state - state changes over time, Eventually consistent - with time all users will see the same output) properties which are less strict than ACID properties. Due to the loose consistency guarantee, NoSQL systems scale out easier than RDBMS. Further, unlike RDBMS systems, NoSQL systems do not have fixed schemas to work with. Developers decide the schema in their code which may lead to faster development. 

As you can see, RDBMS and NoSQL systems demonstrate different set of properties. When you want to have fast data at the expense of providing inaccurate data at times (e.g. Twitter/FB feeds) and/or scale based on changing demand, NoSQL is a better solution. However, when you want to have super valid/consistent data and support complex and dynamic queries, RDBMS is a better solution. In fact, there are already attacks on NoSQL based systems due to lack of consistency. The recent attack on flexcoin is one such example. This teaches us to use the right tool for the job at hand. Hence, it is less like to see the demise of RDBMS technology in the near future.

Life is like a camera man

Life is like a camera man
Focus on what is important .. you can neither do everything nor make everyone happy
Capture the good times .. positive attitude helps succeed in life
Share the captures with the loved ones.. sharing is caring
If you miss an opportunity, keep looking for the next one .. when one door closes, many other opens
Sometimes you have to wait patiently .. but the reward is high
With time you become better at taking shots .. practice makes you perfect
Develop from the negatives .. mistakes are what makes you stronger
And if it does not look good, take another shot.. never give up, try try and try
Recharge the camera regularly .. life is not a rat race .. stop and take a moment to enjoy as you shoot away
Take good care of the camera as it is too precious to loose .. your health is wealth

(Taken from elsewhere and altered)

[Thoughts] Why are we so reluctant to change?

I am no motivational speaker or a self-help guru .. but I was pondering about the fact that how reluctant we are to change. We are very comfortable to continue to what we do even if there is a new and better way of doing it. Why are we so reluctant to change? In my opinion, it is the culture and education that we are brought up. Our culture has taught us that failure is a bane; failure is look down upon. We fear failure when thinking about change. Further, our education system has taught us to follow the path others had followed before us. Taking risks or trying something new is not rewarded unless you succeed. It is so deeply rooted that we sometimes say "the known devil is better than the unknown angel" to avoid change. This has to change! Even the cooperate culture is no different. Organizations are also very reluctant to change even though the change provides many benefit. The following picture depicts this fact nicely:

[Cloud] Cloud Computing Definition

Since the term cloud computing is a relatively new term compared to other technologies in the IT field, different people/organizations have come up with different definitions. Out of all of them, in my opinion, NIST definition is by far the best in terms of clarity and completeness.

NIST definition of cloud computing is quite simple and upto the point:

• Essential characteristics
• On-demand self-service - provisioning of computing capabilities without any human intervention
• Broad network access - capabilities are available over the network/any device
• Resource pooling - multi-tenant model to serve multiple customers from pooled resources
• Rapid elasticity - capabilities can be elastically provisioned or released based on demand
• Measured service - resource usage can be monitored to provide metering capability
• Service models
• Software as a Service - consumers have access to applications running on provider's cloud - zero software installation; e.g. Salesforce, Oracle CRM
• Platform as a Service - consumers have control over deployed apps - consumers are provided middleware running in provider's cloud
• Infrastructure as a Service - consumers have control over OS, storage and deployed apps - consumers are provided virtual machines

The following diagram illustrate the above three service models:

(Source: Wikipedia)

• Deployment models
• Private cloud - provisioned for exclusively used by one organization
• Community cloud - provisioned for used by one community with similar interest
• Public cloud - provisioned for open use by general public - e.g. Amazon cloud service
• Hybrid cloud - any combination of the above three

The following diagram from Microsoft provide some more information on the deployment models:

(Source: Microsoft.com)