A good part of this decade had been ruled by Hadoop with its different software suits as far as big data was concerned. The Hadoop distributed file system or HDFS revolutionized data storage all by itself. The concept of utilizing data for various avails was quite old.
The technology required to store and process large amounts of data was unavailable at that point. When big data did come into industrial practice, it was confined within the elite sphere of enterprises owing to the sheer lack of affordability.
HDFS changed the scenario for the small and midsized enterprises which wanted to capitalize on the data. Hence, it was pretty big for the better part of the decade.
The data processing unit of Hadoop, MapReduce, was never as popular as the HDFS, but still, many people used it as their primary tool.
The decline of Hadoop
We can divide the journey of some software into a few steps, which can help you imagine the graph of its rise, stagnancy and decline. Once the software is introduced, there is a peak of expectation; then, there is usually a form of disillusionment that the software is incapable of actually creating the value expected from it.
Here we can see the curve drop. After this, there is a period of stagnation and a plateau in productivity. In this phase, there is parity between the expectations and the result. People start investing with caution. Then comes the inevitable decline prompted by the arrival of new technologies and changes in the market.
Hadoop’s position is arguably in between the second and the third stage. Some people would say that Hadoop is as good as dead. But I would say there is still value to be acquired from a Hadoop certification, especially in Southeast Asia.
The real situation
With the advent and popularity of the cloud, the distributed file system has taken the back seat, and quite naturally. With the elasticity and simple handling that the cloud provides, distributed storage does seem a backdated option.
Hence most organizations are getting their data out of HDFS, and those still using it will probably not continue for very long.
Does this mean that Hadoop does not have any value? Nothing could be further away from the truth. As part of the Hadoops compute unit, Apache Spark, which works in tandem with Hadoop, is one of the best-known names in the data analytics sphere.
Spark is famous for its processing speed, and with the elasticity of data storage provided by the cloud, Spark is killing it in the market.
So, if you are planning to learn Hadoop, go for it by all means because there are a handful of people on the planet who are skilled in Hadoop. But make sure you are learning stuff that is in tune with the industry requirements.