Data analytics and machine learning are making great progress. But it’s simply the next adjacent possible in the on-going evolution of data production, storage, sharing and networked learning.
Evolution of Data…
- Data Scarcity – In the early 1990’s and prior, data was fairly scarce, both because data production was low (relative to today) and because the mechanisms to store and share data were more limited.
- Data Production – the 2000’s became the age of sensors. With advances in electronics, sensors became more affordable. Consequently, the production of sensors and the variety of new sensors increased significantly. With the vast scale of deployed sensors, data production has grown – and continues to grow – exponentially.
- Data Storage – with the surge of data production came the increased need for cost-effective data storage. At scale, data storage has now become so inexpensive, it approaches a negligible cost until data sizes are enormous.
- Data Sharing – in the 1990’s, companies generated their own internal data, but it was largely not shared outside of the company. Nor was general knowledge, except in printed form. With the advent of the internet and its subsequent broad adoption (1993+), data sharing became more prevalent than ever before (especially with the proliferation of APIs), but it remained largely unstructured.
- Structuring Data – tech-savvy industry experts saw repeatable patterns within the available information sets and began to structure it, one industry vertical at a time. Collect and collate was the next “adjacent possible”, (circa 2004 – 2016). This is in fact how PrivateEquityInfo.com got started in 2004. I identified an opportunity to structure data across private equity and related M&A firms & events. This saved people a tremendous amount of time in information discovery and professional networking and increased the efficiency in which M&A deals could be done.
- Data Analytics – with structured data, we now had the ability, through algorithms, to analyze and spot trends across data over time like never before. This helped us better understand historical contexts, enabling future projections from the historical trends, (2008+). It also provides actionable insights, at least directionally.
- Big Data – as data became increasingly more useful in decision making, we began to collect more and more of it. With data (and sufficient processing power), more is better (assuming data quality does not degrade with data expansion… often a faulty assumption).
- Smart Data & Machine Learning – as the structured data collection from sensors and various inputs began to significantly outpace our (human) ability to keep up with it, we began to employ machine learning techniques to establish relationships within the data and to create better predictors and indicators of the future. Machine learning, fed with large volumes of data is much more powerful than human-coded algorithms. From vast amounts of data, machines can tease out patterns humans can’t. They can learn faster, spot causal correlations in data and thus predict outcomes with better accuracy than humans. Smart, machine-learned, data driven algorithms, coupled with loads of data now approximate intelligence – artificial intelligence (2015+).
Networked Learning – in 2018+, machines create data, make it available in pre-determined, structured formats, share this data with other machines and have the network of machines learn from each other. Machine super-intelligence. It more closely models the complicated network of the human brain, except with more processing power, more storage capacity and faster information retrieval. This isn’t the singularity, but it will be a huge leap forward – to the point that the intelligence produced by the machines won’t feel so artificial.