‘Big data’ is not a new trend, but the analytics behind modern data is revolutionising industries.
Data in its natural state, both structured and unstructured, is useless unless it is developed into a tangible asset which can be governed, migrated, cleansed and even traded between businesses. The emergence of ‘data lakes’ enables this. In this digital age companies are frequently depositing large Corporate datasets into these virtual ‘data lakes’; the vast data which will fill this ecosystem is precious to their existence. These wise waters are often the source of intelligent decision-making.
Given the scale of data being collected, it is perhaps not surprising that 90% of data will never be used. This underlines the ambiguity of what to catch and how to digest it, and form just some of the challenges facing businesses. Can it be made searchable, usable or even available for monetization?
What companies are trying to discover in the data lakes are the most prized assets of value-adding data; the big catch, so to say. To do this, businesses are taking on solutions which encompass the latest technologies, such as ‘Hadoop’ and ‘Spark’. These open-source frameworks are used to store and process big data sets, and are fundamental to enabling companies to react to the ‘story of the data’, especially between consumer and business. One very desirable output is for interactive and real-time data visualisation, ensuring operational and customer-facing teams are always working with accurate and reliable data – essential for “plain sailing” on the data lake!
These solutions are allowing business end-users to quickly visualise patterns and relationships, inform interpretation and decision making, and generate richer insights. To note, the Global Data Visualisation Applications Market is expected to reach $8.33 billion by 2022. Its use is prominent in many industries from cellular communications to the life-sciences industry. The popular urban transport app, ‘Citymapper’, has recently embarked on a trial in London of a new data-driven bus system. Using the wealth of data it has collected since 2011 and integrating it with its original functioning app, the ‘Smart Bus’ is to occupy new routes, monitor real-time traffic information and modify routes to service the greater flow of users in the City. Further afield, biomedicine puts the use of ‘big data’ to the test when retracing previous medical records, easily and effectively sharing information for future prescriptions and diagnosis.
This abundance of data will grow the digital universe to 175 zettabytes by 2025, creating new strategies for business. Those who can adopt techniques like algorithms and machine learning will be the ones who thrive. Consumer facing businesses can monitor activities that provide an open source digital footprint which can in turn supply raw quantifiable material which allow advancements in analytical techniques to breed innovation. L’Oréal’s charming collaboration with ‘The Hair Coach’ app is testament to this theory, generating recommendations and top hair tips by drawing upon personalised analysis on brushing techniques and hair quality, as well as intelligence on localised weather – humidity and temperature. All the while subtly encouraging users towards their products in the process.
Admittedly, we have had much of this data available for many years and there needs to be an emphasis on closing the gap between big raw data and useful (value-adding) outcomes. Through new analytical techniques such a machine learning, anomalies in the data can be minimised to prevent outliers causing unintended outcomes. Furthermore, the technique of federated data managements avoids having to store the data in one single repository. When combining Corporate data with open source data , companies can take the first step in casting a more intuitive net over their data, in the hope of catching the big fish.