Data for dummies and the dodgy data dilemma

Data is the new global currency. Infinite in its scale and application and priceless in its value. Done right, it’s more influential to businesses and markets than gold or oil. A handful of datapoints can help leaders form a hypothesis on the current state of business affairs, but expanded exponentially, you can predict and influence a future that would make you the envy of Nostradamus and Marty McFly.

Data, in its simplest form, is a singular point of information captured in a large storage space known as a database. To quote Sting from The Police…

“Every breath you take (tracked by your heart rate app),
every move you make (tracked on your maps application),
every bond you break (tracked via your social media unfriend or unfollow actions),
every step you take (tracked via your smartwatch),
[someone] is watching you (tracked across millions of databases)”…and trying to predict or influence what you’ll do, say, or purchase next.

For data newbies, the response to this realisation is usually a mix of “how do I protect myself?” and “how can I use this to my advantage?”.

For New Zealand's $13.6 billion dollar agricultural, forestry, and fishing industry, good data can mean the difference between a bountiful harvest profit and a poor harvest peril.

In 2012 the United States witnessed one of the worst droughts in 50 years. One third of US counties across 29 states were considered disaster areas. Water reservoirs were barren. The farming models didn’t anticipate the severity of the drought and farmers who followed the model’s recommendations were left with drastically reduced yields exacerbated by planted crops that were less drought resistant.

While the drought may be an extreme example, Kiwis are neither immune to the detrimental impacts of bad data or the financial opportunities of good data. In manufacturing, a customised digital twin – aka a virtual copy that takes the data from the physical world and simulates it in a virtual one – can be used to predict when machines might fail or need maintenance, when to change inventory levels, and how to train staff on using complex machinery without risking breaking the machine, the person, or disrupting the supply chain. In agriculture, data can predict the best planting seasons, identify the most suitable crop varieties, and mitigate risks associated with the poor application of environmental data that leads to issues with water quality. In healthcare, the data opportunities are endless with automated and equalized resource allocation for rostering schedules, improved tracking of disease outbreaks across the system, and enhanced electronic health records to improve hospital triaging and urgent care wait times. The opportunities are limited only by the few resources required and a little bit of imagination to see the potential outside of the routine.

Navigating data management can feel like herding sheep with a chihuahua. Many organisations with a bit of legacy behind them are often herding that data via dozens of Microsoft Excel spreadsheets sitting across multiple Windows folders with very little connectivity between them. Now imagine if, instead of using a chihuahua, you could quickly build a pathway where every data point constantly evolves and gets properly positioned without the need of a yappy dog or a manager asking you where it is. That pathway is the data lifecycle and it consists of eight stages.

Figure out what data is important to help you make decisions right now, what will help make decisions later, and what data will only lead to analysis paralysis because it adds no value. If you have it already, great!

Now that you know what you want, work with a partner or on your own to find the best way to collect it. This can be done through hardware (physical systems like point of sale devices), or software (programs).

To use the sheep example, in data processing, you can check the total number of sheep herded by the chihuahua (validation), herd those sheep into their appropriate pens (sorting), wash the sheep to clear out the dirt you don’t need (data cleaning), shear the sheep to make it presentable (transformation), and finally drop all the good looking sheep wool together to create a fashionable cashmere sweater (aggregation).

Simply put, it’s where it all gets saved. Best to have a backup in case you lose the key to the barn.

This is the governance portion. It deals with who has access, how they access, how thick the lock has to be, and how to continually ensure that the data you need is the data you’re getting.

Step 6: Data analysis

Data analysis is about processing the data to look for patterns, correlations, anomalies, or trends. It has a quantitative focus centred around structure and exploration. This can be automated through iterations and machine learning (AI), or done manually with independent models.

Data visualisation is the “user experience” side of the data lifecycle. It allows the decision maker to quickly visualize and communicate what is happening often in real time.

The final step is the piece that assigns meaning to it all. It combines the subjective with the objective. The goal here is to translate the findings and create actionable insights, conclusions, and decisions based on expertise, data, and external factors like domain knowledge (and sometimes politics).

There’s no doubt that “Data”, “machine learning”, and “artificial intelligence” are highly complex subjects that have created an industry out of the world’s most brilliant minds including those at Company-X. Though it doesn’t have to be complicated. You can work with people that make it simple. Good data creates an environment for informed decisions, more time to focus, and the ability to run the world better and it all begins with a single cell. Managed correctly, you’ll find yourself unlocking a priceless value that takes you back to a profitable, predictable future.

Interested in learning more about data? damian.bartolomucci@companyx.nz and let's have a chat.