The last year has seen the world of big data consumed by intrigue, as accusations abound that dark forces are using it to subvert democracy and tear apart the very fabric of society. The collection and use of our private data by the likes of Facebook has come under increasing scrutiny and its public perception is generally negative, stringent regulations, such as GDPR, have been implemented to try and correct this but has also complicated how organisations approach their data strategy.
Earlier in the year a survey by NewVantage reported that 48.4% of companies reported achieving measurable results from their big data investments – the first time the survey has found a near majority since it began in 2012. Furthermore, a massive 80.7% of executives characterised their big data investments as ‘successful,’ compared to just 1.6% who said they believed they had failed.
Companies are collecting more data than ever, and there are new technologies and strategies to help exploit it to its full potential.
Analytics Adoption Rises Among Small Companies
Large companies long ago realised the importance of data and analytics. In an IDG study last year, 78% of larger employers agreed data collection and analysis have the potential to completely change the way they do business. However, small companies are still behind in their efforts. According to an SAP-sponsored global survey of small businesses, many are still in the early stages of digital transformation. Further research from One Poll found that 56% of SMEs rarely or infrequently check their business’s data, while 3% have never looked at it at all.
This has started to change in 2018. In the SAP survey, 73% and 87% of SME surveyed indicated that their expectations regarding technology investments were met or exceeded. With the cost of data analysis and visualisation technologies falling and investments bearing fruit, the adoption of data analytics has started to extend to even the smallest of companies.
Outsourcing Of Analytics Increases
One option for SMEs unable to fund full-scale programs is to outsource them to an outside agency that specialises in data analytics.
Outsourcing analytics is an excellent way of enhancing your data capabilities when you lack the necessary funds, making it ideal for small companies. In 2018, though, the ever-growing problem of a lack of qualified data practitioners has also led to many larger organisations looking to the expertise of specialist third party companies and contract hires. According to a recent PwC study, 69% of employers by the year 2021 will demand data science and analytics skills from job candidates, yet at the moment just 23% of graduates have the necessary skills to compete at the level employers demand. In a recent report sponsored by Dun & Bradstreet, 27% cited skills gaps as a major obstacle to their current data and analytics efforts. Of these, 60% said they are already using third parties to support their data project work and 55% are outsourcing some or all of their analytics needs.
Qualitative Data Is On The Up
Organisations are starting to give more credence to qualitative data, realising that quantitative data alone cannot help you truly understand customer behaviour and market trends as it does not allow you to properly understand human emotion. It does not account for the ebbs and flows of people’s motivations and feelings and its insights can quickly become invalid as a result. Qualitative data bridges these knowledge gaps. It is the information found in the unstructured data of online reviews, social media, and so forth, that provides the context to help understand why something is the way it is and if it is changing.
When we rely solely on big data, we end up with a warped sense of the world in which human beings are simply numbers to be fed into an algorithm. This is not to say it is useless, nor that in many cases it can be used alone. It is still a powerful and helpful tool that companies should invest in. However, companies are also investing in gathering and analysing qualitative data to uncover the deeper, more human meaning behind big data.
GDPR Leads To New Opportunities
On the 25th May 2018, the EU General Data Protection Regulation (GDPR) came into full effect after years in the making. The impact of GDPR has been felt by organisations across the globe, applying to every piece of data that touches the countries that signed – regardless of where in the world the data has been captured and analysed. This has had tech giants up in arms at what they perceive to be a war on their power, and they will have their work cut out for them continuing to do so with the new rules.
This has had many ramifications for companies in terms of the data they can collect and how they use it, but it also presents an opportunity to companies that have prepared fully and embraced GDPR, not just in terms of compliance but in rebuilding some of the trust with consumers that has been lost in recent years.
It will also help companies become better at managing their data as the new legislation has put data front of mid and forced companies to rethink their processes. As CFO of SAP, Emmanuelle Brun Neckebrock recently wrote,
‘Data is one of your company’s most valued resources, yet one of the most poorly managed. It’s the golden thread that runs through the entire organisation, and in most instances, it’s managed casually and inconsistently, depending on individual employees and departments. You wouldn’t let your revenue, products, or equipment assets be handled that way, so data (given its inherent value) shouldn’t be any different. It warrants the same due care and attention. GDPR legislation is unique in that it allows you – OK, forces you – to transform the way you handle data across the whole organisation, managing associated risks and compliance. In doing so, it’s actually strengthening your ability to compete on the digital playing field, making you more agile for long-term success.’
Companies Look To External Data
External data is any data generated from outside an organisation. It can come from a variety of sources. For example, the UK government alone makes available more than 46,000 datasets on the government-run website data.gov.uk. Another example is weather, which is particularly useful in supply chain management. Tesco is renowned for its use of weather data to drive richer insights that help them to predict sales and stock requirements. They reported in 2013 that they had managed to save £6m per year and reduced out-of-stock by 30% on special offers. In fact, in a recent survey of supply chain professionals by the Met Office, 47% cited weather as one of the top three factors external to their business that drives consumer demand. Of these, 57% said they had better sales forecast accuracy, 51% that they had better on-shelf availability, and 43% that they had reduced waste.
In 2012, Forbes writer Dan Woods argued that companies were suffering from what he called ‘data not invented here syndrome’, and were failing in their data initiatives because they focused solely on using data created inside the four walls of their business. This is a situation that has largely not improved and to some to some extent, understandable. There are real risks around the use of external data and it is not right for every industry, but it can be a real driver of growth when applied correctly. In 2018, greater accessibility to external information and greater maturity in data programs should see those organisations for whom it makes sense far better positioned to use it.
