How to take care of the number one problem getting value out of big data

Get the benefits of big data without overhauling your IT architecture

By Gavin Morrison, MD of Cubic Blue, a Knowledge Integration Dynamics company


Data-driven insights give you a competitive edge but they must be quick, incorporate more information about your markets, customers, competitors, and products.

That’s why big data is so crucial. Simplistically, big data includes social media posts, e-mails, Word documents, photographs, and information from a variety of sensors known as the Internet of Things (IoT). Check out this Data Science Series from EMC, called Ten Practical Big Data Benefits, for some practical applications of big data.

One of the problems you’ll encounter as you broach big data in your business is what to actually do with the data. Forget project management, change management and business intelligence (BI) for the moment – where do you put all the data so that you can use it?

In most businesses the answer would be the data warehouse. It’s where all the other data goes and the BI environment runs off that just fine. But data warehouses don’t do unstructured data well – and big data is a lot of unstructured data.

Unstructured data is stuff like photos, Word documents, e-mails, plain text, audio and video files, social media posts, and data from sensors that make up the IoT.

Data warehouses have another problem: they rely on IT to collect, collate, analyse, and create views into the data. That can be a slow process that takes a couple of weeks at best. You need information now. It’s how you become agile, flexible, responsive – all those strategic goals that senior executives like you talk about but which can be frustratingly difficult to achieve.

Folders versus metadata
Metadata-driven, federated databases make big data possible through clarity and transparency.

Image Credit: John Norris

The IT vendors tell you they have the solution. Often they do but it requires a great deal of change: new systems, new databases, new warehouses, new servers, new storage, processing capacity, in-memory computing – but it’s expensive, it takes a lot of time to deploy, your IT department’s not always keen on the changes, the enterprise architect wants months to survey the impacts – and all the while the business units clamour for more information, now, so they can deal with the issues that keep them from meeting KPIs and overall business objectives.

That’s precisely the reason a federated, metadata-driven database was developed. It’s responsive, it works with your existing technology investments, it retains the integrity of your IT architecture and framework, yet it also allows you to bring big data into your business quickly, effectively, and inexpensively.