Sudipa DuttaRoy_why big data projects fail.png

Big data is no longer a technological fad. Analytics and big data will drive major innovation and disrupt established business models in the coming years across all industries. The key to big data success is to take an iterative approach with incremental investments in resources and learn by failing early and often.

A major part of big data is experimentation and building up gradually, which requires patience from stakeholders as well as those involved in the build process. But more often than not big data projects do not seem to achieve the expected results and frequently fail for the same reasons.

So why do big data projects fail? Some of the frequent reasons for big data failures are:

1. Data silos

Most organisations have their business data compartmentalised into silos like financial data, CRM data, web analytics data etc. Moreover, the data is often inaccessible or unusable which is contradictory to the essence of big data.

Fragmented analytics initiatives like Marketing Analytics, CRM Analytics, Social Media Analytics, HR Analytics yield little benefit and the insights are confined to one particular business area.

Big data does not necessarily have to be big, but the purpose of big data is to be able to present a holistic view of every aspect of the business which demands comprehensive end-to-end architecture. Robust big data solutions take in data from relational databases, multi-dimensional data, flat files, Excel sheets, XML, JSON, as well as REST and SOAP API calls from multiple data sources and in the process, bridge the silos. The raw data is then cleansed and harnessed by applying business rules and the insights are then presented in visual interfaces for the business to consume and make fact-based decisions.

2. Lacking the right skills

Big data is a relatively new field and hence there is a shortage of people with the right skill-set. When organisations place analytics under IT and expect wonders it does not work because IT is not always business savvy and may not be capable of asking right questions that could lead to business insights.

Placing analytics solely under business does not work either as the business side lack the technical know-how and it becomes a tedious affair to implement ideas.

Big data is quite an ambiguous field, resting somewhere in between business, engineering and statistics, making it tricky to find people knowledgeable in all of the above three but the solution is finding people who have both technical as well as business understanding. And because it is difficult to find people who fit the bill, it might be a good idea to train business people in the field of big data and analytics and vice versa.

3. Wrong use cases

If companies are new to big data, the chances are that the data is not in a useable format and requires time and effort. But since every other organisation is doing it, most companies are in a haste to catch up without thorough soul searching – i.e. setting priorities, creating a conducive environment by hiring the right people and having the right technological enablers.

Companies start with an overly ambitious project that they're not yet ready for like AI but leapfrogging analytics for AI will undoubtedly fail. It is important to have the basics in place as far as big data is concerned, like a robust data platform which can integrate multiple data sources and multiple data formats, and which is scalable and ensures data quality.

Once the data platform is established and it becomes easier for the business to retrieve data for decisions, companies can then embark on initiatives that involve more complex usage of data.

When starting the process of a big data journey it is important to define the right strategy, considering the current and future technology landscape and business goals, have the right people on board and adopt an iterative approach to big data.

If you are interested in further discussions on this topic or require advice regarding your data or innovation strategy, do not hesitate to contact me.

Want us to keep you updated with our latest news? Sign up for our newsletter.

We help you find the right expertise.

Want to collaborate with us?

Want to become one of us?

Sign up for our newsletter

How can we help you?

Write to us, we will guide you to the right person.