Anyone who works in data knows that challenges come up every day. As a data consultant, I love working with organizations to find solutions to their challenges. In this article, I’ll share the most common data challenges our data consulting firm sees in our work with clients. First, you’ll get to see real-world examples of organizations experiencing these data challenges. Then I’ll provide advice on how to solve them and the benefits you can expect as a result.
Let’s dig in!
Data Challenge 1: Data Silos
Siloed data refers to a situation where different departments or systems store data in disparate sources. Modern organizations run on dozens of software solutions because no one vendor is a best-in-class solution to manage all the various aspects of a business. Unfortunately, these systems are often not well integrated, so the data is not easily accessible or shared across the organization.
Siloed data can negatively impact an organization in several ways:
- Missed sales opportunities. A company’s marketing team often stores data on customer behavior and preferences in a separate database from the sales team’s data on customer interactions and purchases. This makes it difficult to analyze the complete customer journey and identify patterns or opportunities for improvement. In turn, this can lead to missed sales opportunities and ineffective marketing.
- Increased costs. When multiple teams collect and maintain similar data sets separately, it leads to duplication of effort and increased costs.
- Poor data quality. Siloed data makes it difficult to ensure data consistency and accuracy.
- Lack of data trust. Poor data quality, inconsistent data, and data that is not readily available to business consumers all lead to lack of trust in company data. This lack of trust can impact decision making and business growth.
Real-World Example
99% of the clients Data Clymer works with struggle with siloed data. It’s especially common amongst fast-growing companies, businesses that have undergone M&A activity, and large organizations where different departments operate independently.
Here is a simple example to bring it to life. One client we worked with wanted to build a report that listed all their customers and showed two specific data points. First, they wanted to show the total value of invoices sent. This data was stored in their accounting system. Second, they wanted to show the partners involved in the projects for the customer. This data was stored in their CRM, Hubspot. They also wanted to be able to analyze that dataset and slice and dice it as needed.
This analysis would provide numerous benefits. Most significantly, they would be able to see which partners were producing the most value. This, in turn, would help them drive more revenue by focusing their strategic efforts there. But because the two systems weren’t “talking to each other” (i.e., siloed data), they couldn’t analyze the data side-by-side without a time-consuming spreadsheet exercise.
Solution
Implement a cloud data warehouse and ELT solution that centralizes and models all the data into a 360-view of the business. When done correctly, every employee will have self-service access to the insights they need to make data-driven decisions. Making a concerted effort to break down data silos will help drive a data-driven culture that prioritizes collaboration and sharing.
How Data Clymer Helped Kentik Eliminate Data Silos and Democratize Data
Data Challenge 2: Limited Reporting Capabilities
Without a modern data stack that includes a business intelligence (BI) tool, employees are left with substandard reporting options. They must either use the limited analytics capabilities included in their business software solutions or rely on manual spreadsheet applications. For instance, companies running Hubspot or Salesforce must rely on the reporting capabilities built into those solutions, which are often cumbersome and limited in functionality. Furthermore, they can only analyze the data in that specific tool at any one time. (This is the data silo referred to above.)
Real-World Example
Imagine your company wants to analyze and report on customer retention. Although it may seem straightforward, it can be nearly impossible to build in all of the business logic and visualizations required to easily tell the story and understand results. Instead, companies are usually forced to download raw data into a spreadsheet. Then they spend days or weeks developing the analysis and telling the story.
The next time that analysis needs to be updated with fresh data, it’s yet another manual effort. Depending on the size of the dataset involved, this approach may take too long, rendering the entire analysis too costly to even try.
Solution
A BI tool like Sigma or Looker can help businesses make more informed decisions by providing rapid insights from any size data set that has been ingested and curated into a data warehouse. BI tools allow businesses to present the results in a meaningful way that can be easily understood and acted upon. BI tools can also facilitate collaboration by allowing teams to share insights and data in real-time.
Implementing a BI tool can help businesses gain a competitive edge by providing timely insights. Employees can then make better decisions, improve processes, and increase efficiency.
Vertical IQ Saves Almost 1 Month Of Manual Labor Annually With Automated Product Analytics
Data Challenge 3: Slow Analytics
Data silos and limited reporting capabilities will naturally lead to slow analytics and insights. In today’s dynamic business environment, data needs to be ready to apply to a problem at a moment’s notice.
Slow analytics can hurt a company in several ways:
- Missed opportunities. Slow analytics can cause delays in identifying market trends, customer preferences, or operational inefficiencies. This delay can lead to a number of missed opportunities. Most notably, businesses may miss the chance to capitalize on emerging trends, optimize processes, or make timely decisions that can impact their bottom line.
- Poor decision-making. When data is not analyzed in a timely manner, stakeholders may have to rely on incomplete or outdated information. In these cases, decisions are often based on intuition or past experience, rather than data-driven insights.
- Reduced efficiency. Slow analytics can hinder business processes, making it harder for employees to get the information they need to perform their jobs effectively. This can lead to reduced productivity, increased frustration, and decreased morale.
- Increased costs. If an operational inefficiency is not identified and addressed in a timely manner, it can lead to increased expenses. Additionally, delayed decision-making can result in missed opportunities for cost savings.
Real-World Example
The CFO at Omnigo Software was struggling to get timely financial insights to stakeholders. The team used a massive spreadsheet to do monthly revenue reporting, which meant stakeholders couldn’t access up-to-date insights when making decisions. During strategic discussions, recent revenue insights weren’t readily available.
Slow analytics can hinder nearly every department across an organization.
For instance, consider the following examples:
- A sales rep wants to know how many customers purchased multiple products, but there is no self-service ad-hoc reporting system.
- A marketing team is having an important meeting and needs quick answers on campaign performance. But because the dataset has multiple sources, they can’t answer that question during the meeting.
- A bookings report is incorrect because the team forgot to update some Hubspot fields for a few deals. After they update the fields, they have to wait until the next data ingestion runs for the changes to show up in the report.
Solution
Implement a modern cloud data stack that includes an ELT-focused data transformation tool like dbt or Matillion, as well as an easy-to-use, self-service BI tool like Sigma or Looker.
Deploying an ELT process helps scale a data infrastructure and expedite data updates in the warehouse. A modern BI tool helps remove bottlenecks from the data team and puts the power of analysis in the hands of non-technical business users. Be sure to use data streaming or high-velocity periodicity in data ingestion jobs to help ensure the latest data is always available.
In the case of our client Omnigo Software, we created a Sigma dashboard with near real-time revenue reporting. Now, the executive team can get instant insights on where revenue is coming from so they can make better decisions. In addition, other teams are beginning to define new dashboards to drive insights and revenue opportunities. A few examples include the product development, engineering, and client success teams.
Omnigo Advances Business Intelligence Reporting with Data Clymer and Sigma
Data Challenge 4: Poor Quality / Missing Data
Even if a company has solved the previously mentioned challenges, they need to take steps to ensure the quality of their data. Poor data quality can quickly eliminate trust in data, which can lead to a complete failure of a data system roll-out.
Poor data quality can have several negative effects, including:
- Loss of revenue. Businesses may miss opportunities to increase sales or identify cost savings. For example, if customer data is inaccurate or incomplete, businesses may not be able to identify their best customers or target them with relevant promotions, resulting in lost revenue.
- Increased costs. If businesses have to spend time and resources cleaning and fixing data, this can drive higher costs. Additionally, if data quality issues are not identified early, it can lead to costly errors and rework.
- Poor customer experience. If customer data is inaccurate, customers may receive irrelevant marketing messages or incorrect billing statements, leading to frustration and dissatisfaction.
- Compliance risks. If businesses rely on inaccurate data to comply with regulations such as GDPR or CCPA, they may face penalties or legal action.
Real-World Example
The San Francisco Giants knew they had a problem with duplicate fan data. Because data was siloed across ticketing, merchandise sales, and ballpark concession purchases, they struggled to gain a clear view of their fan base. Furthermore, fans sometimes purchased tickets online without logging into their account. This meant that there were thousands of duplicate records, creating a huge barrier when trying to understand their customers, ticketing sales, partner campaign success, and venue data.
Using a powerful algorithm to normalize and match data, Data Clymer de-duplicated all of the fan engagement data and built a fan “Golden Record” with a unique customer ID assigned to each record. The result was a whopping 15% reduction in the number of records in the Giants marketing list.
Now, the Giants have a consistent and reliable source that is accessible to multiple departments. Projects that were once impossible due to level of effort are now within reach. They increased their time-to-insights by 50% by leaning on Data Clymer to implement a modern data system.
San Francisco Giants Hit A Home Run With Snowflake And Master Data Management
Solution
Data quality starts at the source. Apply data governance policies to ensure the highest quality source data. Then, in your ELT data pipelines and workloads, harden data engineering code to test for low quality or missing data and prevent issues from making it to end users.
Finally, de-duplicate and map data across multiple sources to ensure a single source of truth, as we illustrated above with the San Francisco Giants’ fan golden record.
Data Challenge 5: Failed Data Initiative
As mentioned in the previous section, lack of trust in the data can lead to a failed data initiative. In fact, this is the leading cause of failed data projects that we see.
Just like in any relationship, once trust is broken it can take a long time to recover. Once end users lose trust in the data, there can be a long period of disillusionment, which we often refer to as the “trough of disillusionment,” as shown in the Gartner hype cycle below.
To get back on track, the data team will have to work very hard to prove the quality problems have been resolved. Data projects fail for a variety of reasons, such as:
- Lack of clear goals. If a data project does not have clear and specific goals, it is likely to fail. It’s important to define what the project aims to achieve and how success will be measured.
- Lack of expertise. Data projects require expertise in statistics, programming, and data analysis. If a company lacks the necessary skills, it may struggle to design, execute, and interpret the results of a data project.
- Resistance to change. Data projects can challenge established processes and procedures, and some employees may resist the changes. This can create friction and undermine the project’s success.
- Poor communication. Data projects involve multiple stakeholders, including data engineers, analytics engineers, managers, and executives. Poor communication between these groups can lead to misunderstandings and misaligned expectations, undermining the project’s success.
Solution
A data project’s overall success depends on having a clearly defined data strategy from the outset. We’ve worked with a number of organizations that started a big data modernization project with the best of intentions—only to realize they didn’t have the right tools or skillsets to finish the job. Developing a data strategy will give you a roadmap so you can identify those gaps at the outset.
Checklist: What You Need to Implement a Data-Driven Approach
Data Challenge 6: No Data Team
Although the technology behind a modern cloud data system has improved by leaps and bounds over the past decade, working with data and extracting actionable insights is still challenging. It’s very hard to be successful without a team of data professionals to support the effort.
A data team is essential for any company that wants to make data-driven decisions and stay competitive in today’s market.
Here are the key roles and responsibilities for a data team:
- Solution architects. Modern cloud data solutions often involve 5-10 vendor tools integrated together to form a complete solution. In order to choose the right vendor to meet a specific organization’s needs, you need a solution architect. This person has a deep understanding of the pros/cons of each vendor and how they all integrate together.
- Data engineers. Data engineers have deep technical skills. They write code and configure tools to build data pipelines that ingest data into a data warehouse.
- Analytics engineers. Once data is landed and centralized in a data warehouse, analytics engineers transform the data into a business-friendly state that is ready for analytics. They understand the data and the business, so they know how to best prepare the data to answer business questions. Analytics engineers also perform analyses and develop reports and dashboards.
- Engagement manager. Engagement managers are a lot like project managers. They ensure that each data project has clear goals, a well thought-out plan, and remains with established budgets. They also ensure effective communication across the team, stakeholders, and end users.
Finding skilled data professionals with the right experience is nearly impossible in today’s market. And keeping them is even harder. Many of our clients need 6-12 months to build up a team.
Solution
Instead of building your own data team, consider hiring a data consulting firm like Data Clymer. We’ve already done all the hard work of finding and retaining excellent data professionals that have seen it all. Hiring a consultancy like us will allow you to move extremely fast; we can instantly install a full team that knows how to work together.
Need Help Solving Common Data Challenges?
No matter what data challenges you’re facing, Data Clymer can help. We offer a variety of data solutions to help organizations unlock the value of their data. Popular services include:
- Data strategy consulting. Create the roadmap for a successful data journey with data strategy consulting
- Data architecture services. Develop a customized cloud data architecture that optimizes data management, performance, and accessibility.
- Data engineering services.Our data experts specialize in providing end-to-end data engineering services across a variety of industries to make your data analytics-ready.
- Data visualization services. Develop interactive dashboards and reports to effectively convey information, break down data silos, and facilitate data-driven decision making.
In today’s challenging economy, it’s harder than ever to gain a competitive edge. Join hundreds of data-driven organizations that are finding success with Data Clymer. Contact us for a 15-minute consultation.
