One of the most common issues that we hear from prospective clients is that they struggle to scale their data and analytics functions to meet the needs of their business. In today’s fast moving digital economy, companies can ill afford to build a data department that becomes slower and more difficult to manage as the company grows. 

Data Clymer has worked with over a hundred different company data teams including Macy’s, Autodesk, and the San Francisco Giants giving us a candid glimpse into the inner workings of real world data teams. This has shown us that the most high performing teams employ at least one (if not several) Analytics Engineer(s). This important role proactively catalogs, controls, develops and maintains the data warehouse to maximize its ROI to the business. While the term “Analytics Engineer” (or AE for short) is a relatively new title, effective data organizations have long employed data engineers, analysts, and data scientists to fulfill the function of AE’s.

Conversely, what we find with struggling data teams is that they spend too much time searching, cleaning, and validating data for individual tasks. This can lead to repeated wasted effort, slow project velocity, and (worst of all) a distrust of the data. As a company continues to scale up, so does the number of data sources and stakeholders. This increase in data volume and business complexity put data teams in a bad position as they must choose between reactively taking on shorter-term low ROI projects or face long lead-time development that often fail to get off the ground at.  

How we got to Analytics Engineers.

Data and analytics have undergone immense changes in the past 10 years. Cloud Data Warehouses such as Snowflake, BigQuery, and Redshift have enabled data transformations to happen faster and easier than ever before. This has led to a paradigm shift away from Extract/Transform/Load (ETL), towards the Extract/Load/Transform (ELT) pattern employed by modern data stacks.  Simultaneously, we have seen the development of easy-to-use cloud data extraction services such as Fivetran and Stitch. These services have nearly eliminated the need for data engineers to build and maintain custom data connectors to the most common data sources.

In 2016, dbt Labs (then called Fishtown Analytics) open sourced the now ubiquitous data transformation tool called “dbt”. dbt has empowered data-teams to implement software development principles to the transform layer using a language already familiar to all data teams: SQL. Since this development, there has been a growing need for people who understand software development principles such as version control, continuous integration / continuous development, separation between development and production environments, testing, and technical documentation. 

These changes in technology have significantly shifted the balance of work across all roles on the data team. Ownership of the data warehouse transformation layer is not well suited for most data analysts and scientists because of their lack of experience with software development principles. Data engineers are better suited for productionalizing the transformation layer, but are in most cases too far removed from the day-to-day needs of stakeholders to build data models that serve the business. A new class of data professional was needed to own these business critical processes and drive analytics success.

Enter the “AE”. 

The data warehouse sits squarely at the center of the AE’s universe, existing at the cross section of Data Engineers and Data Analysts.  AE’s are expected to take on many responsibilities and thus must possess deep knowledge of many data warehousing and business intelligence tools. Typical responsibilities might include:

  • Being data visualization experts.
  • Building complex dashboards and trusted reporting outputs.
  • Evaluating and troubleshooting data pipelines for SaaS to Python solutions. 
  • Creating and maintaining the CI/CD pipeline.
  • Ensuring that all best practices are being tightly adhered to.

Locating your next AE 

Due to the AEs role being relatively new and such a wide breadth of skills are required to fulfill the role, it can be extremely difficult to find skilled experts. One approach is to manage all the necessary steps to find, train and retain skilled employees.  Or, you can take advantage of Data Clymer’s team of experienced Analytics Engineers. We can implement a team for you or even help supply you with the talent you need long term through our in house recruiting services. 

Whether you are just now embarking on your company’s data journey, are looking for guidance on how to improve the effectiveness of your data team, or just need an extra hand, Data Clymer’s team of experts are ready to help you. Start your business transformation today with a discovery call with one of our experts and get set on the path towards maximizing your analytics ROI. 



About Allen

Allen is a hands-on data analytics professional with a background in finance and accounting. His experience includes full data stack engineering, creating financial models, and designing data visualizations and dashboards to provide insights for data-driven decision making.