One of the most common issues we hear from prospective clients is that they struggle to scale their data and analytics functions to meet the needs of their business. In today’s uncertain economic environment, companies can ill afford to build a data department that becomes slower and more difficult to manage as the company grows.

Data Clymer has worked with the data teams at over a hundred different organizations including Macy’s, Autodesk, and the San Francisco Giants, giving us a candid glimpse into the inner workings of real-world data teams. We’ve seen that the most high-performing teams employ at least one (if not several) Analytics Engineer(s).

What is an Analytics Engineer?

This important role proactively catalogs, controls, develops, and maintains the data warehouse to maximize its ROI to the business. While the term “Analytics Engineer” (or AE for short) is a relatively new title, effective data organizations have long employed data engineers, analysts, and data scientists to fulfill the function of AE’s.

Conversely, what we find with struggling data teams is that they spend too much time searching, cleaning, and validating data for individual tasks. This can lead to repeated wasted effort, slow project velocity, and (worst of all) a distrust of the data. As a company continues to scale up, so does the number of data sources and stakeholders. This increase in data volume and business complexity put data teams in a bad position as they must choose between reactively taking on shorter-term low ROI projects or face long lead-time development that often fail to get off the ground.

Analytics Engineers Sit at the Cross Section of Data Engineers and Data Analysts

Why should you hire Analytics Engineers?

Data and analytics have undergone immense changes in the past 10 years. Cloud Data Warehouses such as Snowflake, BigQuery, and Redshift have enabled data transformations to happen faster and easier than ever before. This has led to a paradigm shift away from Extract/Transform/Load (ETL), towards the Extract/Load/Transform (ELT) pattern employed by modern data stacks.  Simultaneously, we have seen the development of easy-to-use cloud data extraction services such as Fivetran and Stitch. These services have nearly eliminated the need for data engineers to build and maintain custom data connectors to the most common data sources.

How dbt Led to the Rise of the Analytics Engineer

In 2016, dbt Labs (then called Fishtown Analytics) open sourced the now ubiquitous data transformation tool called “dbt.” dbt has empowered data teams to implement software development principles to the transform layer using a language already familiar to all data teams: SQL. Since this development, there has been a growing need for people who understand software development principles such as version control, continuous integration / continuous development, separation between development and production environments, testing, and technical documentation.

These changes in technology have significantly shifted the balance of work across all roles on the data team. Ownership of the data warehouse transformation layer is not well-suited for most data analysts and scientists because of their lack of experience with software development principles. Data engineers are better suited for productionalizing the transformation layer, but in most cases they are too far removed from the day-to-day needs of stakeholders to build data models that serve the business.

A new class of data professional was needed to own these business critical processes and drive analytics success.

Enter the Analytics Engineer

The data warehouse sits squarely at the center of the Analytics Engineer’s universe, existing at the cross section of Data Engineers and Data Analysts. Analytics Engineers are expected to take on many responsibilities and thus must possess deep knowledge of many data warehousing and business intelligence tools.

Typical responsibilities might include:

  • Being data visualization experts
  • Building complex dashboards and trusted reporting outputs
  • Evaluating and troubleshooting data pipelines for SaaS to Python solutions
  • Creating and maintaining the CI/CD pipeline
  • Ensuring that all best practices are being tightly adhered to

Locating your next Analytics Engineer

Due to the Analytics Engineer role being relatively new and requiring such a wide breadth of skills, it can be extremely difficult to find skilled experts. One approach is to manage all the necessary steps to find, train, and retain skilled employees. Another option is to hire a data consulting firm like Data Clymer to instantly tap into a team of experienced Analytics Engineers. Our team can serve as an “instant data team” to accelerate your data initiatives. We can even help supply you with the talent you need long term through our in-house recruiting services.

Whether you are just now embarking on your company’s data journey, are looking for guidance on how to improve the effectiveness of your data team, or just need an extra hand, Data Clymer’s team of experts is ready to help you. Start your business transformation today with a discovery call with one of our experts and get set on the path towards maximizing your analytics ROI.

About Allen

Allen is a hands-on data analytics professional with a background in finance and accounting. His experience includes full data stack engineering, creating financial models, and designing data visualizations and dashboards to provide insights for data-driven decision making.