Clients that reach out to request my services typically fall into one of the categories below. Many have seen data projects consume large budgets with uncertain returns and want a more efficient approach. They know that the technical leaders develop and operate data and AI features orders of magnitude more efficiently than typical enterprises.
Companies that have unrealised potential in their data and want to raise their data capabilities and move beyond analytics or prototypes to operational production-quality data and AI products. These clients have typically relied on some combination of data warehousing, lakehouses (the new technical incarnation of data warehousing methods), or other data processing based on SQL and relational databases. While these methods were state of the art and used by the technical leaders of the 1990s, they were developed to solve analytical use cases, and are less suitable for data-powered production services and AI functionality.
Data processing that is part of a machinery and serves functionality with high reliability expectations requires quality, predictability, and reproducibility. When the technical leaders evolved from data warehousing and adopted functional data engineering methods, machine learning at scale became feasible and the AI winter ended.
I help companies evolve from data methods made for analytics to data engineering for products and AI.
These companies have organically grown data management structures, with a mix of databases, microservices, queues, etc. Implementing data use cases takes increasing effort as the structure grows. Inconsistencies and data quality problems are increasingly common. These companies seek a structured approach to data management that addresses data quality and enables them to efficiently build data features.
Companies with a mature data platform and established DataOps practices, but that need experienced data engineers to build features and improve processes and platforms. They typically understand what good data engineering looks like and want someone who can contribute from day one.
These companies want data or AI features developed and operated as a service. For some of these companies, software/data engineering lies outside their core business and they want a partner with complementary competence. Some want to use an external supplier to get a quick start, with the option to educate their staff and take over operations at a later stage.