Platform development

I lead technical work on a lightweight, opinionated data platform using standard cloud and open source components. I guide teams on pipeline development, architectural decisions, and compliance.

I have developed a lightweight playbook to achieve data engineering efficiency that matches the leading tech companies, but with minimal technical investment. If this playbook is followed, my clients can expect > 10x improvement on key data efficiency factors in comparison to typical data pipeline development in data warehouses or lakehouses:

• Operating 100s of data pipelines per developer, not dozens
• First value in weeks, not months
• Return on investment in months, not years
• 1–10 KEUR/month cloud costs, not 100–1000 KEUR

This engagement mode is suitable for organisations with software engineering competence and ambition to evolve their data processing capabilities to build a foundation that enables sustained data and AI innovation. The numbers above are from before the era of AI coding agents; with an AI-powered pipeline factory (see below), the development speed is higher still.

AI-powered pipeline factory

Recent AI evolution has enabled “software factories”, where AI agents autonomously build, test, and evaluate software from specifications, guided asynchronously by humans. Data pipelines are a particularly good fit for this way of working. Functional data pipelines are deterministic functions from input datasets to output datasets. This makes them straightforward to specify, test end-to-end, and evaluate, so they can be made safe to hand over to agents.

I built my first customised, fully automated software development bot in 2021, based on the development automation principles we used in Productivity Engineering at Google, and presented it at Jfokus 2022 in “Secure software supply chain on a shoestring budget”. Before large language models (LLMs), it could only perform simple changes, such as upgrading or removing dependencies, but the architecture fit coding agents perfectly. Today, it autonomously creates, tests, and evaluates complex data pipelines and surrounding software from specifications.

This is still an experimental field and there are few solid established practices. I can use the experience I have obtained running an automated software development bot for five years, including one year enhanced with LLM agents, to help clients build their agentic software delivery pipelines, or enable client development teams with AI-assisted development practices.

Pipeline development

In companies with an existing data platform, I build solutions for data-powered features, complex analytics, or AI applications. I work within clients’ infrastructure and integrate with one of their data teams.

This works best for companies with mature DataOps practices or an ambition to develop them. I can contribute data and AI feature design, data pipeline development, data architecture guidance, and risk management processes for compliance and quality assurance.

Hosted custom data pipelines

For companies where software or data engineering is outside core business, I develop and host custom data processing as an external service.

The custom data pipelines run on Scling’s data platform Orion, which is designed to avoid lock-in. The platform and data pipelines are portable between clouds, built on open source components, and designed to allow handing over operations to another supplier if desired.

This engagement mode provides data and AI capabilities without requiring clients to build and maintain digital infrastructure.

Business process automation

The companies that are most successful with data and AI today started with automating their chores and business processes. Data engineering, amplified by AI agents where applicable, is more cost-efficient for automation than robot process automation (RPA) or commercial tools, and it is more reliable than AI agents alone.

This mode is applicable for any company and yields swift return on investment.

Strategy and advisory

For companies shaping their data and AI strategy, I offer advice, presentations, and workshops on the following topics:

• Realising data and AI value in practice
• Data architecture and operations
• Organising teams for data success
• Privacy and compliance without compromising innovation and velocity
• AI-assisted development of data pipelines and software

See the list of conference presentations for further potential topics.

My engagements often start with strategic or advisory work, but I primarily provide advice when there is potential for a future implementation engagement.