I am a veteran data engineer. I help companies raise their data innovation capabilities beyond data warehouses, lakehouses, analytics, and prototypes, to enable robust data and AI products. My work is founded in the data architecture and operational patterns that we pioneered at Spotify, which became a key factor in the company’s success.
In 2013, I joined Spotify’s core data team where we built and rolled out a unified data platform that democratised data for both analytics and product features. We pioneered continuous deployment, statistical process control, and other operational processes in data lake environments – practices that later became known as DataOps.
Most contemporary platforms were enforcing and strongly prescriptive, but we instead built a platform and data engineering process around documented guidelines and incentives, inspired by the success of Google’s bottom-up process improvement program “Test Certified,” which I had been part of a few years earlier. The concept of an enabling, documented, incentive-based process spread within the company, was named “Golden Path,” a term that eventually became a core platform engineering concept and spread to other companies – including Google, closing the circle. I also wrote the first draft for Spotify’s own Test Certified program, which later was forked to a “Test Certified for Data”.
The team behind Discover Weekly, Spotify’s most popular AI feature, later attributed their success to the fact that “the company had empowered bottom-up innovation” – which is what we aimed to achieve with the enabling platform and process. The first usable version of Discover Weekly was built in a few weeks.
Few Scandinavian companies can match this level of innovation with data and AI. My aim is to spread these capabilities beyond a small elite of technical leaders.
After Spotify, I set out to replicate that innovative capability in smaller environments. At Bonnier News, I led construction of a common data platform with a different approach: minimal technical investment, removing process waste, adopting agile practices driven by use cases, while strictly adhering to the functional data engineering methods that had been successful at Spotify. We had satisfied customers after weeks and return on investment after months – a pace that was very rare at the time. Bonnier News’ data platform is today one of Sweden’s most efficient and successful.
I have used this architecture in every platform since. One distinctive feature is automated end-to-end pipeline testing – something we recognised as important at Spotify but never implemented.
I built my first neural network in 2004, and have implemented applications with all generations of AI, from rule-based systems, machine learning, and classic natural language processing, to deep learning and large language models. I prioritise end-user value over technology, however, and strive to build balanced systems, using simple, deterministic, and robust components where they suffice, and applying AI technology where it shines.
The new generation of AI coding agents fits well with my passion for fully automated DevOps, which enables agents to validate their work, and with functional data engineering / DataOps, which enables agents to automatically evaluate pipeline changes on production data without operational risk. Reliable agentic systems use agents for dynamic and challenging tasks, supported by deterministic machinery to offload predictable tasks and to enforce guardrails.
My focus at Google was productivity engineering: automating the development process so that engineers could spend their time on the work that machines cannot do. Google had “software factories” long before LLMs enabled more complex automated changes, and by 2015, automated systems committed more changes than humans at Google. Based on the code automation principles I learnt at Google, I built a customised, fully automated software development bot for our data platform in 2021, which I presented at Jfokus 2022. Before LLMs, it could only make simple changes, such as upgrading or removing dependencies, but its architecture turned out to fit AI coding agents perfectly.
Today, the bot is a software factory that autonomously creates, tests, and evaluates complex data pipelines and software from specifications. There are other upgrade bots and LLM-based software factory structures, but they typically focus on what code changes to make and less on the more difficult challenge: assessing whether changes are safe and beneficial, including automated evaluations on production data.
Pure functional data pipelines enable automated offline evaluation, and the automated end-to-end pipeline tests in my playbook give agents the same safety net that they give human developers. Most data processing environments are built on data warehouses, lakehouses, or streaming services, where many data assets are “pets” that need to be handled with prudence. They are not naturally suited for automated, agent-driven data engineering.
I use this data pipeline factory in my own work, and use the five years of experience from evolving it to help clients accelerate their data and AI development with AI-assisted engineering.
I have led the development of 9 data platforms since 2013, in media, retail, IoT, construction, healthcare, and telecom. Notable engagements include:
Consumer products With 1.5 developers over 3 years, we built all the company’s user-facing offline data processing – 162 pipelines producing 700 datasets/day. We averaged 80 commits and 35 production deployments per month, and operated 8 Kubernetes clusters at 15–20 KEUR/month – a fraction of the cost of the existing legacy solutions, which were built by a larger team with contemporary technology from major providers, but found inadequate for operational data processing. These productivity numbers predate the era of AI coding agents (commits and deployments are no longer a reliable productivity metric).
Retail With 1–3 developers over one year, we built 70 pipelines producing 3,700 datasets/day covering transactions from e-commerce and physical stores, procurement, and demand forecasting. Cloud costs were 2–3 KEUR/month.
Telecom With 2–5 developers, we built a data platform and ca 100 pipelines for SMS route optimisation, financial reporting, and malice detection (fraud, phishing, etc.) with machine learning, producing 12,000 datasets/day at cloud costs below 10 KEUR/month.
Construction Primary technology provider in a three-year method innovation effort. Out of approximately 5 tools and data-driven approaches tried, one had massive measurable financial impact. Construction projects using the tool improved margins by several percent, a significant number in a vertical with small margins.
AI safety Built AI risk monitoring and mitigation for LLM-powered consumer features in media, using LLM-as-a-judge methods to detect inappropriate user input and agentic system output, in order to provide safe user experiences.
Media Founded a team that built financial data processing pipelines to strengthen compliance and financial and operational risk management in preparation for one of the largest initial public offerings among Swedish startup unicorns.
Address: Scling AB, Kivra: 559151-6231, SE-106 31 Stockholm, Sweden Tel: +46 70 768 71 09 Email: info@scling.com
Theme adapted from Assembly by Templated.co
Header photo by Jon Flobrant on Unsplash
Site built with Hugo
This site sets no cookies.