- Every new data source is a small project, because the platform has no self-service path for one.
- The platform works, but two people know how it works, and neither of them is on your team.
- Access control, logs and dashboards disagree with each other, so nobody trusts what a query shows.
- A Trino, NiFi or Kafka footprint has grown quietly and the dependencies are now guesswork.
Service
Data and integration-heavy internal systems
I make internal data platforms operable: one path for a new source, one place for the logs, and an ownership model that survives the next reorganisation.
When teams call
The situation this fits
These platforms rarely fail loudly. They get slower to change.
How the work runs
Three phases, overlapping
Nothing here starts with a target architecture. It starts with what the platform actually does today.
-
01
Map the flows
I follow data from source to dashboard and note every place it is transformed, cached or copied. The output is a diagram of what depends on what, which is usually the first one anyone has made.
-
02
Decide the front door
What a data team asks for, and what they get without asking: provisioning, credentials, registry entries, query routing, access policy. This is the contract that replaces the ticket queue.
-
03
Deliver one vertical slice
One source, provisioned end to end through the new path while the team watches. Self-service that has not been used once is a diagram, not a capability.
Deliverables
What you are left with
Documentation that matches the system, and a path that removes the manual step.
- A map of the platform: sources, jobs, storage, query layer and the ownership of each.
- A self-service path for the common request, with the operators or automation that make it work.
- Query routing and access policy that match the organisation, including impersonation where that is expected.
- Logs and alerts that end up in one place with a retention story, rather than three overlapping tools.
- Operational documentation written for the people who will be on call, not for the wiki.
- A written list of what to stop running, which is usually shorter and more useful than the list of what to add.
Engagement shapes
Three ways to bring it in
From a fortnight of reading to a platform build that runs alongside your team.
Short assessment
Two to three weeks. A written map of the platform, its ownership and the three changes that would remove the most manual work.
Platform build
Three to six months, part-time. Operators, self-service paths and the query layer, built with the platform team rather than handed over.
Stabilisation
A few weeks of focused work on a platform that is failing under load, losing data, or cannot be upgraded without downtime.
Evidence
What it looks like in practice
One anonymized reference from recent work.
Case study
Industrial data: self-service integration on Kubernetes
Questions
Before a first conversation
The three questions I get asked first.
Do you replace our platform or work with it?
Work with it. These platforms are expensive to rebuild and usually close to working; the gap is normally the self-service path, the ownership, or one component nobody dares upgrade.
Do you work with specific tools?
Apache NiFi, Trino and Starburst, Kafka, PostgreSQL, ClickHouse, BigQuery, Dagster and Kubernetes come up most often. The pattern matters more than the vendor, and I say so when a tool is the wrong fit.
Can this start while the platform is in production?
It has to. Everything I do here happens on a running system, which is why the deliverables include the rollback and the on-call notes rather than just the target state.
Start the conversation
Describe the platform and where it hurts
A short note with the stack, the constraint and the timing is enough for me to say whether this is the right engagement.