Case study · Sensor and automation technology

A self-service data platform on Kubernetes

A data platform is only a platform when the data teams can use it without asking. Most of this work was removing the manual step, and then removing the next one.

The situation

Every connector was a small project

NiFi, Trino and the services around them, on Kubernetes, maintained by too few people.

The platform ran the company's internal data integration: NiFi for flows, Trino for queries, Kafka and object storage underneath. Onboarding a new source meant a series of manual steps across several systems, each with its own credentials and its own conventions.

Logging had grown the same way. Some services wrote to one stack, some to another, and the dashboards that were supposed to explain the platform could not see across them. Nothing was broken; everything was slower than it should have been, including the team that ran it.

What I built

The platform, the operators, the papercuts

Seven areas of work over two years, most of them ending up as an operator or a self-service path.

On-demand Kubernetes clusters with k3d and k3s for local development, a shared tooling image, and the Kubernetes trainings that made them useful.
The platform itself in Pulumi with Go, configured through CUE, with GitLab pipelines moved to Dagger so the complex cases stayed maintainable.
A SCEP issuer for cert-manager, connecting the corporate PKI to cluster certificates instead of running a second certificate authority.
An operator on vcluster that provisioned short-lived clusters for integration tests, plus the open-source tools that came out of the work: k3i, k3droot and k3dnifi.
NiFi registry, credential and connector provisioning, with MinIO and a Hive metastore behind it, so a data team could register a source themselves.
A Trino gateway with its own operator, a PostgreSQL operator with a pod-spec mutator for the security settings it did not expose, and an OCI gateway for data that lived outside the cluster.
Logging reworked end to end: Fluent Bit into S3, Grafana reading it through a Trino plugin, with user impersonation and OPA so a dashboard respected the permissions of whoever opened it.

What changed

Fewer handovers

A data team could onboard a source without opening a ticket, which was the point of the platform.
Certificates, secrets and policy came from the same place as everything else, instead of four manual processes.
Logs and dashboards answered questions with the permissions of the person asking.
The platform ran on-premise and on AKS from the same definitions.

Stack

What it runs on

Kubernetes and AKS · Apache NiFi · Trino and Starburst · Pulumi (Go) · CUE · Dagger · Kyverno and Cilium · Vault and External Secrets · Grafana and Prometheus · Parquet on S3

Data and integration-heavy internal systems

Start the conversation

Running a platform like this?

A short note with the stack, the constraint and the timing is enough for me to say whether this is a fit.

Discuss your project All selected work