Automating Data Quality with AI agents
Description
In this session, we’ll explore how AI agents can automatically think of data tests, generate them, and execute them - transforming a traditionally manual, repetitive task into a self-improving system.
We’ll walk through how autonomous agents learn the structure of the data, propose meaningful validations, and even implement the code behind them.
You’ll see how this approach dramatically reduces the workload on data engineers while increasing data trust, consistency, and observability across the pipeline and across data consumers.
By the end, you'll understand what it really looks like when data begins to test itself.
Speaker
Miky Schreiber
Data Engineering platform TL, Next Insurance
Miky Schreiber is a DE platform team lead in Next Insurance, doing both management of the team and hands-on programming. They're accelerating their Data Engineering group by providing them with the best technologies, tools and data processing methodologies. As they quickly became the DevOps and SecOps team of the DE, they're building the scalable, reliable and cost-effective infra for the whole DE group. To do that, they're building in Terraform, k8s and Jenkins. They're using Spark on EMR and Redshift as our main data processing and DWH engines.
He has 24 years of experience in Data engineering and DWH development and management in many technologies - on-prem, AWS and GCP.