Data Engineer - Databricks Lakehouse

About the role

Catena Media is looking for a Data Engineer to join its Data & Analytics team in Gzira. The team designs, builds, tests and maintains scalable data pipelines and data models on a Databricks lakehouse on Azure, governed through Unity Catalog, within an AI-first team where agentic AI is used every day.

Key responsibilities

• Design and develop data pipelines in Python (PySpark) on Databricks, applying best practices for performance, reliability and maintainability.
• Ingest and integrate data sources into the lakehouse in batch, following the bronze, silver and gold layering and ensuring data quality and accuracy.
• Help introduce event streaming and real-time ingestion: design, build and operate streaming pipelines on Databricks from sources such as Kafka, handling late, duplicate and out-of-order events.
• Maintain data models and the data estate codebase, governed, secured and documented in Unity Catalog.
• Write automated tests: unit tests for pipeline logic, integration tests against real data and data quality checks.
• Follow engineering standards: clean, modular code, peer code review, CI/CD, infrastructure as code (Databricks Asset Bundles) and documentation.
• Use agentic AI tools daily to build, test, review and troubleshoot, and help design and improve AI agents that automate data workflows.
• Monitor and operate the cloud data platform day to day, and collaborate with engineers, analysts and business stakeholders, supporting ad hoc analysis requests.

Required profile

• Degree in IT, Computer Science or a related field.
• Proven experience as a Data Engineer building and maintaining scalable ETL/ELT pipelines for data lakes or lakehouses, with batch and real-time data exposure.
• Experience with agentic AI, using AI coding agents in daily work and designing or integrating LLM-powered agents.
• Comfortable in a fast-paced, agile environment, with strong analytical, problem-solving and communication skills.

Required skills

• Databricks on Azure and Unity Catalog (catalogs, schemas, permissions, lineage)
• Spark Structured Streaming and Kafka
• Strong SQL and relational databases
• Databricks Workflows (Jobs) and Databricks Asset Bundles; Azure Data Factory is an asset
• PySpark and Python
• Unit and integration testing (e.g. pytest) and data quality checks
• Git, CI/CD (Azure DevOps, GitHub Actions or equivalent) and documentation