MS Azure Data Architect.
About the organisation.
Our client is a leading global computer software company with a large-scale enterprise data and analytics practice. They are known for delivering sophisticated data transformation programmes to large organisations across multiple industries. The culture emphasises technical excellence, ownership and continuous innovation.
About the role.
We are migrating a large enterprise data platform from AWS to Azure, and the organisation is looking for a hands-on Data Architect to own the architecture of all ETL workloads moving to Azure Databricks. The role covers migration strategy, Lakehouse design, governance with Unity Catalog and integration with MS Fabric. You will combine deep architectural ownership with active engineering and technical leadership of the data engineering team.
What you will do.
- Define and execute the migration of Databricks workloads from AWS to Azure, including refactoring and re-architecting notebooks, jobs and Delta tables.
- Design Lakehouse architecture using Delta Lake, Unity Catalog and Databricks Workflows, and define best practices for cluster configuration, autoscaling and cost optimisation.
- Architect complex ETL pipelines using PySpark, Spark SQL and Delta Live Tables, ensuring data quality, error handling and performance.
- Implement Unity Catalog for fine-grained access control, lineage and auditing, ensuring compliance with enterprise security standards.
- Define how Databricks Delta tables will be exposed to MS Fabric via OneLake shortcuts, Delta sharing or data writes to a Fabric Lakehouse.
- Implement enterprise-grade security including RBAC, row- and column-level security, Azure Key Vault integration and encryption at rest and in transit.
- Mentor data engineers, conduct code reviews and act as the Databricks technical authority for the team.
- Design and govern Azure data services integration, including Azure Data Factory, ADLS, Event Hubs, IoT Hubs and Azure Stream Analytics.
What you bring.
- 10+ years of experience in Big Data engineering, with 5+ years specifically in Azure Databricks.
- Expert knowledge of Spark internals, Delta Lake and Unity Catalog.
- Proven track record of migrating Databricks workloads from AWS to Azure.
- Deep hands-on experience building ETL with PySpark, Spark SQL and Delta Live Tables.
- Hands-on experience with Azure Data Factory, ADLS, Azure Event Hubs, IoT Hubs or Azure Stream Analytics.
- Strong understanding of Azure networking (VNETs, Private Endpoints), IAM and cost management.
- Proficiency in Python, PySpark and SQL.
- Preferred: Databricks Certified Data Engineer Professional or Databricks Architect certification.
- Apply with your résuméUpload it and verify your email.
- It reaches the search teamYour application goes to the team running this role.
- They contact you if it fitsYour details stay confidential throughout.
