Responsibilities

1. Solution Architecture & Design

1. Lead the design and implementation of end-to-end data platforms on Azure Databricks, integrating with Azure Data Lake, Synapse, and other Azure services.
2. Define data architecture blueprints for ingestion, transformation, storage, and consumption layers, ensuring scalability, performance, and cost-efficiency.
3. Translate business requirements into technical solutions with well-defined architecture diagrams, data flow models, and solution roadmaps.
4. Establish best practices for ETL/ELT pipelines, data modeling, and governance within Databricks environments.

2. Technical Leadership & Delivery

1. Guide engineering teams in developing big data and machine learning pipelines on Azure Databricks using PySpark, Delta Lake, and MLflow.
2. Provide hands-on leadership in building reusable frameworks for data ingestion (batch/streaming), transformation, and orchestration (ADF, Synapse, Event Hub, Kafka).
3. Ensure data quality, lineage, and security are embedded into all solutions delivered.
4. Troubleshoot performance bottlenecks and optimize compute clusters for efficiency.

3. Governance, Security & Compliance

1. Define and enforce governance standards for access control, cluster policies, and workspace management.
2. Align data solutions with GDPR and UK-specific compliance standards.
3. Implement monitoring, logging, and cost-control mechanisms across Azure resources.

4. Stakeholder Engagement & Collaboration

1. Work closely with business stakeholders in the UK timezone to gather requirements, validate designs, and present architectural recommendations.
2. Collaborate with cross-functional teams (Data Engineers, Data Scientists, BI Analysts, Cloud Engineers) to deliver end-to-end solutions.
3. Act as a trusted advisor to clients, presenting technology choices, cost implications, and roadmap options.
4. Mentor junior engineers and architects, building internal capability in Azure Data & AI stack.