Data Engineer Lead

eNGINE is seeking a Data Platform Manager to lead the design, evolution, and optimization of a modern cloud-based data ecosystem built on Microsoft Azure and Databricks. This individual will serve as both a technical leader and hands-on contributor, driving enterprise data initiatives that support analytics, reporting, machine learning, and business intelligence across the organization.

This role combines architecture, engineering leadership, and platform ownership. The ideal candidate brings deep expertise in Databricks and Azure data services, along with a passion for building scalable solutions, mentoring technical teams, and delivering reliable data products that enable informed business decisions.

Working closely with senior leadership, this individual will help shape the organization's data strategy while ensuring the platform remains secure, performant, and ready to support future growth.

Responsibilities

Data Platform Leadership

• Provide technical leadership and mentorship to a team of data engineers and platform specialists.
• Establish engineering standards, development best practices, and code review processes.
• Partner with business stakeholders, analysts, and data science teams to translate business requirements into scalable technical solutions.
• Support production operations by helping resolve data platform incidents and identifying opportunities for continuous improvement.

Cloud Data Engineering

• Design and deliver enterprise-grade data pipelines within Azure Databricks and the broader Azure ecosystem.
• Build efficient data ingestion frameworks capable of processing both batch and real-time data sources.
• Develop scalable ELT/ETL solutions that ensure data quality, availability, and consistency across multiple systems.
• Enable seamless integration of data originating from APIs, databases, event streams, and third-party platforms.

Data Architecture & Lakehouse Strategy

• Define and maintain a Lakehouse architecture leveraging Delta Lake and Databricks best practices.
• Implement and optimize Bronze, Silver, and Gold data layers to support analytics and operational workloads.
• Develop logical and physical data models that support reporting, advanced analytics, and future scalability.
• Drive architectural decisions around storage, partitioning, performance, and long-term maintainability.

Platform Performance & Optimization

• Continuously improve Spark workloads, Databricks clusters, and query performance to maximize efficiency and reduce cloud spend.
• Identify opportunities for workload optimization through partitioning, indexing strategies, and execution plan analysis.
• Monitor platform health, scalability, and reliability while implementing proactive performance improvements.
• Establish governance around cluster configuration, auto-scaling policies, and resource utilization.

Governance, Security & Compliance

• Implement enterprise-grade data governance utilizing Unity Catalog and Azure-native security services.
• Ensure appropriate role-based access controls, data lineage tracking, and data protection measures are in place.
• Promote data quality through validation frameworks, monitoring processes, and standardized governance practices.
• Collaborate with security and compliance teams to align data management practices with organizational requirements.

Advanced Analytics & Machine Learning Enablement

• Partner with data scientists to operationalize machine learning solutions and deploy models into production environments.
• Support MLOps initiatives using modern tooling and automation practices.
• Deliver data products that empower business intelligence, predictive analytics, and data-driven decision making.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Systems, or a related technical discipline.
• 5+ years of experience designing and building enterprise data platforms in cloud environments.
• 2+ years of hands-on experience with Azure Databricks and large-scale Spark-based data processing.
• Previous experience leading, mentoring, or managing data engineering teams.
• Strong understanding of modern data architecture patterns, including Lakehouse, Data Mesh, and Master Data Management concepts.
• Demonstrated ability to work directly with business stakeholders and translate business needs into technical solutions.

Technical Expertise

Azure & Cloud Services

Experience with:

• Azure Databricks
• Azure Data Factory (ADF)
• Azure Data Lake Storage (ADLS Gen2)
• Azure SQL Database
• Azure Synapse Analytics
• Azure Networking (VNETs)
• Azure Key Vault
• Azure Identity and Access Management

Programming & Development

Strong hands-on experience with:

• Python (PySpark)
• SQL
• Scala
• PowerShell
• Java

Data Engineering & Big Data

Experience building and supporting solutions utilizing:

• Apache Spark
• Apache Kafka
• Airflow
• dbt
• MLflow
• Distributed data processing frameworks
• Batch and streaming data architectures

Databases & Storage

Knowledge of:

• Databricks
• SQL Server
• Teradata
• BigQuery
• Relational and NoSQL database technologies
• Parquet, Avro, and ORC file formats
• Data compression and storage optimization techniques

Preferred Certifications

• Microsoft Certified: Azure Data Engineer Associate (DP-203)
• Databricks Certified Data Engineer Professional
• Microsoft Certified: Azure Solutions Architect Expert

Next Steps

No C2C, relocation, referral, or sponsorship candidates for this role.

For finer details on how eNGINE can impact your career, apply today!