Senior Databricks Platform Engineer (AWS)

Responsibilities
• Environment Provisioning & Configuration
• Set up Databricks workspaces in designated AWS regions.
• Configure networking (VPC, subnets, security groups) and establish secure connectivity.
• Implement storage and credential configurations, including KMS encryption keys for data security.
• Access Management & Security
• Address WIZ Violations
• Define and implement role-based access controls (RBAC) using Unity Catalog.
• Integrate Databricks with Vanguard’s Identity Provider (SSO) and manage authentication flows.
• Ensure compliance with security standards and governance controls.
• Platform Integration & Data Access
• Validate external data access strategies and configure connectors for S3 and other data sources.
• Enable ingestion frameworks and orchestration of notebooks for data pipelines.
• Operational Enablement
• Support CI/CD deployment pipelines for Databricks notebooks and jobs.
• Implement monitoring for cluster lifecycle, cost optimization, and performance tuning.
• Assist with troubleshooting and error handling during execution.
• Compliance & Governance
• Apply data governance and ML/GenAI governance controls as per enterprise standards.
• Ensure audit logging and lineage tracking for all data assets.
• Collaboration & Documentation
• Work with internal teams (EADP, CTO, ES&F) to align with architecture strategy and best practices.
• Document configurations, processes, and operational runbooks for future reference.

Qualifications
• Strong Databricks Platform knowledge with AWS, Proficiency in AWS services such as CFT, EC2, S3, Lambda, and RDS, along with the AWS SDK and CLI and Terraform.
• Familiarity with AWS infrastructure as code tools like CloudFormation for automating cloud deployments.
• Understanding of serverless architecture and microservices, with hands-on experience in implementing scalable AWS solutions.
• Should have grasp of security best practices within AWS, including IAM roles, VPCs, and encryption mechanisms.
• Strong analytical and problem-solving skills, essential for debugging and optimizing AWS resource utilization.
• Good communication skills, enabling clear and effective interaction with both technical and non-technical team members and stakeholders.