We are looking for a skilled Databricks Infrastructure Engineer to design, build, and manage the cloud infrastructure that supports Databricks development efforts. This role will focus on creating and maintaining scalable, secure, and automated infrastructure environments using Terraform and other Infrastructure-as-Code (IaC) tools. The infrastructure will enable data engineers and developers to efficiently create notebooks, pipelines, and ingest data following the Medallion architecture (Bronze, Silver, Gold layers). The ideal candidate will have strong cloud engineering skills, deep knowledge of Terraform, and hands-on experience with Databricks platform provisioning.
Key Responsibilities:
Infrastructure Design & Provisioning:
Design and implement scalable and secure infrastructure environments to support Databricks workloads aligned with the Medallion architecture.
Develop and maintain Infrastructure-as-Code (IaC) scripts and templates using Terraform and/or ARM templates for provisioning Databricks workspaces, clusters, storage accounts, networking, and related Azure resources.
Automate the setup of data ingestion pipelines, storage layers (Bronze, Silver, Gold), and access controls necessary for smooth data operations.
Platform Automation & Optimization:
Create automated deployment pipelines integrated with CI/CD tools (e.g., Azure DevOps, Jenkins) to ensure repeatable and consistent infrastructure provisioning.
Optimize infrastructure configurations to balance performance, scalability, security, and cost-effectiveness.
Monitor infrastructure health and perform capacity planning to support evolving data workloads.
Implement and maintain backup, recovery, and disaster recovery strategies for Databricks environments.
Optimize performance of Databricks clusters, jobs, and SQL endpoints.
Automate routine administration tasks using scripting and orchestration tools.
Troubleshoot platform issues, identify root causes, and implement solutions.
Security & Governance:
Implement security best practices including network isolation, encryption, identity and access management (IAM), and role-based access control (RBAC) within the infrastructure.
Collaborate with governance teams to embed compliance and audit requirements into infrastructure automation.
Collaboration & Support:
Work closely with data engineers, data scientists, and platform administrators to understand infrastructure requirements and deliver solutions that enable efficient data engineering workflows.
Provide documentation and training on infrastructure setup, usage, and best practices.
Troubleshoot infrastructure issues and coordinate with cloud and platform support teams for resolution.
Stay up to date with Databricks features, releases, and best practices
Required Qualifications:
8-12 years of experience in Databricks and cloud infrastructure engineering, preferably with Azure
Strong hands-on experience writing Infrastructure-as-Code using Terraform; experience with ARM templates or CloudFormation is a plus.
Practical knowledge of provisioning and managing Databricks environments and associated cloud resources.
Familiarity with Medallion architecture and data lake house concepts.
Experience with CI/CD pipeline creation and automation tools such as Azure DevOps, Jenkins, or GitHub Actions.
Solid understanding of cloud networking, storage, security, and identity management.
Proficiency in scripting languages such as Python, Bash, or PowerShell.
Strong collaboration and communication skills to work across cross-functional teams.
Preferred Skills:
Prior experience working with Databricks platform, including workspace and cluster management.
Knowledge of data governance tools and practices.
Experience with monitoring and logging tools (e.g., Azure Monitor, CloudWatch).
Exposure to containerization and orchestration technologies such as Docker and Kubernetes.
Understanding of data ingestion frameworks and pipeline orchestration tools like Apache Airflow or Azure Data Factory.
It is the policy of AT&T to provide equal employment opportunity (EEO) to all persons regardless of age, color, national origin, citizenship status, physical or mental disability, race, religion, creed, gender, sex, sexual orientation, gender identity and/or expression, genetic information, marital status, status with regard to public assistance, veteran status, or any other characteristic protected by federal, state or local law. In addition, AT&T will provide reasonable accommodations for qualified individuals with disabilities. AT&T is a fair chance employer and does not initiate a background check until an offer is made.
This one's for the grads and early careerists: Our leading internship and development program recruiters weigh in on how to prepare for and handle your interview.
Learn more
September 19, 2024ArticleCareer AdviceRelated Content
T&T’s India Development Centers (IDC) plays a pivotal role in AT&T’s connectivity strategy, and no one is better suited to speak to that importance more than Santosh Bijur, Vice President of the India Development Center
In our India Development Center (IDC), we’re building a talented technology team. By offering essential resources and the chance to work alongside industry leaders, our goal is to support the next generation of innovators in India.
