Job Description

Key Responsibilities: • Azure Databricks Management: Oversee and optimize Azure Databricks environments to ensure efficient and effective data processing and analytics. Implement best practices for configuration, scaling, and performance tuning of Databricks clusters.
• Hive Metastore Management: Manage and optimize Hive Metastore integrations within Azure environments. Ensure data consistency, security, and performance for metadata management across big data applications. Configure or adjust heap memory for metastore and other configurations for increasing performance and troubleshoot jvm related issues for hive.
• Presto Operations: Administer and enhance Presto deployments to support fast and scalable SQL query processing. Implement best practices for optimizing query performance and resource utilization.
• Automation & CI/CD: Develop and maintain automation workflows for managing Databricks, Hive Metastore, and Presto deployments. Implement and manage CI/CD pipelines using Azure DevOps and GitHub to streamline deployments and updates.
• Optimization & Performance: Continuously monitor and analyze the performance of Databricks, Hive Metastore, and Presto. Provide recommendations for system improvements, resource optimization, and cost-efficiency.
• Security & Compliance: Ensure that Azure Databricks, Hive Metastore, and Presto environments adhere to security policies and regulatory requirements. Conduct security assessments and implement data protection best practices.
• Documentation & Handover: Create and maintain detailed documentation for Databricks, Hive Metastore, and Presto configurations and processes. Ensure comprehensive and up-to-date handover of documentation to support teams for ongoing maintenance.
• Knowledge Sharing: Provide training and share expertise with internal teams to enhance their understanding of Azure Databricks, Hive Metastore, and Presto. Ensure effective communication of best practices and operational procedures.
• Stakeholder Communication: Engage with stakeholders, including project managers and developers, to provide updates on platform performance, technical issues, and solutions. Ensure clear and effective communication throughout project and operational activities.
• Continuous Improvement: Stay informed about emerging Azure technologies and industry trends. Identify and implement opportunities for process improvements and contribute to the development of new tools and strategies.
Job Requirement

Required Qualifications: • Technical Expertise: Strong experience with Azure services, including advanced knowledge of Azure Databricks, Hive Metastore, and Presto. Basic understanding of Azure Kubernetes Service (AKS) is required. Proficiency in Terraform and Ansible for infrastructure management is a plus.
• Experience: 5+ years of experience in cloud technologies, including at least 3 years working with Azure Databricks and data management technologies. Demonstrated experience in managing and optimizing large-scale data processing environments.
• Skills: Excellent problem-solving skills with the ability to optimize and manage complex cloud environments. Proficiency in using Azure DevOps for CI/CD pipelines and GitHub for version control.
• Certifications: Microsoft Certified: Azure Solutions Architect Expert (AZ-305) or equivalent advanced certification is preferred. Additional certifications in Azure Databricks or related data technologies are a plus.

Desired Attributes: • Customer Focus: Commitment to delivering high-quality solutions that meet the needs of stakeholders and enhance overall operational success.
• Innovative Thinking: Ability to apply creative solutions to complex data processing and cloud challenges, continuously improving platform performance and processes.
• Team Collaboration: Strong interpersonal skills with the ability to work effectively with cross-functional teams and support collaborative operational environments.