Job Description

REQUIREMENTS Minimum 8 to 13 years of experience in Databricks data engineering with overall 10+ years IT experience Must have experience in Pyspark, Scala, python codes Must have good experience in data bricks architecting Must have experience in Azure data factory and synapses Must have good experience in analysis code and downstream Good understanding on Ingestion optimization methodologies Good understanding on CDC, SCD and delta data analysis in data bricks Must have good exposer on delta lake Must know target data model or Lift and shift methods Good understanding of data partitioning strategies Must have experience in converting business requirement into technical concepts Azure certification is preferred. (optional) Must have Unix and Windows background. Good to have experience with Azure Synapse, PowerBI. Proven ability to work with large cross functional teams with good communication skills. RESPONSIBILITIES Design and implement Azure Databricks best practices Design and implement Databricks data Ingestion methodologies Design and implement CDC, SCD and Delta-Lake Understand/assessment of larger Azure Databricks ecosystem Identify Databricks Inter dependency with ADF and Synapse from the client team Induce watermarks and process pathways based on Data Volume and velocity in Databricks. Redesign partitioning strategy based on the assessment Define Strategy roadmap for Data Ops optimization Understand Databricks data volume movement to the destination Understand data consumption for reporting Identify prerequisites for launching an Azure Databricks workspace. Deploy an Azure Databricks workspace in the Azure portal. Launch the deployed Azure Databricks workspace. Access the Admin Console in the deployed Azure Databricks workspace. Describe identity provider and Azure Active Directory integrations and access control configurations for an Azure Databricks workspace. Describe security standards and configurations for compliance, including cluster policies, Bring Your Own Key, and audit logs. Design and implement Azure Databricks security best practices Enable additional cluster configurations and purge deleted objects from workspace storage. Configure access control for workspace objects, such as notebooks and folders. Configure access control for clusters, pools, and jobs. Ability to build from good relationships with clients/operational managers and colleagues Design, implement, configure, and maintain Azure SaaS/PaaS/IaaS Cloud Services for hosting wide range of complex projects - desirable. Resolve issues and provide technical assistance on the Azure cloud-hosted environment Create and maintain Cloud security strategies, policies, procedures, and documentation Perform and evaluate cost analyses and vendor comparisons of data systems to ensure cost-effective, efficiency and measures feasibility of various approaches and makes recommendations Needs to have a continual improvement mindset while working within a team centric environment. Explain encryptions and permissions available for data protection, such as identity provider authentication, secrets, and table access control. Provide work guidance to less experienced personnel.

foundit