Job Description

Lead Databricks Engineer (Job ID ZR_176_JOB)Kadel Labs is a first-of-its-kind Venture Studio in India. It is a global software technology system incubator with synergetic blend of IT Services and SaaS products under its portfolio. Since 2017, KL has grown organically (180+ employees and expanding) and served 200+ Customers. At the core, KL is a people\'s company. We serve clients by practicing an employee-centric culture and creating an environment where ideas, creativity, free-flowing communication and innovation is encouraged to create better products and services.Role: Lead Databricks EngineerExperience: 3-5 yearsLocation: Udaipur/Kolkata/Bangalore/Udaipur:We are seeking an experienced Databricks Engineer to join our team of innovators. As a Databricks Engineer, you will be responsible for designing, developing, and maintaining large-scale data pipelines and architectures using Databricks, Apache Spark, and other big data technologies. You will work closely with cross-functional teams to drive business growth through data-driven insights and solutions.Responsibilities:Cloud technologies (Azure data services Azure ADLS SHIR ADF Integration with Active directory masking and encryption Governance activities Lineage Catalogue classifications.Infrastructure setup in Azure) Experienced with multiple data-ingestion methods for loading data to Databricks:ADF python Copy Into for large volume of data as well as incremental data handling Data lake and Data Warehouse Design and optimization Data Analysis CDCMust Have Technical understanding of digital customer experience layers Been Leading and built at least 1-2 Projects on Azure cloud with DatabricksExperience with CDC and metadata driven based processing.Have build DWH models Must Have Agile methodology Nice to understand Agile and devops methodology for cloud deployment.Work with cross functional teams including QA Platform Delivery and DevOps. Not afraid of refactoring existing system and guiding the team about same.Must Have Delivery Management Able to plan and deliver project deliverable on time with quality.Requirements/Skills:8+years of demonstrable experience in enterprise level data platforms involving implementation of end-to-end data pipelinesHands-on experience in using DatabricksHands-on experience with at least one of the leading public cloud data platforms (Amazon Web Services, Azure or Google Cloud)Experience with column-oriented database technologies (e.g., Big Query, Redshift, Vertica), NoSQL database technologies (e.g., DynamoDB, BigTable, Cosmos DB, etc.) and traditional database systems (e.g., SQL Server, Oracle, MySQL)Experience in architecting data pipelines and solutions for both streaming and batch integrations using tools/frameworks like Glue ETL, Lambda, Google Cloud DataFlow, Azure Data Factory, Spark, Spark Streaming, etc.Certification on Databricks Data Engineer.Must have Delta Lake Concepts, Delta Live tables and Streaming Data.Understanding on CICD and repo.Experience in Apache Spark Configuration and optimization techniquesEducation and Experience:BTech or relevant educational field requiredVisit us:

Kadel Labs