Job Description

:Job Title: Data Engineer xe2x80x93 Azure StackJob Overview:We are looking for a highly skilled Data Engineer with expertise in the Azure Stack and proficiency in Python, PySpark, SQL, Databricks, Snowflake, Azure Data Factory, Azure Synapse, and Matillion. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines and architecture that support large-scale data solutions. The role will require collaboration with data scientists, analysts, and business stakeholders to deliver real-time analytics and robust data infrastructure.Key Responsibilities:

• Design, develop, and maintain scalable ETL pipelines using Azure Data Factory, Azure Synapse, and Databricks, ensuring data integrity and performance.
• Architect and implement Snowflake data warehouse solutions, leveraging Azure Blob Storage and ensuring scalability and optimization for large-scale analytics.
• Utilize Azure Databricks and PySpark for efficient data transformation and large-scale data processing, including performance tuning and advanced querying using Spark SQL.
• Implement Medallion-based Delta Lake (DLT) pipelines to enhance data reliability and reduce processing time by optimizing data workflows.
• Build and maintain CI/CD pipelines for data code deployment using Databricks Asset Bundles, ensuring seamless transitions between environments.
• Collaborate with business analysts, data scientists, and IT teams to ensure data pipelines meet business needs and deliver high-quality data solutions.
• Write and optimize SQL queries for large-scale data processing and complex analytics in Snowflake and Azure Synapse.
• Develop and maintain comprehensive documentation for data pipelines, data flow, and configurations, and perform UAT testing to validate data transformation accuracy.
• Perform performance tuning and data optimization techniques to ensure pipeline efficiency and scalability.
• Monitor and optimize data pipelines in Azure Data Factory and Databricks for continuous improvement and performance monitoring.
• Facilitate cross-team collaboration with other departments, such as SAP, to resolve data integration issues and ensure synchronization accuracy.

• Databricks certified professional with expertise in Databricks, Python, PySpark, SQL, and Snowflake.
• Hands-on experience with Azure Data Factory, Azure Synapse, and Azure Blob Storage for scalable data architecture and ETL pipeline development.
• Expertise in building Medallion-based Delta Lake pipelines in Databricks for optimized data processing and real-time analytics.
• Proficiency in Spark SQL for optimizing queries and enhancing data transformation processes.
• Solid experience in implementing CI/CD pipelines for code deployment and environment management using Databricks Asset Bundles.
• Strong documentation skills, including creating data flow diagrams, pipeline design specifications, and UAT testing procedures.
• Experience working with large datasets and creating high-performance, scalable data architectures.
• Familiarity with Matillion for data orchestration and integration in the Azure ecosystem.
• Proven experience in data validation and quality assurance, including testing and ensuring data accuracy and integrity.
• Strong problem-solving skills, particularly in troubleshooting data processing and transformation issues.

• Certifications in Microsoft Azure and DataBricks (e.g., Azure Data Engineer, Azure Solutions Architect) are a plus.
• Experience with SAP integration for data migration and synchronization.
• Knowledge of Agile methodologies and experience working in Agile teams.
• Familiarity with additional cloud data services (AWS, GCP) is a plus.

• Strong attention to detail with a focus on delivering high-quality, production-grade solutions.
• Ability to work in a fast-paced environment and handle multiple tasks simultaneously.
• Enthusiastic about learning and exploring new data engineering tools and technologies.

EXL Service