Job Description

Job requisition ID :: 69682
Date: Jul 18, 2024
Location: Bengaluru
Designation: Deputy Manager
Entity:
SkillSet - PySpark developer


Job Type: Full-time


Experience Required: 4-6 years




Job Summary:


We are seeking a talented and experienced Deputy Manager - PySpark Developer with 4 to 6 years of experience to join our dynamic team. The ideal candidate will possess extensive expertise in PySpark, big data technologies, and data engineering practices. As a Deputy Manager, you will play a critical role in developing and optimizing data processing pipelines, ensuring data quality, and collaborating with cross-functional teams to deliver high-impact data solutions.




Key Responsibilities:


• PySpark Development:
+ Design, develop, and maintain scalable and efficient PySpark applications to process large datasets.
• Write clean, maintainable, and efficient code following best practices and coding standards.
• Optimize and tune PySpark jobs to improve performance and reduce execution times.

• Big Data Technologies:
+ Work with big data tools and frameworks such as Hadoop, Hive, and Spark to manage and process large-scale data.
+ Implement and maintain data pipelines using Apache Airflow or similar orchestration tools.
+ Ensure data is ingested, processed, and stored in an optimal manner for further analysis and reporting.
• Data Engineering:
+ Design and implement ETL processes to extract, transform, and load data from various sources.
+ Ensure data quality and consistency through robust validation and testing procedures.
+ Collaborate with data scientists and analysts to understand data requirements and deliver solutions that meet their needs.
• Cloud Integration:
+ Deploy and manage big data solutions on cloud platforms such as AWS, Azure, or Google Cloud.
+ Leverage cloud services for storage, computing, and data processing to enhance scalability and performance.
+ Optimize cloud resource utilization to reduce costs and improve efficiency.
• Collaboration and Communication:
+ Work closely with cross-functional teams, including data scientists, analysts, and business stakeholders, to understand requirements and deliver data solutions.
+ Communicate effectively with team members and stakeholders, providing regular updates on project status.
+ Document technical designs, processes, and procedures to ensure knowledge sharing and continuity.
• Continuous Improvement:
+ Stay up-to-date with the latest industry trends and technologies in big data and data engineering.
+ Identify opportunities for process improvement and implement best practices to enhance productivity and efficiency.
+ Participate in code reviews and provide constructive feedback to ensure code quality and maintainability.
Qualifications:


• Bachelor's degree in Computer Science, Data Science, or a related field.
• 4-6 years of professional experience in PySpark development and big data technologies.
• Strong understanding of distributed computing and parallel processing concepts.
• Proficiency in SQL and experience with data warehousing solutions.
• Familiarity with data modeling, ETL processes, and data pipeline orchestration.
• Experience with cloud platforms (e.g., AWS, Azure, GCP) and related services.
• Solid understanding of data structures, algorithms, and software engineering principles.
• Excellent problem-solving skills and attention to detail.
• Strong communication and interpersonal skills.


Preferred Qualifications:


• Relevant certifications (e.g., Databricks Certified Associate Developer for Apache Spark).
• Experience with real-time data processing and streaming technologies (e.g., Kafka, Flink).
• Knowledge of machine learning frameworks and libraries.
• Familiarity with containerization technologies (e.g., Docker, Kubernetes).
• Experience with version control systems (e.g., Git).