Job Description

JOB DESCRIPTION Principal Data Engineer As a Data Engineer, your primary responsibility is to design, develop, and maintain the infrastructure necessary for the collection, storage, and processing of large volumes of data. You will collaborate with data scientists, analysts, and other stakeholders to understand data requirements and implement data solutions that support business objectives. Key Responsibilities: Data Pipeline Development: Design and build scalable data pipelines to efficiently extract, transform, and load data from various sources into data warehouses or data lakes. . Data Modeling: Create and maintain data models to ensure data is organised, optimised for querying, and easily accessible for analytics purposes. . Data Integration: Integrate data from different systems and sources, ensuring data consistency, accuracy, and reliability. Data Quality and Governance: Implement data quality checks, monitoring, and governance processes to guarantee data integrity and compliance with regulations. . Performance Optimization: Continuously improve data processing and storage performance to ensure efficient data retrieval and analysis. . Automation: Develop and maintain automated data workflows and processes to streamline data operations and reduce manual intervention. . Infrastructure Management: Manage data infrastructure, including databases, cloud-based services, and storage systems, to ensure smooth functioning and scalability. . Collaboration: Work closely with cross-functional teams to understand their data needs and provide data engineering support for various projects and initiatives. . Troubleshooting and Support: Identify and resolve data-related issues, ensuring data availability and reliability for endusers. . Emerging Technologies: Stay up-to-date with industry trends and best practices, adopting new technologies and methodologies that enhance data engineering processes. . EXPERTISE AND QUALIFICATIONS Requirements Bachelor\'s or Master\'s degree in Computer Science, Engineering, or a related field. Proficiency in programming languages like Python, Java, or Scala. Familiarity with big data technologies (e.g., Hadoop, Spark) and cloud-based data platforms (e.g., AWS, Azure, GCP). Experience with data modelling, ETL processes, and data integration techniques. Strong understanding of database systems (e.g., SQL, NoSQL) and data warehousing developments. Problem-solving skills and ability to optimise data processes for performance and efficiency. Excellent communication and teamwork abilities to collaborate effectively with cross-functional teams. Overall 4 to 8 years of experience in managing data platforms

foundit