Job Description

:Essential Duties and Responsibilities:Create and maintain optimal data pipeline architecture
Overseeing activities of the junior data engineering teams, ensuring proper execution of their duties and alignment with business vision and objectives.
Assemble large, complex data sets that meet functional / non-functional business requirements.
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Work closely with the businesss Data and Analytics teams, gathering technical requirements for exceptional data governance across the department and the business at large
Learn how to write more efficient code and support the development of predictive data-driven solutions.
Create systems and procedures to acquire, organize, and clean raw data that data scientists and MIS teams.
Improves efficiency through workable thinking and the efficient use of new technologies and concepts
Create UT cases to ensure that the codebase maintains system stability and documentation.
Implement best practices and learn innovative approaches to keep codebases up to date on a regular basis.Technical Skills:Working with cloud services, data lake storage, Hadoop, Spark, Python/ Scala, Hive, HDFS, SQL and No-SQL databases.
Knowledge on creating Batch and Realtime Data pipelines with different cloud services, ETL tools and Kafka etc.
Perspective to always create optimized complex ETL mappings for relational and non-relational workloads.
Hands on Unix scripting or PowerShell.
Design DWH model, ensuring data design follows the prescribed reference architecture framework while reflecting appropriate business rules built for logical, physical and conceptual model.
Knowledge on setting up code versioning with GitHub and CICD pipelines.
Facilitate data integration, data conformity, data integrity and ensure data structures are designed for flexibility to support future business needs and enhancements.
Understand the relationships across business information and units of data, Reverse engineer data from various source systems.
Exceptional data analysis, critical thinking, communication, problem solving and cross-group collaboration skills.

TecQubes Technologies