Job Description

We are looking for a talented engineer to develop and maintain tools/products aimed at making the data science development work easier. The right candidate will be passionate about learnings, delivering high-quality code and cutting-edge technologies in Bigdata landscape. The candidate will have evangelical mindset to identify, practice and promote better working methods in the fields of distributed computing to be used in data science engineering.What we expect from you

• Develop and maintain engineering tools/products needed for simple and efficient data science development
• Analyse complex data pipelines to identify performance bottlenecks, and suggest robust ways to optimize the workload in reasonable costs
• Proficiency in at least one programming language python/java/scala
• Good knowledge of data engineering techniques, e.g. data ingestion, data processing , data validation, data publishing, data quality etc
• Proficiency in Hadoop, PySpark, Pandas, NumPy and python version > 3.5
• Good Working experience of Partitions, Joins, cache, HDFS, handling data
• skewness and code optimisation in general
• Good knowledge of Airflow or any other data process orchestration tools like Nifi , Lungi
• Good knowledge of SQL
• Good knowledge of shell scripting and GIT workflows
• Experience in re-engineering, automating and productionising code
• Great communication skills, both written and oral
• Should have experience in handling/optimising various file formats like parquet, Avro etc
• Should have good experience in working with Hadoop Ecosystem components (e.g. YARN, Spark UI, Hive, HDFS etc.) and cloud equivalents possibly in GCP
• Working knowledge of web application development and application integration (REST APIs, Web services)

dunnhumby