Job Description

Job Title: Middleware Engineer
Position: Data Engineer
Experience: 5-6yrs
Category: IT Infrastructure
Main location: India, Karnataka, Bangalore
Employment Type: Full Time
Qualification: Bachelor's degree in Computer Science or related field or higher.
Roles and ResponsibilitiesData Engineer - 5-6 years experience.
Responsibilities

• Design, develop, and maintain data architectures, pipelines, and workflows for the collection, processing, storage, and retrieval of large volumes of structured and unstructured data from multiple sources.
• Collaborate with cross-functional teams to identify and prioritize data engineering requirements and to develop and deploy data-driven solutions to address business challenges.
• Build and maintain scalable data storage and retrieval systems (e.g., data lakes, data warehouses, databases), fault-tolerant, and high-performance data platforms on cloud infrastructure such as AWS, Azure, or Google Cloud Platform.
• Develop and maintain ETL workflows, data pipelines, and data transformation processes to prepare data for machine learning and AI applications.
• Implement and optimize distributed computing frameworks such as Hadoop, Spark, or Flink to support high-performance and scalable processing of large data sets.
• Build and maintain monitoring, alerting, and logging systems to ensure the availability, reliability, and performance of data pipelines and data platforms.
• Collaborate with Data Scientists and Machine Learning Engineers to deploy models on production environments and ensure their scalability, reliability, and accuracy.

Requirements:

• Bachelorxe2x80x99s or masterxe2x80x99s degree in computer science, engineering, or related field.
• At least 5-6 years of experience in data engineering, with a strong background in machine learning, cloud computing and big data technologies.
• Experience with at least one major cloud platform (AWS, Azure, GCP).
• Proficiency in programming languages like Python, Java, and SQL.
• Experience with distributed computing technologies such as Hadoop, Spark, and Kafka.
• Familiarity with database technologies such as SQL, NoSQL, NewSQL.
• Experience with data warehousing and ETL tools such as Redshift, Snowflake, or Airflow.
• Strong problem-solving and analytical skills.
• Excellent communication and teamwork skills.

Preferred qualification:

• Experience with DevOps practices and tools such as Docker, Kubernetes, or Ansible, Terraform.
• Experience with data visualization tools such as Tableau, Superset, Power BI, or Plotly, D3.js.
• Experience with stream processing frameworks such as Kafka, Pulsar or Kinesis.
• Experience with data governance, data security, and compliance.
• Experience with software engineering best practices and methodologies such as Agile or Scrum.

Must Have Skillsdata engineer with expertise in machine learning, cloud computing , and big data technologies.
Data Engineering Experince on multiple clouds one of them , preferably GCP
data lakes, data warehouses, databases
ETL workflows, data pipelines,data platforms
Hadoop, Spark, or Flink
Hadoop, Spark, and Kafka
SQL, NoSQL, NewSQL
Redshift, Snowflake, or Airflow

ScaleneWorks People Solutions