Job Description

Who We AreArtmac Soft is a technology consulting and service-oriented IT company dedicated to providing innovative technology solutions and services to customers.Job Title : Databricks Solution Architect (AWS, Pyspark)Job Type : Full Time / ContractExperience : 10-20 yearsLocation : RemoteResponsibilities :

• Must have 10-20 years of experience as a Data Engineer with a strong focus on AWS cloud services with Databricks and Pyspark.
• Extensive hands-on experience with Databricks, AWS, and PySpark.
• Proven experience in designing and implementing large-scale data solutions on cloud platforms, particularly AWS.
• Strong experience in building and managing data pipelines using tools like PySpark, Apache Spark, and Delta Lake.
• Experience in data modeling, ETL/ELT processes, and data warehousing solutions.
• Expertise in integrating Databricks with AWS services (e.g., S3, Redshift, Glue, Lambda).
• Deep knowledge of PySpark and its application in data processing and transformation.
• Experience with other big data technologies (e.g., Hadoop, Kafka, Snowflake).
• Familiarity with machine learning and data science workflows.
• Proficiency in AWS services including S3, Redshift, Glue, EC2, Lambda, and others.
• Strong understanding of cloud security, networking, and cost management.
• Experience with CI/CD pipelines and DevOps practices in the context of data engineering.
• Lead the architecture, design, and development of scalable data solutions using Databricks on AWS.
• Design and implement data pipelines, data lakes, and data warehouses to support complex analytical needs.
• Architect solutions that leverage PySpark for large-scale data processing and transformation.
• Ensure high performance, scalability, and security of the Databricks environment.
• Develop data management strategies for ingestion, storage, processing, and access to large datasets.
• Lead the implementation of data governance, security, and compliance best practices.
• Optimize data storage and processing workflows to ensure efficiency and cost-effectiveness.
• Provide technical leadership and mentorship to data engineering and analytics teams.
• Collaborate with cross-functional teams including data scientists, data engineers, and business stakeholders to understand requirements and deliver robust solutions.
• Lead the troubleshooting and resolution of complex technical issues related to Databricks and AWS environments.

• Certifications: AWS or Databricks.
• Bachelor\'s or Master\'s degree in Computer Science, Information Technology, or a related field.

Artmac Soft LLC