Data Engineer

Job Description

Data EngineerResponsibilities:
• Develop, implement, and maintain efficient and scalable data engineering solutions to support the companys data initiatives.
• Collaborate with cross-functional teams to understand business requirements and translate them into technical data engineering solutions.
• Build and maintain data pipelines to ingest, transform, and store data from various sources, ensuring data quality, integrity, and reliability.
• Deploy and operationalize machine learning models for real-time inference and prediction on edge devices.
• Optimize data processing workflows to achieve high performance and low latency in handling large volumes of data.
• Design and implement data warehouse and data lake solutions to support data analytics and reporting needs.
• Collaborate with data scientists and data analysts to understand their data requirements and assist in the design and implementation of data solutions.
• Monitor and troubleshoot data pipelines, proactively addressing any performance issues, bottlenecks, or failures to ensure uninterrupted data flow.
• Stay updated with industry trends and emerging technologies in data engineering, edge computing, cloud services, data warehousing, and data lakes, and evaluate their potential for improving existing systems and processes.
• Continuously improve data engineering processes and systems through the adoption of best practices and innovative approaches.
• Leading a team of Data Engineers, grooming
Skills and Qualifications:
• Bachelors or masters degree in computer science, Data Engineering, or a related field.
• 6-8 years of experience as a Data Engineer, with expertise in developing scalable data solutions.
• Strong experience in data engineering, including data ingestion, transformation, storage, and data pipeline optimization.
• Proficiency in programming languages such as Python, Java, SQL for data engineering and analytics tasks.
• In-depth/ HANDS ON knowledge of cloud services, with a minimum of 4 years of experience working with AWS. Familiarity with AWS fundamental services, data services, security services, IAC services, IIOT services such as S3, Glue, Redshift, Lambda, Glue DataBrew, Athena, EMR, Kinesis, AWS Glue Crawlers, QuickSight, CloudFormation, AWS IOT Sitewise, AWS IOT Core, AWS IOT GreenGrass, etc.
• Experience in designing and implementing data warehouses and data lakes for analytics and reporting.
• Basic Understanding of MES, MIS, SCADA, etc systems that are relevant to manufacturing environment.
• Knowledge and experience with NoSQL databases (e.g., MongoDB) and graph databases.
• Understanding of data governance policies, data security measures, and relevant regulatory requirements.
• Strong problem-solving skills and the ability to optimize data processing workflows for performance and scalability.
• Excellent communication, collaboration, and attention to detail to deliver high-quality, reliable data solutions.
Key Performance Indicators (KPIs):
• Successful development and implementation of efficient and scalable data engineering solutions.
• Timely delivery of data engineering solutions that meet business requirements.
• Efficient utilization of cloud services and technologies to achieve optimal performance and cost-effectiveness.
• Ensuring data quality and reliability through robust data validation and cleansing techniques.
• Effective collaboration with cross-functional teams to understand and address their data requirements.
• Proactive identification and resolution of performance issues or bottlenecks in data processing workflows.
• Compliance with data governance policies, data security measures, and regulatory requirements.
• Continuous improvement of data engineering processes and systems through the adoption of best practices and emerging technologies.

Keyskills :
visualizationsqlpythondata engineeringsparkapachecloudetl