Senior Cloud Data Engineer

Optomi

Senior Cloud Data Engineer (Hybrid in Minneapolis)

Optomi, in partnership with a flagship manufacturing company, is seeking a Senior Cloud Data Engineer for their Minneapolis, MN location! The right Data Engineer will come into this role with 6+ years of professional cloud data and analysis experience with relevant technologies such as SQL, PySpark, and Scala. The right candidate will enjoy working on highly visible projects for a Fortune 500 employer!

What the right candidate will enjoy:
• Hybrid role (Remote / On-Site)!
• The opportunity to work in a fast-paced retail environment!
• Highly competitive compensation!
• A collaborative team culture with the chance to work independently!
• Growth opportunities!

Experience of the right candidate:
• Bachelor's Degree in computer science, Information Technology, or a related field.
• 8+ years’ experience with cloud data and analytics platforms such as AWS, Azure, or GCP.
• 5+ years of experience working with data & analytics concepts such as SQL, ETL, ELT, reporting and report building, data visualization, data lineage, data importing & exporting, and data warehousing.
• 3+ years of experience working with general IT concepts such as integrations, encryption, authentication & authorization, batch processing, real-time processing, CI/CD, automation.

Responsibilities of the right candidate:
• Develop and implement strategies for processing and analyzing large volumes of IoT data, ensuring scalability and performance.
• Design, develop, and maintain streaming and batch data pipelines in our AWS cloud environment
• Participate in the planning and prioritization of development tasks, ensuring efficient use of resources and timely delivery of projects
• Collaborate with data analysts and data scientists to understand their data requirements and develop solutions to meet their needs.
• Optimize and tune data pipelines for performance and scalability.
• Develop and maintain monitoring and alerting solutions to ensure data quality and availability.
• Collaborate with our Platform team to ensure data analytics infrastructure is secure, reliable, and highly available.
• Continuously research and evaluate new tools and technologies to improve data processing and analytics capabilities.
• Conduct code reviews to ensure adherence to coding standards and best practices.
• Capture and document data lineage, metadata, and data dictionaries to ensure comprehensive data governance.

Key Competencies:
• Advanced knowledge of cloud technologies and services, specifically around Amazon Web Services (AWS) Data Analytics tools (ex: Glue, Redshift, Lambda, Kinesis).
• Broad understanding of data engineering methodologies and tools, including data warehousing, DevOps/DataOps, data ingestion, ELT/ETL, and data visualization tools.
• Knowledge of database management systems, data modeling, and data warehousing best practices.
• Strong coding skills in languages such as Pyspark, Scala, and SQL.
• Experience in data streaming technologies such as Kafka, Kinesis, or Pub/Sub.
• Experience in handling unstructured streaming data is highly desired.
• Experience in software development on a team using Agile methodology.
• Knowledge of Business Intelligence Dimensional Modeling, Star Schemas, Slowly Changing Dimensions.
• Knowledge of DevOps tools and techniques (Ex: CI/CD, Azure DevOps, etc.).
• Knowledge of data governance and security practices.

Location: United States

Posted: Aug. 19, 2024, 7:10 p.m.