Job Description

Sr. Data Engineer is needed to perform the following duties: Build streamline pipelines leveraging various AWS services including Lambda and Glue for data processing, ingestion, transformation, feature engineering, and business analysis. Leverage AWS Event Bridge, EMR and Step Functions to create a streamline data pipeline to invoke Lambda and Glue jobs with batch based and event-based triggers, finally ingesting transformed data into Snowflake. Develop custom PySpark and Scala coding to perform data transformations such as filtering, cleansing, mapping, handling null values and improving data quality. Improve efficiency by optimizing complex SQL queries in Snowflake, implementing data partitioning, and indexing strategies for parallel processing, and reduced pipeline complexity. Leverage python and its multiple packages to create functions to create derived variables from existing variables, performed end-to-end testing, unit testing, integration tests, and exceeded the required sonar coverage.Bachelor's Degree is required in Computer Science or Computer Engineering or Information Systems or Business Analytics.

Job Tags

Senior data engineer, sr. data engineer