Anicca Data Science Solutions s is seeking a Data engineer to join our Data Operations team. Successful candidates will partner with our business partners to understand their data needs, and build data pipelines using cutting edge technologies. Additionally, you will provide engineering leadership to create and enhance data solutions enabling seamless integration and flow of data across our data ecosystem. You will provide technical consulting to peer data engineers during design and development for highly complex and critical data projects. Some of these projects will include designing and developing data ingestion and processing/transformation frameworks leveraging AWS and open source tools such as Java, Python, Spark, Scala, Hive, Python, etc.
• Designs and develops data ingestion frameworks, real-time processing solutions, and data processing/transformation frameworks, leveraging open-source tools.
• Provides senior-level technical consulting to peer-data engineers during design and development for highly complex and critical data projects
• Deploys application codes and analytical models using CI/CD tools and techniques and provides support for deployed data applications and analytical models.
• Provide system support as part of a support rotation with other team members.
• Be part of Data Engineering team translating business and technology requirements into our ETL/ELT architecture.
• Collaborate with cross functional teams such as AWS cloud & platform engineering, DBAs and business teams.
• Design and develop real-time and batch data ingestion and stream-analytic solutions leveraging technologies such as Kafka, Apache Spark, Kinesis, Java, NoSQL DBs, AWS EMR.
• Develop data driven solutions utilizing current and next generation technologies to meet evolving business needs.
• Ability to quickly identify an opportunity and recommend possible technical solutions.
• Develop Custom Data pipeline (Cloud and locally hosted).
• Provide support for deployed data applications and analytical models by being a trusted advisor to Data Scientists and other data consumers by identifying data problems and guiding issue resolution with partner Data Engineers and source data providers.
• Provide subject matter expertise in the analysis, preparation of specifications and plans for the development of data processes.
• Ensure proper data governance policies are followed by implementing or validating Data Lineage, Quality checks, classification, etc.
• Bachelor's Degree in Computer Science, or related field
• 2+ years of experience in Data Platform Administration/Engineering
• 4+ years of experience in Data Platform Administration/Engineering
• Experienced in Agile/Lean/SAFe methodologies.
• Proficiency in one of the scripting languages such as Shell/Python/Scala/Java.
• Good understanding of Big Data technology trends, with knowledge of technologies such as Kinesis, Kafka, Spark, Hive, and Python.
• Experience with one of the ETL tools Abinitio, Informatica or Data Stage.
• Hands on experience with AWS based solutions such as RDS, Lambda, DynamoDB, Redshift, Snowflake or S3.
• Experience in building & utilizing tools and frameworks within the Big Data ecosystem including Kafka, Spark, HBase, NoSQL.
• Experience with relational databases such as MySQL or Postgres and understanding of columnar data stores such as Redshift or Snowflake; and expert skills in writing complicated SQL statements.
• Ability to work in a fast-paced, rapidly changing environment.