Data Engineer

Data Engineer II

A Data Engineer II works alongside Information Architects, Engineers, Data Scientists, and other stakeholders to design, develop, deploy, and end-to-end testing of moderate to advanced data pipelines. The Data Engineer II is responsible for building and maintaining the data pipelines and the data in relational and non-relational databases that support various data engineering applications, tools. The Data Engineer II is responsible for developing and supporting reports that provide accurate and timely data for internal and external clients. The incumbent will develop and maintain the data infrastructure that powers our ability to make timely and data-driven decision. This role requires familiarity of data architecture and extraction & manipulation of data sets of various sizes.

Position Responsibilities

Design and Methodology

• Design, implement and operate medium to large-scale, high-volume, high-performance data structures for reporting, analytics, and data science.
• Write and refine code to ensure performance and reliability of data extraction and processing.
• Implement data ingestion routines both real time and batch using best practices in data modeling, ETL/ELT processes by leveraging big data tools.
• Create data products for analytics and Data Scientist team members to improve their productivity.
• Document and test moderate data systems that bring together data from disparate sources, making it available to Data Scientists, and other users using scripting and/or programming languages.
• Design, develop and build real-time data pipelines from a variety of sources (streaming data, APIs, data warehouse, messages etc.)
• Leverage the understanding of software architecture and software design patterns to write scalable, maintainable, well-designed, and future-proof software.
• Manage existing pipelines and create new pipelines from a variety of sources (relational, XML, etc.)
• Design and build solutions to track data quality, stabilize data pipeline, etc. to ensure reliable operations.
• Document and test data processes including performance of through data validation and verification.
  
  

Technical Consulting

• Gather business and functional requirements and translate these requirements into robust, scalable operable solutions with a flexible and adaptable data architecture.
• Collaborate with engineers to help adopt best practices in data system creation, data integrity, test design, analysis, validation, and documentation.
• Collaborate with Data Scientists to create fast and efficient algorithms that exploit our rich data sets for optimization, statistical analysis, prediction, clustering, and machine learning.
• Coordination with other teams to design optimal patterns for data ingest and egress, as well as lead and coordinate data quality initiatives and troubleshooting.
• Participate in sprint planning meetings as needed.
• Foster a culture of sharing, re-use, design for scale stability, and operational efficiency of data and analytical solutions.
• Optimize support for ad-hod analysis across various data sources.
  
  

Planning & Organizing

• Ensure best practices are followed across architecture, codebase, and configuration.
• Continually improve ongoing reporting and analysis processes, automating or simplifying self-service modeling and production support for customers.
• In collaboration with the Data Engineer III, contribute to the exploration and understanding of new tools and techniques for improvements to the data pipeline.
  
  

Risk Management

• Review and are familiar with automated processes for performance and fault tolerance.
• Review functional and technical designs to identify areas of risk and any missing requirements.
• Design and implement security measures to protect data from unauthorized access or misuse.
• In collaboration with the Data Engineer III, design backup and recovery procedures to ensure data integrity is maintained.
  
  

Administration/Support

• Keeps management informed of status of on activities through accurate, timely, and appropriate reporting.
• Contribute to Data Governance, system documentation and sharing of data asset knowledge.
• Actively participates in committees representing the department and/or planning unit.
• Keeps abreast of leading-edge technologies in the Data Engineering space.
  
  

Position Qualifications

• Bachelor's Degree from an accredited university in Computer Science, Engineering or in a Technology related field OR equivalent through a combination of education and/or technology experience OR 12 years of technology experience
• 5 years of experience in Data Engineering, BI Engineering or related field in architecting and developing end-to-end scalable data applications and data pipelines
• 5 years of development experience in extracting, transforming/manipulating, and loading data sets of various sizes using Informatica IICS cloud or Snowflake or similar skills
• 5 years of experience using programming languages Python and querying languages (SQL)
• 5 years of experience working knowledge of different databases (e.g., SQL & NoSQL)
• 4 years of experience implementing data warehouse solutions using CI/CD automation, Jenkins etc.
• 4 years of experience developing strong collaborative relationships with key partners in data engineering, business intelligence, software development, finance, modelling, and product teams
• 3 years of experience working with software engineering best practices across the development lifecycle, including agile methodologies, coding standards, code reviews, source management, build processes and testing
• 2 years of experience using AWS or Azure cloud technologies leveraging S3, Lambda, EC2 resources
  
  

Preferred Qualifications

• Preferred, Amazon Web Services (AWS) Practitioner or Solutions Architect
• Hands-on work experience using DataIku
  
  

17 Cowboys Way

8:00am - 5:00pm Monday - Friday