Job description
You will be a contributor in a new dedicated core data engineering team tasked with developing and owning standard data engineering patterns that BAM's data engineers, analysts, scientists, and quantitative researchers depend on for ingesting and transforming data throughout our ecosystem. This central team is responsible for owning core ETL/ELT pipelines for shared general-use datasets, as well as owning and stewarding the core shared libraries and design patterns used across the large number of other data engineering teams in the organization. Our Data Engineering systems span a cloud-native ecosystem heavily leveraging kubernetes, airflow, dbt and a number of cloud data warehouse technologies with a focus on snowflake.
The ideal candidate is passionate about designing high-quality resilient data architectures, thrives on working with data in all its forms, and comes to work with an open, collaborative design mindset. You have deep experience as a data engineer as well as rigorous foundational software development skills, allowing you to contribute across a range of high and low level design and implementation tasks. You bring to bear extensive knowledge from prior experiences but lead with an open learning-focused attitude to each new challenge.
This team will be aligned both with business stakeholders as the owners of pipelines that produce critical shared datasets as well as cross-organization stakeholders that will depend on the shared libraries and patterns evangelized and supported by the team. As such, team members must blend a pragmatic results-oriented attitude with a client-service mindset; you don’t just build the best systems but go above and beyond in documenting, evangelizing and supporting adoption of your systems and patterns across the organization.
Responsibilities of the Data Engineer – Data Platform include:
- Devising and developing solutions that capitalize on cloud-based data and distributed computing technologies to create BAMs next-gen data quality platform.
- Contribute thought-leadership through the full software development lifecycle from requirements gathering, through architecture, prototyping, development rollout and support.
- Partnering with end-users and collaborating with team members to clarify and expand upon requirements, as well as providing first-class support and guidance on best practices.
- Employing creativity and lateral thinking to engineer high-performance solutions for observing, measuring and alerting on data quality issues.
- Working within a globally-distributed yet tightly-knit engineering team, actively contributing to feature brainstorming, technical design discussions, code reviews, and overall team engagement.
- Producing software that is comprehensible, thoroughly commented, rigorously tested, and meticulously documented.
QUALIFICATIONS AND REQUIREMENTS:
- A degree in Computer Science or a closely related field.
- Over 5 years of professional data and software engineering experience.
- A fervent desire to build robust systems.
- Profound knowledge of engineering best practices and the data engineering technology ecosystem.
- Experience with complex data ingest and onboarding architectures, and experience in building complex distributed batch and streaming data processing systems.
- Advanced knowledge of data orchestration and data warehouse systems. Airflow, dbt, redshift, snowflake are a plus.
- Advanced knowledge of SQL and experience working with a variety of different SQL systems.
- Familiarity with fundamental DevOps practices, including continuous integration/continuous deployment (CI/CD).
- Experience in at least one cloud environment, with a preference for those familiar with AWS.
- A self-starter mindset, with a penchant for experimentation, a habit of asking probing questions, and the ability to thrive in a fast-paced, agile setting.
- A continuous learner, eager to gain a comprehensive understanding of large and complex systems at every level, who enjoys delving into nuanced problems, conceiving innovative solutions, and conducting thorough tests to grasp both the positive and negative outcomes.
