May 26, 2023
Apply Now
Brooklyn, NY, United States
May 26, 2023
Apply NowJob description
Title: Sr. Data Engioneer 1
Location: (Remote) Brooklyn, NY
Duration: 1 Year (1820 Hours)
Interview: Virtual
Visa: No restrcition as long as eligible to work
Job Description
This role is for client's project. Client will improve and replace the existing Job search service. The Senior Data Engineer will assist the Application Engineering in building a robust, secure, and modern data pipelines to ingest, process, transform data using Informatica Intelligent Cloud Services or a comparable ETL/ELT tool employing modern data movement strategies and methodologies. The data needs to be store in an Azure Data Lake and eventually brought into a cloud-based data store such as Snowflake or a similar data store. Then, the Analytics reporting solution needs to be built either using cloud-based Google Looker or Microsoft Power BI or another similar tool.
Using the Cloud Data Platform, the Senior Data Engineer will build highly available, robust, concrete data pipelines and reporting solutions following industry best practices, adhering to security guidelines, modifying, and running automated CI/CD pipelines used for releasing code, and modifying and executing terraform modules for deploying infrastructure components, working collaboratively under the direction of client's data engineering management and leads. The resource will be a person with integrity who is dependable and is fully focused on delivering optimal solutions with little to no maintenance and operations overhead.
SCOPE OF SERVICES
Tasks
Candidates who do not have the mandatory skills will not be considered
Location: (Remote) Brooklyn, NY
Duration: 1 Year (1820 Hours)
Interview: Virtual
Visa: No restrcition as long as eligible to work
Job Description
This role is for client's project. Client will improve and replace the existing Job search service. The Senior Data Engineer will assist the Application Engineering in building a robust, secure, and modern data pipelines to ingest, process, transform data using Informatica Intelligent Cloud Services or a comparable ETL/ELT tool employing modern data movement strategies and methodologies. The data needs to be store in an Azure Data Lake and eventually brought into a cloud-based data store such as Snowflake or a similar data store. Then, the Analytics reporting solution needs to be built either using cloud-based Google Looker or Microsoft Power BI or another similar tool.
Using the Cloud Data Platform, the Senior Data Engineer will build highly available, robust, concrete data pipelines and reporting solutions following industry best practices, adhering to security guidelines, modifying, and running automated CI/CD pipelines used for releasing code, and modifying and executing terraform modules for deploying infrastructure components, working collaboratively under the direction of client's data engineering management and leads. The resource will be a person with integrity who is dependable and is fully focused on delivering optimal solutions with little to no maintenance and operations overhead.
SCOPE OF SERVICES
Tasks
- Collaborate with stakeholders to define and understand data needs.
- Design and develop efficient data architectures that can support large-scale data processing and storage requirements.
- Develop and maintain data pipelines, data models, and ETL processes that align with business requirements, data quality standards, and industry best practices.
- Work closely with other data engineering teams to build and maintain reusable data pipelines and tools, enabling faster time-to-market for data-driven solutions.
- Monitor, troubleshoot, and optimize data pipelines and processes for performance, reliability, and scalability.
- Ensure the quality and integrity of various datasets across different platforms and data sources.
- Continuously evaluate and recommend emerging technologies and methodologies to improve data engineering processes, workflows, and performance.
- Mentor and guide junior data engineers on technical best practices, code reviews, and design patterns to ensure high-quality, scalable, and maintainable data engineering solutions.
- Perform special technology-related projects, as assigned.
Candidates who do not have the mandatory skills will not be considered
- 12+ years of experience in data engineering with a focus on designing and building scalable data platforms using cloud technologies.
- Strong experience designing and implementing data models and schemas in Snowflake.
- Define Snowflake databases objects to support efficient data storage and retrieval.
- Expert proficiency in SQL and one or more programming languages such as Python, Java, or Scala.
- Knowledge of cloud-based data platforms such as AWS, Azure, or GCP, including experience with cloud-based storage, compute, and data processing services.
- Experience designing and implementing ETL and data integration pipelines, and familiarity with data modeling concepts and database design principles.
- Excellent communication skills and the ability to collaborate effectively with cross-functional teams.
- Experience with Agile methodologies and working in an Agile team environment.
- Experience developing production-grade, large-scale data solutions using cloud technologies.
- Experience managing data orchestration at scale using tools such as Airflow and Dagster.
- Familiarity with version control systems (e.g., Git) and CI/CD principles.
- Experience developing dashboards and reports in applications such as Oracle Analytics Server (OAS), Microsoft Power BI, and Google Looker.
- Thorough experience with data integration tools such as Informatica Intelligent Cloud Services and MuleSoft.
- Experience using Azure services for Security, Blob Storage, Data Lake, Databricks, Data Factory etc.
- Experience with Azure Monitoring services
- Microsoft Certified Azure Solutions Architect Expert or a Snowpro Certification or a similar one
