Data Engineer/Architect

About North East Link

The Spark Consortium has been selected for the Central Package of the North East Link Project - Victoria's biggest road project.

The Spark Consortium’s Design and Construction Contractor - which comprises Webuild, CPB Contractors, GS Engineering and Construction and China Construction Oceania - will deliver the twin three-lane tunnels and key interchanges for the project.

The Central Package will be delivered as a Public-Private Partnership and provides three-lane twin tunnels that will close the missing link in Melbourne’s freeway network from the M80 Ring Road to an upgraded Eastern Freeway, reducing journey times and diverting heavy vehicles away from local roads. Up to 135,000 vehicles will use North East Link every day, reducing congestion in the north-east while maintaining local roads.

About The Role

We are seeking a skilled and detail-oriented Data Engineer with strong expertise in Azure, Databricks, and Power BI to join our Data & Analytics team. This role is ideal for professionals who have a deep understanding of data engineering processes and solution architecture within the Azure ecosystem, coupled with experience in data governance and reporting. As a Data Engineer, you will be responsible for designing, building, and maintaining scalable data solutions that drive business insights and decision-making, leveraging Azure and Databricks technologies.

Key Responsibilities

• Azure and Databricks Solution Architecture:
• Lead the design and implementation of robust, scalable data pipelines and architectures on Azure and Databricks platforms to support business intelligence, analytics, and reporting needs.
• Collaborate with cross-functional teams to understand data requirements and translate them into effective architectural solutions within Azure.
• Ensure that data solutions are designed for performance, reliability, scalability, and security, with a focus on leveraging Azure-native services.
• Data Engineering and Development:
• Develop, test, and maintain data pipelines, ETL processes, and data integration workflows using Azure Data Factory, Databricks, and other related tools.
• Work with large datasets, ensuring efficient data storage, processing, and retrieval on Azure and Databricks to support analytics and reporting functions.
• Implement and optimize data models in Databricks to improve query performance and facilitate complex data analysis.
• Governance and Compliance:
• Establish and enforce data governance standards within the Azure environment, ensuring data quality, consistency, and compliance with regulatory requirements.
• Implement data security measures, including data masking, encryption, and access controls, to protect sensitive information in Azure.
• Collaborate with stakeholders to define and enforce data governance policies and procedures across the Azure ecosystem.
• Power BI Reporting and Analytics:
• Design and develop reporting solutions using Power BI that provide insights into key business metrics and support data-driven decision-making.
• Create and maintain dashboards, visualizations, and reports in Power BI that meet the needs of various stakeholders, leveraging Azure data sources.
• Ensure that Power BI reporting solutions are accurate, timely, and aligned with business objectives.
• Collaboration and Leadership:
• Serve as a subject matter expert on Azure, Databricks, and Power BI, providing guidance to junior team members.
• Collaborate with data analysts, and business stakeholders to deliver comprehensive data solutions within the Azure platform.
• Lead and participate in design reviews, ensuring that data solutions align with best practices and organizational standards.
• Documentation and Continuous Improvement:
• Maintain comprehensive documentation of Azure and Databricks architectures, processes, and governance policies.
• Continuously evaluate and improve data engineering processes, tools, and technologies within the Azure ecosystem to enhance efficiency and effectiveness.
• Stay current with industry trends and advancements in Azure, Databricks, and Power BI.
  

Qualifications

• Education: Bachelor’s or Master’s degree in Computer Science, Data Engineering, Information Technology, or a related field.
• Experience: 3-5 years of experience in data engineering with a focus on Azure, Databricks, and Power BI; experience with data governance and reporting is essential.
• Technical Skills:
• Proficiency in SQL, Python, with a focus on Azure and Databricks environments.
• Experience with Azure data services (e.g., Azure Data Factory, Azure Data Lake, Azure Synapse Analytics) and Databricks.
• Strong understanding of data modeling, ETL processes, and data warehousing within Azure.
• Expertise in creating and optimizing Power BI dashboards and reports using Azure data sources.
• Soft Skills: Strong analytical thinking, excellent problem-solving abilities, effective communication skills, and the ability to work collaboratively in a fast-paced environment. Ability to mentor associate data engineers and other team members.
  

Why Join Us?

• Innovative Environment: Engage in cutting-edge projects that leverage Azure, Databricks, and Power BI technologies.
• Professional Growth: Opportunities for learning, development, and career advancement in a supportive and dynamic team.
• Impactful Work: Play a crucial role in shaping the data strategy of a forward-thinking organization.
  
  

If this opportunity is of interest to you, please click on the ‘Apply now’ button to register your interest and please also subscribe to ‘Job Alerts’ to be notified of upcoming opportunities with SPARK.

SPARK is committed to equal employment opportunity (EEO) and actively seeks to remove barriers people may face in gaining a successful position on the North East Link Central Package. We value diversity, inclusion and equity through providing a working environment where all employees feel included, valued, and free to bring their different skills, experiences and perspectives.