Senior IT Big Data Developer

Project Description :

We are seeking an experienced, detail-oriented IT Big Data Developer to expand and optimize our data and data pipeline architecture, as well as optimize data flow and collection for economic policy and research teams. The ideal candidate is a hands-on data modeler with working knowledge of database design and administration, data pipeline building, and data wrangling who thrives on improving existing data systems and building new ones from the ground up. This role will support economists and technical experts, ensuring optimal data delivery architecture is designed and developed. The right candidate will be excited by the prospect of optimizing or re-designing the division's data architecture to support the next generation of data initiatives.

Qualification Requirements :

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related technical field with at least 7 years of related experience; advanced degree preferred
• Advanced working knowledge of SQL and relational database platforms including PostgreSQL, Microsoft SQL Server, and MySQL
• Advanced working knowledge of Python, R, and other scripting languages used for data engineering and analytics
• Experience with large-scale data systems including distributed computing, scalable data processing, data storage architecture, and high-volume data workload optimization
• Experience designing, developing, and automating ETL/ELT workflows and data integration pipelines
• Experience with workflow orchestration and pipeline automation tools such as Apache Airflow, Prefect, Dagster, or AWS Step Functions
• Experience migrating workflows and data pipelines between on-premises and cloud environments
• Experience developing in Linux environments and using source control platforms such as GitLab and/or GitHub
• In-depth experience designing and implementing database, data lake, and enterprise data platform solutions
• Strong hands-on software engineering experience including development, testing, and deployment of data applications and services
• Experience working in a research environment and/or with economic or financial data
• Working experience with cloud technologies such as AWS, Microsoft Azure, and Snowflake
• Experience implementing data warehouses utilizing Change Data Capture (CDC) methodologies
• Experience implementing and maintaining CI/CD pipelines and DataOps platforms

Skills Requirements :

• SQL - advanced proficiency (PostgreSQL, Microsoft SQL Server, MySQL)
• Python and R for data engineering and analytics
• ETL/ELT workflow design, development, and automation
• Workflow orchestration tools: Apache Airflow, Prefect, Dagster, or AWS Step Functions
• Distributed computing and scalable data processing architecture
• Database, data lake, and enterprise data platform design and implementation
• Cloud platforms: AWS, Microsoft Azure, Snowflake
• Linux development environments and source control (GitLab, GitHub)
• NoSQL and graph database technologies
• Machine learning model development, training, deployment, and maintenance
• Change Data Capture (CDC) methodologies
• CI/CD pipelines and DataOps platforms
• Time series data analysis and forecasting techniques
• Structured and unstructured data processing and integration
• Additional programming languages: Java, Scala, JavaScript, or Perl
• Excellent oral and written communication with a strong customer service orientation
• Exceptional analytical, problem-solving, and troubleshooting skills

Responsibilities :

• Expand and optimize data and data pipeline architecture to support economic policy and research teams
• Design and develop optimal data delivery architecture to meet multi-team and multi-system data needs
• Build, optimize, and maintain scalable databases, data pipelines, and data processing frameworks
• Design, develop, and automate ETL/ELT workflows and data integration pipelines
• Process, analyze, and integrate structured and unstructured data sources
• Design and communicate enterprise information architecture at conceptual, logical, and physical levels
• Perform root cause analysis on internal and external data and business processes to identify opportunities for improvement
• Migrate workflows and data pipelines between on-premises and cloud environments
• Develop, train, deploy, and maintain machine learning models
• Implement and maintain CI/CD pipelines and DataOps platforms
• Support economists and technical experts with their data needs across multiple teams and systems

Job ID : 1553