Data Engineer (PROJ-4784)

Key duties and responsibilities

• Identify and undertake analysis to define, structure and solve complex data engineering challenges. 

• Understand, use and build tools and techniques to develop and test data engineering solutions to achieve business outcomes. 

• Provide advice on requirements, current and future state models and plans, designs and solutions to improve data interoperability, storage, analysis and transmission. 

• Build, maintain and deploy data pipelines, analytical environments and data stores to store, transform, integrate, analyse and transport data and metadata from a variety of sources to enable the use and analysis of data. 

• Conduct diagnostic procedures to identify system vulnerabilities and improvements. 

• Identify process improvements, apply metadata frameworks and assist with the governance of data engineering design patterns and practices. 

Essential criteria

• Data engineering: Level 4 (SFIA)
  Designs, implements, and maintains complex data engineering solutions to acquire and prepare data. Creates and maintains data pipelines to connect data within and between data stores, applications and organisations. Carries out complex data quality checking and remediation.

• Data modelling and design: Level 3 (SFIA)
  Applies standard data modelling and design techniques based upon a detailed understanding of requirements. Establishes, modifies and maintains data structures and associated components. Communicates the details of data structures and associated components to others using the data structures and associated components.

• Data use and re-use: Level 1 (DCF)
  Understands the data they work with can be used more widely and is aware of relevant data sharing legislation Is familiar with basic data sharing guidelines and organisational procedures to support the re-use of their data by others

• Improvement and Innovation - Data processes/systems and tools/products: Level 1 (DCF)
  Can identify a successful process system or tool/process/product Can identify deficiencies in current processes/ systems or tools/products Can support systems testing under guidance and following test plans

• Measurement: Level 3 (SFIA)
  Applies standard techniques to support the specification of measures and the collection and maintenance of data for measurement. Generates, produces and distributes reports. Uses measurement tools for routine analysis of data. Identifies and implements improvements to data collection methods.