Data Engineer’s daily job #

MetaCroc is a tool that brings data engineering close to business, i.e. it reduces requirements for technical skills of data engineers. Typically, only one really technical developer is necessary in the data engineering team, the rest can be more business oriented.

Pure technical developer #

Requirements:

• knowledge of the data platform technologies
• experience in scripting, CI/CD
• experience in data engineering
• knowledge of data transformation patterns

Responsibility:

• preparation of data environment
• care of deployment processes and pipellines
• preparation of templates for data transformation patterns
• performance tuning

Data engineers #

Requirements:

• knowledge of data warehousing principles
  • data modelling
  • data transformation patterns
• advanced SQL knowledge
• ability to talk to business users and gathering their needs

Responsibility:

• analysis of data users requirements
• preparation of SQL statement that computes target data set
• validation of the data set
• input metadata to MetaCroc (based on the SQL statement)
• generation of output scripts for deployment
• testing of data transformation results