Settle DQ Workflow

Working with Settle DQ is a continuous process.

  • The Data User and the Data Engineer settle on DQ Agreements - a short description of condtions that the data should satisfy.
  • The Data Engineer then develops DQ Expectations - declarative definitions that check whether data satisfy the given conditions.
  • Then the Data Engineer creates validation notebooks that validate the data against the expectations (this only needs to be done for new checked data sources).
  • The Data User then sees the validation results in a web app every time new data are processed. When conditions change, Data User and the Data Engineer discuss to update and/or create new DQ Agreements.