Source Code:

• Codeberg: https://codeberg.org/era/malleable-checker
• Microsoft Github: https://github.com/era/malleable-checker

Current status:

• Work In Progress
  • Re-writing the whole project using Rust and WebAssembly. The details written in this page are based on the original project that was using python. The new version using WebAssembly as runtime is very raw and it’s badly explained at the project readme

Overview

• A PoC of a malleable system that let’s you build rules using a GUI or a script to assert your production data. If something does not pass an assertion, an alarm is triggered. The idea is to have rules written as code if you need something complex and custom, but also allow users who don’t know how to code to use an UI. A alarm can trigger any user defined action, such as: sending messages on Slack, Telegram.
• This is a toy/pet 🐈‍⬛ project done on my free time, it’s going to take a while to be ready for a demonstration.

How it works?

• You define the datasource (e.g. an SQL query)
• Using either a HTML form or a python script you assert what you expect of that data.
• If the assertions fail, the alarm is triggered

For example, with the following datasource:

// Data source id = 32, name = 'users_with_email_null'
SELECT username from users where email is null;

You can create a new rule:

CheckerCase().assertLessThan(datasets['users_with_email_null'], 
  42, 
  'There should be less than 42 users in that state') # Otherwise we are going to alarm

The interesting bits is that you can use Python code for it, so you could define your own helpers and write things like:

if bussiness_day():
  CheckerCase().assertLessThan(datasets['users_with_email_null'], 42, 'There should be less than 42 users in that state')
else:
  CheckerCase().assertLessThan(datasets['users_with_email_null'], 5, 'There should be less than 5 users in that state')

This is executed hourly by a cron job, if the assertion throws an exception an Alarm will be fired. After the alarm is fired an action is triggered as defined by the user (email, slack, page).

After saved, the alarm has two values red and green. Red means the checker failed and the alarm was triggered. Green means everything is ok.

flowchart TD
    CheckerScheduler -->|every 5 minutes| FetchDataSources
    FetchDataSources --> EvaluateRules
    EvaluateRules --> RunAssertions{Assertion Failed?}
    RunAssertions --> |False| SaveGreenState
    RunAssertions --> |True| TriggerAlarm
    TriggerAlarm --> |Send Message to a queue so user can consume and trigger other actions| MessageQueue
    TriggerAlarm --> SaveRedState

On Images

Basically, you type your SQL query:

Then you can either use the advanced option and write the rule in python or use a normal HTML form to create the rule: