The Mayday Framework: Birth of a Centralised Anomaly Detection System

Since we published the anomaly detection system, adjoe has more than doubled its daily active users. But that is not the only thing that grew. 

More users means more publishers, campaigns, countries and more advertisers. The number of data slices you have to monitor grows faster than the traffic itself, because it grows with the product of those dimensions rather than the sum. Our alerting system had to evolve with it, and it did.

That post ended with a promise: move from a daily statistical sweep to something that fires every few minutes and watches over our core KPIs continuously. We kept the promise. 

The statistics are the half that gets talked about. The other half is everything that happens after a value is flagged: getting it to the one person who can act on it. This post covers both: the anomalies that are detected and the engine that helps deliver it to the responsible person.

Together they are what we call the Mayday Framework.

Why the Initial Detection System Ran Out of Road

The original design worked. It bootstrapped each data slice until a Kolmogorov-Smirnov test agreed the old and recent windows came from the same distribution. 

Then it flagged a recent value as an outlier if it failed either a Z-test or an interquartile-range check. Failures went out over a webhook.

What it could not do was scale in the organisational dimension. Each new alarm meant a new pipeline: new SQL, a near-copy of the detection code, its own delivery snippet, and schedule.  

Without shared standards, our alarms caused constant noise, fragmented communication, and ambiguity during silent periods.

Published work on anomaly detection concentrates almost entirely on the first row. Making sure the right grouping and formatting, or the external watchdog, is where the Mayday Framework kicks in.

How the Mayday Framework is Put Together

Mayday is one Airflow DAG that runs every 15 minutes, and takes care of all three points mentioned above. 

Today, it carries more than 50 active detectors across more than 15 domains.
It is delivered through 10+ independently scheduled routes to email, Google Chat and Slack.

The framework splits cleanly into two: a declaration layer and an engine.

framework/
---Declaration
├── config.yaml            # every detector + every delivery route
├── queries/<domain>/      # one SQL file per detector
├── python_transforms/     # optional post-SQL hook, one per detector
├── cards/<domain>/        # chat card templates
---Engine
├── math_library.py        # the statistical models
├── helper.py              # query loading, staging, dedup, rendering, sending
├── runner.py              # builds the pipeline from config.yaml
└── env.py                 # the one place production and staging differ

Each detector declares five things:

Those declarations project into a pipeline. Each detector delivery route becomes an individual task. Three execution gates decide which one runs on a given cycle: 

The three gates are the reason a fifteen-minute cadence remains affordable. And fanning out delivery tasks ensures a broken detector cannot silence the rest. 

The key here is automation: none of this workflow is wired by hand. The entire task graph is derived from the config file on every deploy.  Adding a detector means writing one SQL file and one config block. The Airflow task is auto-generated.

One Configuration File as the Source of Truth

Our core design choice: a detector is a declaration, not a program. A single config file specifies the measurement, tests, frequency, deduplication, and routing. Airflow tasks, queries, and final alerts are all projected automatically from that setup.

Here is a real detector: the view-to-click conversion rate, checked hourly across four dimensions. Comments are added for readability: 

V2C:
  display_name: "V2C (View-to-Click Rate)"
  query_file:   queries/tech_alerts/master_data_hourly.sql

  # The metric — a ratio. Both sides are summed per hour, then divided.
  numerator_col:   clicks
  denominator_col: views

  # Run the same test independently across four slicings of the data
  dim_cols: [sdkhash, platform, mmp_name, osversion]

  # How and when to test
  hourly_check:
    cron:   "45 * * * *"                  # every hour, at :45
    method: historical_baseline_iqr_anomaly_detection

    baseline_hours:               336     # 14 days of history
    min_baseline_hours:           336     # ...and require it to be complete
    min_baseline_samples_per_hod:  14     # ...with 14 same-hour samples

    iqr_upper_multiplier:         4.0     # spike fence, above p90
    iqr_lower_multiplier:         3.0     # drop fence, below p25
    min_baseline_avg:           0.001     # ignore slices averaging under 0.1%

  # Stay quiet about a repeat of the same slice for 4 hours
  dedup:
    key: [dimension, dimension_val]
    lookback_hours: 4

  # Where it goes
  alerts:
    gchat:
      card_file: cards/tech_alerts/tech_alerts.json

That is the whole detector. No Python, no scheduling code, no delivery code. 

The same test runs independently across four slices of the data, per SDK, per platform, per attribution partner, per OS version. So a regression confined to one iOS version surfaces on its own rather than being averaged away in a global number.

The method line is the pluggable part. Swap it, adjust the parameters under it, and the same query and chat card now feed a completely different statistical test.

Part 1 Detection: Maths, Revisited

Dual-baseline IQR

Our workhorse point detector, historical_baseline_iqr_anomaly_detection  began as a straight port of the original. The original combination was Z-test and IQR: two different statistics over the same window. That guards against distributional assumptions, which is real but not our main source of false positives.

Our main source of false positives was time of day. Offerwall traffic is deeply diurnal. Views at 03:00 UTC look nothing like views at 18:00 UTC. So any detector that compares “now” against a flat 14-day average fires every single night, then again every morning.

So we changed what should agree. For every metric and every dimension value, we build two baselines from the trailing 14 days:

1. A flat baseline: every hour in the window.
   
2. An hour-of-day baseline: only those hours whose clock hour matches the current one. Yesterday’s 14:00, the day before 14:00, and so on.

Each baseline yields its own fence, computed from percentiles rather than mean and standard deviation, so a single past incident in the baseline cannot inflate the threshold:

upper = p90 + 4.0 × (p90 − p25) 

lower = p25 − 3.0 × (p75 − p25)

An alert fires only when the current value breaches the same side of both fences simultaneously. A 03:00 trough breaches the flat fence but sits comfortably inside its own hour-of-day fence, so it is silent. A genuine collapse at 14:00 breaches both, so it fires.

Blindspots the First System Missed

Then, we found the incidents our detector was structurally blind to. 

Consider a conversion rate that erodes from 30% to 20% smoothly over three weeks. That is a serious regression, roughly a third of the conversion gone. At no individual hour does it breach any fence, because the baseline is also drifting down. It is always within tolerance of its recent self. The detector reports health the entire way down.

Percentage-change thresholds cannot catch this. Neither can point-anomaly detection of any kind, including the dual-baseline detector above. The signal is not in any single observation; it is in the slope. A detector that only ever compares now against recently is unable to see a decline that moves recently along with it.

So we added a second detector class,  logistic_regression_trend_anomaly_detection.
For a ratio metric over a 21-day window (504 hourly observations) we fit a Binomial GLM:

logit(numerator / denominator) ~ hours_since_start + C(hour_of_day)

The hours_since_start coefficient is the trend. 

The 23 hour-of-day dummies absorb the diurnal cycle, so the slope estimate is the trend after controlling for time of day. This matters significantly, because without it the fitted slope mostly reflects where in the daily cycle the window starts and ends. 

We predict the rate at the start and end of the window at the same reference hour-of-day for the same reason. 

An alert fires only when three conditions hold together:

1. The slope’s p-value is below 0.01. Deliberately strict: over 504 observations a random walk will clear p < 0.05 often enough to be a nuisance.
2. The predicted decline is at least 20% relative.
3. The predicted decline is at least N percentage points absolute.

All of which is one config block. Same metric as the hourly detector earlier, same query, same chat card only the test and its parameters change: 

V2C Gradual Decline:
  display_name: "V2C — Gradual Decline"
  query_file:   queries/tech_alerts/master_data_hourly.sql   # same query
  numerator_col:   clicks
  denominator_col: views
  dim_cols: [sdkhash, platform, mmp_name, osversion]

  trend_check:
    cron:   "0 3,9,15,21 * * *"           # 4× a day 
    method: logistic_regression_trend_anomaly_detection

    window_hours:              504        # 21 days of hourly observations
    hour_of_day_fixed_effect: true        # control for the diurnal cycle
    relative_drop_threshold:  0.20        # ≥ 20% decline, end vs start
    p_value_threshold:        0.01        # ...and a slope we're confident in
    min_absolute_drop:        0.02        # ≥ 2pp — relative alone is scale-blind
    min_denominator:         10000        # ignore thin slices

  dedup:
    key: [dimension, dimension_val]
    lookback_hours: 72                    # a slow bleed needn't be re-announced daily

  alerts:
    gchat:
      card_file: cards/tech_alerts/tech_alerts.json          # same card

Two detectors, one metric, watching for two genuinely different kinds of failure, and the difference between them is twelve lines of configuration. 

Significance Gates are not just a Detail

Roughly half the tuning effort in this system went into deciding what not to test. Every detector carries floors: a minimum baseline average, minimum denominator, baseline hours, same-hour samples.

They matter more than the test statistic. A slice with 10 views has a wildly unstable conversion rate and will breach any fence. Without floors, your alert channel fills with statistical noise from irrelevant slices. The real signal is buried, which returns you to the “nobody acts” failure mode by a different route. 

Detection thresholds are per-metric, and any framework that pretends otherwise will silently mislead you.

Part 2 Delivery: From Flagged to Acted On

Detection and delivery are not the same 

The single most consequential architectural decision in Mayday: 

There is one deliberate exception. If a group’s delivery fails, its files stay in pending/, which means every channel for that group is retried, including ones that already succeeded, so a duplicate is possible.

When correctness and tidiness conflict in a monitoring system, correctness wins every time.

Deduplication is a first-class feature, not a filter 

The most common reason alerting systems get ignored is not inaccuracy. It is repetition. An incident lasting four hours, on a 15-minute detector, is sixteen identical messages. People mute the channel and then miss the next real one. 

So deduplication is declared alongside the detector:

dedup:
  key:            [sdkhash, alert_type]   # what makes this alert "the same alert"
  lookback_hours: 4                       # ...and how long we stay quiet about it

Before an alert is staged, the framework reads recent sent/ history, builds the set of identities already delivered inside the lookback window, and drops rows that match. That same four-hour incident becomes one message. The window is per-detector: hourly volume checks use 4 hours; three-week trend detectors use 72, because a slow bleed does not need re-announcing daily.

We also thread chat messages. Every Google Chat message is posted with a stable thread key of project : detector : group : UTC date, so a follow-up alert for the same thing on the same day replies in-thread rather than opening a new top-level card. A multi-hour incident is one readable conversation. Threads roll over daily, so a week-long issue is seven threads rather than one unreadable one.

Route by who acts, not by what fired

The original system posted everything to one webhook. That is fine with three alarms and unusable at scale.

In Mayday, every detector declares a grouping key, and delivery is built around it. 

A budget report grouped by account manager produces one email per manager, containing only their campaigns, with their own reporting line automatically copied in, from one detector run, with no per-manager configuration anywhere.

Then the channel is matched to the decision:

The image above shows us how informative can one Google Chat card be. This one alert already gives you insight into “what’s the acceptable range of values” vs. “what is the current anomalous value. You can also directly add a deep link to your monitoring dashboard.

Knowing when the detector is broken

This closes the third failure mode, and it is the piece we would tell anyone to build first.

Every run writes a small heartbeat object at start and overwrites it at finish. A separate watchdog pipeline reads those objects and reports when a run is still marked. Starting past its allotted slot, naming the detectors that were consequently skipped.

Two design points earn their keep:

• It is a separate pipeline. A watchdog cannot reliably detect that the system it monitors failed to run if it is part of that same system.
  
  
• The finish marker is written unconditionally: Whether the run succeeds, fails, or is fully skipped, the marker is recorded. “Still running” must mean exactly that – not “failed before the bookkeeping happened.”

Isolation, because one broken detector must not silence the others

Originally, one task delivered everything. Its logic was correct, and its blast radius was the entire platform: a single failing detector left every other domain’s alerts sitting undelivered in pending/.

Now each delivery route is its own task, waiting only on the detectors inside its own scope. A detector that has not fired this cycle does not block anything. A detector that has failed blocks only its own route.

In an alerting system, resilience means independent failure domains. A shared delivery path is a single point of failure with extra steps.

Moving Forward 

Two directions we are actively working on:

• Thresholds that tune themselves. Every floor and multiplier is currently hand-set from a backtest. That does not scale to hundreds of detectors. We want per-metric parameters fitted from historical alert precision, so a detector calibrates itself against its own track record.

• Correlated incidents. When one upstream cause breaks four metrics, we currently send four alerts. Grouping causally-related alerts into a single narrative is the next big reduction in noise.

Our conclusion from the first article still holds. With large-scale systems like these, we never reach perfection, but you get better. What we would add is that “better” turned out to mean something different from what we expected. 

We assumed the road ahead was better statistics. It was mostly better engineering around ordinary statistics, and that is where the returns were.

We’ll keep sharing what we learn along the way. Stay tuned for more engineering stories from adjoe. 

Build products that move markets