Definition
The percentage change in a guardrail metric observed during an experiment or rollout meant to improve a different primary metric.
Formula
(Guardrail metric in test - Guardrail metric in control) / Guardrail metric in control x 100
How to measure
Compare the guardrail metric's value in the test group against the control group during an experiment, expressed as a percentage difference.
Why track it
Monitoring guardrail deviation catches situations where a metric win is actually a hidden loss elsewhere, like retention dropping while signups rise.
Example
A signup flow change lifts conversion 12% but also shows a 4% guardrail deviation in 7-day retention, prompting the team to hold off shipping it broadly.
FAQ
- What is Guardrail Metric Deviation?
- The percentage change in a guardrail metric observed during an experiment or rollout meant to improve a different primary metric.
- How do you calculate Guardrail Metric Deviation?
- (Guardrail metric in test - Guardrail metric in control) / Guardrail metric in control x 100
- Why track Guardrail Metric Deviation?
- Monitoring guardrail deviation catches situations where a metric win is actually a hidden loss elsewhere, like retention dropping while signups rise.