Would you keep this release live?
Choose a response to a regression before planning a refactor.
The question
Patchwell is a fictional team that builds scheduling software. This is an authored practice case. A release changes shift-access checks. A 10% canary returns authorization errors for managers who should retain access. The old version remains deployable.
| Requests | Unexpected denials | |
|---|---|---|
| Old version | 9000 | 9 |
| Canary | 1000 | 80 |
Authored example. Values are supplied for this case.
No evidence of unauthorized access. The change is reversible. One manager reports being locked out. The canary rollout can be stopped immediately.
“What do you do now, and what evidence do you need before trying the change again?”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: Compare rates
What would you do here? Open the move you’d make.
Compare denial rates and user impact
The strong move. The canary has a much larger observed failure rate.
Wait because 80 errors is a small count
Tempting, but it misses something. The count comes from only 1,000 requests.
See how a strong candidate says it
“Old version: 9/9000 = 0.1%. Canary: 80/1000 = 8%. That is an 80-fold rate increase in this window. I would check that cohorts are comparable, but the observed access failure already merits action.”
Why it works: Different denominators make raw counts misleading.
Step 2: Contain the reversible failure
What would you do here? Open the move you’d make.
Keep ramping to collect more failures
Tempting, but it misses something. The existing evidence is enough to contain the regression.
Stop and roll back while checking recovery
The strong move. It addresses a live failure before diagnosis expands.
See how a strong candidate says it
“I would stop the rollout and return traffic to the old version, with the incident owner informed. I would preserve version and role diagnostics, then confirm manager access recovers.”
Why it works: The supplied reversible option limits further harm.
Step 3: Find the missing contract test
What would you do here? Open the move you’d make.
Reproduce allowed and denied role decisions
The strong move. Both false denials and false grants matter.
Make every manager request return success
Tempting, but it misses something. That may hide the defect by removing authorization.
See how a strong candidate says it
“I would compare old and new authorization decisions for the affected manager role using a minimal fixture. I would test permitted and denied roles, tenant boundaries, and any migrated role representation.”
Why it works: A fix must protect both availability and access control.
Step 4: Restart with a bounded check
What would you do here? Open the move you’d make.
Declare success when the build is green
Tempting, but it misses something. A build does not demonstrate restored access.
Prove the repair and repeat the canary
The strong move. The test and live observation cover different risks.
See how a strong candidate says it
“After the regression test fails on the broken version and passes on the repair, I would repeat a small canary. I would monitor unexpected denials by role and version and stop on a new regression.”
Why it works: Recovery and prevention need separate evidence.
The answer, in short
- Decision
- Stop the rollout and roll back, then reproduce the role failure.
- Because
- The canary has an 8% denial rate versus 0.1% on the old version.
- What would change it
- Evidence that rollback has a larger security risk would change the containment choice.
Common traps
- Comparing raw error counts across unequal traffic.
- Removing authorization to reduce denials.
Check your answer
- Did you calculate rates correctly?
- Did you contain the reversible failure?
- Did you test false denials and false grants?
- Did you distinguish rollback recovery from repair proof?
Last reviewed Nov 10, 2026