Incident Response Planning¶
Problem Statement¶
A complex system of internal and customer-facing services with many interdependencies should be brought back online as efficiently as possible after an outage.
Full example scenario
As a DevOps engineer, you are responsible for planning the recovery of a set of interconnected systems after a major outage, in order to prioritize critical systems and allocate recovery resources efficiently. Each system has dependencies on other systems that must finish recovering first, and some — the customer-facing ones — carry a priority score. To induce urgency, the value retained from bringing a customer-facing system back online decays the longer it stays down, following \(V(t) = V_0 \cdot e^{-0.0398 \cdot t}\), where \(V_0\) is the system’s priority score and \(t\) is its completion (recovery finish) time in minutes from the start of the incident.
Recovery starts at \(t=0\), and a system cannot begin its own recovery until every system it depends on has finished.
The on-call crew can recover at most 2 systems at once — one recovery per engineer, and once a system’s recovery starts it runs to completion without pausing.
Internal infrastructure systems carry no priority score and contribute nothing to the objective directly, but still take time to recover and can block other systems as dependencies.
The goal is to choose the start time of every system’s recovery so as to maximize the total retained value across all customer-facing systems.
The systems, their dependencies, priority scores, and recovery times can be found in
incident_response_systems.json, with the fieldsSystem, Name, Priority, Dependencies, Recovery Time (minutes).
What an optimal solution looks like
Dependency graph — who has to finish before whom.
Arrows point from a prerequisite to the system it unblocks. Node color is the system's priority score V0.
Engineer timeline — who recovers what, and when.
Gray blocks are internal infrastructure (zero customer value, but can still block others). Colored blocks are customer-facing systems, colored the same way as the dependency graph above.
Built with Modeler¶
The problem above is stated as a single shot: you hand over the whole problem and get a model back. Modeler takes the other route — you start with the business question you actually have, and it works with you until the problem is pinned down, the data is in, the code runs, and the answer holds up when the problem changes.
The tabs below walk the same incident-recovery problem through that route, one Modeler capability at a time.
Instead of writing the whole prompt up front, you start with the business question you actually have. Modeler searches the knowledge base, asks one clarifying question at a time — always the most important one first — and tells you which parts of the model are still missing. The options it offers come from examples in the knowledge base and from how similar problems are usually modeled in your industry, so you can answer with a number instead of prose.
I’m on the DevOps/SRE team. When we have a major outage, we have a bunch of interconnected systems that need to be brought back online — some systems depend on others being restored first before they can start recovering. Some of these systems are customer-facing and have a priority/importance score; the sooner we bring an important one back online, the more value we retain, and that value decays the longer the system stays down. I want to figure out the best order and schedule for bringing everything back online to maximize the total value we retain. Can you help me build this as an optimization model?
This is a precedence-constrained scheduling problem with a time-decaying reward — recovery order matters, and the DAG of dependencies will drive most of the structure. Before anything else I need to know how much recovery work can happen at once, because that’s what decides whether this is even a scheduling problem with real choices, or just a critical-path calculation with one forced answer. Let me check the knowledge base for how similar problems are modeled before asking.
Great start. Here is the first clarifying question:
Question 1 of ~7: How many systems can be recovered in parallel at the same time?
In other words, do you have a fixed number of recovery teams or engineers that can each work on one system at a time, or is there only one team working through systems sequentially?
Only 1 recovery team — systems are restored one at a time, in a strict sequence.
A fixed number of teams (e.g., 2, 3, N) — up to N systems can be in recovery simultaneously.
Unlimited parallelism — any number of systems can recover at the same time (subject only to dependency constraints).
Resource-constrained parallelism — recovery requires shared resources (e.g., DBAs, network engineers) with limited capacity, and different systems may need different resource types.
Let’s start with option 3 — unlimited parallelism, just as a first pass, so I can see what the dependency graph alone implies before we layer in the real crew limits. One more thing: only some of the systems are customer-facing and carry a priority score — the rest are internal infrastructure with no direct customer value, so they don’t count toward the objective, but they still have recovery times and can block other systems as dependencies.
That’s a deliberately simplified starting point on parallelism — worth revisiting once I can see what it implies — and it volunteers the internal-vs-customer-facing distinction, which fixes the shape of the objective (a sum over a subset of systems, not all of them). What’s still open is exactly how “value decays over time” should be modeled mathematically — that determines whether this is a linear program or something with a nonlinear or piecewise term, so it’s the next thing to pin down before writing anything.
Good. Now a critical design question about how value decays over time.
Question 2 of ~6: How does the retained value of a customer-facing system decrease the longer it stays down?
Linear decay — value = priority_score − decay_rate × completion_time.
Weighted completion time — value = priority_score × (1 / completion_time), equivalently minimizing the sum of priority-weighted completion times.
Step / deadline-based — full value before a deadline, zero (or reduced) value after.
Custom / piecewise — each system follows its own decay curve (e.g., exponential), defined per system.
The most natural fit for “maximize value retained, sooner is better” is option 2, but if you already have a specific curve in mind, option 4 lets you plug it in directly.
Option 4, but with a specific known formula rather than a generic curve: retained value follows V(t) = V0 * e^(-0.0398 * t), where V0 is the system’s priority score and t is its completion (recovery finish) time in minutes. This applies uniformly to every customer-facing system — it’s not a per-system custom curve, just the one exponential decay formula applied to each one’s own V0 and finish time. Recovery times per system are known and fixed (no uncertainty). I have the full list of systems, their priority scores, dependencies, and recovery times ready to share as a data file.
Also, please go ahead and share your data file whenever you are ready — I will incorporate it directly into the specification and use it as the basis for the test scenarios.
Modeler updates the specification after every answer, even while it is still incomplete — so you can always see what it has understood and what it is still waiting on.
Modeler asked for the systems, dependencies, priorities, and recovery times as a real file rather than typed-out numbers, so it can check the dependency graph and recovery times directly against what you actually run.
incident_response_systems.json:
[
{"System": "S1", "Name": "Core Network", "Priority": 0, "Dependencies": [], "Recovery Time (minutes)": 5},
{"System": "S2", "Name": "Auth Database", "Priority": 0, "Dependencies": ["S1"], "Recovery Time (minutes)": 8},
{"System": "S3", "Name": "Cache Cluster", "Priority": 0, "Dependencies": ["S1"], "Recovery Time (minutes)": 4},
{"System": "S4", "Name": "User Database", "Priority": 0, "Dependencies": ["S1"], "Recovery Time (minutes)": 10},
{"System": "S5", "Name": "Auth Service", "Priority": 0, "Dependencies": ["S2"], "Recovery Time (minutes)": 6},
{"System": "S6", "Name": "Billing Database", "Priority": 0, "Dependencies": ["S1"], "Recovery Time (minutes)": 7},
{"System": "S7", "Name": "API Gateway", "Priority": 0, "Dependencies": ["S5", "S3"], "Recovery Time (minutes)": 5},
{"System": "S14", "Name": "Admin Portal", "Priority": 0, "Dependencies": ["S2"], "Recovery Time (minutes)": 6},
{"System": "S12", "Name": "Recommendations", "Priority": 15, "Dependencies": ["S4"], "Recovery Time (minutes)": 4},
{"System": "S13", "Name": "Notifications", "Priority": 10, "Dependencies": ["S5"], "Recovery Time (minutes)": 3},
{"System": "S9", "Name": "Search Service", "Priority": 25, "Dependencies": ["S7", "S4"], "Recovery Time (minutes)": 4},
{"System": "S11", "Name": "Mobile API", "Priority": 30, "Dependencies": ["S7", "S4"], "Recovery Time (minutes)": 5},
{"System": "S8", "Name": "Checkout Service", "Priority": 40, "Dependencies": ["S7", "S6"], "Recovery Time (minutes)": 6},
{"System": "S10", "Name": "Web Storefront", "Priority": 60, "Dependencies": ["S8", "S9"], "Recovery Time (minutes)": 3}
]
Here’s the data file — 14 systems from a recent outage post-mortem. “Recovery Time (minutes)” is how long that system’s own recovery takes once it’s able to start; “Dependencies” lists systems that must finish first; “Priority” is the V0 score for customer-facing systems (0 for internal infrastructure that has no direct customer value but can still block others as a dependency).
With the actual DAG in hand I can compute the critical path under the “unlimited parallelism” assumption from the last answer. If that critical path already pins every system’s completion time uniquely, there’s no scheduling decision left to make — the model would have nothing to optimize. That’s worth checking and flagging before going any further, rather than building a model that turns out to be degenerate.
Orhan, I need to flag a critical structural finding before asking the next question.
Important: Unlimited Parallelism Removes All Scheduling Decisions
I ran the critical path analysis on your 14-system DAG. Here are the results under your original “unlimited parallelism” assumption:
System |
Name |
Priority |
Fixed completion |
Retained value |
|---|---|---|---|---|
S12 |
Recommendations |
15 |
19 min |
7.04 |
S13 |
Notifications |
10 |
22 min |
4.17 |
S9 |
Search Service |
25 |
28 min |
8.20 |
S11 |
Mobile API |
30 |
29 min |
9.46 |
S8 |
Checkout Service |
40 |
30 min |
12.12 |
S10 |
Web Storefront |
60 |
33 min |
16.13 |
Total |
57.12 |
With unlimited parallelism, every system starts the instant its last dependency finishes. The completion times above are the only possible outcome — the dependency graph leaves zero room for any scheduling choice. There is nothing to optimize: there is only one feasible schedule and it is automatically optimal.
For a meaningful optimization problem, you need a finite parallelism limit. This is exactly what creates the interesting tradeoff: when multiple systems become eligible to start at the same time but only N can run concurrently, the model must decide which ones to prioritize — and that decision directly impacts when high-value systems like Web Storefront (S10, V0=60) come back online.
Question 3 of 5: What is the maximum number of systems that can be recovering simultaneously?
max_parallel = 1 — Fully serialized.
max_parallel = 2 — Two concurrent recoveries. Realistic for a small SRE team.
max_parallel = 3 — Three concurrent recoveries.
max_parallel = N — you tell me exactly how many teams/engineers you have.
max_parallel = 14 — effectively unlimited, usable as a theoretical upper-bound benchmark.
Which option reflects your real operational capacity?
Option 2 — max_parallel = 2. We run a rotating on-call crew of two recovery engineers during an incident; each can work on one system’s recovery at a time, occupying that engineer for the system’s full recovery duration once started.
Specification updated. Here is the next question.
Question 4 of 5: Should recovery start times be continuous (real-valued minutes) or discretized into fixed time slots?
Continuous start times — the standard approach for scheduling with fixed processing times; since all your recovery durations are whole-number minutes, optimal start times will naturally land on integers anyway.
Discrete time slots (e.g. 1-minute slots).
Discrete with a coarser grid (e.g. 5-minute slots).
Given your data, option 1 is both more general and computationally leaner.
Option 1, continuous start times.
Specification updated with continuous start times confirmed. One final question before moving to implementation.
Question 5 of 5: What is the desired interface and output format for the model?
Python class with a solve() method — a self-contained, reusable, testable class.
Standalone script — reads the data file, solves, prints to console.
Both — a class-based implementation plus a thin CLI wrapper.
Notebook / report style.
Given that you already have a structured JSON file and this is for an SRE team, option 3 is the most practical.
Option 3 — a testable class plus a thin CLI wrapper.
Where your files live
Every file that comes out of this conversation — the data you attached, the specification, the code, the tests, the solve log, and the formulation — is saved into a workspace tied to this conversation, so nothing has to be re-attached or re-derived later.
File |
Produced by |
|---|---|
|
You (attached above) |
|
The specification (next tab) |
|
The implementation (Code tab) |
|
The test suite (Tests tab) |
|
The cloud solve run (Cloud solve tab) |
|
The mathematical formulation, on request (Formulation tab) |
Starting from files you already have
You don’t have to start from a blank conversation, either. If you already have a project — a spec, some code, a folder of CSVs — you can attach the whole thing and ask Modeler to “check the project.” It will read what’s there, tell you what’s consistent, what’s missing, and what looks contradictory between the spec, the implementation, and the tests, before making any changes.
Once the clarifying questions settle, Modeler writes the specification — the artifact everything downstream is generated from. It’s written in business terms, with no math, so anyone on the team can read and sanity-check it.
Overview
After a major service outage, a DevOps/SRE team must bring a set of interdependent systems back online as quickly as possible while maximizing the total retained customer value. Systems form a directed acyclic graph (DAG) of dependencies: a system cannot begin recovery until every system it depends on has finished recovering. A limited on-call crew — a maximum number of engineers working in parallel — constrains how many systems can be in active recovery at any given moment.
Objective
Maximize the total retained customer value, defined as the sum over all customer-facing systems of their priority score multiplied by an exponential decay factor applied to that system’s completion time. Internal infrastructure systems have no priority score and contribute nothing to the objective, but still consume engineer time and can block dependent systems.
Constraints
C1 — Dependency ordering. A system cannot begin its recovery until every system it directly depends on has completed its own recovery. This is the non-negotiable rule that makes the recovery order matter at all.
C2 — Pairwise sequencing. For every pair of systems that could plausibly overlap (neither is a prerequisite of the other), exactly one order between them is chosen and enforced — either the first finishes before the second starts, or vice versa, whenever they end up competing for the same engineer.
C3 — Overall engineer capacity. At most max_parallel systems may be
in active recovery at any instant, reflecting the size of the on-call crew.
Once a system’s recovery starts, it occupies one engineer for its full
duration with no preemption.
C4 — Decay variable definition. Each customer-facing system’s decayed value is tied directly to its completion time through the fixed exponential formula, so the objective is always evaluated against the real schedule Modeler is choosing.
C5 — Exponential value constraint. The retained value of a customer-facing system is exactly V0 × e^(−0.0398 × completion_time) — not an approximation of it — so the schedule that maximizes the objective is the schedule that truly maximizes retained business value.
Input
Entity |
Fields |
Values |
|---|---|---|
System |
system, name, priority, dependencies, recovery_time |
14 systems; 6 customer-facing (priority > 0), 8 internal (priority = 0) |
Scheduler parameter |
max_parallel |
2 (on-call engineers) |
Decay rate |
Global constant |
0.0398 per minute, applied uniformly |
Six of the fourteen systems carry real business value; the other eight exist purely to unblock those six, so the crew’s two engineers spend most of an incident’s first 20 minutes on work that doesn’t show up in the objective at all — until it clears the path for what does.
Output
The optimal start and finish time for every system’s recovery.
The total retained customer value achieved by that schedule.
Whether the schedule is feasible and provably optimal, or infeasible (e.g. a circular dependency).
Test scenarios
ID |
Scenario |
Acceptance criteria |
|---|---|---|
T1 |
Baseline — 5-system DAG, max_parallel=2 |
Optimal solution found; all precedences satisfied |
T2 |
Priority perturbation that changes the decision — a system’s priority drops from 100 to 10 |
The system’s completion time increases (it’s no longer worth rushing) |
T3 |
Priority perturbation that does not change the decision — a priority drops from 100 to 95 |
Schedule (start times) unchanged; objective value changes only slightly |
T4 |
Infeasible — circular dependency between two systems |
Solver reports infeasible; no objective value returned |
T5 |
Optimal vs. naive baseline — compare against a “recover in system-ID order” greedy heuristic |
Optimal objective is at least as good as the naive one |
T6 |
Full 14-system production dataset |
Optimal solution found; all precedences and the max_parallel=2 limit respected |
Out of scope
Preemption — once a system’s recovery starts, it cannot be paused
Engineer heterogeneity — every engineer is treated as interchangeable
Partial or phased system recovery
Stochastic recovery durations
Dynamic re-scheduling during an active recovery
This is the version that came out of the run-and-fix loop shown in the Cloud solve tab, after Modeler ran the tests and confirmed everything passed — not the first draft.
"""
Incident-Response Recovery Scheduling Optimizer
================================================
Maximizes total retained customer value after a service outage by
scheduling the recovery of interconnected systems subject to:
- Dependency ordering (DAG precedence constraints)
- Parallel engineer capacity (at most max_parallel simultaneous recoveries)
- Exponentially decaying customer value: V(t) = V0 * exp(-DECAY_RATE * t)
Key modelling efficiency: binary sequencing variables are created ONLY for
incomparable pairs of systems (neither is a transitive ancestor of the other).
For ancestor-descendant pairs the existing chain of precedence constraints
already enforces the required ordering, so no binary variable is needed.
Usage
-----
from incident_response_modeler import DisasterRecoveryScheduler, load_systems_from_json
import gurobipy as gp
systems = load_systems_from_json("incident_response_systems.json")
with gp.Env(empty=True) as env:
env.setParam("OutputFlag", 1)
env.start()
with DisasterRecoveryScheduler(env) as scheduler:
scheduler.set_data(systems, max_parallel=2)
scheduler.build_model()
result = scheduler.solve()
scheduler.display_results(result)
"""
import collections
import json
from itertools import combinations
import gurobipy as gp
from gurobipy import GRB
# Global exponential decay rate (per minute)
DECAY_RATE = 0.0398
# ---------------------------------------------------------------------------
# Data loading helper
# ---------------------------------------------------------------------------
def load_systems_from_json(filepath: str) -> list:
"""
Load system definitions from the incident-response JSON file.
Parameters
----------
filepath : str
Path to the JSON file produced by the SRE team.
Returns
-------
list of dict
Each dict has keys: system, name, priority, dependencies, recovery_time.
"""
with open(filepath) as fh:
raw = json.load(fh)
return [
{
"system": item["System"],
"name": item.get("Name", item["System"]),
"priority": float(item["Priority"]),
"dependencies": list(item["Dependencies"]),
"recovery_time": int(item["Recovery Time (minutes)"]),
}
for item in raw
]
# ---------------------------------------------------------------------------
# Optimizer class
# ---------------------------------------------------------------------------
class DisasterRecoveryScheduler:
"""
MIP-based recovery scheduler for interdependent systems.
Decision variables
------------------
start[i] Continuous start time (minutes) for system i.
seq[i,j] Binary: 1 if system i completes before system j starts.
Created only for *incomparable* pairs (neither is a
transitive ancestor of the other).
decay[i] Auxiliary: -DECAY_RATE * completion_time_i (< 0).
expval[i] Auxiliary: exp(decay[i]) in (0, 1].
Objective
---------
Maximize sum_i priority_i * expval[i] (customer-facing systems only)
"""
def __init__(self, env: gp.Env):
"""Initialise with a Gurobi environment (must already be started)."""
self.env = env
self.model: gp.Model = None
# Data
self.systems: list = []
self.max_parallel: int = 2
# Internal model components (populated by build_model)
self._ids: list = []
self._sys_map: dict = {}
self._bigM: float = 0.0
self._ancestors: dict = {}
self._incomparable: list = []
self._incomparable_set: set = set()
self._start: gp.tupledict = {}
self._seq: gp.tupledict = {}
self._decay: gp.tupledict = {}
self._expval: gp.tupledict = {}
# ------------------------------------------------------------------
# Context-manager support
# ------------------------------------------------------------------
def __enter__(self):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
if self.model is not None:
self.model.dispose()
return False
# ------------------------------------------------------------------
# Data assignment
# ------------------------------------------------------------------
def set_data(self, systems: list, max_parallel: int = 2) -> None:
"""
Load system records and parallelism limit.
Parameters
----------
systems : list of dict
Required keys per record: system, priority, dependencies, recovery_time.
Optional key: name.
max_parallel : int
Maximum simultaneous recoveries (= number of on-call engineers).
"""
self.systems = systems
self.max_parallel = max_parallel
self._ids = [s["system"] for s in systems]
self._sys_map = {s["system"]: s for s in systems}
# ------------------------------------------------------------------
# Transitive ancestor computation — iterative (handles cycles gracefully)
# ------------------------------------------------------------------
def _compute_ancestors(self) -> dict:
"""
Return a dict mapping each system id to its set of transitive ancestors.
Uses Kahn's topological-sort algorithm (fully iterative, no recursion).
If a dependency cycle is detected (no valid topological ordering exists),
an empty ancestor set is returned for every system. The contradictory
precedence constraints (C1) that arise from the cycle will cause Gurobi
to correctly report the model as infeasible.
"""
sys_map = self._sys_map
ids = self._ids
# Build: for each dependency 'dep', list the systems that depend on it
dependents_of: dict = {sid: [] for sid in ids}
in_degree: dict = {sid: 0 for sid in ids}
for sid in ids:
for dep in sys_map[sid].get("dependencies", []):
if dep in sys_map:
in_degree[sid] += 1
dependents_of[dep].append(sid)
# Kahn's BFS topological sort
queue: collections.deque = collections.deque(
sid for sid in ids if in_degree[sid] == 0
)
topo_order: list = []
while queue:
node = queue.popleft()
topo_order.append(node)
for dependent in dependents_of[node]:
in_degree[dependent] -= 1
if in_degree[dependent] == 0:
queue.append(dependent)
# If not all nodes were processed, a cycle exists
if len(topo_order) != len(ids):
# Return empty ancestors; C1 constraints will expose the infeasibility
return {sid: set() for sid in ids}
# Compute transitive ancestors in topological order
ancestors: dict = {sid: set() for sid in ids}
for sid in topo_order:
for dep in sys_map[sid].get("dependencies", []):
if dep in sys_map:
ancestors[sid].add(dep)
ancestors[sid] |= ancestors[dep]
return ancestors
# ------------------------------------------------------------------
# Model construction (public entry point)
# ------------------------------------------------------------------
def build_model(self) -> None:
"""Build the complete MIP model (variables, constraints, objective)."""
self.model = gp.Model("DisasterRecovery", env=self.env)
# Non-convex setting required: we maximise a sum of convex exp functions
self.model.Params.NonConvex = 2
# Big-M: total sequential work — safe upper bound on any feasible finish time
self._bigM = float(sum(s["recovery_time"] for s in self.systems))
# Compute incomparable pairs (binary variables only for these)
self._ancestors = self._compute_ancestors()
self._incomparable = [
(i, j)
for i, j in combinations(self._ids, 2)
if j not in self._ancestors[i] and i not in self._ancestors[j]
]
self._incomparable_set = set(self._incomparable)
self._create_variables()
self._add_constraints()
self._set_objective()
self.model.update()
# ------------------------------------------------------------------
# Variable declarations
# ------------------------------------------------------------------
def _create_variables(self) -> None:
"""Declare all decision and auxiliary variables."""
ids = self._ids
# Start time of each system's recovery (continuous, non-negative)
self._start = self.model.addVars(ids, lb=0.0, name="start")
# Binary sequencing variables — only for incomparable pairs
# seq[i,j]=1 means system i completes before system j starts
ordered_pairs = (
[(i, j) for i, j in self._incomparable] +
[(j, i) for i, j in self._incomparable]
)
if ordered_pairs:
self._seq = self.model.addVars(ordered_pairs, vtype=GRB.BINARY, name="seq")
else:
self._seq = {}
# Auxiliary variables for the nonlinear retained-value objective
priority_ids = [s["system"] for s in self.systems if s["priority"] > 0]
if priority_ids:
# decay_i = -DECAY_RATE * (start_i + p_i) always in (-inf, 0]
self._decay = self.model.addVars(
priority_ids, lb=-GRB.INFINITY, ub=0.0, name="decay"
)
# expval_i = exp(decay_i) in (0, 1]
self._expval = self.model.addVars(
priority_ids, lb=0.0, ub=1.0, name="expval"
)
# ------------------------------------------------------------------
# Constraints
# ------------------------------------------------------------------
def _add_constraints(self) -> None:
"""Add all business constraints to the model."""
ids = self._ids
sys_map = self._sys_map
start = self._start
seq = self._seq
M = self._bigM
# C1: Dependency ordering — j cannot start before dependency i finishes.
for sid in ids:
for dep in sys_map[sid]["dependencies"]:
p_dep = sys_map[dep]["recovery_time"]
self.model.addConstr(
start[sid] >= start[dep] + p_dep,
name=f"prec_{dep}_{sid}",
)
# C2: Big-M sequencing — activates non-overlap when seq[i,j]=1.
# Applied only to incomparable pairs.
for i, j in self._incomparable:
p_i = sys_map[i]["recovery_time"]
p_j = sys_map[j]["recovery_time"]
self.model.addConstr(
start[j] >= start[i] + p_i - M * (1 - seq[i, j]),
name=f"bigm_{i}_{j}",
)
self.model.addConstr(
start[i] >= start[j] + p_j - M * (1 - seq[j, i]),
name=f"bigm_{j}_{i}",
)
# C3: Concurrency limit — for any (max_parallel+1) mutually incomparable
# systems, at least one pair must be sequenced.
K = self.max_parallel
for combo in combinations(ids, K + 1):
combo_pairs = list(combinations(combo, 2))
if all(
(a, b) in self._incomparable_set or (b, a) in self._incomparable_set
for a, b in combo_pairs
):
self.model.addConstr(
gp.quicksum(
seq[a, b] + seq[b, a] for a, b in combo_pairs
)
>= 1,
name="cap_" + "_".join(combo),
)
# C4: Decay variable definition
for sid, decay_var in self._decay.items():
p = sys_map[sid]["recovery_time"]
self.model.addConstr(
decay_var == -DECAY_RATE * start[sid] - DECAY_RATE * p,
name=f"decay_def_{sid}",
)
# C5: Exponential retained-value constraint (nonlinear)
for sid in self._expval:
self.model.addConstr(
self._expval[sid] == gp.nlfunc.exp(self._decay[sid]),
name=f"exp_def_{sid}",
)
# ------------------------------------------------------------------
# Objective
# ------------------------------------------------------------------
def _set_objective(self) -> None:
"""Maximise total retained customer value."""
obj = gp.quicksum(
self._sys_map[sid]["priority"] * self._expval[sid]
for sid in self._expval
)
self.model.setObjective(obj, GRB.MAXIMIZE)
# ------------------------------------------------------------------
# Solve and solution extraction
# ------------------------------------------------------------------
def solve(self) -> dict:
"""
Solve the model and return the recovery schedule.
Returns
-------
dict
status : int — Gurobi status code
is_optimal : bool — True when an optimal solution was found
objective_value : float or None
schedule : dict mapping system_id ->
{name, start, finish, recovery_time, priority}
"""
if self.model is None:
raise RuntimeError("Model not built. Call build_model() first.")
self.model.optimize()
status = self.model.Status
result = {
"status": status,
"is_optimal": status == GRB.OPTIMAL,
"objective_value": None,
"schedule": {},
}
if status == GRB.OPTIMAL:
result["objective_value"] = self.model.ObjVal
for sid in self._ids:
s_val = self._start[sid].X
p = self._sys_map[sid]["recovery_time"]
result["schedule"][sid] = {
"name": self._sys_map[sid].get("name", sid),
"start": round(s_val, 4),
"finish": round(s_val + p, 4),
"recovery_time": p,
"priority": self._sys_map[sid]["priority"],
}
return result
# ------------------------------------------------------------------
# Output / display
# ------------------------------------------------------------------
def display_results(self, result: dict) -> None:
"""Print a formatted recovery schedule to stdout."""
print("\n" + "=" * 68)
print(" DISASTER RECOVERY SCHEDULE — OPTIMAL PLAN")
print("=" * 68)
if not result["is_optimal"]:
print(f" No optimal solution found. Gurobi status: {result['status']}")
print("=" * 68)
return
print(f" Total Retained Customer Value : {result['objective_value']:.4f}")
print("-" * 68)
print(f" {'ID':<6} {'Name':<26} {'Start':>7} {'Finish':>7} {'V0':>7}")
print("-" * 68)
ordered = sorted(result["schedule"].items(), key=lambda kv: kv[1]["start"])
for sid, info in ordered:
print(
f" {sid:<6} {info['name']:<26} "
f"{info['start']:>7.2f} {info['finish']:>7.2f} "
f"{info['priority']:>7.0f}"
)
print("=" * 68)
# ------------------------------------------------------------------
# Model diagnostics
# ------------------------------------------------------------------
def model_summary(self) -> dict:
"""Return variable and constraint counts for the built model."""
if self.model is None:
return {}
self.model.update()
return {
"num_vars": self.model.NumVars,
"num_constrs": self.model.NumConstrs,
"num_bin_vars": self.model.NumBinVars,
"num_incomparable_pairs": len(self._incomparable),
}
# ---------------------------------------------------------------------------
# Main — solve the production 14-system dataset
# ---------------------------------------------------------------------------
if __name__ == "__main__":
import os
data_file = os.path.join(os.path.dirname(__file__), "incident_response_systems.json")
systems = load_systems_from_json(data_file)
with gp.Env(empty=True) as env:
env.setParam("OutputFlag", 1)
env.setParam("TimeLimit", 120)
env.start()
with DisasterRecoveryScheduler(env) as scheduler:
scheduler.set_data(systems, max_parallel=2)
scheduler.build_model()
summary = scheduler.model_summary()
print(f"\nModel summary: {summary}")
result = scheduler.solve()
scheduler.display_results(result)
A pytest suite with one test per scenario in the specification, each test
referencing its scenario identifier, using small hard-coded data and a short
solver time limit so the suite runs in well under a second.
"""
pytest suite for DisasterRecoveryScheduler
==========================================
Covers all six scenarios defined in incident_response_spec.md.
T1-T5 use small (5-6 system) instances.
T6 uses a 7-system representative subset and also attempts the full 14-system
dataset (skips gracefully on a size-limited license).
"""
import math
import time
import pytest
import gurobipy as gp
from gurobipy import GRB
from incident_response_modeler import DisasterRecoveryScheduler, DECAY_RATE
def _make_env(time_limit: int = 30) -> gp.Env:
env = gp.Env(empty=True)
env.setParam("OutputFlag", 0)
env.setParam("TimeLimit", time_limit)
env.start()
return env
def _solve(systems, max_parallel=2, time_limit=30) -> dict:
with _make_env(time_limit) as env:
with DisasterRecoveryScheduler(env) as sched:
sched.set_data(systems, max_parallel=max_parallel)
sched.build_model()
return sched.solve()
def retained_value(priority: float, finish_time: float) -> float:
return priority * math.exp(-DECAY_RATE * finish_time)
def compute_naive_objective(systems: list, max_parallel: int) -> float:
"""Greedy naive schedule in given system-list order."""
sys_map = {s["system"]: s for s in systems}
done_at: dict = {}
eng_avail = sorted([0.0] * max_parallel)
remaining = [s["system"] for s in systems]
while remaining:
for sid in remaining:
if all(d in done_at for d in sys_map[sid]["dependencies"]):
deps_done = max((done_at[d] for d in sys_map[sid]["dependencies"]), default=0.0)
eng_avail.sort()
start = max(eng_avail[0], deps_done)
finish = start + sys_map[sid]["recovery_time"]
eng_avail[0] = finish
done_at[sid] = finish
remaining.remove(sid)
break
return sum(
retained_value(sys_map[sid]["priority"], done_at[sid])
for sid in done_at if sys_map[sid]["priority"] > 0
)
# T1 -----------------------------------------------------------------------
T1_SYSTEMS = [
{"system": "SA", "name": "Core", "priority": 0, "dependencies": [], "recovery_time": 2},
{"system": "SB", "name": "Auth", "priority": 0, "dependencies": [], "recovery_time": 2},
{"system": "SC", "name": "Gateway", "priority": 100, "dependencies": ["SA"], "recovery_time": 3},
{"system": "SD", "name": "Checkout", "priority": 80, "dependencies": ["SB"], "recovery_time": 2},
{"system": "SE", "name": "Storefront", "priority": 60, "dependencies": ["SC", "SD"], "recovery_time": 1},
]
def test_t1_baseline():
"""T1: 5-system baseline — optimal solution found and all precedences satisfied."""
t0 = time.time()
result = _solve(T1_SYSTEMS, max_parallel=2, time_limit=30)
assert time.time() - t0 < 35
assert result["is_optimal"]
assert result["objective_value"] > 0
sch = result["schedule"]
assert set(sch.keys()) == {"SA", "SB", "SC", "SD", "SE"}
assert sch["SC"]["start"] >= sch["SA"]["finish"] - 1e-6
assert sch["SD"]["start"] >= sch["SB"]["finish"] - 1e-6
assert sch["SE"]["start"] >= sch["SC"]["finish"] - 1e-6
assert sch["SE"]["start"] >= sch["SD"]["finish"] - 1e-6
for sid, info in sch.items():
assert info["start"] >= -1e-6
# T2 -----------------------------------------------------------------------
def _t2_systems(se_priority: float) -> list:
return [
{"system": "SA", "name": "Core", "priority": 0, "dependencies": [], "recovery_time": 1},
{"system": "SB", "name": "Auth", "priority": 0, "dependencies": [], "recovery_time": 2},
{"system": "SC", "name": "Cache", "priority": 0, "dependencies": [], "recovery_time": 3},
{"system": "SD", "name": "Checkout", "priority": 100, "dependencies": ["SA", "SB"], "recovery_time": 2},
{"system": "SE", "name": "Mobile", "priority": se_priority, "dependencies": ["SB"], "recovery_time": 1},
{"system": "SF", "name": "Reports", "priority": 60, "dependencies": ["SC"], "recovery_time": 1},
]
def test_t2_priority_perturbation_changes_decision():
"""T2: SE priority 100 -> 10 shifts the optimal schedule (SE finish time increases)."""
t0 = time.time()
r_high = _solve(_t2_systems(100), max_parallel=2, time_limit=30)
r_low = _solve(_t2_systems(10), max_parallel=2, time_limit=30)
assert time.time() - t0 < 70
assert r_high["is_optimal"] and r_low["is_optimal"]
se_finish_high = r_high["schedule"]["SE"]["finish"]
se_finish_low = r_low["schedule"]["SE"]["finish"]
assert se_finish_high < se_finish_low + 1e-3
assert abs(r_high["objective_value"] - r_low["objective_value"]) > 1e-3
# T3 -----------------------------------------------------------------------
def test_t3_priority_perturbation_no_schedule_change():
"""T3: SE priority 100 -> 95 leaves every system start time unchanged."""
t0 = time.time()
r_100 = _solve(_t2_systems(100), max_parallel=2, time_limit=30)
r_95 = _solve(_t2_systems(95), max_parallel=2, time_limit=30)
assert time.time() - t0 < 70
assert r_100["is_optimal"] and r_95["is_optimal"]
for sid in r_100["schedule"]:
assert abs(r_100["schedule"][sid]["start"] - r_95["schedule"][sid]["start"]) < 1e-3
assert abs(r_100["objective_value"] - r_95["objective_value"]) > 1e-4
# T4 -----------------------------------------------------------------------
T4_SYSTEMS = [
{"system": "SA", "name": "Alpha", "priority": 0, "dependencies": ["SB"], "recovery_time": 2},
{"system": "SB", "name": "Beta", "priority": 10, "dependencies": ["SA"], "recovery_time": 3},
]
def test_t4_circular_dependency_infeasible():
"""T4: SA<->SB circular dependency must result in infeasibility."""
result = _solve(T4_SYSTEMS, max_parallel=2, time_limit=10)
assert not result["is_optimal"]
assert result["objective_value"] is None
# T5 -----------------------------------------------------------------------
T5_SYSTEMS = [
{"system": "SA", "name": "Core", "priority": 0, "dependencies": [], "recovery_time": 2},
{"system": "SB", "name": "Auth", "priority": 0, "dependencies": [], "recovery_time": 2},
{"system": "SC", "name": "Cache", "priority": 0, "dependencies": [], "recovery_time": 3},
{"system": "SD", "name": "Checkout", "priority": 100, "dependencies": ["SA", "SB"], "recovery_time": 2},
{"system": "SE", "name": "Mobile", "priority": 100, "dependencies": ["SB"], "recovery_time": 1},
{"system": "SF", "name": "Reports", "priority": 60, "dependencies": ["SC"], "recovery_time": 1},
]
def test_t5_optimal_beats_naive():
"""T5: MIP-optimal objective must be >= naive system-list-order greedy."""
t0 = time.time()
result = _solve(T5_SYSTEMS, max_parallel=2, time_limit=30)
assert time.time() - t0 < 35
assert result["is_optimal"]
naive_obj = compute_naive_objective(T5_SYSTEMS, max_parallel=2)
assert result["objective_value"] >= naive_obj - 1e-4
# T6 — Full 14-system production dataset -----------------------------------
FULL_14_SYSTEMS = [
{"system": "S1", "name": "Core Network", "priority": 0, "dependencies": [], "recovery_time": 5},
{"system": "S2", "name": "Auth Database", "priority": 0, "dependencies": ["S1"], "recovery_time": 8},
{"system": "S3", "name": "Cache Cluster", "priority": 0, "dependencies": ["S1"], "recovery_time": 4},
{"system": "S4", "name": "User Database", "priority": 0, "dependencies": ["S1"], "recovery_time": 10},
{"system": "S5", "name": "Auth Service", "priority": 0, "dependencies": ["S2"], "recovery_time": 6},
{"system": "S6", "name": "Billing Database", "priority": 0, "dependencies": ["S1"], "recovery_time": 7},
{"system": "S7", "name": "API Gateway", "priority": 0, "dependencies": ["S5", "S3"], "recovery_time": 5},
{"system": "S8", "name": "Checkout Service", "priority": 40, "dependencies": ["S7", "S6"], "recovery_time": 6},
{"system": "S9", "name": "Search Service", "priority": 25, "dependencies": ["S7", "S4"], "recovery_time": 4},
{"system": "S10", "name": "Web Storefront", "priority": 60, "dependencies": ["S8", "S9"], "recovery_time": 3},
{"system": "S11", "name": "Mobile API", "priority": 30, "dependencies": ["S7", "S4"], "recovery_time": 5},
{"system": "S12", "name": "Recommendations", "priority": 15, "dependencies": ["S4"], "recovery_time": 4},
{"system": "S13", "name": "Notifications", "priority": 10, "dependencies": ["S5"], "recovery_time": 3},
{"system": "S14", "name": "Admin Portal", "priority": 0, "dependencies": ["S2"], "recovery_time": 6},
]
def test_t6_full_14_system_dataset():
"""T6: Complete 14-system production dataset, max_parallel=2. Skips on a size-limited license."""
summary = {}
t0 = time.time()
try:
with _make_env(60) as env:
with DisasterRecoveryScheduler(env) as sched:
sched.set_data(FULL_14_SYSTEMS, max_parallel=2)
sched.build_model()
summary = sched.model_summary()
result = sched.solve()
except gp.GurobiError as exc:
if "too large" in str(exc).lower() or "size-limited" in str(exc).lower():
pytest.skip(f"14-system instance requires unrestricted license: {exc}")
raise
elapsed = time.time() - t0
assert result["is_optimal"], f"status={result['status']}, elapsed={elapsed:.1f}s"
assert result["objective_value"] > 0
sch = result["schedule"]
assert set(sch.keys()) == {s["system"] for s in FULL_14_SYSTEMS}
sys_map = {s["system"]: s for s in FULL_14_SYSTEMS}
for sid, info in sch.items():
for dep in sys_map[sid]["dependencies"]:
assert info["start"] >= sch[dep]["finish"] - 1e-4
Modeler doesn’t just generate code — it runs it, in an isolated cloud sandbox created fresh for this conversation, solving against Gurobi in the cloud. No local install or license is needed; it reads the output and fixes what it finds.
Please finalize the specification, implement it, write tests, run them, and report the results on the actual 14-system dataset.
The specification is complete — dependency precedence, the max_parallel=2 capacity limit, and the exponential decay objective are all pinned down. Time to write the sequencing-variable MIP, generate one test per scenario, and run the suite in the sandbox before reporting anything back.
All 6 tests pass in 0.86 s and the full 14-system dataset solves to proven global optimality.
============================= test session starts ==============================
collected 6 items
test_t1_baseline PASSED
test_t2_priority_perturbation_changes_decision PASSED
test_t3_priority_perturbation_no_schedule_change PASSED
test_t4_circular_dependency_infeasible PASSED
test_t5_optimal_beats_naive PASSED
test_t6_full_14_system_dataset PASSED
============================== 6 passed in 0.86s ===============================
Model statistics: 120 variables (26 continuous + 94 binary), 191 constraints, 47 incomparable pairs.
Optimize a model with 191 rows, 120 columns and 612 nonzeros
Model has 6 general nonlinear constraints (exp)
Variable types: 26 continuous, 94 integer (94 binary)
Presolve removed 38 rows and 12 columns
MIP has 6 nonlinear function transformations
Explored 4087 nodes (11842 simplex iterations) in 0.86 seconds (0.71 work units)
Thread count was 8 (of 8 available processors)
Solution count 4: 48.9233 48.8951 48.6104 ...
Optimal solution found (tolerance 1.00e-04)
Best objective 4.892330000000e+01, gap 0.0000%
Total Retained Customer Value: 48.9233
System |
Name |
Start |
Finish |
V0 |
|---|---|---|---|---|
S1 |
Core Network |
0.00 |
5.00 |
0 |
S2 |
Auth Database |
5.00 |
13.00 |
0 |
S3 |
Cache Cluster |
5.00 |
9.00 |
0 |
S6 |
Billing Database |
9.00 |
16.00 |
0 |
S5 |
Auth Service |
13.00 |
19.00 |
0 |
S4 |
User Database |
16.00 |
26.00 |
0 |
S7 |
API Gateway |
19.00 |
24.00 |
0 |
S8 |
Checkout Service |
24.00 |
30.00 |
40 |
S9 |
Search Service |
26.00 |
30.00 |
25 |
S11 |
Mobile API |
30.00 |
35.00 |
30 |
S10 |
Web Storefront |
30.00 |
33.00 |
60 |
S12 |
Recommendations |
33.00 |
37.00 |
15 |
S13 |
Notifications |
35.00 |
38.00 |
10 |
S14 |
Admin Portal |
75.00 |
81.00 |
0 |
File |
|---|
|
|
|
|
|
|
Test |
Result |
|---|---|
test_t1_baseline |
PASSED |
test_t2_priority_perturbation_changes_decision |
PASSED |
test_t3_priority_perturbation_no_schedule_change |
PASSED |
test_t4_circular_dependency_infeasible |
PASSED |
test_t5_optimal_beats_naive |
PASSED |
test_t6_full_14_system_dataset |
PASSED |
Everything is built and verified. Let me know if you’d like to change anything about the model, or if you’d like the mathematical formulation.
This run’s solver log lands in the workspace as execution.log. The loop
ends one of two ways: every test passes, as it did here, or Modeler finds a
genuine contradiction between the specification, the implementation, and the
tests, and stops to ask rather than guessing — that second path is what the
Changing the model tab demonstrates.
What the cloud environment gives you, and what it does not
The sandbox that ran this suite is created fresh for the conversation and torn down afterward. It solves against a real Gurobi installation in the cloud, so there’s nothing to install and no license file to manage — but it does have limits:
A trial license cap of 2,000 variables and 2,000 constraints per model. The 14-system production model above uses 120 variables and 191 constraints, well within that cap.
A 60-second execution limit per run.
No extra packages beyond what’s already in the sandbox, no internet access, and no connection to a Compute Server or cluster — this is a self-contained environment for building and validating a model, not for production-scale solves.
Business problems don’t hold still. The specification’s “Out of scope” section explicitly called out engineer heterogeneity — every engineer treated as interchangeable. In practice, some recoveries need a specific kind of engineer, and that assumption doesn’t survive first contact with a real incident.
Change request: it turns out not every recovery can be done by just any engineer. The three database systems in our dataset — Auth Database (S2), User Database (S4), and Billing Database (S6) — specifically require a database specialist (DBA) to recover, not a generalist engineer. Of our 2 on-call engineers, only 1 is DBA-qualified. Please update the specification, then the implementation, then the tests to reflect this.
This is a scope change, not a data correction — it moves “engineer heterogeneity” out of the out-of-scope list and introduces a second, independent capacity resource layered on top of the existing overall limit. The spec needs a new constraint (C6), a new input field (required_skill), and a new scheduler parameter (skill_capacities) before anything else changes. One subtlety worth surfacing: this new constraint can only ever make the schedule worse or the same, never better — since it’s a restriction added to an already-optimal model — so the user should expect the retained value to drop, not improve.
Got it. Here’s what changes in the specification:
Constraints — new C6: Skill-specific capacity. For each engineer skill type with declared capacity K_skill, at most K_skill systems requiring that skill may be in active recovery at once.
Input — systems gain a
required_skillfield ("dba"or none); scheduler parameters gainskill_capacities(e.g.{"dba": 1}).Out of scope — “Engineer heterogeneity” is removed from this list; “Multiple simultaneous specialist qualifications per engineer” is added in its place, since that’s the next layer of realism we’re deliberately still not modeling.
One thing to flag before I implement this: adding a capacity restriction can never improve the objective — the DBA constraint sits on top of the existing engineer-count limit, so the best this change can do is leave the schedule unchanged, and the more likely outcome is that it pushes the retained value down. Shall I proceed?
Yes, go ahead.
Specification, then code, then tests
C6 — Skill-Specific Capacity — Specialist Concurrency Limit
For each engineer skill type with declared capacity K_skill, and for any
group of (K_skill + 1) mutually incomparable systems carrying that skill,
at least one pair must be sequenced. Production configuration: skill
"dba", K_dba = 1. Systems S2, S4, S6 carry required_skill = "dba".
def set_data(
self,
systems: list,
max_parallel: int = 2,
+ skill_capacities: dict = None,
) -> None:
self.systems = systems
self.max_parallel = max_parallel
+ self.skill_capacities = skill_capacities if skill_capacities is not None else {}
self._ids = [s["system"] for s in systems]
self._sys_map = {s["system"]: s for s in systems}
+ # Build skill groups: skill -> [sys_id, ...]
+ self._skill_groups = {}
+ for s in systems:
+ skill = s.get("required_skill")
+ if skill:
+ self._skill_groups.setdefault(skill, []).append(s["system"])
def _add_constraints(self) -> None:
self._add_precedence_constraints() # C1
self._add_bigm_sequencing_constraints() # C2
self._add_overall_capacity_constraints() # C3
self._add_decay_definition_constraints() # C4
self._add_exp_value_constraints() # C5
+ self._add_skill_capacity_constraints() # C6
+def _add_skill_capacity_constraints(self) -> None:
+ """C6: at most K_skill specialist systems active at once, per skill."""
+ for skill, cap in self.skill_capacities.items():
+ skill_ids = self._skill_groups.get(skill, [])
+ if len(skill_ids) <= cap:
+ continue
+ for combo in combinations(skill_ids, cap + 1):
+ combo_pairs = list(combinations(combo, 2))
+ if all(
+ (a, b) in self._incomparable_set or (b, a) in self._incomparable_set
+ for a, b in combo_pairs
+ ):
+ self.model.addConstr(
+ gp.quicksum(
+ self._seq[a, b] + self._seq[b, a] for a, b in combo_pairs
+ ) >= 1,
+ name=f"skill_cap_{skill}_" + "_".join(combo),
+ )
# T7: DBA specialist constraint — serialization enforced
T7_SYSTEMS = [
{"system": "DA", "name": "DB Server A", "priority": 0, "dependencies": [], "recovery_time": 3, "required_skill": "dba"},
{"system": "DB", "name": "DB Server B", "priority": 0, "dependencies": [], "recovery_time": 4, "required_skill": "dba"},
{"system": "GS", "name": "Cache Svc", "priority": 0, "dependencies": [], "recovery_time": 2, "required_skill": None},
{"system": "PA", "name": "App Layer A", "priority": 100, "dependencies": ["DA"], "recovery_time": 2, "required_skill": None},
{"system": "PB", "name": "App Layer B", "priority": 80, "dependencies": ["DB"], "recovery_time": 2, "required_skill": None},
]
def test_t7_dba_constraint_serialization():
"""T7: DBA constraint forces DA and DB to be serialized."""
r_dba = _solve(T7_SYSTEMS, max_parallel=2, skill_capacities={"dba": 1}, time_limit=30)
r_free = _solve(T7_SYSTEMS, max_parallel=2, skill_capacities={}, time_limit=30)
assert r_dba["is_optimal"] and r_free["is_optimal"]
sch = r_dba["schedule"]
da_start, da_finish = sch["DA"]["start"], sch["DA"]["finish"]
db_start, db_finish = sch["DB"]["start"], sch["DB"]["finish"]
assert (da_finish <= db_start + 1e-4) or (db_finish <= da_start + 1e-4), (
"DBA systems must not overlap"
)
assert r_free["objective_value"] >= r_dba["objective_value"] - 1e-4
Where it stops and asks
Adding the DBA constraint touches more than S2, S4, and S6 — it also touches T5, the existing “optimal beats naive” scenario, because that test’s naive-baseline heuristic silently assumes every engineer is interchangeable. Modeler didn’t just implement the change and move on; it audited every existing scenario against the new rule first.
Reading |
What it would mean |
Verdict |
|---|---|---|
A — Naive ignores DBA, MIP enforces it |
Naive schedule can be infeasible under the real rules; the “optimizer wins” comparison becomes meaningless |
Wrong — silently invalid |
B — Both sides enforce DBA |
A legitimate test, but already covered by T6 and T7 |
Redundant, not wrong |
C — Neither side has DBA systems (T5 as built) |
Isolates one claim only: does smart ordering beat naive ordering under capacity alone? |
Correct, and what T5 already does |
T5’s data has no DBA-tagged systems, so the comparison is already
clean — no code change needed there. But there’s a real risk: if
someone edits T5_SYSTEMS later and adds a DBA system without
re-reading the spec, the naive baseline would silently become
infeasible under the real rules, and the test would compare against a
schedule that could never actually happen. Shall I add an explicit
guard assertion to test_t5_optimal_beats_naive that fails loudly if
that ever happens, instead of leaving it to a comment?
Good catch — yes, please add that guard.
Done. All 7 tests still pass. The guard names the offending systems in its failure message and points at the correct fix (“add a new T8 scenario instead of editing T5”), so a future contributor can’t accidentally break the isolation between “does smart ordering help?” and “does the DBA constraint work?” without an immediate, explicit error.
The answer flips
The DBA constraint is a pure restriction on top of an already-optimal schedule, so it can only hold value steady or push it down — and on the real 14-system dataset, it pushes it down. Auth Database, Billing Database, and User Database now have to be serialized onto the one DBA-qualified engineer instead of running two-at-a-time, which delays User Database’s downstream customer-facing dependents (Search Service, Mobile API, Recommendations).
Rule |
User Database finish |
Total retained value |
|---|---|---|
Before (any engineer can do DB work) |
t = 26 |
48.9233 |
After (DBA-only, 1 of 2 engineers qualified) |
t = 30 |
45.0021 |
A 3.92-point drop — about 8% of the retained value — for a constraint that never shows up in the objective function itself, only in which schedules are even reachable.
The specification, code, and tests all deliberately avoid math — they’re written for a business audience. This artifact exists for the audience that thinks in constraints rather than business rules: review, documentation, or handoff to someone auditing the model. Modeler derives it from the current specification and implementation on request, and saves it to the workspace as Markdown with LaTeX.
Sets and indices
\(\mathcal{N}\) — set of all systems to be recovered, indexed by \(i\) or \(j\)
\(\mathcal{E} \subseteq \mathcal{N} \times \mathcal{N}\) — directed dependency edges: \((i,j) \in \mathcal{E}\) means system \(j\) depends directly on system \(i\)
\(\mathcal{A}(i)\) — transitive ancestors of system \(i\): every system that must complete before \(i\), directly or transitively
\(\mathcal{I} \subseteq \binom{\mathcal{N}}{2}\) — incomparable pairs \(\{i,j\}\), where neither is a transitive ancestor of the other (the only pairs needing a sequencing variable)
\(\mathcal{P} \subseteq \mathcal{N}\) — customer-facing systems, \(\mathcal{P} = \{i \in \mathcal{N} \mid V_i > 0\}\)
\(\mathcal{S}\) — set of specialist skill types (production: \(\mathcal{S} = \{\text{dba}\}\))
\(\mathcal{N}_s \subseteq \mathcal{N}\) — systems requiring specialist skill \(s \in \mathcal{S}\)
Parameters
\(p_i \in \mathbb{Z}_{>0}\) — recovery duration of system \(i\), in minutes
\(V_i \geq 0\) — base customer priority; \(V_i > 0\) iff \(i \in \mathcal{P}\)
\(\lambda = 0.0398\) — exponential decay rate per minute, uniform across all systems
\(K \in \mathbb{Z}_{>0}\) — overall engineer capacity (\(K=2\) in production)
\(K_s \in \mathbb{Z}_{>0}\) — specialist capacity for skill \(s\) (\(K_{\text{dba}}=1\) in production)
\(M = \sum_{i \in \mathcal{N}} p_i\) — Big-M coefficient, a safe upper bound on any feasible completion time
Decision variables
\(x_i \geq 0,\ x_i \in \mathbb{R}\) for all \(i \in \mathcal{N}\) — start time of system \(i\). The completion time is the derived quantity \(C_i = x_i + p_i\).
\(z_{ij} \in \{0,1\}\) for every \(\{i,j\} \in \mathcal{I}\), both directions — \(z_{ij}=1\) encodes “system \(i\) completes before system \(j\) starts.” Restricted to incomparable pairs only: for any ancestor-descendant pair, the precedence constraint C1 already forces the ordering, so no binary variable is needed there. On the 14-system production instance this cuts binary variables from 182 (all ordered pairs) to 94, a 48% reduction.
\(d_i \in (-\infty, 0]\) for \(i \in \mathcal{P}\) — decay auxiliary
\(e_i \in [0,1]\) for \(i \in \mathcal{P}\) — retained-value auxiliary
Objective function
Constraints
C1 — Dependency precedence.
C2 — Big-M disjunctive sequencing, for every unordered incomparable pair \(\{i,j\} \in \mathcal{I}\):
C3 — Overall engineer capacity. For every \((K{+}1)\)-element subset \(T \subseteq \mathcal{N}\) in which every pair is incomparable:
C4 — Decay variable definition, for \(i \in \mathcal{P}\):
C5 — Exponential retained-value constraint (nonlinear), for \(i \in \mathcal{P}\):
C6 — Skill-specific specialist capacity. For each skill \(s \in \mathcal{S}\) with capacity \(K_s\), and every \((K_s{+}1)\)-element subset \(T \subseteq \mathcal{N}_s\) in which every pair is incomparable:
For the production configuration (\(K_{\text{dba}}=1\), \(\mathcal{N}_{\text{dba}} = \{S_2, S_4, S_6\}\)), this reduces to complete pairwise serialization: every incomparable DBA pair must be fully sequenced.
Complete model
Additional notes
Incomparable-pair restriction. Binary variables and both C2 and C6 are instantiated only over \(\mathcal{I}\), computed via Kahn’s topological sort. On the 14-system instance \(|\mathcal{I}|=47\) versus \(\binom{14}{2}=91\) total pairs.
Nonlinearity. C5 is a nonlinear equality; maximizing a sum of convex exponential functions is globally non-convex, so the model sets
NonConvex = 2to enable Gurobi’s spatial branch-and-bound, which certifies global optimality rather than a local one. The 14-system instance solves to a proven 0.00% gap in under one second.C3 and C6 share one pattern. Both are “at least one sequencing pair per oversized clique” constraints; C6 is C3 restricted to a specialist subset under its own, tighter capacity. Since \(K_s < K\) in any heterogeneous-crew deployment, C6 adds genuinely tightening constraints beyond what C3 alone provides.
Note
This formulates the current model, after the DBA specialist constraint (C6) introduced in the Changing the model tab. The pre-change model is identical except for the omission of C6 and the \(\mathcal{N}_s\) / \(K_s\) parameters — every other set, variable, and constraint is unchanged.
### Sets and indices
- $\mathcal{N}$ — set of all systems to be recovered, indexed by $i$ or $j$
- $\mathcal{E} \subseteq \mathcal{N} \times \mathcal{N}$ — directed dependency edges: $(i,j) \in \mathcal{E}$ means system $j$ depends directly on system $i$
- $\mathcal{A}(i)$ — transitive ancestors of system $i$: every system that must complete before $i$, directly or transitively
- $\mathcal{I} \subseteq \binom{\mathcal{N}}{2}$ — incomparable pairs $\{i,j\}$, where neither is a transitive ancestor of the other (the only pairs needing a sequencing variable)
- $\mathcal{P} \subseteq \mathcal{N}$ — customer-facing systems, $\mathcal{P} = \{i \in \mathcal{N} \mid V_i > 0\}$
- $\mathcal{S}$ — set of specialist skill types (production: $\mathcal{S} = \{\text{dba}\}$)
- $\mathcal{N}_s \subseteq \mathcal{N}$ — systems requiring specialist skill $s \in \mathcal{S}$
### Parameters
- $p_i \in \mathbb{Z}_{>0}$ — recovery duration of system $i$, in minutes
- $V_i \geq 0$ — base customer priority; $V_i > 0$ iff $i \in \mathcal{P}$
- $\lambda = 0.0398$ — exponential decay rate per minute, uniform across all systems
- $K \in \mathbb{Z}_{>0}$ — overall engineer capacity ($K=2$ in production)
- $K_s \in \mathbb{Z}_{>0}$ — specialist capacity for skill $s$ ($K_{\text{dba}}=1$ in production)
- $M = \sum_{i \in \mathcal{N}} p_i$ — Big-M coefficient, a safe upper bound on any feasible completion time
### Decision variables
- $x_i \geq 0,\ x_i \in \mathbb{R}$ for all $i \in \mathcal{N}$ — start time of system $i$. The completion time is the derived quantity $C_i = x_i + p_i$.
- $z_{ij} \in \{0,1\}$ for every $\{i,j\} \in \mathcal{I}$, both directions — $z_{ij}=1$ encodes "system $i$ completes before system $j$ starts." Restricted to incomparable pairs only: for any ancestor-descendant pair, the precedence constraint C1 already forces the ordering, so no binary variable is needed there. On the 14-system production instance this cuts binary variables from 182 (all ordered pairs) to 94, a 48% reduction.
- $d_i \in (-\infty, 0]$ for $i \in \mathcal{P}$ — decay auxiliary
- $e_i \in [0,1]$ for $i \in \mathcal{P}$ — retained-value auxiliary
### Objective function
$$
\max \quad \sum_{i \,\in\, \mathcal{P}} V_i \cdot e_i
\;=\; \sum_{i \,\in\, \mathcal{P}} V_i \cdot e^{-\lambda (x_i + p_i)}
$$
### Constraints
**C1 — Dependency precedence.**
$$
x_j \;\geq\; x_i + p_i \qquad \forall\, (i,j) \in \mathcal{E}
$$
**C2 — Big-M disjunctive sequencing**, for every unordered incomparable pair $\{i,j\} \in \mathcal{I}$:
$$
x_j \;\geq\; x_i + p_i \;-\; M\,(1 - z_{ij})
$$
$$
x_i \;\geq\; x_j + p_j \;-\; M\,(1 - z_{ji})
$$
**C3 — Overall engineer capacity.** For every $(K{+}1)$-element subset $T \subseteq \mathcal{N}$ in which every pair is incomparable:
$$
\sum_{\{a,b\} \subseteq T} \bigl(z_{ab} + z_{ba}\bigr) \;\geq\; 1
$$
**C4 — Decay variable definition**, for $i \in \mathcal{P}$:
$$
d_i \;=\; -\lambda\,(x_i + p_i)
$$
**C5 — Exponential retained-value constraint (nonlinear)**, for $i \in \mathcal{P}$:
$$
e_i \;=\; \exp(d_i)
$$
**C6 — Skill-specific specialist capacity.** For each skill $s \in \mathcal{S}$ with capacity $K_s$, and every $(K_s{+}1)$-element subset $T \subseteq \mathcal{N}_s$ in which every pair is incomparable:
$$
\sum_{\{a,b\} \subseteq T} \bigl(z_{ab} + z_{ba}\bigr) \;\geq\; 1
$$
For the production configuration ($K_{\text{dba}}=1$, $\mathcal{N}_{\text{dba}} = \{S_2, S_4, S_6\}$), this reduces to complete pairwise serialization: every incomparable DBA pair must be fully sequenced.
### Complete model
$$
\begin{aligned}
\max \quad & \sum_{i \,\in\, \mathcal{P}} V_i \cdot e_i \\[6pt]
\text{s.t.} \quad
& x_j \geq x_i + p_i
&& \forall\, (i,j) \in \mathcal{E}
&& \text{(C1)}\\[4pt]
& x_j \geq x_i + p_i - M\,(1 - z_{ij})
&& \forall\, \{i,j\} \in \mathcal{I}
&& \text{(C2a)}\\[4pt]
& x_i \geq x_j + p_j - M\,(1 - z_{ji})
&& \forall\, \{i,j\} \in \mathcal{I}
&& \text{(C2b)}\\[4pt]
& \sum_{\{a,b\}\subseteq T}(z_{ab} + z_{ba}) \geq 1
&& \forall\, T \subseteq \mathcal{N},\, |T|=K{+}1
&& \text{(C3)}\\[4pt]
& d_i = -\lambda\,(x_i + p_i)
&& \forall\, i \in \mathcal{P}
&& \text{(C4)}\\[4pt]
& e_i = \exp(d_i)
&& \forall\, i \in \mathcal{P}
&& \text{(C5)}\\[4pt]
& \sum_{\{a,b\}\subseteq T}(z_{ab} + z_{ba}) \geq 1
&& \forall\, s\in\mathcal{S},\, T\subseteq\mathcal{N}_s,\, |T|=K_s{+}1
&& \text{(C6)}\\[6pt]
& x_i \geq 0
&& \forall\, i \in \mathcal{N}\\[4pt]
& z_{ij} \in \{0,1\}
&& \forall\, \{i,j\} \in \mathcal{I}\\[4pt]
& d_i \in (-\infty, 0],\ e_i \in [0,1]
&& \forall\, i \in \mathcal{P}
\end{aligned}
$$
### Additional notes
- **Incomparable-pair restriction.** Binary variables and both C2 and C6 are instantiated only over $\mathcal{I}$, computed via Kahn's topological sort. On the 14-system instance $|\mathcal{I}|=47$ versus $\binom{14}{2}=91$ total pairs.
- **Nonlinearity.** C5 is a nonlinear equality; maximizing a sum of convex exponential functions is globally non-convex, so the model sets `NonConvex = 2` to enable Gurobi's spatial branch-and-bound, which certifies global optimality rather than a local one. The 14-system instance solves to a proven 0.00% gap in under one second.
- **C3 and C6 share one pattern.** Both are "at least one sequencing pair per oversized clique" constraints; C6 is C3 restricted to a specialist subset under its own, tighter capacity. Since $K_s < K$ in any heterogeneous-crew deployment, C6 adds genuinely tightening constraints beyond what C3 alone provides.
> This formulates the **current** model, after the DBA specialist constraint (C6) introduced in the "Changing the model" tab. The pre-change model is identical except for the omission of C6 and the $\mathcal{N}_s$ / $K_s$ parameters — every other set, variable, and constraint is unchanged.