Testing Strategy Optimization

Problem Statement

Smartly decide which machines to run tests on and what kind of testing environment to simulate.

Full example scenario

A DevOps engineer wants to optimize the testing strategy for a software application that must be tested on four operating systems — Linux64, Armlinux64, MacOS, and Windows — each exactly once. There are 10 testing environments, numbered 1-10, and 40 testing machines with 10 dedicated to each OS, where each machine can only handle a subset of the testing environments. Which {machine, testing environment} combination should be chosen for each operating system, in order to maximize the total score?

The score for a given combination is the number of days since that machine/environment pair’s last testing date, multiplied by its modifier. The number of days matters because you do not want to keep testing on the same machine over and over — repeatedly leaning on the same machine risks uneven wear, degraded reliability, and results that no longer reflect a fresh environment, so the model favors machine/environment pairs that have sat idle the longest. The modifier is not spelled out in the data, but a multiplier layered on top of idle-days is a natural way to encode something else that matters besides recency — for example, how capable a machine is (a higher-spec rig is worth prioritizing sooner), how critical the environment it is wired to is (a production-like environment may be worth refreshing more urgently), or a general reliability/priority weighting from past runs. Whatever its exact business meaning, the model treats it the same way: a multiplier on top of days idle, so a fresher-but-lower- priority pair can still lose out to an older-but-higher-priority one.

  • Each operating system must be tested exactly once, on one of its OS-specific machines.

  • Each testing environment can be chosen at most once across all operating systems.

  • A {machine, testing environment} combination can only be chosen if that machine actually supports the given environment.

  • The compatibility and scoring data can be found in testing_strategy_optimization.csv, with the columns OS,Testing Environment,Machine,Testing Date,Modifier.

What an optimal solution looks like

The compatibility grid — which environment each machine is wired to, and how long it has sat idle.

Every OS has its own 10 dedicated machines, numbered #1 – #10. Each machine is wired to exactly one testing environment (colored badge), so picking a machine also picks its environment. Showing #1 – #3 and #10 here; #4 – #9 follow the same pattern.

OS Machine #
#1 #2 #3 … #10
Linux64 Env #1
32d idle		 Env #1
18d idle		 Env #2
23d idle		 … Env #4
3d idle
Armlinux64 Env #1
32d idle		 Env #2
18d idle		 Env #3
23d idle		 … Env #2
3d idle
MacOS Env #2
32d idle		 Env #1
18d idle		 Env #4
23d idle		 … Env #1
3d idle
Windows Env #4
32d idle		 Env #4
18d idle		 Env #3
23d idle		 … Env #1
3d idle

✓ Results — the plan the optimizer actually picked.

OS Machine # Environment Days idle Modifier Score
Armlinux64 #5 Env #1 21 2 42
MacOS #5 Env #2 21 2 42
Linux64 #5 Env #3 21 2 42
Windows #2 Env #4 18 2 36
Max total score 162

Built with Modeler

The problem above is stated as a single shot: you hand over the whole problem and get a model back. Modeler takes the other route — you start with the business question you actually have, and it works with you until the problem is pinned down, the data is in, the code runs, and the answer holds up when the problem changes.

The tabs below walk the same testing-strategy problem through that route, one Modeler capability at a time.

Instead of writing the whole prompt up front, you start with the business question you actually have. Modeler searches the knowledge base, asks one clarifying question at a time — always the most important one first — and tells you which parts of the model are still missing. The options it offers come from examples in the knowledge base and from how similar problems are usually modeled in your industry, so you can answer with a number instead of prose.

I’m a DevOps engineer. Before every release we need to test our application on several operating systems, and we have a pool of testing machines to run those tests on, plus a number of testing environment “slots” we can assign work to. Each machine can only be plugged into certain slots (not all combinations work), and I want to pick, for each OS, which machine and which slot to run it on, so as to maximize some kind of overall score for the choices we make — the score depends on how long it’s been since that particular machine/slot combo was last used for testing and a modifier value tied to it. Can you help me build this as an optimization model?

This has the shape of a resource assignment problem — OS, machine, and slot form one combined decision, with two separate uniqueness constraints in play (slot and machine). Let me check the knowledge base for how similar assignment problems are formulated before asking anything.

Good — the knowledge base points at the standard resource-assignment pattern. The one thing that changes the entire variable and constraint structure is how “recency” and “modifier” actually combine into the number being maximized, so that has to be the first question.

Good start. Here is my first question:

Question 1 of several: How is the assignment score calculated?

You mentioned it depends on two things: (1) how long ago the machine-slot pair was last used, and (2) a modifier value. Which of the following best describes how those two factors are combined into a single score?

  1. The score is the product of the two factors: days_since_last_use x modifier

  2. The score is a weighted sum: w1 * days_since_last_use + w2 * modifier

  3. The modifier is applied on top of a base score derived from recency (e.g., a lookup table maps recency to a base score, then multiplied by the modifier)

  4. The score is just the modifier value, and recency is used only as a tiebreaker

  5. Something else — please describe

Learn More:
• Resource Assignment Problem formulation ★★☆

Option 1 — the score is the product: days_since_last_use * modifier. Here’s the rest of the picture: we have 4 operating systems (Linux64, ArmLinux64, MacOS, Windows), and each one must be tested exactly once. There are 10 testing environment slots, numbered 1-10, and each slot can be used by at most one OS overall. We have 40 physical testing machines, 10 dedicated to each OS, and each machine is only wired up to a specific subset of the 10 slots — so the machine and slot choice for an OS are linked, not independent; you can only pick a (machine, slot) pair if that machine actually supports that slot. One more requirement: maximize the total score summed over the 4 assignments we make (one per OS).

That resolves the score formula and volunteers the entire rest of the structure at once: OS count, slot uniqueness, machine-to-slot wiring as the source of validity, and a single additive objective. Nothing about the business rules is still open — what remains is purely a data-shape question (how the compatibility and score data will actually arrive), which decides variable and constraint construction just as much as anything asked so far.

The specification draft is saved. Here is the next question:

Question 2: How will the input data be provided to the model?

Specifically, for the valid (machine, slot) pairs along with their days_since_last_use and modifier values — which of the following best describes the expected input format?

  1. As pandas DataFrames passed directly into the solver class or function (data is generated or loaded by the caller)

  2. As CSV or JSON files that the model reads from disk at runtime

  3. As plain Python lists or dictionaries hardcoded or constructed in the calling code

  4. The model should include a built-in data generator that creates realistic sample data for the 4 OSes / 40 machines / 10 slots setup, so it can be run standalone

  5. A combination — for example, a built-in sample dataset for testing, but the solve function accepts DataFrames for production use

Modeler updates the specification after every answer, even while it is still incomplete — so you can always see what it has understood and what it is still waiting on. Here, the one open item left is not a business rule but a data-shape choice, which is exactly what the next tab walks through.