Quick answer
Orchard-SWE is the software-engineering recipe in Microsoft Orchard. The published workflow combines distilled agent trajectories, credit-assignment supervised fine-tuning, reinforcement learning, denser process feedback, and value-model reranking.
Microsoft reports a 69.7% SWE-bench Verified result for the dense-reward recipe and 73.0% with value-model reranking. These numbers describe a particular model, dataset, harness, benchmark version, and selection procedure. They do not predict performance on a private repository or establish that generated patches are secure.
Training sequence
| Stage | Purpose | Review boundary |
|---|---|---|
| Trajectory distillation | Collect multi-turn issue-solving behavior | Check source-model terms, repository licenses, leakage, and task provenance |
| Credit-assignment SFT | Learn from productive parts of partial attempts | Inspect how steps receive credit and whether errors are retained |
| Balanced Adaptive Rollout | Use sparse pass/fail feedback more efficiently | Reproduce sampling, task balance, and hidden-test isolation |
| Dense feedback | Score decisions or process before final success | Treat teacher and judge scores as fallible labels |
| Value reranking | Select among several candidate solutions | Report candidate count and selection cost with the result |
The Microsoft Research post says the SWE dataset was distilled from 107,000 interactions across GitHub issues. The current repository describes a swe dataset configuration with 107,185 multi-turn rollouts. Record the exact dataset revision used by your experiment rather than merging rounded and repository counts.
Reproduce before adapting
Start with a bounded evaluation:
- pin the Orchard code, model checkpoint, dataset revision, harness, and container image;
- confirm the benchmark split and remove any training overlap;
- run the untrained or pre-SFT baseline under the same inference budget;
- reproduce one training stage at a time;
- report pass rate, attempted samples, retries, candidate count, and compute;
- manually review a sample of passing and failing patches;
- test on an internal holdout with security and regression gates.
A patch passing benchmark tests can still introduce insecure behavior, hidden regressions, license problems, or an unacceptable implementation. Keep human review and CI independent of the reward model.
Choose the relevant method guide
For a deeper breakdown of credit-assignment SFT and rollout feedback, use the Orchard SFT and RL guide. For sandbox lifecycle and repository isolation, read the Orchard Env guide. The real-harness guide covers train-deploy mismatch.
Frequently asked questions
What is Orchard-SWE?
Orchard-SWE is Microsoft’s software-engineering recipe for training and evaluating an agent that works across real repositories using the shared Orchard Env substrate.
What result does Microsoft report for Orchard-SWE?
Microsoft reports 69.7% on SWE-bench Verified for the dense-reward recipe and 73.0% when a separately trained value model reranks candidate solutions. These are reported research results under the documented setup, not a production guarantee.
Why use partial failed trajectories in Orchard-SWE?
Credit-assignment supervised fine-tuning attempts to retain productive portions of a trajectory even when the final issue is not resolved, increasing useful training signal without labeling every failed step as worthless.
Official sources
Source check: August 4, 2026. Verify the paper, code, dataset, model, harness, benchmark revision, inference budget, and reranking setup before comparing results.