Quick answer
SageMaker model customization uses datasets and evaluators as versioned assets. AWS describes training examples such as prompt-response pairs, domain text, or labeled data. Evaluators can be:
- code-based reward functions, used for verifiable scoring and custom scorers;
- natural-language reward prompts, used for RLAIF and LLM-as-a-judge evaluation.
Keep training, validation, and final holdout data separate. A scorer that influenced training cannot provide an independent deployment decision by itself.
Dataset contract
For every dataset version, record:
- source, owner, license, consent, and allowed purpose;
- collection window and population;
- schema, task type, and expected output;
- deduplication and contamination checks;
- sensitive-data detection and redaction;
- train, validation, and holdout split method;
- known gaps, label quality, and removal process.
Do not place secrets or unrelated personal data in training examples. Confirm S3, KMS, IAM, account, and Region boundaries before upload.
Evaluator contract
| Evaluator | Good fit | Required check |
|---|---|---|
| Exact or code scorer | Structured, testable outcomes | Edge cases, gaming, deterministic behavior |
| Domain rubric | Multi-dimensional quality | Reviewer agreement and rubric coverage |
| LLM judge | Scalable qualitative comparison | Bias, prompt sensitivity, position effects, calibration |
| Human review | High-impact or ambiguous outputs | Expertise, sampling, privacy, and escalation |
Use more than one signal. A reward function can be incomplete, a reward prompt can prefer fluent errors, and humans can disagree. Inspect false positives and false negatives against an expert-reviewed sample.
Evaluation and lineage
AWS documents logged models, registered models, model package groups, and automatically captured lineage between jobs, input datasets, output models, evaluations, and deployments. Registration supports governance; it does not itself approve the model.
Compare the custom model with its base model on task quality, regressions, safety, latency, throughput, and cost. Define a rejection threshold before looking at final results. If an evaluation feature uses Amazon Bedrock Evaluations, review AWS’s note about possible cross-Region processing within the same geography.
Use the setup guide for job submission and the deployment guide after approval.
Frequently asked questions
What are SageMaker model customization assets?
AWS defines datasets and evaluators as model customization assets. Datasets provide training or evaluation examples, while evaluators use reward functions or reward prompts to assess outputs.
What is the difference between a reward function and a reward prompt?
A reward function is code-based scoring logic, while a reward prompt instructs an LLM to judge response quality. Both require validation and neither should be treated as unquestionable ground truth.
Does a successful customization job mean the model passed evaluation?
No. Job completion creates artifacts and logged-model records. Evaluation on an untouched holdout plus human and deployment-specific review is a separate approval step.
Official sources
- SageMaker model customization concepts
- SageMaker open-weight model evaluation
- SageMaker open-weight job submission
Source check: August 4, 2026. Verify supported dataset formats, scorer interfaces, processing locations, permissions, lineage behavior, and evaluation limits before use.