# Themis data curation — themis-v0

This is an automated, reproducible pilot release. Model training has not started.

## Release

| Split | Decisions | Requests | Multi-question requests |
| --- | ---: | ---: | ---: |
| train | 44,577 | 33,573 | 5,324 |
| validation | 3,170 | 2,275 | 343 |
| calibration | 2,606 | 1,945 | 218 |
| test | 3,693 | 2,781 | 420 |

## Training coverage

Targets are maximum decision counts. Missing coverage is not padded with unrelated tasks.

| Category | Target | Selected | Shortfall |
| --- | ---: | ---: | ---: |
| engineering | 7,000 | 7,000 | 0 |
| ai_agents_evals | 7,000 | 7,000 | 0 |
| trust_safety | 4,000 | 4,000 | 0 |
| customer_support | 5,000 | 5,000 | 0 |
| sales_commerce | 4,000 | 3,595 | 405 |
| finance | 5,000 | 3,500 | 1,500 |
| legal | 5,000 | 5,000 | 0 |
| product_management | 3,000 | 482 | 2,518 |
| data_analytics | 5,000 | 5,000 | 0 |
| documents_meetings | 4,000 | 4,000 | 0 |
| design | 1,000 | 0 | 1,000 |

## Question types and labels

| Type | Training decisions |
| --- | ---: |
| choice | 30,400 |
| noul | 7,292 |
| score | 6,885 |

Soft targets: 12,900 training decisions. Distributions are preserved; only rounding drift within 1e-5 is normalized.

| Label origin | Training decisions |
| --- | ---: |
| objective_commit_pair | 5,853 |
| procedural_rule | 6,215 |
| source_annotation_or_transformation | 20,285 |
| teacher_distribution | 12,224 |

## Sources

Revisions and every downloaded shard's SHA-256 are pinned in `configs/sources.lock.json` and copied into the release.

- [tasksource/tasksource-jev-typed-decisions](https://huggingface.co/datasets/tasksource/tasksource-jev-typed-decisions) — `263e8c75c6a90c8bb43e2c8993b7f911e1ee5722`; candidate training/holdout source.
- [tasksource/procedural-typed-decisions](https://huggingface.co/datasets/tasksource/procedural-typed-decisions) — `e4b21b7b55f6cc9e76e33563e9d604ac5fcd25ad`; candidate training/holdout source.
- [tasksource/synthetic-typed-decisions](https://huggingface.co/datasets/tasksource/synthetic-typed-decisions) — `7ec5a61d5db158f637c30adeb6e66660f1dfee87`; candidate training/holdout source.
- [LocalLLaMA/typed-decisions](https://huggingface.co/datasets/LocalLLaMA/typed-decisions) — `7189f46b02248bd2569df7325c9df8e3b24e90a7`; candidate training/holdout source.
- [bigcode/commitpackft](https://huggingface.co/datasets/bigcode/commitpackft) — `fc56fe33c030c6daa414c2b112c932b8eed085e6`; candidate training/holdout source.
- [atlanai/decision-bench](https://huggingface.co/datasets/atlanai/decision-bench) — `353cc17388ca67913320a39dacd043524a35090a`; evaluation exclusion reference.

Tasksource sources are deliberately mapped to coverage categories. Noncommercial, unspecified, bare `other`, and custom-scoped reuse terms are excluded. Per-record source terms remain in metadata; share-alike and attribution obligations are not removed.

CommitPackFT Python diffs are paired with their original commit message among four options. Distractors come from the same derived partition, with candidate provenance retained. Fork aliases are joined before assigning project-level splits. Stack Overflow popularity prediction is excluded from engineering.

## Automated validation

Audit passed: **True**. The audit independently checks every exported decision against its pinned original target and re-tokenizes every single- and multi-question request.

- output checksums
- input/target separation
- probability validity
- source target alignment
- Qwen payload token budget
- state/group/fingerprint split isolation
- literal Decision Bench overlap
- Multi-question pack coverage: every selected decision appears exactly once in its split's request export.
- Native held-out state matches in training: 0.

## Exclusions

Upstream rows scanned: 1,492,387. Unique non-conflicting candidate decisions: 300,513.
Repeated decisions collapsed: 244. Conflicting duplicate observations: 1; affected decision fingerprints are excluded.

Counts below are filter events. Some filters operate on source rows and others on individual decisions; they are not additive record totals.

| Filter | Events |
| --- | ---: |
| unmapped_source | 714,798 |
| procedural_case_subsample | 351,043 |
| procedural_duplicate_collection | 123,013 |
| redundant_augmentation_variant | 91,895 |
| unmapped_workflow | 12,595 |
| reuse_terms_excluded | 8,403 |
| normalization:commitpack_terms_or_message_quality | 7,228 |
| over_token_budget | 6,006 |
| benchmark_overlap | 28 |
| normalization:duplicate_option_text | 7 |
| native_holdout_overlap | 2 |

## Files and schema

- `data/curated/themis-v0/<split>.jsonl`: one decision per row, with `input`, `target`, `option_keys`, and separate provenance `metadata`.
- `data/curated/themis-v0/<split>-requests.jsonl`: same-state decisions packed into bounded multi-question `input`, with keyed `targets`, `option_keys`, and original `decision_ids`.
- `manifest.json`: counts, exclusion evidence, output checksums, configuration and code fingerprints.
- `validation.json`: completed audit results.
- `<split>-review-samples.json`: deterministic examples for semantic review.

The `input` field alone is model-visible. Never pass the complete training row to a model. Probability vectors align exactly with `option_keys`; yes/no always uses `[true, false]` and ordered scales retain their original order. These are trainer-independent files; the training adapter must consume the soft targets explicitly.

## Limits and next curation work

- Category mappings are curated rules, not independent human annotations.
- Literal overlap checks do not detect all paraphrases or upstream pretraining exposure.
- Token budget is canonical request JSON plus 128 reserved tokens; final trainer serialization must be checked.
- Native held-out records are preserved; only the synthetic collection guarantees workflow-disjoint native splits.
- Source reuse classifications are metadata evidence, not independent legal clearance.
- Automated structural validation does not establish semantic label correctness. Teacher distributions are teacher judgments, not a correctness oracle; rule labels have not been independently recomputed from generator code.
- Synthetic application/skill categories are heuristics; product coverage currently means feedback triage, not feature prioritization or release planning.
- Documents coverage includes document entailment and dialogue acts; it is not a dedicated AMI action-item extraction set.
- Add independently sourced finance, commerce, and product decisions for the remaining quotas. Design needs a separately labeled text UI/accessibility dataset; the original benchmark's icon-image task cannot be reproduced with text alone.
- Re-check the exact final trainer prompt and tokenizer budget. The current budget is canonical API request JSON plus a 128-token reserve, not a guarantee for every future head's serialization.