Understanding Themis, from zero to the completed experiment

A learning guide, not a launch post. Verified against the local implementation and saved experiment artifacts on 16 October 2026.

This guide covers the task, architecture, data, training, evaluation, actual results, and the proposed vision-capable 2B model. It starts with the basic concepts and then connects them to the code. Invented numerical examples are marked illustrative. Experiment numbers come from the saved run. Proposed work is explicitly identified as proposed.

You do not need to know calculus to follow the main explanation. The equations give precise definitions; the paragraphs underneath explain what they mean.

How to use this guide

Read Parts 1–3 before inspecting the charts. Read Parts 4–6 before drawing conclusions about the model. Part 7 explains the next model without treating it as already built.

At the end, there is a source map, a glossary, and questions you can use to check your understanding. The best preparation for your own writeup is to explain each worked example aloud without copying the wording here.

The experiment in one sentence: we adapted the text network of Qwen3.5-0.8B to score supplied answer options directly, trained it with LoRA on a curated mixture of typed decisions, selected a checkpoint using validation distribution loss, and calibrated its probabilities on a separate split.

In this map, targets connect to the loss, never to the model-visible request. During prediction, the loss and optimizer path is absent.

Part 1 — What problem are we solving?

1.1 A decision is an answer inside a specified answer space

Consider a support ticket:

Invoice INV-4411 was charged twice. Please refund one charge.

An application might need three outputs: whether this is a refund request, which team should handle it, and its urgency. The answers are bounded by the application’s definitions. The model does not invent a new department or an undefined urgency level.

The input has two essential parts:

• State: the case or evidence to inspect. Here, the ticket text. It can also be a string containing JSON, a code diff, a policy excerpt, or document text.
• Questions: instructions defining the decisions, their types, and their allowed options or scale descriptions.

The application supplies the meaning of the decision. The model estimates a distribution over its allowed answers.

An illustrative choice output is:

{"billing": 0.94, "technical": 0.04, "sales": 0.02}

This is a categorical distribution: the values are nonnegative and sum to one. The highest-probability answer is billing. It does not mean billing is the objectively correct answer with a proven 94% chance on this individual ticket.

1.2 Why use a model if rules can make decisions?

Use a rule when the decision is already a reliable rule. A condition such as amount > limit is better computed directly than estimated by a neural network.

A learned decision model becomes useful when the evidence is varied language and the mapping is difficult to enumerate: interpreting a customer’s intent, checking whether a passage supports a claim, matching a code change to its description, or choosing a next action from contextual evidence.

The model remains one part of the application. It does not execute refunds, establish legal truth, or determine a company’s policy. Application code controls thresholds, escalation, access, and actions.

1.3 The three types implemented in Themis v0

noul: yes/no. The name comes from the typed-decision ecosystem. Our internal order is always true, then false: [P(true), P(false)]. A predicted distribution [0.8, 0.2] means the model favours yes. Other implementations can use different orders; never assume a universal convention.

choice: exactly one of the supplied options. Options can change between requests. Today the model may choose between departments; tomorrow it may choose between five proposed code fixes. The model reads the option descriptions as part of its input.

score: one level on an ordered scale. The levels are supplied in order, from lowest to highest. The API returns both the most likely level index and an expected score computed from the distribution. For three levels with illustrative probabilities [0.1, 0.3, 0.6], the expected index is 0×0.1 + 1×0.3 + 2×0.6 = 1.5.

That expected value is in index units. If the descriptions refer to real amounts such as £0, £100, and £1,000, our implementation does not automatically convert the expected index into an expected monetary amount. Equal spacing between level indices is an assumption of this summary.

The current implementation does not provide multi-label selection, span extraction, or region pointing. Those exist elsewhere in the AMOS family but were outside Themis v0’s scope.

1.4 How this differs from asking a chat model for JSON

A chat model normally predicts output tokens successively. To produce a JSON answer, it generates keys, punctuation, strings, and possibly numbers. Constrained decoding and schema enforcement can make this reliable; structured output is not unique to decision models.

Themis instead produces numerical answer scores inside one forward pass over the rendered request. Ordinary Python code constructs the output object. The neural network does not generate its JSON syntax, answer names, or decimal strings.

The useful distinction is the prediction objective and output path. We train the model to match decision distributions directly, and we avoid an autoregressive output-generation loop. We have not run a matched experiment proving this architecture is universally better or faster than every structured-generation alternative.

1.5 What “System 1” means here

It is an analogy for fast, direct decisions, not a demonstrated theory of the model’s cognition. Themis does substantial computation inside many neural-network layers. It does not expose a generated chain of thought. Neither fact proves that it does, or does not, perform an internal reasoning process.

Call it a direct decision model when you want to be precise.

1.6 A probability and a good action are different things

Choosing the highest-probability answer is a sensible default for a classification task where mistakes have equal cost. Business actions often have unequal costs. Failing to flag fraud and incorrectly flagging a legitimate payment may have very different consequences.

If the output estimates an event such as fraud, application code can choose an action by expected cost: multiply each outcome's probability by that action's cost under the outcome, add them, and choose the lowest-cost action. This requires relevant probabilities and explicit cost assumptions. A support-routing label or a teacher's preferred action distribution is not automatically an event-risk estimate suitable for such a calculation.

Similarly, “send to human review below 80% confidence” is a proposed operating policy, not something our current evaluation has validated. Threshold quality depends on calibration, coverage, error cost, and distribution. Themis produces scores; choosing and testing an operating policy is another layer of work.

Part 2 — From text to scores: the architecture

2.1 Tokens are the network’s input units

A tokenizer converts text into integer IDs. A token might correspond to a word, a fragment, punctuation, or whitespace. Token count is not word count, and the same text can have different lengths under different tokenizers.

The tokenizer and the input template are part of the model interface. Changing them can change the network’s input even if the human-readable case appears similar.

Our rendered context limit is 2,048 tokens, including system text, case, all question instructions, all option descriptions, chat-template control tokens, and answer placeholders. It is not a 2,048-token allowance for the case alone.

2.2 Embeddings turn IDs into vectors

An embedding table contains a numerical vector for each token ID. Looking up an ID yields its vector. These are learned coordinates, not an explicit dictionary of meanings such as “dimension 3 means refund.”

Initially, identical token IDs look up identical vectors. After the network layers process the sequence, their resulting representations differ according to context. The word “charge” in a refund ticket and “charge” in an electrical specification need not have the same final representation.

2.3 Hidden states are contextual representations

At each token position, the network maintains a vector called a hidden state. Attention and feed-forward transformations update it through successive layers. The final sequence of hidden states represents the processed input in a form that a readout can use.

For a batch with B requests, padded sequence length L, and hidden width d, the final hidden tensor has shape:

[B, L, d]

A shape describes the dimensions of a numerical array. It does not tell you what the network knows or why it made a particular decision.

2.4 We start with a trained model, not random weights

The base is Qwen/Qwen3.5-0.8B, pinned to revision:

2fc06364715b967f1860aea9cf38778875588b17

We use its existing text network and output-embedding weights. The knowledge and representations acquired during its prior training are the starting point. Our relatively small decision dataset adapts that network; it does not teach language understanding from scratch.

The model name is a size designation. Do not infer the exact retained parameter count, image-encoder size, or memory requirement simply by multiplying the name by a storage dtype. The recorded count we can state exactly for our adaptation is 21,645,312 trainable parameters.

For a more concrete scale, the pinned 0.8B configuration has 24 text layers and hidden width 1,024. Its active text network contains 752,393,024 parameters, including the shared input/output embedding. The stored checkpoint also contains vision and auxiliary multi-token-prediction weights that are not part of our retained text path. The source notes explain why the complete checkpoint's stored parameter count is larger than the active text network's count. LoRA adds parameters on top; it does not replace the base with a 21.65-million-parameter model.

The base Qwen checkpoint includes vision machinery. Our loader retains the text decoder and output embeddings, releases the encompassing multimodal model, and never moves the vision tower to the training GPU. Themis v0 is text-only.

Source: model loader, especially load_model().

2.5 Attention: what can a token read?

Attention lets a token’s representation incorporate information from other token positions. A mask determines which positions are permitted to contribute.

In causal attention, a token can read its own position and earlier positions. This is useful for next-token generation: when predicting a continuation, future answer tokens must not be visible.

In bidirectional attention, a real input token can also read later real input tokens. This is useful for processing an already supplied case and set of questions as a whole.

Qwen3.5 is a hybrid: it combines full-attention layers with gated linear-attention/recurrent layers. Our code changes the mask for full-attention layers to allow all real input positions and exclude padding. It preserves the forward recurrence in linear-attention layers.

Specifically, the text stack repeats three linear-attention layers followed by one full-attention layer, six times: 18 linear-attention layers and 6 full-attention layers. Full attention compares token positions through queries, keys and values. The recurrent components carry and update a state along the sequence; they do not use precisely the same all-pairs computation. This hybrid is why changing one attention mask does not automatically change all information flow in the network.

Therefore, “we converted the entire model into a fully bidirectional encoder” would be inaccurate. Later full-attention layers can mix information across the input, but we did not reverse or replace every recurrent layer.

There is also a distribution shift: the base network was not originally trained under precisely our decision-input and attention contract. Fine-tuning adapts to that change. We have not isolated the benefit of the altered mask with a separate causal-attention ablation.

Source: bidirectional_masks().

Inside a network block: attention, positions, and the feed-forward path

For conventional full attention, learned projections make queries, keys, and values from the incoming representations. A query describes what a position is looking for; keys provide features against which queries are compared; values carry information to mix. These are explanatory roles, not human-written semantic fields.

The core operation is:

attention(Q,K,V) = softmax(Q × transpose(K) / sqrt(d_key) + mask) × V

The query/key comparison produces scores between positions. Scaling controls their magnitude. The mask blocks forbidden interactions. Softmax makes mixing weights, and the weighted sum of values yields the attention output. Multiple attention heads can compute different mixtures in parallel; grouped-query designs can share keys/values across query heads. The actual Qwen projection shapes should be read from its configuration rather than assumed to be a simple width-divided-by-heads layout.

Position information matters because the same words in a different order can mean something different. Qwen uses rotary position embeddings, or RoPE, which rotate query/key features according to positions so their comparisons can depend on relative position. RoPE is part of attention computation, not a separate database containing word order.

Linear-attention blocks use a different recurrent mechanism. Conceptually, each new position reads and updates a compact state instead of forming the same explicit all-pairs attention table. Gates control how information is retained or updated; the gated delta mechanism uses a corrective update. This is a rough explanation of its role, not the exact mathematical recurrence. Our attention-mask change leaves that recurrent algorithm forward-running.

Attention is only part of a block. A gated feed-forward network applies learned linear transformations and nonlinearities to each position's features. Normalization controls feature scale, and residual connections add transformed features back to the running representation. Together, many layers repeatedly mix contextual information and transform it into features useful for the output objective.

A linear projection here means matrix multiplication, optionally with a bias. It is different from the term linear attention, which names the sequence-processing mechanism. LoRA adapts many such projections inside both kinds of block. Themis does not replace every block with a small classifier.

For these ideas, an attention heatmap can illustrate a computation but is not a complete causal explanation of why the model chose an answer. Many layers and transformations contribute.

2.6 The exact input contract

Our codec renders system instructions, a user message containing the case and every question with options, and an assistant message containing fixed placeholders. A simplified readable version is:

System: You make decisions about a case ...

Case:
Invoice INV-4411 was charged twice. Please refund one charge.

Questions:
q1. Does the customer request a refund?
A) yes
B) no

q2. Which team should handle this?
A) billing
B) technical
C) sales

Assistant:
q1: <answer>
q2: <answer>

This sketch omits Qwen’s actual chat-template delimiters. The implementation applies the tokenizer’s chat template with thinking disabled. The inputs are not just a concatenation of raw JSON.

Question names are retained outside this rendering for mapping results back. The rendered questions use sequential names such as q1 and q2.

There are no gold answers in those placeholders. Training targets are separate numerical arrays attached to the batch after encoding. This matters particularly because our full-attention layers can read future positions. A gold answer accidentally inserted into the input could become a shortcut.

2.7 Why read immediately before <answer>?

For each question, the codec records the position immediately before its <answer> token. That is the position from which a conventional language-model readout would score the next token.

Our network processes the complete placeholder-containing input, then gathers the final hidden states at these recorded positions. The placeholders provide a stable format; their presence does not mean that the model generates or fills them.

We add <answer> as a special token and initialize its input embedding from the mean of existing embeddings. The embedding is frozen, and its value is explicitly saved for consistent reloads. Only LoRA parameters train. The new marker is not a learned answer classifier.

Source: Codec.encode() and DecisionModel.forward().

2.8 Answer codes are temporary addresses, not permanent labels

For one request, code A might mean billing. For another, it might mean “reject the proposed change.” The option text in that request establishes the correspondence.

The codec checks candidate code strings using the actual tokenizer. It accepts only codes that are a single token, have distinct token IDs, and are not special tokens. It verifies 255 usable codes. Candidates start with uppercase and lowercase letters, followed by uppercase pairs where those pairs tokenize as one token.

This verification matters: treating AA as one answer code when it is actually two tokens would break a one-token readout. We do not assume that all character pairs work.

For a decision with three options, only the first three code embeddings are allowed. A 255-code vocabulary does not imply every decision has 255 options. Long option descriptions can exhaust the context window before that maximum becomes useful.

2.9 The scoring equation

Let h_q be the selected hidden-state vector for question q. Let e_k be the frozen output-embedding vector for answer code k.

z[q,k] = dot(h_q, e_k)
p[q,k] = exp(z[q,k]) / sum_j exp(z[q,j])

z is a logit: an unnormalized score. p is the resulting probability after softmax. A dot product multiplies corresponding vector components and adds the products.

Illustratively, logits [2, 1, 0] become approximately [0.6652, 0.2447, 0.0900]. Adding 100 to every logit changes none of those probabilities; relative differences matter.

The code computes this as a matrix operation, in float32 at the readout, then masks nonexistent options with negative infinity before softmax. If one question has three options and another has five, padding the score tensor to width five does not give the three-option question two extra choices.

The output embeddings are frozen. Training changes the hidden states through LoRA so that their scores align with the right answer-code embeddings.

At prediction time these probabilities are conditional on the supplied answer set. Softmax always distributes all probability among valid options. If the correct real-world answer is absent, the model can still assign a very high probability to one of the bad choices. An explicit “none of these” option or application-level abstention policy needs definition and evaluation; normalized output alone does not detect missing answers or unfamiliar cases.

2.10 Why not score the entire vocabulary?

A conventional language-model output layer can score every vocabulary token at every output position. We only need scores for a small allowed set at the selected question positions.

If there are Q questions, K allowed codes, and hidden width d, this selected readout requires work on the order of Q × K × d, rather than applying a vocabulary-wide projection to every input position. This saves output-projection work and intermediate storage.

It does not remove the cost of processing the input through the network. A long case is still expensive. A single forward pass is not constant-time inference, and full-attention layers still have quadratic sequence interactions.

2.11 Multiple questions share computation, but are not isolated

The case appears once in a packed request. We obtain multiple selected hidden states and multiple distributions from the same pass. That can avoid repeatedly encoding the same case for separate questions.

Our mask does not isolate each question into its own attention branch. In full-attention layers, questions can influence one another through the supplied input. The implementation also does not explicitly enforce logical consistency between their answers.

Consequently, we should not claim that adding another question can never change an existing answer. Question-set invariance and cross-question consistency would need their own evaluations.

2.12 What “AMOS-style” does and does not mean

Here it describes the approach of rendering typed decisions and scoring selected answer codes without generating prose. It does not mean our weights, data, training recipe, or results reproduce AMOS exactly.

AMOS Mini uses a different mechanism: an Ettin text encoder with pointer-style heads that compare question representations against contextual option representations. Its vision path uses a frozen SigLIP2 encoder and a projector. That is different from our fixed answer-code embedding readout.

The AMOS family’s repositories can evolve. Our experiment is defined by its own pinned Qwen revision, dataset revision, configuration, and source hashes. A current upstream model card is not evidence of what every AMOS release used when we started.

Primary sources: AMOS Mini configuration, Mini decision heads, and the separately recorded primary-source notes.

Part 3 — The data: what the model learns from

3.1 A training example is a supervised comparison

A decision example contains the state, question instructions, allowed options, and a target distribution. The model predicts a distribution, and training compares it with the target.

The target expresses what the dataset asks the model to match. Its evidential strength depends on where it came from. A deterministic rule, a human annotation, a transformed benchmark label, and a teacher model’s probability estimate are not interchangeable sources of truth.

We preserve label-origin metadata so that distinction remains inspectable.

3.2 Hard targets and soft targets

For a three-option decision, a hard target might be:

[0, 1, 0]

This is a one-hot vector: all target probability is on the second answer.

A soft target might be:

[0.15, 0.70, 0.15]

Training on the soft vector asks the model to preserve the alternative possibilities, not merely choose the middle option. Converting every target to its argmax would discard that information.

However, a teacher-generated soft target is not a proven measure of real uncertainty. It can encode the teacher’s bias or miscalibration. Matching it closely is evidence of matching the reference, not automatic evidence of factual correctness.

3.3 Alignment is more important than the array looks

Suppose the keys are [billing, technical, sales] and the target is [0.8, 0.1, 0.1]. The vector only makes sense in that order.

If training shuffles the options to [sales, billing, technical], the aligned target must become [0.1, 0.8, 0.1].

Shuffling the option text without moving the probabilities would train incorrect supervision while still producing perfectly valid arrays. This is why our schema carries explicit option_keys, and why tests check alignment rather than only checking that probabilities sum to one.

Yes/no and score ordering stay fixed. Choice options are deterministically shuffled during training, with targets moved alongside them.

3.4 Where our mixture came from

The curated mixture combines records derived from:

• tasksource/tasksource-jev-typed-decisions;
• tasksource/procedural-typed-decisions;
• tasksource/synthetic-typed-decisions;
• LocalLLaMA/typed-decisions;
• Python code changes from bigcode/commitpackft, transformed into diff/message matching choices.

atlanai/decision-bench is an evaluation exclusion reference, not a training source. Its text is used to help filter literal overlap.

Exact revisions and downloaded-file hashes are recorded in sources.lock.json. The published mixture used for training is lothnic/themis-decision-mixture, pinned at:

92843bdf66de205d6b4238c6a0d70701fd6737a7

The model name is Themis. The dataset name describes a reusable mixture; it is a separate artifact with its own version and provenance.

3.5 What curation actually did

The pipeline normalizes different upstream formats into the same decision schema, checks distributions and option text, assigns categories using explicit mappings, filters unsuitable reuse metadata, excludes literal benchmark overlap, deduplicates examples, groups related examples for split assignment, samples for coverage, and packs same-state questions into requests.

It also excludes conflicting duplicate fingerprints instead of arbitrarily choosing one label. Fingerprints normalize the relevant state/question/option content; choice fingerprints account for option ordering.

These are automated and rule-based checks. They do not establish that every question is meaningful, every category tag is perfect, or every label is correct. A semantic review of weak and ambiguous examples remains valuable.

The curation report records 1,492,387 upstream rows scanned and 300,513 unique non-conflicting candidate decisions before final sampling. Those are not the training counts. Filter-event totals also operate at different levels and must not be added as if they were mutually exclusive rows.

Sources: curation report, normalization, selection.

3.6 The engineering examples explain one important result

For CommitPack-derived examples, the state is a Python code diff. The correct option is its original commit message; three distractors are messages sampled from the same derived partition, excluding near-identical text.

Repository/fork aliases are joined so related projects do not leak across derived splits. Nevertheless, identifying an original message among random alternatives can be easier than choosing between four plausible fixes to a subtle bug.

This distinction helps explain why very high in-mixture engineering agreement should not be advertised as broad software-engineering competence. The label is grounded in provenance, but the difficulty also depends on the distractors.

Source: CommitPack adapter.

3.7 The four splits serve different jobs

Training: 44,577 decisions in 33,573 requests. These examples contribute gradients that update LoRA parameters.

Validation: 3,170 decisions in 2,275 requests. We use these to compare checkpoints and monitor training. They do not contribute parameter-update gradients.

Calibration: 2,606 decisions in 1,945 requests. After choosing the checkpoint, we use these to fit three temperature values. They do not train the LoRA weights.

Test: 3,693 decisions in 2,781 requests. These measure the selected model and its calibrated probabilities. They do not select the checkpoint or fit temperatures.

Total: 54,046 decisions across all four splits. Saying “trained on 54,046 examples” would obscure that 9,469 decisions were held out from weight training.

Splits are group-aware rather than a naive shuffle of individual rows. Upstream test records remain held out, upstream validation records supply validation/calibration, and connected state/group identities are kept out of competing splits. Different source collections have different grouping guarantees; not every held-out task is a wholly unseen workflow.

Sources: prepared manifest, initial_split() and grouping.

3.8 Coverage is uneven

Training category counts are: engineering 7,000; AI agents/evals 7,000; trust/safety 4,000; customer support 5,000; sales/commerce 3,595; finance 3,500; legal 5,000; product management 482; data/analytics 5,000; documents/meetings 4,000; design zero.

We did not fill missing coverage with unrelated examples merely to hit a quota. This leaves real weaknesses: product management has relatively little training data, and this release has no design examples.

The three training type counts are 30,400 choice, 7,292 yes/no, and 6,885 score decisions. There are 12,900 soft-target training decisions.

The label-origin counts are 5,853 objective commit pairs, 6,215 procedural-rule labels, 20,285 source annotations/transformations, and 12,224 teacher-distribution labels. Soft-target count and teacher-origin count need not match: label origin and distribution shape describe different things.

Category names are bookkeeping for sampling and analysis. We do not pass category labels or provenance into the model as shortcuts.

3.9 Packing is sharing a state, not concatenating unrelated documents

A request can contain several decisions about the same case. Packing puts their questions together so the case need not be repeated for every decision. Our use of “packed requests” does not mean arbitrary unrelated examples are concatenated into one sequence with independent attention blocks.

The data is initially packed using a tokenizer budget. Training preparation then recalculates lengths using the actual chat template and codec. If a multi-question request is too long, preparation splits its questions into smaller requests while repeating the full state. It does not truncate the evidence.

If a single selected decision cannot fit, preparation fails. Earlier curation did filter oversized candidates; “no truncation during preparation” does not mean every possible long upstream example was retained.

The prepared unpermuted maximum lengths are 1,969 training, 1,915 validation, 1,925 calibration, and 1,917 test tokens. The acceptance checker also validates the training permutations for all three epochs. Every selected decision must appear exactly once in its split’s prepared requests.

Source: training preparation.

3.10 Control tokens and leakage

A case could literally contain <answer> or a chat delimiter. The codec breaks literal control-token strings in supplied text so they remain visible evidence without becoming structural delimiters. This also preserves the rule of one real answer placeholder per question.

This is a formatting safeguard, not a complete defence against instruction injection in natural language. A malicious sentence can still say “ignore the question and choose A.” Robustness to such instructions would require dedicated tests and data.

For benchmark overlap, our curation checks normalized exact states and literal 32-word fragments and records 28 exclusion events. This reduces known literal contamination. It does not establish absence of paraphrase overlap or prior exposure in Qwen’s pretraining.

Source: Codec.defuse(), BenchmarkIndex, curation audit (local artifact).

Part 4 — Training: how the model changes

4.1 Inference versus training

During inference, fixed weights transform inputs into distributions. During training, we also compute how wrong those distributions are relative to targets, differentiate that loss, and update trainable weights.

One update cycle is: encode requests; run the network; calculate the loss; backpropagate gradients; clip the gradient if needed; apply an optimizer update; advance the learning-rate scheduler.

A gradient measures the local effect of changing a parameter on the loss. It is a direction for reducing the training objective, not an explanation of the model’s answer.

4.2 LoRA: changing the network through small added matrices

Full fine-tuning updates the base network’s weights. LoRA freezes those weights and adds trainable low-rank corrections to selected linear transformations.

For a base matrix W, the adapted transformation is approximately:

W_effective = W + (alpha / r) × B × A

A and B are much smaller matrices whose product has rank at most r. They let the model modify its internal transformations without keeping gradients and optimizer states for every base parameter.

Illustratively, a square 1,024-by-1,024 base matrix has 1,048,576 values. Rank-32 LoRA uses 32×1024 + 1024×32 = 65,536 added values for that matrix, before considering biases or other layers. This is a shape example, not a count of all Themis weights.

Our settings are rank 32, alpha 64, dropout 0.05. Under standard LoRA scaling, alpha/r is 2. Dropout randomly suppresses part of the adapter input during training; evaluation disables it.

The loader targets decoder linear modules with PEFT’s all-linear setting. It excludes the released vision tower and the separately held output embeddings. The final code checks that every trainable parameter name belongs to LoRA. Recorded trainable count: 21,645,312.

Freezing a base matrix does not make its computation free. Gradients still have to propagate through the network to train adapters, and intermediate activations still consume memory.

LoRA is also not the same as quantization. QLoRA combines low-rank adaptation with a quantized frozen base; our run uses BF16 rather than a four-bit QLoRA base. At inference, an unmerged adapter adds its own small matrix operations. Merging compatible LoRA updates into base weights can remove those separate operations, but our current deliverable is an adapter export, not a merged or quantized release.

Primary background: LoRA paper. Implementation: model loader.

4.3 The actual loss is soft cross-entropy

For target probabilities t[k] and predicted probabilities p[k]:

CE(t,p) = -sum_k t[k] × ln(p[k])

Natural logarithms give loss in nats. Lower is better.

For a one-hot target, only the correct answer’s term remains. If the correct answer has probability 0.8, loss is about 0.2231; if it has probability 0.01, loss is about 4.6052. Confident wrong answers are heavily penalized.

For a soft target [0.7, 0.3], predicting exactly [0.7, 0.3] gives loss about 0.6109, not zero. That remaining loss is the target distribution’s entropy. Predicting [0.99, 0.01] gives about 1.3886—even though the highest-probability answer is still the same.

This is the first reason accuracy and loss can disagree. Accuracy asks which option came first. Cross-entropy evaluates the entire probability distribution.

One useful derivative makes the learning signal intuitive: with softmax and cross-entropy, the gradient with respect to each logit is p[k] - t[k], before the batch-normalization factor. For illustrative predictions [0.6, 0.3, 0.1] and target [0.8, 0.1, 0.1], that signal is [-0.2, +0.2, 0]. Gradient descent therefore pushes the first score up and the second down, locally. The optimizer does not edit the logits permanently; backpropagation translates that signal into changes to adapter parameters that produced the hidden state.

Our training objective uses soft cross-entropy for all three types. We did not add an ordinal-distance loss for score decisions or a Brier-loss term. Ordered descriptions and the reported expected-score metric do not mean the optimizer explicitly penalizes a two-level error more than a one-level error in an ordinal-specific way.

Source: soft_ce_sum() and optimize_window().

4.4 Microbatches, accumulation, and the correct denominator

A microbatch is the group of requests processed together before one backward pass. Gradient accumulation combines several backward passes before an optimizer update.

We start with 8 requests per microbatch × 2 accumulation = 16 requests per optimizer step. The final window of an epoch can be smaller.

Requests contain different numbers of decisions. If one has one question and another has five, giving both requests equal average weight would reduce the weight of each question in the five-question request.

Instead, we sum per-decision losses and divide by the actual number of decisions across the full accumulation window. Every decision in that window has equal weight. This does not mean every category or source has equal weight: their frequencies still affect their contributions.

Illustratively, one microbatch has two decisions and another has six. The denominator is eight, not the number of requests and not an average of two separately normalized microbatch means.

Gradient accumulation approximates the intended larger batch, but dropout and floating-point reduction order mean differently partitioned microbatches need not be bit-for-bit identical.

4.5 What happens if memory runs out?

The implementation catches an out-of-memory error at microbatch eight, clears incomplete gradients and failed graph references, clears the CUDA cache, restores the saved random-number state, and replays the whole accumulation window at microbatch four.

The effective window remains 16 requests, so that generally becomes four microbatches. It does not retain gradients from the partially failed attempt. Another failure at microbatch four is raised rather than silently shrinking indefinitely.

This is an implemented fallback, not evidence that the completed run necessarily used it. It is also different from automatically retrying a fresh cloud training experiment.

4.6 Optimizer, learning rate, and schedule

AdamW maintains running gradient statistics to scale updates and applies decoupled weight decay. We use weight decay 0.01. The implementation applies the configured optimizer to the trainable adapter parameters.

The learning rate is the update-size control. Our main rate is 0.0001, also written 1e-4. It is not a fixed promise that each parameter moves by that exact amount: gradients, AdamW’s state, and scaling determine the update.

The scheduler warms up for 3% of the planned optimizer steps—188 steps under this run’s integer calculation—then follows a cosine decrease toward 0.00001, or 1e-5. Warmup gradually increases the scale of early updates; decay makes late updates smaller.

The gradient-norm clip threshold is 1.0. If the combined gradient is too large, clipping scales it down before the update. A logged pre-clipping norm above one is not itself evidence that clipping failed.

4.7 Precision and GPU memory

CUDA execution uses BF16 for much of the network computation. BF16 stores values more compactly than float32 and has a broad exponent range, but less numerical precision. Our selected logits and loss calculations use float32; metric calculations use float64 in the analysis functions.

GPU memory includes weights, trainable gradients, optimizer state, saved activations, temporary workspaces, and allocator overhead. The model’s parameter-storage size alone is not a training-memory estimate.

Gradient checkpointing saves memory by retaining fewer forward activations and recomputing some during backward. It trades extra computation for reduced memory use. It has nothing to do with saving a resumable training checkpoint to disk; the shared name is confusing.

The paid worker was an NVIDIA A100-SXM4-40GB, requested as A100-40GB, with four CPU cores and 24 GiB host RAM. Local smoke testing used the detected RTX 4060 Laptop GPU, not the originally discussed hypothetical 16 GB card.

Source: saved provenance, smoke report (local artifact).

4.8 Steps and epochs: derive the run length

An epoch is one traversal of the training requests. An optimizer step is one weight update after accumulation.

ceil(33,573 requests / 16 requests per window) = 2,099 steps per epoch
2,099 × 3 epochs = 6,297 steps
44,577 decisions × 3 epochs = 133,731 decision presentations

Decision presentations are repeated exposures, not 133,731 unique labelled decisions. Choice permutations change their presentation order, not their underlying identity.

The epoch sampler shuffles requests deterministically from the seed and epoch. Choice permutations use a deterministic seed derived from the global seed, epoch, and request ID. The recorded global seed is 3407.

4.9 Validation and early stopping

The program evaluates validation before training, every 500 optimizer steps, at epoch boundaries, and when finalization requires it. It saves the checkpoint with the lowest observed validation soft cross-entropy.

The patience rule counts evaluations without an improvement of at least 1e-4; after the first epoch, three such evaluations can stop normal training. Checkpoint selection is subtly separate: any strictly lower loss can become best, even if the improvement is too small to reset patience.

You later explicitly requested completion of all three epochs. The authorized continuation preserved the same experiment and bypassed the patience stop. It still retained validation-based best-checkpoint selection. Completing more epochs did not force the exported model to be the final step.

Step 4,153 is an additional recorded validation point near an earlier stopping/finalization boundary, not a standard multiple of 500. It is included in the saved history and has the lowest recorded validation loss.

4.10 What a resumable checkpoint must contain

Saving adapter weights is enough to load a predictor, but not enough to continue the same training trajectory.

Our training checkpoints include adapter weights and tokenizer/configuration, the frozen marker embedding, optimizer state, scheduler state, Python/NumPy/PyTorch/CUDA random states, the epoch and sampler position, step count, best-loss and patience state, and the current microbatch setting. A durable separate budget ledger retains cumulative runtime and estimated cost.

Writing uses a temporary directory, per-file hashes, a completion manifest, and pointer files for best/latest/previous. The pointers only reference completed checkpoints. Retention removes other checkpoint directories, so a metrics point at every validation step does not imply all corresponding weights still exist.

Resume restores the stochastic and optimizer state as well as the adapter. Code/config/data identity checks prevent silently continuing a different experiment. This is reproducible state management, not a guarantee of universal bitwise determinism across GPU models or library changes.

If a worker dies after step 5,900 but its latest committed checkpoint is step 5,750, recovery starts from 5,750. Work after that saved boundary may be repeated and still costs money. Preserving the cumulative ledger avoids treating that lost work as free. Duplicate log points can consequently exist even though the resumed sampler correctly starts from the checkpoint position.

Source: checkpoint code.

4.11 Modal, budget control, and durable execution

Local preparation and uploads happen before the A100 is requested. A CPU-only preflight verifies data, model-file checksums, and the accepted code snapshot. Inputs and outputs live in separate Modal volumes.

The experiment has no automatic GPU substitution or application-level retry that starts another training run. Later recovery uses an explicitly resumed checkpoint. Durable asynchronous submission records a function-call ID so the training call does not depend on a client holding a synchronous wait open.

The original target was US$10. You approved raising the same experiment’s cumulative cap to US$15 to finish three epochs. That extension preserves prior spending; it does not reset the counter. The continuation’s estimated stop threshold is US$14, with a finalization reserve and a five-hour cumulative worker allowance.

The ledger combines configured GPU, CPU, and RAM rates, preparation cost, a 10% safety multiplier, and a conservative two-minute allowance after an unclean interruption. It is a spending guard based on runtime estimates, not a complete cloud invoice that necessarily captures every storage/build/network charge.

The completed estimate is US$11.7023. It includes the recorded continuation and finalization under the project’s estimator. We should describe it as estimated experiment spend, not as a verified Modal bill or a prediction of future 2B cost.

Sources: Modal launcher (local artifact), budget ledger implementation, continuation limits, completed summary.

4.12 Tracking and acceptance checks

W&B was originally planned. The implemented default is TensorBoard, with persistent JSON metrics and saved prediction files. It requires no tracking API key. Modal itself still uses the existing authenticated account; “no tracking key” does not mean “no authentication anywhere.” W&B remains optional, not the source of a completed online dashboard here.

Before paid execution, checks covered all decision types, 255 code IDs, option/target alignment, control tokens, input/label separation, attention/padding behaviour, actual template lengths, and every training permutation for three epochs. Tests also exercised checkpoint corruption, resumed optimizer/RNG behaviour, accumulation weighting, out-of-memory replay, budget stopping, and the authorized continuation.

The real local smoke report records 20 tiny-batch overfit steps, finite computation, a changed adapter parameter, final loss approximately 1.15e-6 on that deliberately tiny set, zero measured reload probability difference, and approximately 1.97 GiB peak allocated memory for that smoke workload.

Tiny-batch overfitting is a wiring test: the model can learn those examples. It does not establish generalization or imply the full training workload fits in the same memory. The saved training-test report confirms all 54,046 prepared decisions were checked across their splits.

Part 5 — Evaluation: what each number means

5.1 Start with the population and the checkpoint

Every metric needs context: which checkpoint, which split, how many examples, which types, what context limit, which calibration, and what metric definition?

Validation at step 6,297 is not the same measurement as test at step 4,153. A held-out mixture drawn from the curation sources is not the same distribution as an external benchmark. A text-only subset is not the full multimodal benchmark.

All reported in-mixture and external test results in this guide refer to the selected checkpoint, step 4,153, unless explicitly labelled otherwise.

5.2 Argmax agreement

For each decision, compare the index of the model’s highest probability with the index of the target’s highest probability. Agreement is the fraction that match.

For hard labels, this is ordinary top-1 classification accuracy. For soft targets, it is agreement with the reference’s preferred answer. It discards how much probability either distribution assigns elsewhere.

Calling the mixed-target result “argmax agreement” is more precise than implying that every answer has independently established objective ground truth. It also inherits tie-breaking behaviour if a distribution has tied maxima.

5.3 Cross-entropy and KL divergence

Cross-entropy includes the uncertainty already in the target. KL divergence subtracts that target entropy:

H(t) = -sum_k t[k] × ln(t[k])
KL(t || p) = sum_k t[k] × ln(t[k] / p[k])
CE(t,p) = H(t) + KL(t || p)

On the same fixed evaluation set, average target entropy is constant. Improving cross-entropy improves KL by the same amount. Reporting both helps interpret the remaining loss, but they are not independent discoveries.

On different datasets, target entropy changes. Comparing a soft-label cross-entropy with a hard-label loss without that context can be misleading. For a uniform K-option prediction, the cross-entropy is ln(K); tasks with different option counts also have different reference levels.

5.4 Our Brier convention

The in-mixture code reports:

half-Brier = 0.5 × sum_k (p[k] - t[k])²

Then it averages over decisions. Some sources omit the factor 0.5, and binary implementations sometimes use only the positive-class error. Those values are numerically different conventions.

This squared-distance metric penalizes mismatch across the distribution without the same logarithmic penalty for confidently assigning near-zero probability to a correct answer.

Our external benchmark’s reported Brier is kept separate from the mixture half-Brier. Never put them on one scale as though only the dataset name changed.

5.5 Calibration: are confidence numbers trustworthy in aggregate?

Suppose a model makes 100 predictions near 90% confidence. If only 65 are correct, it is overconfident on that group. If about 90 are correct, those confidence estimates are well calibrated on that group and distribution.

Calibration and accuracy are different. A model can be accurate and overconfident, or modestly accurate but honest about uncertainty. Calibration is a population property, not a certificate that any particular 90% prediction is right.

Our hard ECE uses 15 confidence bins and only one-hot-target decisions. Within each populated bin, it measures the absolute difference between mean confidence and empirical accuracy, then weights that difference by bin count.

ECE = sum_bins (n_bin / N) × |mean_confidence_bin - accuracy_bin|

In the test set, the hard-label subset contains 2,134 of 3,693 decisions. Its ECE does not summarize every soft-target decision. The code separately records a soft reliability view using the target mass assigned to the predicted answer; that is agreement with a soft reference rather than observed event frequency.

ECE depends on binning, finite sample size, and the evaluated distribution. It can hide subgroup errors. Our external benchmark uses 10 bins, so even its ECE convention differs from the mixture report.

5.6 Temperature scaling

After selecting the adapter, we fit one positive temperature for each type:

p_calibrated = softmax(logits / T)

T > 1 softens a distribution. T < 1 sharpens it. T = 1 leaves it unchanged. The fit uses only calibration examples and minimizes cross-entropy. Our numerical implementation optimizes log-temperature with a bounded positive range and falls back to one if fitting is invalid or fails to improve that calibration objective.

Multiplying every logit in a question by the same positive inverse temperature preserves its ranking. Therefore, this calibration cannot change the top-1 choice or argmax agreement. It can change confidence, threshold-triggered actions, expected scores, loss, and calibration metrics.

The fitted temperatures are 1.157726 for choice, 1.107598 for yes/no, and 1.020533 for score. They soften this selected model’s probabilities slightly.

Temperature scaling cannot repair an incorrect answer ranking, missing evidence, a bad label, or a domain shift. A type-level temperature is also too coarse to guarantee good calibration in every category.

Primary background: On Calibration of Modern Neural Networks. Implementation: fit_temperatures().

5.7 Expected-score error

For score decisions, we compare the expected level index under the model with the expected index under the target:

error = |sum_k k × p[k] - sum_k k × t[k]|

We average the error over score decisions. This is measured in scale-index units. It does not mean the same real-world error across differently described scales, and two quite different distributions can share the same expectation.

This metric complements exact-level agreement. Neither replaces inspection of whether the scale rubric itself is meaningful.

5.8 Option-order sensitivity

If an option’s position changes while its meaning stays fixed, ideally the probability associated with its original key should stay stable.

We select 256 deterministic, category-stratified choice examples from the test set. Each is evaluated as a single-question request, then evaluated again after rotating options by one position. Probabilities are aligned back by original option key before comparison.

We measure whether the winning key flips, and total variation distance:

TV(p,q) = 0.5 × sum_k |p[k] - q[k]|

Observed result: 6.25% answer flips, or 16 of 256, and 0.06115 mean TV. The TV calculation uses raw probabilities in this function. A positive scalar temperature would preserve the winning answer but can change the numerical TV.

This is one cyclic rotation per selected question, not an exhaustive test of all permutations, not a paraphrase test, and not a test of changing the surrounding question set. Training-time shuffling reduced an incentive for position shortcuts but did not make the architecture permutation-invariant.

5.9 Test discipline after this experiment

The model-selection rule uses validation, and temperature fitting uses calibration. The test set is not passed into the weight-training objective.

Once we inspect the test and external-benchmark failures to design the next model or curate its data, those evaluations begin to influence research decisions. A fresh final holdout becomes valuable. Repeatedly tuning to the same public benchmark can overfit research choices even without gradient training on its rows.

Part 6 — What happened in this experiment?

6.1 The run completed; the exported model came from earlier

Run themis-amos-v0, ID 4d614aa34ab5, finished three epochs and 6,297 optimizer steps. Stopping reason: epochs_complete. Estimated experiment spend: US$11.70 under the saved estimator.

The selected/exported checkpoint is step 4,153, approximately 1.979 epochs. It has the lowest recorded validation soft cross-entropy, 0.656687. The final-step model was evaluated on validation but was not separately evaluated on the test split in the saved report.

“Training completed” describes execution. “Best checkpoint” describes selection by a metric. They answer different questions.

Source: completed run summary.

6.2 The validation progression

The validation set contains 3,170 decisions at each recorded point; its one-hot ECE subset contains 1,794 decisions.

The step-zero baseline is our modified decision architecture before LoRA training, not Qwen’s ordinary generated-answer performance. Its attention and readout already differ from a stock chat run.

6.3 How can agreement improve while loss worsens?

From selected step 4,153 to final step 6,297, agreement rises by about 1.07 percentage points, but soft CE worsens from 0.6567 to 0.7163, and hard ECE rises from 3.79% to 9.01%.

An illustrative two-example hard-label calculation explains the mechanism:

• Model A gives the correct answer probability 0.60 on one example and 0.40 on another. It gets one of two right. Mean loss is about 0.714.
• Model B gives the correct answer probability 0.99 on the first and 0.01 on the second. It also gets one of two right, but mean loss is about 2.308.

The same accuracy can coexist with much worse loss because the remaining mistake becomes extremely confident. Across a larger dataset, a model can also correct a few borderline decisions while becoming overconfident on others. Soft targets add another failure mode: becoming too certain about the preferred answer can be wrong even when the argmax agrees.

Our late-training pattern is consistent with overfitting and overconfidence. The aggregate curves alone do not identify the exact examples or prove the sole cause. We would inspect per-example losses and category/type changes to establish more detail.

The selected checkpoint is not the one with the lowest ECE either. It was selected by the predefined cross-entropy objective. Choosing whichever metric looks nicest afterwards would change the experiment’s selection rule.

6.4 Training loss is not validation loss on training data

The logged training scalar is the loss of an individual accumulation window, sampled generally every ten optimizer steps. It is not a full evaluation of the entire training set every ten steps, and not an average of all ten intervening windows.

The report smooths those logged samples with an exponential moving average, alpha 0.12. It removes 78 replayed duplicate log rows and keeps the latest per step. Smoothing helps reveal a trend; it can hide spikes and should always be labelled.

Training also uses dropout and shuffled choice options; validation uses evaluation mode and fixed rendering. Batch composition and task difficulty vary. Therefore, the difference between a displayed training window loss and whole-validation loss is not a perfectly controlled generalization-gap estimate.

Source: report generator (local artifact), analysis snapshot (local artifact).

6.5 Selected-checkpoint test results

The test evaluation contains 3,693 decisions. Raw → calibrated values are:

• Soft cross-entropy: 0.708560 → 0.704117.
• KL divergence: 0.393148 → 0.388705.
• Half-Brier: 0.097691 → 0.096900.
• Argmax agreement: 77.34% → 77.34%, unchanged as expected.
• Hard-label 15-bin ECE: 4.21% → 2.28%, on 2,134 hard-label decisions.
• Expected-score MAE: 0.45708 → 0.45846, on 585 score decisions; this slightly worsened.

Calibration improves several distribution metrics here, but not every metric. Its objective was calibration-split cross-entropy, not minimizing test expected-score error. We did not tune the temperatures to the test outcomes.

Calibrated type results are choice 78.62% agreement on 2,526 decisions, yes/no 83.16% on 582, and score 65.98% on 585. Their option counts and task difficulty differ, so these are coverage diagnostics rather than proof that one output type is intrinsically easier.

Source: machine-readable evaluation.

6.6 Category results need their denominators

Selected-checkpoint in-mixture test agreement is:

There is no product-management or design test score in this mixture evaluation because those categories do not have examples in this test split. They must not be shown as zero accuracy or silently treated as tested.

An overall average is weighted by the examples present. It does not demonstrate equal competence across categories. The engineering result is especially dependent on the source/task construction discussed earlier.

6.7 External Decision Bench results

A saved local text-only evaluation of Atlan Decision Bench, bench-v4, reports 688 correct out of 949, or 72.50% accuracy, for selected checkpoint 4,153 with calibration applied. Image-bearing examples are excluded.

This extended evaluation raises the runtime context limit to 8,192 tokens. The longest evaluated input is 4,696 tokens, and 37 inputs exceed the 2,048-token training contract. That makes it useful evidence of behaviour under a longer runtime cap, not proof that the trained model is equally reliable at every length up to 8K.

The strict 2,048-token evaluation scores 912 examples and counts the 37 oversized errors as incorrect. It reports 670/949 = 70.60%. Both numbers are legitimate descriptions of their respective protocols; reporting only the larger one would omit a material condition.

The extended result’s 95% Wilson interval is approximately 69.57%–75.24% under a binomial-sampling model. This captures finite-example uncertainty under that model; it does not account for all task clustering, dataset-design effects, contamination, or real-world distribution shift.

The external category results are trust/safety 88.33% on 60, finance 83.52% on 91, support 80.60% on 67, sales 77.78% on 90, legal 75.40% on 126, engineering 74.67% on 150, data 71.28% on 94, agents/evals 69.48% on 154, product management 53.33% on 60, and documents/meetings 38.60% on 57.

In-mixture documents agreement of 75.40% and external documents accuracy of 38.60% are not contradictory. They measure different examples and task distributions. The gap suggests a transfer weakness worth investigating; it does not by itself identify the cause.

The saved external median inference latency is approximately 55.45 ms on the RTX 4060 Laptop GPU for that workload. It is not an end-to-end hosted API SLA or a controlled comparison against another model.

The external predictions and counts were checked from saved artifacts when preparing the earlier report. This guide does not claim a new model rerun or a fresh semantic contamination audit. The original curation records literal overlap exclusions, but paraphrase/pretraining exposure remains unassessed.

Source: verified external analysis, saved predictions (local artifact).

6.8 What was exported?

The export includes the selected LoRA adapter, tokenizer, answer-code mapping, architecture/training configuration, explicitly saved marker embedding, calibration settings, and evaluation summary. A predictor combines these with the pinned base checkpoint.

It is an adapter export, not a fully standalone merged copy of Qwen. Local inference still needs access to the correct base weights through the configured model path or cache.

The downloaded export passed a local reload check. On the saved refund example, it returned true with probability 0.949515 and false with probability 0.050485, using 92 input tokens. That is an actual recorded example, unlike the illustrative billing distribution earlier. A successful example verifies the loading path; it is not an evaluation dataset by itself.

Source: export directory (local artifact), reload verification, predictor.

6.9 What can we honestly conclude?

We have a working, locally reloadable direct-decision adapter whose held-out metrics improve substantially over its modified step-zero baseline. It produces bounded distributions for three decision types, and its probabilities benefit from modest temperature scaling on the evaluated mixture.

We also have evidence of late-training distribution degradation, residual option-order sensitivity, uneven category transfer, weak external documents/product results, and a mismatch between the trained context contract and part of the extended benchmark.

We do not have an architecture sweep, a controlled advantage over Unsloth or standard SFT, a causal-mask ablation, evidence that the 0.8B capacity is exhausted, or a trained multimodal Themis. Unsloth is primarily training/inference tooling; AMOS-style refers to a modelling/readout approach. They are not mutually exclusive architectural categories.

Part 7 — Keeping 0.8B text-only and adding a vision-capable 2B

7.1 Bigger capacity and a new modality are separate changes

Moving from 0.8B to 2B gives a different, larger text backbone. Retaining the vision path enables image input. Neither guarantees better downstream decisions, and combining both changes makes it harder to attribute gains unless evaluation includes text-only comparisons.

The verified 2B configuration still has 24 text layers in the 18-linear/6-full pattern, but hidden width increases from 1,024 to 2,048. Its active text network contains approximately 1.882 billion parameters, and its vision stack including the merger contains approximately 331 million. These are component counts, not trainable LoRA counts or predictions of cloud memory usage. Exact configurations and counting scopes are linked in the primary-source notes.

The sensible proposed split is a small text-only model for low-resource decisions and a larger multimodal model for decisions that also use images. We should keep the existing selected 0.8B adapter as a baseline.

7.2 How a pretrained vision-language model sees images

A vision encoder converts image patches into feature vectors. A connector or merger puts those vectors into a space the text network can consume. The processor prepares pixels, image-grid information, and the correctly placed special tokens. The multimodal forward path combines visual and text representations.

Pretraining teaches a relationship between image features and language. We would retain that existing machinery rather than train a visual system from scratch on our relatively small decision dataset.

For Qwen, the correct integration must follow its actual processor and model implementation. We should not copy AMOS Mini’s fixed 224-pixel/49-token SigLIP path and assume it matches Qwen’s image handling.

Primary sources: Qwen3.5-2B model card, Qwen3.5 Transformers documentation.

7.3 What must change in our code

The current loader releases vision. A multimodal version must retain the needed encoder/connector and use the complete image-aware model path. The codec must accept images, use the correct processor, preserve visual token placement and position information, and gather the right decision states after multimodal processing.

Preparation must measure image-plus-text rendered lengths rather than text tokens alone. Batching must handle image tensors and metadata as well as text IDs. Checkpoint/export logic must preserve any newly trainable connector or vision parameters and the processor configuration. Prediction must accept the same image contract used during training.

Most critically, our bidirectional-mask alteration is not automatically a validated multimodal recipe. We must verify attention, image positions, padding, gradients, and save/reload behaviour under the modified forward path. Keeping pretrained components is a promising starting point, not proof that they will work unchanged with our decision readout.

This is why restoring vision is implementation work, not a configuration flag that turns our existing adapter into an image model.

7.4 A reasonable training starting point

Proposed initial strategy: freeze the pretrained vision encoder, retain the pretrained connector, and adapt the text network with LoRA using a mixture of text and visual decision examples. Whether to train the connector as well should be determined by its compatibility and observed learning, rather than silently assumed.

Mixed text/image batches help retain the text interface while teaching the direct readout to use visual evidence. We must measure text regression rather than assume mixing prevents it. Image processing can substantially increase effective sequence lengths and memory use, so the 0.8B text throughput cannot predict 2B multimodal cost.

Before committing cloud credits, local correctness checks and a short measured throughput/memory preflight should determine feasible resolution, context, microbatch, accumulation, and budget. No 2B training has been launched as part of this guide.

7.5 What visual decisions would we teach first?

Examples include selecting a visible object category, deciding whether a document image contains a required field, choosing which supplied description matches a diagram, or rating a clearly defined visual property. Each requires evidence in the pixels and an appropriate target.

A basic visual-choice dataset should not allow the question text alone to reveal the answer. Otherwise, we may obtain attractive scores while the model ignores the image.

We should compare normal images against blanked, shuffled, or mismatched images and inspect whether performance changes in the expected direction. These controls help establish image use. Separate held-out visual tasks help establish transfer.

OCR and document understanding need resolution-aware evaluation. A broad object-recognition result does not establish reliable reading of tiny invoice text or dense tables.

7.6 Vision does not automatically provide grounding

Answering “is a dog present?” is a categorical decision. Returning where the dog is requires a spatial output contract and grounding supervision.

AMOS Mini’s region head scores image cells. Our current answer-code readout does not provide a bounding-box mechanism. We could later design a region/grid readout or another spatial interface, but that would be a separate implementation and evaluation task.

For a first 2B model, text/image yes-no, choice, and score questions keep the output contract comparable with Themis v0. Region pointing should be described as future work unless actually implemented and tested.

7.7 More 0.8B epochs or a 2B model?

The existing curves do not support paying for more unchanged epochs simply because the training loss still decreases. The selected checkpoint’s validation distribution metrics are better than the final one’s.

But “more unchanged epochs are unlikely to be the best next spend” is different from “0.8B cannot improve.” Better examples, harder distractors, less label noise, different regularization, different weighting, or a revised objective could improve it.

A fresh 2B experiment is a reasonable next hypothesis, especially because you want vision. Its value must be measured against the current baseline with the same decision definitions, explicit text and visual evaluations, calibration checks, and measured latency/memory. Pretraining benchmark differences alone cannot establish the result on our mixture.

Part 8 — Read the code without getting lost

Follow one example through these modules in order:

1. themis_data/schema.py: makes upstream labels and options consistent without inventing new probabilities.
2. themis_data/curate.py, filters.py, commitpack.py: source eligibility, exclusions, split groups, coverage sampling, derived engineering tasks.
3. themis_data/pack.py: combines same-state decisions into bounded requests.
4. themis_train/data.py: downloads the pinned release, verifies checksums, renders with the real template, preserves decision coverage.
5. themis_train/codec.py: safe rendering, answer-code mapping, positions, target-preserving permutations.
6. themis_train/model.py: text-only loader, masks, LoRA, selected-code scores, distribution loss.
7. themis_train/engine.py: sampler, accumulation, optimizer, schedule, validation, selection, finalization.
8. checkpoint.py, budget.py, tracking.py: durable recovery, cumulative accounting, observable metrics.
9. evaluation.py: reference metrics, grouped breakdowns, temperatures, option rotation.
10. predict.py: reloads the export and maps probabilities back to original keys.
11. scripts/train_modal.py (local artifact): CPU preflight, volumes, explicitly requested A100, async submission.
12. scripts/report_training.py (local artifact): builds the charts from saved metrics; it does not train the model.

The definitive run artifacts are summary.json, provenance.json, evaluation.json, and calibration.json. Earlier reports may describe the state at the time they were generated—for example, the curation report still says training had not started. Use completed-run artifacts for current status.

The recorded training environment uses PyTorch 2.14.1, Transformers 5.19.0, PEFT 0.21.2, tokenizers 0.23.3, and huggingface-hub 2.2.0. W&B 0.30.0 is installed but online W&B tracking is disabled. The dependency lock, rather than a list of top-level versions alone, defines the reproducible environment.

Part 9 — Glossary in ordinary language

• Adapter: the small added set of trained parameters loaded alongside a base model.
• Answer code: a temporary token such as A or B associated with an option in the current request.
• Argmax: the index of the largest value.
• Attention mask: rules controlling which token positions can read which others.
• Backbone: the main pretrained network producing contextual representations.
• Backpropagation: computing loss derivatives through the network.
• Batch: a group of examples processed together; here, requests can contain multiple decisions.
• BF16: a compact floating-point format used for much of the GPU computation.
• Calibration: agreement between confidence estimates and reference outcome frequencies on a population.
• Checkpoint: saved state at a point in training; a resumable checkpoint includes more than weights.
• Codec: our shared input-rendering, tokenization, and answer-mapping contract.
• Context window: the maximum input length the configured interface accepts.
• Cross-entropy: a loss evaluating predicted probabilities against a target distribution.
• Decision: one question and its permitted answer space, interpreted against a state.
• Distribution shift: a change between training conditions and evaluation/deployment conditions.
• Embedding: a numerical vector looked up for a token or used as an output scoring vector.
• Epoch: one traversal of the training requests.
• Evaluation mode: network behaviour with training features such as dropout disabled.
• Fine-tuning: adapting an already trained model to a further objective or dataset.
• Forward pass: calculating network representations and outputs for supplied inputs.
• Gradient: a local derivative showing how a parameter change affects the loss.
• Gradient accumulation: combining backward passes before one optimizer update.
• Gradient checkpointing: saving activation memory by recomputing parts of the forward pass during backward.
• Hidden state: a contextual vector at a token position inside the network.
• Holdout: examples withheld from a particular fitting or selection operation.
• Inference: calculating predictions with trained weights.
• KL divergence: mismatch between distributions after accounting for target entropy.
• Label leakage: answer information entering inputs or otherwise compromising evaluation.
• Learning rate: a scale controlling optimizer updates.
• Logit: an unnormalized score before softmax.
• LoRA: low-rank trainable corrections to frozen linear transformations.
• Microbatch: requests handled together in one forward/backward chunk.
• One-hot: a vector with one value equal to one and the others zero.
• Optimizer state: accumulated information, such as AdamW’s gradient statistics, needed to continue updates.
• Overfitting: fitting training-specific patterns in a way that degrades useful generalization.
• Padding: extra token positions used to make sequence lengths compatible in a batch.
• Permutation: a rearrangement; our training permutes choice options together with their targets.
• Readout/head: the operation converting network representations into the desired output scores.
• Request: a state and one or more typed questions processed together.
• Seed: an initial value controlling reproducible pseudorandom choices.
• Soft target: a reference distribution that can assign mass to multiple options.
• Softmax: the transformation from logits to a normalized categorical distribution.
• State: the evidence or case the questions refer to.
• Step: here, an optimizer update after an accumulation window.
• Temperature: a positive scalar dividing logits to soften or sharpen probabilities.
• Tokenizer: the mapping between text and token IDs.
• Validation: held-out evaluation used during model development and checkpoint selection.
• Vision encoder: a network converting image pixels/patches into numerical features.

Part 10 — Check your understanding

Try answering before reading the explanations.

1. Why is Themis not a fixed three-department classifier? Because the question and option descriptions are supplied with each request. The same answer-code embedding can represent different named options in different requests.

2. Does the model fill <answer> with generated text? No. We gather the hidden state immediately before each fixed placeholder and score allowed output-embedding rows directly.

3. How can all output embeddings stay frozen while predictions improve? LoRA changes the text network’s transformations, changing the hidden states and therefore their dot products with those frozen scoring vectors.

4. Why is a perfect match to [0.7, 0.3] not zero cross-entropy? Cross-entropy includes the target’s entropy. KL divergence is zero at that match.

5. If options move, what else must move? The target probabilities and code-to-original-key correspondence. Otherwise, labels become misaligned.

6. Why normalize across decisions rather than requests? Different requests have different question counts. Per-request averaging would give individual decisions unequal weights.

7. Why retain step 4,153 after completing step 6,297? The selection rule is lowest validation soft cross-entropy, which occurs at 4,153. The final checkpoint has higher agreement but worse distribution loss and calibration.

8. Why can temperature scaling improve ECE without increasing accuracy? A shared positive temperature preserves logit ranking but changes confidence values.

9. Why is the 99.6% engineering result insufficient as a broad capability claim? It measures the mixture’s engineering tasks and distractor construction. External engineering evaluates different questions and is substantially lower.

10. Why is 72.50% external accuracy incomplete without its protocol? It covers 949 text-only examples at an extended runtime context cap. The strict 2,048-token protocol reports 70.60%, with oversized inputs counted as failures.

11. Does a base checkpoint with vision make our export vision-capable? No. Our loader removed that path and the adapter was trained on text decisions. The multimodal processor, forward path, data, and evaluation need implementation.

12. Does no generated chain of thought imply no reasoning? No. It specifies the observable output interface, not a proven account of internal cognition.

13. What would prove that a new 2B model actually uses its images? Visual-task performance plus controls such as mismatched or blank images, tested under a defined evaluation protocol. Its model name alone would not prove it.

14. What is still a hypothesis? That 2B improves our decisions, that a frozen vision path works well under our altered mask/readout, and that any proposed data/objective change fixes the observed transfer weaknesses. These require experiments.

A writing compass for later

When you eventually write your article, separate three kinds of sentence:

• Implementation: “We trained only LoRA parameters and scored supplied answer codes directly.” This is supported by code.
• Observation: “The best validation loss occurred near the end of epoch two, while final-step accuracy was slightly higher.” This is supported by saved measurements.
• Interpretation or hypothesis: “The pattern suggests late overconfidence; we want to test a larger multimodal backbone.” This is a reasoned next step, not an established result.

Keeping those distinctions will make the article credible and much easier to condense into shorter posts without losing the conditions attached to its results.