Primary-source notes for learning Themis

Checked 2026-10-16. These notes support an educational guide; they are not new training results. No models were trained or published. Only documentation, configuration and weight metadata were inspected externally.

Qwen: what the small model names include

The official cards describe language-model sizes, alongside a vision encoder. They are not exact counts of all components and should not be interpreted as trained-from-scratch Themis parameter counts. Both releases are post-trained, pretrained multimodal models. 0.8B card, 2B card.

The Themis-pinned 0.8B revision is 2fc06364715b967f1860aea9cf38778875588b17; it remains the current upstream revision at this check. Its text stack has 24 layers: 18 Gated DeltaNet linear-attention layers and 6 full-attention layers, one full layer after every three linear layers. Hidden width is 1,024, FFN width 3,584, vocabulary 248,320, and input/output token embeddings are tied. The native context configuration is 262,144; Themis's actual training limit was 2,048, a separate project decision. Its pretrained vision stack has 12 layers, hidden width 768, 16-pixel patches, spatial merge factor 2, and projects to text width 1,024. Pinned configuration.

The current 2B revision is 15852e8c16360a2fea060d615a32b45270f8a8fc. It also has 24 text layers with the same 3-linear/1-full pattern, but text width 2,048 and FFN width 6,144. Vision has 24 layers, width 1,024, and output width 2,048. The patch and spatial merge sizes remain 16 and 2. Pinned 2B configuration.

Parameter scope cross-check, derived by constructing the official configurations on PyTorch's meta device using the installed Transformers Qwen3_5ForConditionalGeneration class, without loading weights:

The stored checkpoints additionally contain mtp.* tensors; the standard instantiated class above does not expose those tensors. Counts were cross-checked against only the JSON headers of the pinned safetensors shards (59,840 and 76,648 header bytes respectively); no tensor payloads were read. The auxiliary tensors exactly explain the difference. Thus storage totals and active decoder counts have different scopes. The tied output embedding must not be double-counted. 0.8B index, 2B index, 0.8B metadata API, 2B metadata API.

These are architecture counts, not counts of parameters trained by LoRA. Themis uses only the text decoder and added adapter/marker parameters. The guide should use the run's actual trainable-parameter count if explaining how many weights changed.

AMOS is a family, not one interchangeable architecture

AMOS repositories evolve. The current Large main revision is 362f4dc8da538beda4157a882bfabc4bc6a4538a. Its root configuration now names Qwen/Qwen3.8-27B. Its nested model configuration still uses the qwen3_5 implementation and declares amos_noncausal: true and amos_codes: 255. Do not retroactively identify this current release as the exact AMOS version that inspired Themis. Our reproducibility comes from our own pinned Qwen/data revisions and implementation. Current Large root configuration, nested configuration.

Large's active serving path is serve/amos/decoder.py, not the shared encoder implementation in serve/amos/model.py. Decoder lines 141–150 locate the hidden state immediately before each <answer> token. Lines 397–408 multiply those states by selected output embedding rows and apply log-softmax over each question's valid answer codes. The current Large has untied embeddings: saying it scores input embeddings would be incorrect for that checkpoint. Some docstrings retain the historical phrases “52 letters” and “tied”; executable configuration/code permit 255 codes and untied output embeddings. Pinned decoder source.

The noncausal decoder path masks full-attention keys by actual sequence length; forward linear recurrences remain forward. Its image path retains the multimodal model, processor, expanded image tokens and spatial position indices. Region/span capabilities use additional trained heads. This is architectural inspiration, not evidence Themis reproduces Large's scores or complete capabilities. Decoder source, hidden, noncausal, Heads.

AMOS Mini revision 340d8af4bfbf420b2cec31b79d27a528958eaf3a instead names jhu-clsp/ettin-encoder-400m and google/siglip2-base-patch16-224. It is an encoder with typed heads. Choice/score use bilinear scores between a decision representation and mean-pooled option representations. It does not use the decoder's letter-token readout. Mini configuration, Mini model implementation.

Mini resizes images to 224×224, obtains 196 patch vectors on a 14×14 grid, merges 2×2 groups and projects them with a two-layer MLP to 49 image tokens. These enter the text encoder alongside text. Its region head scores those image cells. The vision tower is frozen and downloaded separately from the saved encoder/head checkpoint. Mini vision implementation, Mini model implementation.

LoRA and calibration foundations

LoRA freezes a base matrix W and learns a low-rank update ΔW = (α/r)BA, typically with A shaped [r, input] and B shaped [output, r]. The number of update parameters is r(input + output) instead of input × output. This reduces trainable weights and optimizer states; it does not remove the frozen backbone or its forward/backward activation costs. An unmerged adapter can require extra matrix operations; the original paper's no-added-latency property refers to merging the update into the base weights. LoRA does not intrinsically imply quantization or QLoRA. Original LoRA paper, authors' implementation.

Temperature scaling replaces logits z with z/T before softmax, fitting a positive scalar on held-out calibration data. T>1 softens confidence; T<1 sharpens it. A shared positive temperature preserves the argmax ordering, so accuracy need not change even when probability-based metrics improve. It cannot repair wrong rankings, missing evidence or domain shift by itself. Themis fits a separate positive temperature for each decision type, using its calibration split rather than test labels. Original calibration paper.

What remains unverified for Themis 2B vision

The native Qwen2B checkpoint is multimodal; that is verified. Merely restoring its encoder does not establish that Themis's modified bidirectional full-attention masks preserve its pretrained visual quality. The proposed image decision model needs image-token expansion, pixel preprocessing, image grid metadata, spatial/multimodal positions and identical train/inference attention behavior. It needs real visual smoke tests and held-out visual evaluations. AMOS's code illustrates a working design direction; our 2B compatibility and performance remain unmeasured. A frozen vision encoder is a proposed first experiment, not a completed project result.

Official Qwen general language/vision benchmarks concern their original generation models and protocols. They cannot substitute for Themis's own decision evaluation, and the modified step-0 Themis baseline is not the stock Qwen chat benchmark.