FINAL_EVIDENCE_VERIFIED · generated 2026-10-12T04:41:06.704740+00:00

Strata on a 16GB RTX 5070 Ti

Sonny’s Labs · v0.1.43 · 64GB installed RAM · IQ2_XS versus IQ3_XXS

This local experiment tests whether a large MoE deployment can serve useful coding and agent workloads with limited VRAM and RAM, while moving compilation and downloads out of runtime. The original practical/coding index and long-context reliability are reported separately. Any incomplete results remain explicitly pending in draft builds.

Official upstream measurements

Three measured repetitions; upstream warmup discarded, thinking off. Engine prefill/decode timings and streaming client TTFT retain the original methodology. Full measured ranges. Original upstream files and submission instructions: IQ2_XS · IQ3_XXS (links appear when packages exist).

Upstream actual 32K input is reported in the raw files. The agent session contains an initial turn plus five follow-ups. Prefix reuse is distinct from hot-expert cache occupancy. Two-client statistics include FIFO waiting on one serial slot; the standard and optional results remain separate.

Original Sonny’s Labs v2.1 scoring

Unchanged 24 practical tasks, three repetitions, 2,048 output cap; all 164 original HumanEval+ problems with EvalPlus 0.3.1 in a network-isolated scorer. The retained weights are business 20%, research 20%, knowledge 20%, agent tools 15%, sysadmin 5%, HumanEval+ 10%, executive content 10%. The scorer’s rebuilt transitive dependencies are inventoried, rather than claimed byte-identical to history.

Role maxima use the original practical/coding weights and exclude long-context quality. Separate dossier reliability and speed qualification are available in the structured evidence.

These original clients are nonstreaming: total request latency is measured, while client TTFT is unavailable for these rows. Engine counters provide prefill/decode timing. Draft acceptance combines the exposed speculation counters and must not be called an isolated MTP-only acceptance rate. Short-prompt rates include fixed overhead and should not be compared directly with long-input prefill.

Compatibility and retained failures

Dossier and long-context reliability

Dossier output cap is 3,072 and reasoning budget 512. Repeat prompts may reuse prefixes; first-ingest and warmed-repeat times are separate. A nominal upstream 32K needle produced 32,804–32,805 actual prompt tokens, exceeding a 32,768 context plus output allowance. Those HTTP rejections are retained. Successful allocated contexts are not declared to pass their full capacity merely because shorter inputs worked.

Memory, VRAM, power and SSD activity

One-second UTC-aligned host, LXC, process and Docker cgroup records are preserved separately. Host physical-NVMe reads include all activity; they are not labeled model-specific. Per-process I/O and container block I/O have narrower attribution. Cgroup memory includes charged cache and shared memory; host available RAM and swap activity are separate fields in the structured evidence. Linux Cached includes Shmem/tmpfs, including shared expert allocations, so it must not all be described as evictable filesystem cache. Raw Cached, Shmem, Cached-minus-Shmem, Mlocked and Unevictable, plus overlapping cgroup file/shmem counters, are preserved separately.

This entire-capture summary includes model loading, gaps and isolated scoring during the captured experiment period. It is not a per-model attribution. Per-phase windows below exclude unrelated stages.

Swap activity uses kernel page counters (4 KiB pages on this machine), distinct from swap allocation. Host total-minus-available is an available-memory pressure measure; it is not process RSS. LXC procfs is virtualized and can expose cache counters with different accounting. Docker memory.events and stopped-container OOM flags are retained independently. Measurements during a phase use its first and last one-second samples; boundary traffic and subsecond peaks can be missed.

Requested cache/prefill are auto, LOW_RAM resident on, KV int8, requested MTP spec 4 / minimum probability 0.5, VRAM reserve request 1,024 MiB. Actual free VRAM is the engine-reported resolved value; the reserve request is not a promise of 1 GiB remaining after all allocations. The aggregate speculation field includes lookup behavior and must not be mislabeled as six MTP tokens. Vision, MCP execution and batching are off.

Precompiled Docker and security

The engine was compiled for SM120 using CUDA 13.0.0 from the exact Strata and llama.cpp revisions before runtime. Verified model files and packs live outside the image. The serving entry point checks the engine digest and directly starts the API; it refuses MCP/vision configuration. Inference uses network none, nonroot UID 65534, read-only root/models, zero capabilities, no-new-privileges, and no Docker socket or management mounts. GPU access still shares the host kernel and NVIDIA driver.

Both quantizations passed real offline health and inference tests independently. The initial Syft/Grype assessment reported 550 package matches, including 14 high matches across three vulnerability IDs. A separately tagged runtime updated OS/CUDA packages without changing the engine hash: 281 matches, 253 medium, 26 low, two negligible, zero high/critical. These scans cover the retained inference images; they do not scan the host, NVIDIA driver, scorer or workstation. These are scanner matches, not independently demonstrated exploits or a safety certification. The retained image archive permits exact deployment; byte-identical rebuilding is not guaranteed because apt repositories were live.

Official startup can invoke setup and downloads; this deployment separates preparation from serving. Bob’s retained source captions identify concerns about permissions, executable downloads and agent-driven installation. The measured mitigations address those operational paths, while remaining trust in native parsers, CUDA/driver binaries, dependencies and the host kernel is disclosed. No maliciousness or endorsement is claimed.

What the memory hierarchy actually does

The pinned implementation puts attention/shared weights, MTP and a hot-expert cache on the GPU. In the tested LOW_RAM resident variant, RAM holds the expert-cache complement, not a redundant copy of every GPU-resident expert; CPU expert computation and PCIe GPU work cooperate. IQ2 at 64K resolved 7,031 GPU cache slots (9.42 GiB) and a 27.44 GiB page-locked expert complement. Some prompt-path borrowed slots retain RAM copies. A 32,768-token resident KV window uses VRAM while other KV state lives in pinned RAM.

The separate 28.8 GB PLE n-gram shard remains an SSD workload: the engine’s default direct I/O reads selected rows past the OS page cache. The observed log confirms a direct-read probe and PLE enabled. Zero expert-file blob reads during resident decoding therefore does not mean zero SSD reads. Initial loading and later PLE traffic must be distinguished. Pinned upstream architecture.

Developer claims and measured scope

The pinned README reports IQ2_XS at 79 decode tokens/s and 2,090 prefill tokens/s on a 32K prompt, and IQ3_XXS at 62 and 1,750, on an RTX 5070 12GB/Ryzen 5 7600 using engine 0.1.26 for those rows. Our IQ2 official standard chat median is 81.9 decode tokens/s, and the optional 32K-input prefill median is 2,183.4. These are broadly similar scales, an engineering observation rather than a controlled replication: GPU, CPU, instruction set, PCIe, engine version and output length differ. The same-model quantization comparison in this experiment is the controlled comparison. Model-card AIME/GPQA/LiveCodeBench claims are different evaluations and cannot be confirmed by these Sonny’s Labs tasks alone. Pinned developer speed table.

Provenance and reproducibility

{
  "name": "Strata",
  "version": "0.1.43",
  "commit": "7695859aa13afb248bdb4d36f9b2e95707db079d",
  "llama_cpp_commit": "3cf03257f219afbe7334045ff7c6a06ac68c627d",
  "image": "sha256:f759e6167315dac77292d394a470240493dad78de6d23dd270eaa5c7ffc3e143",
  "driver": "615.71.09",
  "cuda_build": "13.0.0",
  "architecture": "sm_120"
}

Quantization repository revision: ed59f92082b1e93c0e96d60a8b11aab089b52f09. Every GGUF and MTP safetensor was checked against its pinned hash. The PLE shard is shared only after complete hash agreement; native experts and pack index v3 are prepared separately for each quantization. The underlying architecture is described by its creators as a 125B MoE with 6B active parameters plus a 51B n-gram table and 4B MTP component; these are creator claims, not measured active memory requirements. Pinned model card.

Local reproduction tooling and exact commands are retained with the experiment. Apt resolution was from signed live repositories rather than a frozen snapshot; preserve the image archive and wheel/source receipts for strongest repeatability. Build-time networking was allowed. Runtime offline inference was tested separately.

Historical continuity and limitations

report4.firetussin.com is the user-confirmed latest historical report. The verified local 17-configuration source is the comparison baseline; the live endpoint returned HTTP 403 during this experiment. Original datasets, v2.1 scoring functions and category weights are retained. The older generator hardcodes 17 Qwen3.8-27B queue entries and cannot represent this two-model Strata/upstream-suite experiment unchanged, so this wrapper reuses its scoring output, font assets and data/handoff contract.

A fast empty answer is still a failed answer. The original practical/coding index does not measure long-context quality. All 102 report4 fixed/adaptive responses were rechecked against raw artifacts. HauhauCS Q2_K_P retains its 97.61 index and 6.90 s measured adaptive median, but has 0/6 valid final answers and zero validated dossier quality. It is excluded from successful-speed and reliability-qualified recommendations. Thirteen of the 17 historical configurations completed all six responses with all 14 literal markers; this stricter gate is a separately labeled evaluation, not a change to the original index.

Full recalculated report4 reference · Recalculated raw-source audit and rankings. The original report4 publication and saved scores are preserved. Strict recall qualification requires completed nonempty stop responses and 14/14 original literal markers; this heuristic does not establish semantic accuracy. Historical adaptive latency and Strata fixed cached repeats differ in runtime/configuration; ranks are not pooled.

• IQ2_XS vs IQ3_XXS is a same-model, same-hardware comparison. Historical report4 uses Qwen3.8-27B and a different runtime; cross-model differences cannot be attributed solely to quantization.
• Both quantizations use mixed tensor formats and share the verified PLE shard; the labels do not mean every tensor uses two or three bits.
• Model preparation warmed filesystem cache. No storage-cold performance is claimed. Model load, initial uncached prompt processing and prefix-cached follow-ups are separate measurements.
• Dossier repeats retain the existing benchmark prompts and cache policy. First response and warmed repeats are reported separately; cached prefill rates must not be used as full-input throughput.
• The original v2.1 practical rubric and weighted score are retained. Hermes literal-marker heuristic is reported separately, with empty finals explicitly zeroed; it is not an independent broad-quality assessment.
• The practical rubric checks explicit content and structure, not arbitrary semantic correctness. These fixed synthetic workload scores are not a general intelligence or production-agent guarantee.
• The practical/coding index excludes long-context quality. Empty final answers count as dossier failures and receive zero validated dossier quality, even if reasoning exists or the API reports stop. Successful-speed rankings require valid completed answers. A separate full-marker reliability gate does not alter the original weighted index.
• HumanEval+ uses one generation per problem (pass@1); practical tasks have three repeats. Small score differences have not been established as statistically significant or universal quantization effects.
• Strict JSON-schema enforcement is unavailable in the tested image because an optional dependency is absent. Successful JSON examples and forced client tool calls do not prove general schema/tool reliability.
• Server-side MCP execution and vision are disabled. The optional two-client speed test uses one serial service slot, not GPU batching.
• Nonstreaming original Sonny clients measure end-to-end latency and retain engine timings; client TTFT is measured by the upstream streaming suite, not inferred for every Sonny request.
• One-second telemetry can miss subsecond peaks. Host physical-device disk traffic is not model-specific. Docker charged-memory, process RSS, page cache and host memory have distinct definitions.
• The engine retains upstream route-tail-skip default 7. Missing experts may be skipped, so speed/quality results include that approximation. Greedy output can vary with draft grouping, adaptive expert placement and prefix-cache state; fixed seed/temperature do not promise byte-identical responses.
• Apt repositories and some rebuilt scorer transitive dependencies are not snapshot-pinned. Source/image/model hashes and package inventories improve auditability but do not constitute a security certification.

Cost and attribution

Local owned hardware. No cloud rentals or purchases; electricity, amortization, bandwidth and historic quant creation not metered. Zero rental is not zero total project cost.

Credit Strata maintainers and contributors, Qwen model creators, ISTA-DASLab/GSQ-RCO quantization contributors, llama.cpp/ggml, NVIDIA and container-tooling projects, EvalPlus, and Unbiased Bob’s independent security work. No endorsement is implied. The original model and quantization cards carry separate license metadata; do not infer a blanket license from one card.

Structured evidence · Verification

• Pinned Strata source
• Community benchmark methodology
• Quantized model revision
• Original Qwen model
• Unbiased Bob’s independent security discussion
• Bob’s reference fork (not tested engine)

Retained reproducibility and security documents: Exact reproduction commands · Artifact manifest · Supply-chain review · Bob comparison and caption provenance.