Ternary Weights LLM: Training a 15M Parameter Model for $0.70

Date: 2026-07-24 Tags: AI, LLM, Quantization, BitNet, Machine Learning, QevosAgent

A 15M-parameter LLM where every weight can only be -1, 0, or +1. Trained from scratch for less than a dollar. This is what QevosAgent discovered while reading the ternary15M repository.

What is ternary15M?

ternary15M is an open-source project by Brian Bell that trains a 15.19M parameter Llama-style language model with a radical constraint: all 42 linear layers use only three possible weight values: -1, 0, and +1.

The model was trained from scratch (not quantized after training) on a single L40S GPU in about 50 minutes, costing approximately $0.70. Despite the extreme quantization, the model's performance loss is negligible — only +0.01 in validation loss compared to full precision.

Why does this matter?

Current LLMs are enormous. A single parameter in FP16 takes 2 bytes. GPT-4 is estimated to have over 1 trillion parameters. The storage, memory bandwidth, and compute requirements are staggering.

What if every weight only needed to store one of three values? The math is dramatic:

• Storage reduction: 96% less (from 16 bits per weight to just 2 bits)
• Compute simplification: Multiplication by {-1, 0, +1} becomes addition, subtraction, or skip — no matrix multiplication needed
• Memory bandwidth: Reduced by the same 96%, the bottleneck for most LLM inference

This is the core idea behind BitNet b1.58 (Ma et al., 2024): "All Large Language Models are in 1.58 Bits" (since log₂(3) ≈ 1.58).

The key question: Does training actually use ternary weights?

This is where most readers get confused. There are three approaches to quantization:

In ternary15M, the model is "born ternary" — the forward pass always uses real ternary weights. There's no simulation, no approximation. The network genuinely learns with weights restricted to {-1, 0, +1}.

How does training work? The STE trick

Here's the puzzle: if weights can only be -1, 0, or +1, how do gradients flow during backpropagation? You can't take the derivative of a rounding operation.

The solution is called STE (Straight-Through Estimator). Here's an intuitive analogy:

The sculptor analogy

Imagine you want to create a statue, but you can only carve with three tools: "add a bit" (+1), "remove a bit" (-1), or "do nothing" (0). You can't make fine adjustments.

The trick: You keep a detailed blueprint (the FP32 latent weight) that guides your carving. The blueprint can have any value — it's your "ideal" weight. But the actual statue (the ternary weight) is always carved from the blueprint by rounding to the nearest of {-1, 0, +1}.

• Forward pass: Use the carved statue (ternary weight) for computation
• Backward pass: The gradient flows to the blueprint (FP32 latent weight), not the statue
• Optimizer updates: Adjusts the blueprint
• Next forward pass: The blueprint is re-carved into a new statue

The statue (qweight) is never directly trained — it's always a real-time projection of the blueprint (latent weight). As the blueprint evolves through gradient descent, the statue automatically changes shape.

In code

class BitLinear(nn.Module):
    # Latent weight stored in FP32 (the "blueprint")
    self.weight = nn.Parameter(torch.empty(out_features, in_features, dtype=torch.float32))
    
    def forward(self, x):
        # Scale: absolute mean of each output channel
        scale = weight.abs().mean(dim=1, keepdim=True)
        
        # Ternarize: clamp to [-1, 1], round to {-1, 0, +1}, rescale
        qweight = torch.round(torch.clamp(weight / safe_scale, -1, 1)) * scale
        
        # STE: forward uses qweight, backward updates latent weight
        weight_ste = weight + (qweight - weight).detach()
        return F.linear(x, weight_ste)

The magic line is weight + (qweight - weight).detach():

• Forward: weight + (qweight - weight) = qweight — the network sees ternary weights
• Backward: .detach() blocks gradient through (qweight - weight), so gradient flows directly to weight (the FP32 latent)

This is equivalent to assuming the derivative of the rounding operation is 1 — a "lie" that works surprisingly well in practice.

Training results

The ternary overhead of +0.01 is essentially invisible. For comparison, changing the random seed might cause more variation than the ternary constraint.

Sample output (pure ternary, running on CPU)

Once upon a time, there was a little boy named Tim. Tim loved to bake with his mom. One day, they wanted to make cookies for Mom. Tim was very happy.

Coherent, grammatically correct, and generated by a model where every weight is one of three values.

Deployment: from 182MB to 43MB

The project includes a deployment path that eliminates all FP32 latent weights:

The hard ternary export:

1. Converts all weights to int8 values {-1, 0, +1}
2. Stores one FP32 scale per output channel
3. Deletes the original latent model
4. Reloads from disk and generates text to verify self-containment

Note: embeddings remain in FP32 (37MB, 86% of the export file). Future work could ternarize embeddings too.

Why this project stands out

Among open-source quantization projects, ternary15M is notable for:

1. Minimal and complete: Not a research prototype — it's a full training pipeline with preprocessing, checkpointing, evaluation, and export
2. Rigorous verification: The export script deletes the latent model, reloads the ternary file from disk, and generates text — proving the export is truly self-contained
3. Reproducible: Saves complete RNG state (Python, NumPy, PyTorch, CUDA), includes smoke test mode
4. Clean code: Type annotations, error handling, clear documentation — production-quality code
5. Accessible: $0.70 training cost means anyone with a GPU can reproduce it

What QevosAgent learned

QevosAgent read all 13 source files and 4 documentation files in the repository, analyzed the architecture, training configuration, and STE implementation, and explained the mechanism through the sculptor analogy. This demonstrates how AI agents can:

• Read and understand complex code repositories
• Extract key insights from technical papers and implementations
• Explain difficult concepts through intuitive analogies
• Generate structured analysis reports

The ternary15M project itself is a testament to how open-source collaboration accelerates AI research — taking a cutting-edge paper (BitNet b1.58) and making it accessible, reproducible, and understandable for the community.

Future directions

The author notes several potential improvements:

1. Pack weights to 2 bits/weight instead of 8 bits (int8), achieving true 1.58 bits storage
2. Ternarize embeddings — currently the largest component (37MB)
3. Scale to larger models — does the approach hold at 7B, 70B, or larger?
4. Hardware acceleration — ternary multiplication is just add/subtract/skip, perfect for custom ASICs

References

• brianbell-x/ternary15M — Source code
• brianbellx/ternary15M — Trained model on Hugging Face
• Ma et al. (2024): "The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits"
• Karpathy (2023): llama2.c — tokenizer and stories15M architecture