null-pointer.blog

recent posts

about

  • What Is a System One Model? A Builder’s Guide to Jev

    Look at a production agent trace and count how many LLM calls are actually decisions. Is this ticket urgent? Which tool should run next? Is this bash command safe? Does this answer need a stronger model? Each one is a one-bit or few-way judgment, and we pay for it with a full text-generation call: a prompt, a structured-output schema, a parser, a retry loop, and hundreds of milliseconds to seconds of latency.

    Jev, released by TypeSafe AI, is a bet that this is the wrong tool for the job. It is the first of what TypeSafe calls System One models, and it doesn’t generate text at all.

    The core idea

    The name borrows from Kahneman. System 1 is fast, intuitive judgment; System 2 is slow, deliberate reasoning. TypeSafe’s argument is that most decisions inside software are System 1 judgments, yet we have been renting System 2 to make them. (The model name nods to Jevons: his paradox holds that falling cost drives rising consumption, which hints at the intended use. Make decisions cheap enough and you’ll put them everywhere.)

    The launch post frames the motivation as a question rather than a benchmark. If models have been superhuman at chat for years, where is all the automation? Their answer is that the bottleneck was never raw intelligence; a model that replies in prose is simply an awkward component to build software on.

    The interface: state in, typed decisions out

    You don’t prompt Jev. You give it a state (the context) and a set of questions about that state:

    {
      "model": "jev-latest",
      "state": "Checkout has thrown 500s for 20 minutes. Two enterprise customers already emailed.",
      "questions": {
        "is_urgent": {
          "type": "noul",
          "instructions": "This needs a human to look at it immediately"
        },
        "is_billing_related": {
          "type": "noul",
          "instructions": "The issue involves payments or billing"
        }
      }
    }

    Each question comes back as a probability, not a sentence. There are three question types:

    • Noul: a yes/no statement. You get the probability that it’s true.
    • Choice: pick from options you define. You get a probability per option plus an overall confidence.
    • Score: place the input on an ordered scale (low/medium/high). You get a continuous score, the distribution, and a confidence.

    Two properties fall out of this design. First, a response cannot contain a value outside your schema, so schema errors and wrong decisions become separate problems. There’s no JSON repair or “please respond only with yes or no.” Second, the questions are answered together. Jev uses a parallel sampler that produces all outputs in a single query rather than autoregressively, so adding questions to one request is nearly free. TypeSafe’s own cookbook reports that batching a 13-question briefing into one call came out 12.2x cheaper and 10x faster than asking one at a time, with identical answers.

    Why calibration is the actual headline

    The speed numbers get the attention. The more important property is calibration.

    Jev is trained with what TypeSafe calls Reinforcement Learning for Calibrated Decisions (RLCD). RLHF optimizes for responses human raters prefer and RLVR for outputs a program can verify; RLCD instead targets honest probabilities on decision tasks. Concretely, across many predictions, answers given 90% probability should be right about 90% of the time.

    Why this matters for builders: a model that’s right 95% of the time but can’t tell you which calls fall in the other 5% can’t be automated around. One with honest uncertainty can, because you branch on confidence and send low-confidence cases to a human or a bigger model.

    That changes how you design the system. Instead of one global “trust the model” threshold, you set a threshold per action, scaled to what being wrong costs. Auto-archiving spam can act at 0.8. Blocking a production deploy might need 0.99, with everything in between escalated.

    Note the caveat hiding in “across many predictions”: calibration is a property of the aggregate. Any single answer can still be wrong.

    What it costs and how fast it is

    Per TypeSafe, Jev answers in one parallel pass in 70 to 500 milliseconds, at $0.042 per million input tokens, with output free. The headline comparison against frontier LLMs is up to roughly 194x faster and 445x cheaper on these tasks.

    Treat those as vendor claims for now. As one early guide put it, they’re TypeSafe’s own numbers, self-run and unreproduced. (Independently benchmarking them is the next post in this series.)

    What it deliberately can’t do

    Jev can’t write a reply, produce code, summarize a document, or explain its reasoning. If you need text, you need an LLM. There are also practical limits. A choice question supports up to 255 options; beyond that, TypeSafe uses a slower two-stage approach of scoring candidates and then choosing.

    So Jev is not an LLM replacement. It’s a new component.

    Where it fits in an agent

    The useful mental model is a split: Jev decides, the LLM writes. In an agent loop, that means:

    • Routing: decide whether a request needs the cheap model or the expensive one before the run starts.
    • Gating: check a tool call for risk before it executes, and block it or escalate it.
    • Control flow: decide whether to retry, stop, retrieve again, or hand off, instead of asking the main LLM to “reflect.”
    • Triage at scale: classify every inbound email, ticket, or event where per-call LLM cost was prohibitive.

    LangChain already exposes this as middleware (model routing and tool-risk gating), so you can drop it into an existing agent without restructuring it.

    The open questions

    A few things I’ll be testing in upcoming posts:

    1. Do the speed, cost, and accuracy claims hold on independent workloads?
    2. How well calibrated is it out of distribution, like long states, ambiguous instructions, or adversarial inputs?
    3. How sensitive are results to the wording of instructions, and can that field be optimized the way we optimize prompts?
    4. Where’s the line between System 1 and System 2? Some “simple” decisions quietly require multi-step reasoning.

    System One models are a genuinely different primitive. Whether they’re a better one depends on measurements that don’t exist yet. Next post, I’ll start making them.

  • How We Cut LLM Inference Latency by 40% Using vLLM, Tensor Parallelism, and Semantic Caching

    Optimizing retrieval-augmented LLM inference isn’t just about bigger GPUs, it’s about smarter orchestration.

    🚨 The Problem

    A few months ago, our retrieval-augmented generation (RAG) system was hitting its limits.

    We were serving clinical queries from healthcare providers, but:

    • p95 latency: 850 ms
    • GPU utilization: ~35 %
    • Query throughput: barely 300 QPS under load

    The pipeline looked like this:

    User Query
  ↓
1️⃣ Embed query → FAISS search
2️⃣ Lexical retrieval → Elasticsearch (BM25)
3️⃣ Merge + re-rank
4️⃣ Construct context → LLM inference
5️⃣ De-identify + postprocess
  ↓
Response

    Despite using a strong model, our serving stack wasn’t scaling.

    We needed to bring latency below 500 ms while keeping costs in check.

    🧠 Profiling the Bottleneck

    We profiled the full inference path using:

    • Nsight Systems and PyTorch Profiler for GPU traces
    • Prometheus + Grafana for latency and token-level metrics

    Key findings:

    • 40 % GPU idle time between kernels (batching inefficiency)
    • Retrieval (FAISS + Elastic) serialized — ~120 ms wasted
    • KV cache fragmentation consuming unnecessary VRAM
    • Static batching delaying low-traffic requests

    ⚙️ The Redesign

    We re-architected inference around three major improvements:

    (1) Tensor-parallel serving via vLLM, (2) Continuous batching, and (3) Semantic caching.

    🚀 1. Multi-GPU Tensor Parallelism

    We deployed our model using vLLM on AWS p5 instances (8× H100 GPUs):

    python -m vllm.entrypoints.api_server \\
  --model meta-llama/Llama-2-70b-chat-hf \\
  --tensor-parallel-size 4 \\
  --gpu-memory-utilization 0.9 \\
  --port 8000
    • Model weights sharded across 4 GPUs
    • Each GPU stores KV-cache only for its own attention heads
    • Communication handled by NCCL over NVLink

    ✅ Result:

    • Linear scaling of throughput
    • GPU utilization ↑ to 85 %
    • Per-token latency ↓ 35 %

    ⚡ 2. Continuous Batching

    Static batching was replaced with continuous batching, where new requests join an ongoing batch mid-generation.

    This kept GPUs fully occupied even with variable traffic.

    Before: waiting to fill batches → GPU idle gaps

    After: dynamic merge/split at token granularity

    ✅ Result:

    • Throughput ↑ 2.8×
    • Latency variance (p95 – p99) ↓ 50 %

    🧠 3. Semantic Caching

    We introduced a Redis-backed semantic cache keyed by query embeddings.

    cache_key = f"embed:{hash(embedding)}"
if redis.exists(cache_key):
    results = redis.get(cache_key)
else:
    results = run_retrieval(embedding)
    redis.setex(cache_key, 3600, results)

    When a new query arrives, we check for high cosine similarity (> 0.9) with cached embeddings.

    If found, we reuse the retrieval + context immediately — skipping FAISS + Elastic calls.

    ✅ Result:

    • Cache hit rate 71 %
    • Vector DB load ↓ 60 %
    • End-to-end latency ↓ ~90 ms for repeated/semantically similar queries

    🔍 Results

    MetricBeforeAfterΔ
    GPU Utilization35 %85 %+50 pts
    p95 Latency850 ms490 ms↓ 42 %
    Cost per Query—↓ 37 %—
    NDCG@10 (Retrieval Quality)0.730.89+22 %
    Cache Hit Rate0 %71 %—

    🧩 Lessons Learned

    • Batching is a hidden bottleneck — static batching kills latency under variable load.
    • Tensor parallelism isn’t free — one slow GPU throttles the group; NVLink topology matters.
    • Semantic caching > lexical caching — paraphrased queries benefit enormously.
    • Monitoring is critical — token throughput and cache hit dashboards caught regressions early.

    🧠 Takeaway

    Optimizing LLM inference isn’t about throwing bigger GPUs at the problem — it’s about making each token, cache lookup, and GPU cycle count.

    By re-architecting our serving stack with vLLM, tensor parallelism, and semantic caching,

    we now sustain 1 K QPS at < 500 ms p95 — with better retrieval grounding and 35 % lower cost.

    ✍️ Final Thought

    We plan to open-source part of our semantic caching implementation soon.

    If you’re scaling RAG inference or debugging GPU underutilization, I’d love to compare notes — reach out anytime.