Skip to main content

Streaming Engine

Shim’s streaming engine solves three core problems with LLM streaming outputs.

The Three Problems

1. Buffer Bottleneck

Problem: Most JSON parsers require the full string. Waiting for full output adds 2-5 seconds of latency. Solution: Shim attempts JSON.parse() on every chunk. Returns partial object as soon as parseable.

2. Markdown Fence Trap

Problem: LLMs often wrap JSON in markdown fences:
Parsers fail because of the fence. Solution: Shim strips fences early in the pipeline:
Fence patterns detected:
  • ```json ... ```
  • ``` ... ```
  • Leading/trailing text before { or [

3. Numerical Gyrations

Problem: Partial numbers cause UI flicker:
Solution: Shim detects incomplete tokens and holds them:
Incomplete token patterns:
  • Trailing decimal: 0.
  • Leading decimal: .5
  • Partial escape: \u00
  • Partial string: "incomplete

Architecture

Session State Machine

State Transitions


Key Algorithms

Structural Completeness Check

Incomplete Token Detection

Brace Tracking


Performance Characteristics

Time Complexity

Memory Usage

  • Buffer: O(m) where m = total output size
  • State: O(1) constant overhead
  • Circuit breaker: 5MB max buffer

Throughput

  • Chunks/sec: 1M+ per Worker
  • Latency/push: <1ms average
  • Parse attempts: 1 per push

Safety Features

Circuit Breaker

Terminates sessions at 5MB buffer:

Session Expiration

Sessions expire after 60 seconds:

Junk Seek

Ignores data before first { or [:

Comparison: Streaming vs Batch


When to Use Streaming

Use Streaming When:
  • LLM is streaming tokens
  • You want real-time UI updates
  • Output size is large (>1MB)
  • You need progress indicators
Use Batch When:
  • You have the full output
  • Real-time updates aren’t needed
  • Output is small (<100KB)
  • You want simplicity

Next Steps

Streaming Guide

Best practices for streaming

Streaming API

API reference

Confidence Levels

Understand confidence scoring

TypeScript SDK

Use the official SDK