Migrating from OpenAI JSON Mode

OpenAI JSON mode reduces failures but doesn’t eliminate them. Shim catches what JSON mode misses.

The Problem with JSON Mode

OpenAI’s response_format: { "type": "json_object" } improves JSON output but has gaps:

1. Still Produces Invalid JSON

JSON mode can still return:

Truncated objects (missing closing brackets)
Trailing commas
Malformed decimals
Unquoted keys (rare but happens)

2. No Schema Validation

JSON mode produces JSON, but not necessarily your schema:

// You asked for:
{ "name": "string", "age": "number" }

// JSON mode returns:
{ "name": "John", "age": "30" }  // age is string, not number

3. No Repair Feedback

JSON mode either succeeds or fails. No indication of what was fixed.

Why Add Shim

Shim acts as a safety layer on top of JSON mode:

Catch remaining failures: Repair what JSON mode missed
Schema validation: Enforce your schema with type coercion
Confidence scoring: Know when repairs were needed
Streaming support: Repair as tokens arrive

Migration Path

Before: JSON Mode Only

import OpenAI from 'openai';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });

const response = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [
    {
      role: 'system',
      content: 'You are a helpful assistant that outputs JSON.'
    },
    {
      role: 'user',
      content: 'Extract name and age from: "John is 30 years old"'
    }
  ],
  response_format: { type: 'json_object' }
});

const raw = response.choices[0].message.content;
const data = JSON.parse(raw); // Can still fail!

After: JSON Mode + Shim

import OpenAI from 'openai';
import { ShimClient } from 'shim-sdk';

const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
const shim = new ShimClient({ apiKey: process.env.SHIM_API_KEY });

const response = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [
    {
      role: 'system',
      content: 'You are a helpful assistant that outputs JSON.'
    },
    {
      role: 'user',
      content: 'Extract name and age from: "John is 30 years old"'
    }
  ],
  response_format: { type: 'json_object' }
});

const raw = response.choices[0].message.content;

// Repair with Shim
const result = await shim.repair({
  raw_output: raw,
  schema: {
    type: 'object',
    properties: {
      name: { type: 'string' },
      age: { type: 'number' }
    },
    required: ['name', 'age']
  }
});

if (result.success) {
  const data = result.repaired; // { name: "John", age: 30 }
  console.log(`Confidence: ${result.metadata.confidence}`);
}

Benefits

Feature	JSON Mode Only	JSON Mode + Shim
Valid JSON	95-98%	99.9%
Schema validation	No	Yes
Type coercion	No	Yes
Repair visibility	No	Yes
Confidence scoring	No	Yes
Streaming repair	No	Yes

Streaming with JSON Mode

Before: No Streaming Validation

const stream = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [...],
  response_format: { type: 'json_object' },
  stream: true
});

let buffer = '';

for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content || '';
  buffer += content;

  // Can't parse until complete
  console.log('Buffer:', buffer);
}

// Only now can we parse
const data = JSON.parse(buffer); // Still can fail!

After: Streaming with Shim

const stream = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [...],
  response_format: { type: 'json_object' },
  stream: true
});

// Start Shim session
const { session_id } = await shim.stream.start({
  schema: mySchema,
  mode: 'strict'
});

for await (const chunk of stream) {
  const content = chunk.choices[0]?.delta?.content || '';

  // Push to Shim
  const { state } = await shim.stream.push({
    session_id,
    chunk: content
  });

  // Show preview as soon as parseable
  if (state.safe_to_emit && state.partial) {
    console.log('Preview:', state.partial);
  }
}

// Finalize
const result = await shim.stream.finalize({ session_id });
const data = result.repaired;

Schema Enforcement

Problem: JSON Mode Doesn’t Validate Schema

// Prompt asks for:
// { "name": "string", "age": "number" }

const response = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [...],
  response_format: { type: 'json_object' }
});

const raw = response.choices[0].message.content;
// Returns: { "name": "John", "age": "30" }  // Wrong type!

const data = JSON.parse(raw);
console.log(typeof data.age); // "string" (expected number)

Solution: Shim Schema Validation

const result = await shim.repair({
  raw_output: raw,
  schema: {
    type: 'object',
    properties: {
      name: { type: 'string' },
      age: { type: 'number' }
    },
    required: ['name', 'age']
  },
  mode: 'strict'
});

if (result.success) {
  console.log(typeof result.repaired.age); // "number" (coerced)
  console.log(result.metadata.schema_repairs);
  // [{ type: "type_coercion", field: "age", ... }]
}

Confidence Feedback

Before: No Visibility

const raw = response.choices[0].message.content;
const data = JSON.parse(raw);
// Did JSON mode fix anything? No idea.

After: Confidence Scoring

const result = await shim.repair({ raw_output: raw });

if (result.success) {
  switch (result.metadata.confidence) {
    case 'high':
      // Perfect or only structural fixes
      if (result.metadata.was_repaired) {
        console.log('Minor structural repairs applied');
      } else {
        console.log('Perfect JSON, no repairs needed');
      }
      break;

    case 'medium':
      // Minor fixes applied
      console.log('Repairs:', result.metadata.repairs_applied);
      break;

    case 'low':
      // Significant repairs needed
      console.warn('JSON mode failed, Shim recovered');
      break;
  }
}

Error Handling

Before: Try/Catch

try {
  const raw = response.choices[0].message.content;
  const data = JSON.parse(raw);
  return data;
} catch (error) {
  logger.error('JSON mode failed', { error });
  return null;
}

After: Structured Errors

const raw = response.choices[0].message.content;
const result = await shim.repair({ raw_output: raw });

if (result.success) {
  return result.repaired;
}

// Structured error handling
const error = result.metadata.errors[0];

logger.error('Repair failed', {
  code: error.code,
  message: error.message,
  recoverable: error.recoverable
});

if (error.recoverable) {
  // Retry logic
  return retryWithShim(raw);
}

return null;

Cost Comparison

JSON Mode Only

Scenario: 100K requests/month, 2% failure rate

Failed parses: 100K × 2% = 2,000 failures
Retry cost: 2,000 × $0.03 (GPT-4 input) = $60/month

JSON Mode + Shim Pro

Scenario: 100K requests/month, 2% failure rate

Shim cost: $29/month (includes 100K repairs)
Failed parses: 0 (Shim catches all)

Monthly cost: $29/month
Savings: $31/month

Function Calling Alternative

Option 1: OpenAI Function Calling

More reliable than JSON mode for structured outputs:

const response = await openai.chat.completions.create({
  model: 'gpt-4',
  messages: [...],
  functions: [{
    name: 'extract_person',
    parameters: {
      type: 'object',
      properties: {
        name: { type: 'string' },
        age: { type: 'number' }
      },
      required: ['name', 'age']
    }
  }],
  function_call: { name: 'extract_person' }
});

const args = response.choices[0].message.function_call.arguments;
const data = JSON.parse(args); // More reliable than JSON mode

Option 2: Function Calling + Shim

Belt-and-suspenders approach:

const args = response.choices[0].message.function_call.arguments;

const result = await shim.repair({
  raw_output: args,
  schema: functionSchema
});

if (result.success) {
  return result.repaired;
}

When to Use Shim

Use Shim If:

JSON mode failures occur (even 1-2%)
Schema validation needed
Type coercion required
Streaming repair wanted
Repair visibility important

Skip Shim If:

JSON mode has 100% success rate (rare)
No schema validation needed
Extra latency unacceptable (<10ms)
Budget is extremely tight

Fallback Strategy

Use Shim as a fallback only:

const raw = response.choices[0].message.content;

// Try native parse first
try {
  const data = JSON.parse(raw);
  return data;
} catch {
  // Fallback to Shim
  const result = await shim.repair({ raw_output: raw });

  if (result.success) {
    logger.info('Shim recovered failed parse');
    return result.repaired;
  }

  throw new Error('Both parsers failed');
}

Production Checklist

Enable JSON mode on OpenAI calls
Add Shim API key to environment
Install shim-sdk (if using TypeScript)
Wrap JSON.parse with Shim repair
Add schema validation
Implement confidence-based logging
Monitor repair rates in console
Alert on high failure rates

Migration Strategies

Strategy 1: Safety Net (Recommended)

Always use Shim. Catch all failures.

const raw = response.choices[0].message.content;
const result = await shim.repair({ raw_output: raw, schema });
return result.success ? result.repaired : null;

Strategy 2: Fallback Only

Try native parse first, Shim on failure.

try {
  return JSON.parse(raw);
} catch {
  const result = await shim.repair({ raw_output: raw });
  return result.success ? result.repaired : null;
}

Strategy 3: Conditional (Advanced)

Use Shim only for complex schemas.

if (schema.properties && Object.keys(schema.properties).length > 10) {
  // Complex schema, use Shim
  const result = await shim.repair({ raw_output: raw, schema });
  return result.repaired;
} else {
  // Simple schema, native parse
  return JSON.parse(raw);
}

FAQ

Should I still use JSON mode?

Yes. JSON mode reduces failures. Shim catches what remains.

Can I use Shim without JSON mode?

Yes. Shim works with any LLM output (JSON mode or not).

Does Shim add latency?

Yes, <10ms. Worth it for reliability.

What about Structured Outputs (new API)?

Structured Outputs are more reliable than JSON mode. Still, Shim adds schema validation and repair visibility.

Can I use both function calling and Shim?

Yes. Function calling → Shim → your app.

Next Steps

Quick Start

Get your first repair working

Schema Validation

Add JSON Schema validation

Streaming Guide

Repair streaming outputs

Confidence Levels

Understand confidence scoring

​Migrating from OpenAI JSON Mode

​The Problem with JSON Mode

​1. Still Produces Invalid JSON

​2. No Schema Validation

​3. No Repair Feedback

​Why Add Shim

​Migration Path

​Before: JSON Mode Only

​After: JSON Mode + Shim

​Benefits

​Streaming with JSON Mode

​Before: No Streaming Validation

​After: Streaming with Shim

​Schema Enforcement

​Problem: JSON Mode Doesn’t Validate Schema

​Solution: Shim Schema Validation

​Confidence Feedback

​Before: No Visibility

​After: Confidence Scoring

​Error Handling

​Before: Try/Catch

​After: Structured Errors

​Cost Comparison

​JSON Mode Only

​JSON Mode + Shim Pro

​Function Calling Alternative

​Option 1: OpenAI Function Calling

​Option 2: Function Calling + Shim

​When to Use Shim

​Use Shim If:

​Skip Shim If:

​Fallback Strategy

​Production Checklist

​Migration Strategies

​Strategy 1: Safety Net (Recommended)

​Strategy 2: Fallback Only

​Strategy 3: Conditional (Advanced)

​FAQ

​Should I still use JSON mode?

​Can I use Shim without JSON mode?

​Does Shim add latency?

​What about Structured Outputs (new API)?

​Can I use both function calling and Shim?

​Next Steps