Ask a model to return JSON and, sooner or later, it will hand you something a parser can't read. Not because the model is broken, but because language models generate text token by token — and JSON is a strict format that doesn't forgive a single missing comma. If you're building anything that treats LLM output as data, you've hit this. Here's why it keeps happening and how to stop it from breaking your pipeline.
Why models generate invalid JSON
A model isn't running a JSON parser while it writes. It's predicting the next token. It knows JSON should have balanced braces and commas in the right places, but it has no syntax checker to catch mistakes. The result is JSON that's usually right and occasionally broken in predictable ways.
There's also the length problem. Models have an output token limit. If your object is large, the model gets cut off mid-array — and you get a response that ends at "tags": ["a", "b", with no closing bracket. Truncation is probably the single most common cause of broken JSON from LLMs.
The failure modes, in order of frequency
1. Truncation
Cut off mid-object or mid-array, brackets left unclosed:
{
"name": "John",
"tags": ["dev", "ops"2. Extra text around the JSON
The model writes a friendly sentence, then the JSON, or adds a closing remark. Now your JSON.parse sees Here you go: and throws:
Here's the data you asked for:
{"name": "John", "age": 30}3. Missing or trailing commas
The model forgets a comma between fields, or adds one after the last item (which is legal in Python and JSON5, but not in strict JSON):
{
"name": "John"
"age": 30,
}4. Unescaped characters
A string contains a raw double quote or a literal newline, breaking the syntax:
{"text": "He said "hello" and left"}Why "JSON mode" isn't a cure
Most providers now offer a JSON mode or structured outputs. These help a lot — they constrain the model to emit valid syntax most of the time — but they don't eliminate the problem. Truncation still happens at the token limit. Extra text still appears in some modes. And a model can still produce syntactically valid JSON that's simply wrong for your schema. JSON mode raises your success rate, it doesn't make it 100%.
So you can't just trust the output. You need a step in between that validates and repairs — every time, no exceptions.
The fix: validate, then repair
The pattern that makes LLM JSON reliable is simple: never feed model output straight to JSON.parse. Validate it first, and if it's broken, repair it. This catches truncation, strips stray text, fixes commas and brackets, and escapes whatever needs escaping.
That's exactly what the JSON repair tool does. Paste the model's output and it returns clean, parseable JSON — including the Python-style syntax models sometimes drift into (single quotes, None, True).
Let the agent fix its own JSON
If you're working with an AI agent — Claude, Cursor, or a custom agent built on a model — the repair step can happen inside the agent itself. JSONGuy ships an MCP server so the agent can call repair_json and validate_json before it sends anything back. The model produces, then repairs its own output.
Bottom line
LLMs generate text, not data structures. They will keep producing broken JSON — that's a property of how they work, not a bug you can prompt away. The robust approach is to stop trusting the output and add a validate-then-repair step. JSON mode gets you most of the way; a repair step gets you the rest. Do both, and model output becomes safe to parse.
Try it yourself: paste a truncated or messy model response into the JSON repair tool and watch it come back as valid JSON.