TL;DR
Large Language Models generate text probabilistically—they predict "what comes next," not "what is correct." This means every dependent step in a reasoning chain has a chance of error, and those errors compound exponentially. A 98% per-step accuracy drops to just 13.3% over 100 steps. This article explains the math, shows real coding failures, and provides four battle-tested patterns to build reliable systems with LLMs.
The Three Axioms of LLM Behavior
Before diving into failure modes, we need to establish what LLMs actually are from first principles. Large Language Models obey three fundamental axioms that define their capabilities—and limitations:
-
Probabilistic Generation: Every output token is sampled from a likelihood distribution over the vocabulary. LLMs don't derive answers from axioms or execute deterministic algorithms—they predict the most probable next token given context.
-
Contextual Attention: The transformer architecture weights tokens by relevance patterns learned during training. This enables remarkable coherence but doesn't guarantee semantic truth or logical validity.
-
Feed-Forward Irreversibility: Token generation is unidirectional. Once a token is committed to the output, subsequent generation conditions on it. The model cannot "go back" to reconsider—a mistaken token becomes ground truth for everything that follows.
From these axioms flows an unavoidable conclusion: Correctness decays exponentially with dependent reasoning chains. This isn't a bug to be fixed with more parameters or better training—it's a mathematical property of how these systems work.
The Mathematics of Compounding Error in LLMs
The Probability Chain Rule (Formalized)
Consider a task requiring n dependent reasoning steps. Let p_i represent the probability that step i is correct, given that all prior steps succeeded.
The probability of complete task success is:
$$P(\text{complete}) = p_1 \times p_2 \times p_3 \times \ldots \times p_n$$
In an idealized scenario where each step has identical accuracy p, this simplifies to:
$$P(\text{complete}) = p^n$$
However, real LLM behavior is worse. As context grows, attention diffuses, uncertainty accumulates, and the model increasingly conditions on its own potentially-flawed outputs. Empirically:
$$p_1 > p_2 > p_3 > \ldots > p_n$$
True decay outpaces simple exponential models.
Numerical Reality Check
Even with an optimistic 98% per-step accuracy, success probability collapses rapidly:
| Reasoning Steps | Success Probability | Practical Meaning |
|---|---|---|
| 5 | 90.4% | Simple functions usually work |
| 10 | 81.7% | Short scripts often succeed |
| 20 | 66.8% | Medium complexity gets unreliable |
| 50 | 36.4% | Complex logic fails majority of time |
| 100 | 13.3% | System-level reasoning almost always fails |
This mathematical reality explains the "vibes" of AI-assisted programming: short snippets feel magical, while complex systems drift into subtle wrongness.
How Self-Conditioning Amplifies AI Hallucinations
LLMs create irreversible feedback loops. Each generated token becomes "ground truth" for subsequent generation—even when that token is subtly wrong.
The Failure Cascade Pattern
- Step k: Model makes a small error (wrong API name, off-by-one, incorrect assumption)
- Step k+1: Model reasons from the mistake as if it were correct
- Step k+n: Output is internally coherent but externally wrong
The danger: high coherence masks low correctness. The code reads right but behaves wrong.
Real-World Example: API Hallucination Cascade
Consider this prompt: "Parse a JSON API response in Python"
# LLM-generated code
def parse_response(data):
import json
import requests # Hallucinated dependency
response = requests.Response() # Inventing API usage
response._content = data # Using private attributes
return json.loads(response.text)
The model hallucinated that requests.Response() is instantiable and that setting _content directly is valid. Each subsequent line builds coherently on these false premises. A developer might not catch this in review because the code looks plausible.
This pattern appears constantly in LLM code generation: invented methods, hallucinated parameters, confident misuse of real libraries.
Error Tolerance Varies by Domain
Not all tasks suffer equally from compounding error. Understanding domain-specific tolerance helps you deploy LLMs appropriately:
| Domain | Error Tolerance | Why | Typical Failure Mode |
|---|---|---|---|
| Creative writing | High | No single "correct" answer | Style drift, tone inconsistency |
| UI generation | Medium | Visual review catches issues | Layout bugs, accessibility gaps |
| Data transformation | Medium-Low | Must preserve semantics | Silent data corruption |
| Business logic | Low | Edge cases matter | Incorrect conditionals |
| Mathematical proofs | Near-zero | Single error invalidates all | Wrong intermediate step |
| Security code | Near-zero | Attackers find any flaw | Exploitable vulnerabilities |
Key insight: LLMs excel where constraints are soft and multiple outputs are acceptable. They struggle where a single correct answer exists and must be found exactly.
Why LLMs Can't Do Arithmetic Reliably
A particularly striking limitation: LLMs process numbers as text tokens, not mathematical objects.
Tokenization fragments numbers unpredictably:
- "12345" might tokenize as ["123", "45"]
- "3.14159" might become ["3", ".", "141", "59"]
The model has no internal carry mechanism, no numeric registers, no computational state. When arithmetic "works," it's because the model has memorized patterns from training data—not because it's computing.
This explains why:
- Simple arithmetic often succeeds (high training frequency)
- Novel number combinations fail
- Precision degrades with magnitude
- Multi-step calculations compound errors rapidly
Four Engineering Patterns for Reliable LLM Usage
Given these fundamental limitations, how do we build reliable systems? The answer: treat LLMs as proposal generators, not authoritative sources. Verify everything, minimize unverified chains, constrain the solution space.
Pattern 1: Skeleton-First Architecture
Freeze high-level structure before generating implementations. This prevents compounding errors from propagating across abstraction boundaries.
Step 1 — Generate Structure Only:
class PaymentProcessor:
"""Handles payment transactions with retry logic."""
def __init__(self, gateway: PaymentGateway):
self.gateway = gateway
def process(self, amount: Decimal, card: CardInfo) -> TransactionResult:
"""Process payment with automatic retry on transient failures."""
pass
def _validate_card(self, card: CardInfo) -> bool:
"""Validate card details before processing."""
pass
def _handle_retry(self, attempt: int) -> bool:
"""Determine if retry should occur."""
pass
Human reviews structure, types, and signatures. Only after approval:
Step 2 — Implement One Method:
def _validate_card(self, card: CardInfo) -> bool:
if not card.number or len(card.number) < 13:
return False
if not card.expiry or card.expiry < datetime.now():
return False
return self._luhn_check(card.number)
Verify. Test. Repeat for each method.
Why this works: Errors in one method don't propagate to others. Each verification checkpoint resets the error accumulation.
Pattern 2: Constraint-Locked Generation
Collapse the probability space by specifying explicit constraints. Fewer valid outputs means higher probability of correctness.
constraints = [
"Use only Python standard library (no external packages)",
"O(n) time complexity maximum",
"No mutation of input arguments",
"Raise ValueError for invalid inputs, never return None",
"Include type hints for all parameters and return values",
"Maximum 20 lines of code"
]
prompt = f"""
Write a function to find the longest palindromic substring.
CONSTRAINTS (violating any constraint makes the solution unacceptable):
{chr(10).join(f'- {c}' for c in constraints)}
"""
Why this works: Constraints eliminate large portions of the solution space, increasing the probability that the model's output lands in the "correct" region.
Pattern 3: Verification Hooks
Embed runtime assertions that catch errors the moment they occur:
def merge_sorted_lists(list1: list[int], list2: list[int]) -> list[int]:
"""Merge two sorted lists into a single sorted list."""
# Pre-conditions
assert list1 == sorted(list1), "list1 must be sorted"
assert list2 == sorted(list2), "list2 must be sorted"
# Implementation
result = []
i = j = 0
while i < len(list1) and j < len(list2):
if list1[i] <= list2[j]:
result.append(list1[i])
i += 1
else:
result.append(list2[j])
j += 1
result.extend(list1[i:])
result.extend(list2[j:])
# Post-conditions
assert len(result) == len(list1) + len(list2), "All elements preserved"
assert result == sorted(result), "Result is sorted"
assert set(result) == set(list1) | set(list2), "No elements added or removed"
return result
Why this works: Assertions convert silent failures into loud failures. You'll know immediately when something goes wrong rather than discovering corruption downstream.
Pattern 4: Multi-Sample Consensus
Generate multiple independent solutions, compare their structure, and regenerate based on common patterns:
- Generate 3-5 solutions with temperature > 0
- Extract common structural elements
- Identify divergence points (likely error locations)
- Regenerate final solution using consensus structure as constraint
Why this works: Independent samples are unlikely to make identical errors. Agreement across samples suggests correctness; divergence flags risk.
From Open-Loop to Closed-Loop LLM Systems
Traditional LLM usage is "open-loop": input goes in, output comes out, no feedback.
Open Loop (unstable):
User Input → LLM → Output (accept as-is)
Production systems need closed-loop control with error correction:
Closed Loop (stable):
User Input → LLM → Output → Verification → [Pass/Fail]
↓ (on fail)
Error Signal → Regenerate with feedback
Add verification at every feasible checkpoint:
- Type checkers for generated code
- Unit tests for behavioral correctness
- Linters for style and security patterns
- Static analysis for potential bugs
- Human review for semantic correctness
Each verification stage dampens error propagation.
The Fundamental Limit Theorem
Here's the uncomfortable truth: No amount of scaling eliminates compounding error for tasks requiring extended chains of dependent reasoning.
Larger models improve per-step accuracy p, but as long as p < 1, exponential decay guarantees failure on sufficiently long chains. Scale delays the problem; it doesn't solve it.
This is why reasoning models like o1 and DeepSeek R1, despite impressive benchmarks, still fail on novel multi-step problems. They've pushed p higher and extended the "reliable reasoning length"—but the fundamental mathematics remains unchanged.
Appropriate Roles for LLMs in Software Engineering
Given these limitations, where should LLMs fit in your workflow?
| Role | Suitability | Notes |
|---|---|---|
| Boilerplate generation | Excellent | Low reasoning depth, pattern matching |
| Code explanation | Excellent | Analysis, not generation |
| Refactoring assistance | Excellent | Local transformations with verification |
| Test case generation | Good | Verify generated tests actually test something |
| Architecture exploration | Good | Use for ideation, not decisions |
| Bug hypothesis generation | Good | Suggests what to investigate |
| Proof generation | Poor | Extended reasoning chains fail |
| Security-critical code | Dangerous | Adversarial context + zero tolerance |
| Regulatory compliance | Dangerous | Must be exactly right |
The principle: Humans own correctness invariants. LLMs propose; humans verify.
Conclusion: Fluency in Probability Wins
LLMs are not flawed humans or buggy computers. They're probabilistic proposal engines—fundamentally different from deterministic systems. Engineers who understand this distinction build reliable AI-assisted systems. Those who don't chase phantom reliability and ship broken code.
The winning approach:
- Shorten chains: Decompose complex tasks into verifiable steps
- Constrain solutions: Explicit requirements narrow the solution space
- Verify ruthlessly: Every LLM output is a hypothesis until proven
- Design for failure: Assume errors will occur; build systems that catch them
The future belongs to engineers fluent in probability who can orchestrate AI capabilities within principled uncertainty bounds.
Frequently Asked Questions
Q: Will larger models solve the compounding error problem?
A: Larger models improve per-step accuracy but don't eliminate exponential decay. A 99% per-step accuracy still yields only 36.6% success over 100 steps. The fundamental mathematics is unchanged by scale.
Q: Are reasoning models like o1 or DeepSeek R1 immune to these issues?
A: No. These models extend reliable reasoning length through explicit chain-of-thought, but they still exhibit compounding error on sufficiently complex problems. They've moved the failure point, not eliminated it.
Q: How do I know when an LLM output is reliable?
A: You don't—that's the point. Treat every output as a hypothesis requiring verification. Use type checkers, tests, linters, and human review. Reliability comes from your verification system, not the model.
Q: Is "vibe coding" dangerous?
A: It depends on context. For exploratory prototypes with no correctness requirements, rapid LLM iteration is valuable. For production systems with real users, every piece of AI-generated code needs verification before deployment.
Q: Should I stop using LLMs for coding?
A: Absolutely not. LLMs are transformative productivity tools when used appropriately. The key is understanding their limitations and building workflows that verify outputs. Used correctly, they accelerate development dramatically.