Failures

Failures are inevitable when orchestrating LLM-powered workflows — models hit rate limits, agents produce bad output, commands fail, and providers go down. Fabro handles this with multiple layers of defense: automatic retries with backoff, provider failover, failure classification for intelligent routing, circuit breakers to prevent infinite loops, and goal gates to catch quality problems before a run completes.

Failure classification

When a node fails, Fabro classifies the failure into one of six categories. These classes drive retry decisions, circuit breaker logic, and edge routing.

Class	Description	Examples
`transient_infra`	Temporary infrastructure problem — likely to resolve on retry	Rate limits, timeouts, network errors, 5xx responses
`deterministic`	Permanent failure — retrying won’t help	Authentication errors, bad configuration, invalid requests
`budget_exhausted`	Resource limit reached	Context length exceeded, token/turn limits, quota exhausted
`compilation_loop`	Reserved for loop detection	—
`canceled`	User or system cancellation	Cancel signal, abort
`structural`	Reserved for scope enforcement	Write scope violations

Classification happens automatically. Fabro inspects SDK error types, HTTP status codes, and error message patterns to assign the right class. The failure_class is written to context after each stage, so you can route on it in edge conditions:

implement -> fix       [condition="failure_class=transient_infra"]
implement -> escalate  [condition="failure_class=deterministic"]

Human gates fail closed when unanswered

Human approval gates are special: lack of an answer is treated as a failure, not as an implicit approval. If a prompt ends because stdin is closed, the user cancels it, a web session disconnects, or a test/replay interviewer has no answer available, the human stage returns a failure outcome instead of selecting an unconditional branch. That means an unanswered gate will only continue if you model that path explicitly, for example:

approve [shape=hexagon, label="Approve release?"]

approve -> ship          [label="[A] Approve"]
approve -> manual_review [condition="outcome=failed"]

If no outcome=failed edge or retry_target exists, the run stops rather than advancing past the approval gate.

Stop linear workflows on failure

By default, Fabro uses on_failure="route". A failed node can take an unconditional edge when no explicit route matches. This compatibility default lets existing workflows decide how later nodes handle the failure. Set graph-level on_failure="exit" to stop a linear workflow at a failed node:

stop-on-failure.fabro

digraph Build {
    graph [on_failure="exit"]

    start [shape=Mdiamond]
    exit [shape=Msquare]
    plan [prompt="Plan the work"]
    implement [prompt="Implement the plan"]
    verify [prompt="Verify the implementation"]

    start -> plan -> implement -> verify -> exit
}

Fabro still uses an explicit recovery edge, such as condition="outcome=failed", before it applies this policy. Matching preferred labels and suggested next node IDs also remain explicit routes. If no explicit edge matches, exit skips the unconditional edge and checks retry targets. The run ends as failed only when no retry target exists. Set on_failure on a node to control that node alone. The node-level attribute overrides the graph level, in both directions: a node can opt out of a graph-level exit with on_failure="route", or stop the run on its own failure with on_failure="exit" while the rest of the graph keeps the default. A node without the attribute inherits the graph policy. See Failed-node routing policy. The policy applies only to failed. Other outcomes keep their normal routing behavior. For parallel nodes, the policy uses the completed parallel node’s final outcome. It does not stop or cancel individual branches early.

Treat a failed node as succeeded

Set on_failure="succeed" on a best-effort node so its failure never blocks the workflow. This pairs well with a strict graph default:

best-effort-node.fabro

digraph Review {
    graph [on_failure="exit"]

    start [shape=Mdiamond]
    exit [shape=Msquare]
    required_check [script="./required-check"]
    optional_scan [script="./optional-scan" on_failure="succeed"]

    start -> required_check -> optional_scan -> exit
}

When optional_scan fails, Fabro first checks explicit recovery routes with the failed outcome. If none match, it rewrites the outcome to succeeded and routes the node as a success. Retries still run first; only the final outcome changes. The original failure stays on the stage.completed event and in the checkpoint, and the outcome’s notes record the promotion. A promoted outcome satisfies a goal gate. Setting on_failure="succeed" on the graph applies it to every node. succeed applies only to failed. It does not change a partially_succeeded outcome. auto_status=true is the deprecated spelling of this policy; validation warns and suggests on_failure="succeed".

Retry layers

Fabro retries failures at three levels: LLM retries handle transient API errors inside a single model call, turn-level retries recover from dropped streams mid-response, and node retries re-execute the entire node handler when the first two levels aren’t enough. These layers are independent — a node retry re-runs the full handler, which gets its own fresh set of LLM and turn-level retries.

LLM retries

Every LLM call (within an agent session or a one-shot prompt node) has a built-in retry loop for transient API errors. This is invisible to the workflow — it happens inside the model call itself.

Setting	Default
Max retries	3
Initial delay	1 second
Backoff multiplier	2x
Max delay	60 seconds
Jitter	0.5x–1.5x random factor

Only transient errors are retried: rate limits, server errors (5xx), timeouts, network failures, and stream interruptions. Permanent errors like authentication failures or invalid requests fail immediately. If the provider returns a Retry-After header, Fabro respects it — unless the delay exceeds 60 seconds, in which case the call fails rather than blocking the run.

Turn-level retries

When an LLM stream drops mid-response (common under high concurrency), Fabro retries the same agent turn up to 3 times instead of failing the entire session. Conversation history is preserved across retries, and any partial assistant output from the interrupted stream is replayed so the model can continue where it left off. This avoids restarting the full stage from scratch for transient stream failures.

Node retries

When a node handler fails (after LLM and turn-level retries are exhausted), the engine can retry the entire node. This is controlled by retry policies.

Retry policies

Set a retry policy on a node with the retry_policy attribute:

implement [retry_policy="standard"]

Policy	Max attempts	Initial delay	Backoff	Typical delays
`none`	1	—	—	No retries
`standard`	5	200ms	2x exponential	200ms, 400ms, 800ms, 1.6s
`aggressive`	5	500ms	2x exponential	500ms, 1s, 2s, 4s
`linear`	3	500ms	1x (constant)	500ms, 500ms
`patient`	3	2s	3x exponential	2s, 6s

All policies apply random jitter (0.5x–1.5x) and cap individual delays at 60 seconds.

Setting retries without a policy

You can also set just the retry count using max_retries:

implement [max_retries="5"]

This uses the default backoff (5s initial, 2x exponential) with the specified number of retries.

Resolution order

The engine resolves retry configuration in this order:

Node attribute retry_policy — named preset
Node attribute max_retries — count only, default backoff
Graph attribute default_max_retries — applies to all nodes without explicit config (default: 3)

What gets retried

Not all errors trigger a node retry. The handler’s should_retry check must return true — generally, only errors classified as transient are retried. Deterministic errors (auth failures, bad config) fail immediately without consuming retry attempts. When a handler reports a retryable failure, retries always proceed if attempts remain. If retries are exhausted and the node has allow_partial=true, the outcome is promoted to partially_succeeded instead of failed. See Node Outcomes — Retry loop for a detailed flow diagram.

Model fallbacks

When a model provider fails with a provider-local error, Fabro can automatically switch to another target. Configure one fixed chain for each requested model in your run configuration:

run.toml

[run.model]
name = "claude-opus-4-6"
provider = "anthropic"

[run.model.fallbacks]
"claude-opus-4-6" = ["gemini", "openai"]

When Anthropic fails, Fabro tries Gemini first, then OpenAI. Fallback resolution is provider-aware:

A bare provider token such as "gemini" selects that provider’s closest compatible model.
A qualified selector such as "openrouter:gpt-56-sol" resolves only within that provider. The selector may be a canonical model ID, alias, or provider API ID such as "openrouter:moonshotai/kimi-k3".
A bare model slug or alias considers ready providers and uses provider priority.

Qualified fallback references always remain provider pins. For example, "openai:gpt-5.6-sol" pins the direct OpenAI offering. Legacy provider/model fallback references remain accepted for compatibility. Fabro selects the chain by the original requested model. A target in that chain never activates the target model’s own chain. The same chain position is retained across structured-output repairs and cached agent sessions. The primary provider and model were already resolved and persisted when the run was created; resuming does not re-run primary selection. Fallbacks are only considered after an eligible runtime failure. If the fallback model does not support the requested reasoning level, Fabro uses the nearest supported level and rounds equal-distance choices up.

What triggers failover

Failover is a superset of LLM retry eligibility:

Error type	LLM retry	Provider failover
Rate limit	Yes	Yes
Server error (5xx)	Yes	Yes
Timeout / network	Yes	Yes
Quota exceeded	No	Yes
Authentication (401)	No	Yes
Access denied (403)	No	Yes
Model not found (404)	No	Yes
Model refusal	No	Yes
Invalid request (400)	No	No
Context length (413)	No	No
Content filter	No	No

Quota errors are the key distinction — they aren’t retried against the same provider (the quota won’t reset) but are eligible for failover to a provider with its own quota.

Loop detection

Fabro has two independent mechanisms for detecting stuck loops: node visit limits that catch workflow-level cycles, and tool call pattern detection that catches agent-level repetition.

Node visit limits

The max_node_visits graph attribute sets the maximum number of times any single node can execute before the run is terminated:

example.fabro

digraph Example {
    graph [max_node_visits="20"]
    // ...
}

Context	Default
Normal runs	Disabled (unlimited)
Dry runs (`--dry-run`)	10
Explicit `max_node_visits`	The configured value

When a node hits the limit, the run fails immediately:

node "verify" visited 20 times (graph limit 20); run is stuck in a cycle

Per-node overrides

You can set max_visits on individual nodes to override the graph-level limit for that node:

example.fabro

digraph Example {
    graph [max_node_visits="20"]
    fix [max_visits=3]
}

The per-node max_visits takes precedence over max_node_visits (and the dry-run default of 10). This is useful when specific nodes — like a fix-and-verify loop — should have a tighter limit than the rest of the workflow:

node "fix" visited 3 times (node limit 3); run is stuck in a cycle

Tool call loop detection

Inside an agent session, Fabro monitors the last 10 assistant turns for repeating tool call patterns. It detects patterns of length 1 (same call repeated), 2 (A-B-A-B), or 3 (A-B-C-A-B-C). Every complete group in the window must match for detection to trigger. When a loop is detected, Fabro injects a steering message into the conversation:

WARNING: Loop detected. You appear to be repeating the same tool calls. Please try a different approach or ask for clarification.

This gives the agent a chance to break out of the loop without failing the node.

Failure signatures and circuit breakers

Failure signatures are a deduplication mechanism that prevents the same failure from recurring indefinitely across loop iterations. They are particularly important for workflows with retry loops (implement → verify → fix → verify → …).

How signatures work

After each failed node, Fabro constructs a failure signature — a normalized fingerprint combining the node ID, failure class, and error message:

implement|deterministic|handler panicked: index out of bounds

The error message is normalized by lowercasing, replacing hex strings with <hex>, replacing digits with <n>, and truncating to 240 characters. This groups failures with the same root cause even when details like line numbers or timestamps vary.

Circuit breaker

Fabro tracks signature counts across the run. When the same signature repeats 3 times (configurable via loop_restart_signature_limit), the run is terminated:

deterministic failure cycle detected: signature ... repeated 3 times (limit 3)

Only deterministic and structural failures are tracked — transient failures are excluded because they may genuinely resolve on retry.

Failure signature counts are never reset on success. This is intentional — it prevents cycles like “implement succeeds → verify fails → fix → implement succeeds → verify fails” from running indefinitely.

Loop restart edges

Taking an edge marked with loop_restart=true restarts the workflow from the edge’s target node. A restart is more than a jump: the completed-stage history, per-node outcomes, and retry counts are cleared, and the run context is replaced with a fresh, empty context — the target node starts over as if the run had just begun there, with no preamble of prior stages. Node visit counts are the one thing preserved, so max_visits and max_node_visits still bound how many times a restart loop can run. A successful outcome may take a loop_restart edge freely. This is the “start another round from a clean slate” pattern — for example, a self-loop that begins a fresh batch of work and re-derives its remaining work from the repository state rather than from accumulated context. A failed outcome faces an additional guard: only transient_infra failures may cross a loop_restart edge. If the failure class is anything else, the run is terminated:

loop_restart blocked: failure_class=deterministic (requires transient_infra)

Loop restart edges also have their own separate circuit breaker (restart_failure_signatures) that enforces the same signature limit.

Goal gates

Goal gates are quality checkpoints that are enforced when the workflow reaches an exit node. A node marked with goal_gate=true must have completed with succeeded or partially_succeeded — otherwise the run cannot finish. See Node Outcomes — Goal gate interaction for how partially_succeeded and allow_partial interact with goal gates.

verify [shape=box, goal_gate="true"]

When a goal gate is unsatisfied at the exit node, Fabro looks for a retry target — a node to jump back to for another attempt:

Failed node’s retry_target attribute
Failed node’s fallback_retry_target attribute
Graph-level retry_target attribute
Graph-level fallback_retry_target attribute

example.fabro

digraph Example {
    graph [retry_target="plan"]
    verify [shape=box, goal_gate="true", retry_target="implement"]
    // If verify fails, jump to implement
    // If no node-level target existed, would jump to plan
}

If no retry target is found at any level, the run fails:

goal gate unsatisfied for node verify and no retry target

Stall watchdog

Fabro runs a background watchdog that monitors event activity. If no events are emitted for longer than the stall timeout, the run is canceled. This catches cases where a handler hangs indefinitely without producing errors.

Setting	Default
`stall_timeout`	1800 seconds (30 minutes)
Set to `0`	Disables the watchdog

example.fabro

digraph Example {
    graph [stall_timeout="300"]  // 5 minutes
}

When failures become fatal

A node failure does not automatically terminate the run. Fabro follows this escalation path:

LLM retries — transient API errors are retried inside the model call (up to 3 retries)
Turn-level retries — dropped streams retry the same agent turn (up to 3 retries), preserving conversation history
Provider failover — if configured, switch to a fallback provider
Node retries — re-execute the entire handler (per the retry policy)
Direct jump — use jump_to_node when the outcome supplies one
Explicit edge routing — look for a matching condition, preferred label, or suggested next node
Failure policy — with no explicit route, apply the effective on_failure (node-level on_failure first, then graph-level): exit skips the unconditional edge, succeed promotes the outcome to succeeded and routes it as a success, and route (or no attribute) keeps normal fallback routing
Unconditional edge — in route mode, or after a succeed promotion, use an edge without a condition as the fallback
Retry target — if no edge was selected, check retry_target and fallback_retry_target on the node, then on the graph
Run failure — if none of the above produces a path forward, the run terminates

When a retry target sends the run back to a failing path, use graph-level max_node_visits or node-level max_visits to stop an unbounded cycle. The run also terminates immediately for:

Node visit limit exceeded — a node has been visited too many times
Circuit breaker tripped — the same failure signature has repeated too many times
Loop restart blocked — a non-transient failure tried to cross a loop_restart edge
Goal gate failure with no retry target — a required gate was unsatisfied at the exit node
Stall timeout — no events for too long
Cancellation — user or system cancel signal

Getting Started

Core Concepts

Defining Workflows

Executing Workflows

Human Tools

Agents

Integrations

Reference

Failure classification

Human gates fail closed when unanswered

Stop linear workflows on failure

Treat a failed node as succeeded

Retry layers

LLM retries

Turn-level retries

Node retries

Retry policies

Setting retries without a policy

Resolution order

What gets retried

Model fallbacks

What triggers failover

Loop detection

Node visit limits

Per-node overrides

Tool call loop detection

Failure signatures and circuit breakers

How signatures work

Circuit breaker

Loop restart edges

Goal gates

Stall watchdog

When failures become fatal

​Failure classification

​Human gates fail closed when unanswered

​Stop linear workflows on failure

​Treat a failed node as succeeded

​Retry layers

​LLM retries

​Turn-level retries

​Node retries

​Retry policies

​Setting retries without a policy

​Resolution order

​What gets retried

​Model fallbacks

​What triggers failover

​Loop detection

​Node visit limits

​Per-node overrides

​Tool call loop detection

​Failure signatures and circuit breakers

​How signatures work

​Circuit breaker

​Loop restart edges

​Goal gates

​Stall watchdog

​When failures become fatal

Failure classification

Human gates fail closed when unanswered

Stop linear workflows on failure

Treat a failed node as succeeded

Retry layers

LLM retries

Turn-level retries

Node retries

Retry policies

Setting retries without a policy

Resolution order

What gets retried

Model fallbacks

What triggers failover

Loop detection

Node visit limits

Per-node overrides

Tool call loop detection

Failure signatures and circuit breakers

How signatures work

Circuit breaker

Loop restart edges

Goal gates

Stall watchdog

When failures become fatal