Crash Recovery & Safe Mode

Safe Mode

Safe mode activates on its own when Daintree detects a crash loop. There's no manual trigger and no menu entry for it.

What Triggers It

Daintree counts unclean exits in a sliding 30-minute window. The state lives in crash-loop-state.json under user data. Three unclean exits inside that window flip the next launch into safe mode. Five in the same window go a step further: auto-relaunch is disabled entirely, so the app won't try to come back on its own after a crash. You launch it yourself.

The window genuinely slides. Old launches expire lazily on each new launch, so a single crash followed by 30 minutes of clean operation never builds toward safe mode. If crash-loop-state.json itself is unreadable, Daintree moves it aside as crash-loop-state.json.corrupted.{timestamp} (owner-only permissions, up to three siblings retained), resets the counter, and boots normally.

The Safe Mode Banner

Safe mode pins an inline warning banner to the top of the app:

"Safe mode: panels weren't restored"

Beneath the title is a crash-count meta line in the form "3 crashes detected, last 2m ago". The right side of the banner carries two controls: a Show details popover button and a Restart normally primary action. The X icon at the far right dismisses the banner for the current session only. It comes back on the next boot until you restart normally.

Show Details Popover

Clicking Show details opens a popover listing every panel Daintree quarantined as a likely culprit. Each row shows the panel title and its working directory or worktree ID, with a per-row Restore panel button. The popover also repeats the crash-count meta line and, when relevant, a short note on what's been deferred.

Restoring a panel from the popover is a next-launch operation. The panel is dropped from the quarantine ledger so it comes back on the next boot, but it doesn't re-hydrate in the current session. Each row moves through visible states (idle → clearing → cleared or failed) so you can tell whether the request landed.

If there are no quarantined panels but Daintree still skipped some during boot, because they weren't safe to restore in a crash-loop state, the popover shows a simpler message: a count of skipped panels and a prompt to use Restart normally to reload them.

Restart Normally

Restart normally opens a destructive confirm dialog:

• Title: "Restart Daintree normally?"
• Description: "All running terminals and agent sessions will be killed. Scrollback and in-flight agent work will be lost."
• Confirm: Restart normally (destructive variant)

A small View logs text link sits at the bottom of the dialog. It opens the current log file in your system editor so you can check what's been happening before you commit to the restart. The link is in this dialog only. There's no separate View logs action on the banner itself.

Returning to Normal Without Restart

You don't have to use Restart normally to get out of safe mode. The 30-minute window decays on its own. Once the most recent unclean exit ages out, the next clean launch comes up without the banner. If you've been running stably for the rest of the window, relaunching the app eventually does the right thing.

Crash Recovery

Daintree detects unclean exits with a marker file. running.lock is written on launch with a JSON payload (session start time, app version, platform, crash log path, heartbeat timestamp, suspend-start timestamp), refreshed on every backup tick, and removed on a clean shutdown. If the marker is still there at the next start, Daintree knows the previous session ended unexpectedly and runs the recovery flow.

Session backups capture your panel list, working directories, last agent state per panel, and window layout. They write to disk on three triggers: a 60-second periodic timer, window blur (100 ms debounced so dragging the window doesn't cause write storms), and scheduled state changes from the app store (1.5 seconds debounced). The service is sleep/wake aware. Suspend stops the timer and stamps the marker; wake clears the suspend and restarts the timer.

Backups land in backups/session-state.json, with the previous generation rotated to backups/session-state.previous.json before each new write. That rolling pair means a corrupt write doesn't destroy your only snapshot. The payload is small: a capturedAt timestamp plus the serialized app and window state.

Crash Classification

Daintree makes a best-effort attribution of why the last session ended, and says so in plain language rather than showing you an enum. Six causes are checked in priority order:

These are heuristics, and the page is honest about their limits. Power-loss detection is reliable in the positive direction only: sleep pauses the uptime clock, so the absence of the signal doesn't prove the machine didn't reboot.

Separately, if watchdog-kill.flag is present and fresh, the crash log carries a precise watchdog-deadlock annotation that overrides the heuristic, along with the killed PID, the missed-heartbeat count, and the kill timestamp. That one isn't a guess. See Crash Watchdog.

Recovery Dialog

The recovery dialog opens before the workspace loads. Its title is the cause headline from the table above, followed by the one-line description and the date the previous session ended. When a backup is available its timestamp shows below the body.

The panel list is a scrollable set of rows. Each shows the panel kind icon, the panel title, the working directory, the last known agent state, and a checkbox. A Select all / Deselect all toggle handles bulk selection and the header tracks "X of N selected".

Panels created within 30 seconds of the crash are tagged as suspects and marked with an AlertTriangle. The rule is deliberately crude: a panel that came into existence right before the crash is the most likely thing to have caused it. Once a crash has been counted, suspects are pre-deselected and the inline banner reads "{N} panels deselected, created shortly before the crash". On a first crash they're highlighted but left selected, and the banner reads "{N} panels created shortly before the crash".

Two actions sit at the bottom:

• Restore selected (N): restores only the checked panels.
• Continue without restoring: discards the panel list and comes up clean.

If the backup has no panel list, the dialog collapses to a two-button layout: Restore previous session (with the backup date, or a warning when the backup is missing) and Continue without restoring.

A restore that fails is reported as a failure. It is not treated as a success and silently swallowed, which would leave you looking at an empty workspace with no explanation.

Continue Without Restoring

Because this throws work away, it routes through a second destructive confirm before anything is discarded:

• Title: "Reset to clean layout?"
• Description (with panels): "All open panels listed below will be closed and their state will be discarded."
• Description (no panels, backup present): "Your session will start with a clean layout and the existing session backup will be discarded."
• Confirm: Reset to clean layout (destructive variant)

When panels are being discarded the confirm lists them inline, so you know exactly what's going.

Auto-Restore

Auto-restore is on by default. The recovery dialog shows a switch labeled "Restore automatically next time" with the subtitle "Skips this dialog. Shows again if Daintree crashes twice in a row." With it on, a single unclean exit silently restores every panel and shows a confirmation banner instead of the dialog.

After two consecutive crashes it pauses itself, because at that point silently restoring the same layout is more likely to reproduce the crash than to recover from it. The switch is replaced by static text: "Auto-restore paused: too many consecutive crashes." Once the crash counter decays, the switch becomes interactive again.

To change it outside the dialog, go to Settings → Troubleshooting and toggle Auto-restore on crash.

Silent Auto-Restore Banner

When auto-restore handles a crash silently, an inline banner confirms it. Two forms:

• No suspects: "Session recovered after unexpected exit." Auto-dismisses after 10 seconds.
• With suspects: "Session recovered after unexpected exit: N panels created near the crash may be affected." Stays until you dismiss it.

The sticky variant gives you time to deal with the suspect panels before the notice disappears. The auto-dismissing one is a low-noise confirmation that recovery happened at all.

Error Details, the Action Trail, and Crash Reporting

The recovery dialog has a collapsible Error details section. Expanding it shows the context captured at crash time: app version, platform and architecture, OS version, session duration, Electron version, free and total memory, panel count, process uptime, and, when available, the error message and stack trace.

Below that is the pre-crash action trail: "Recent actions (N)", newest first. Daintree keeps a rolling breadcrumb of dispatched actions, and the last ten are written onto the crash log entry. This is usually the fastest route to a reproduction: not "it crashed", but "it crashed right after a rebase and a panel close".

Three buttons sit alongside:

• Open log file: opens the crash log in your system editor.
• Copy stack: copies the stack trace.
• Report this crash: opens an editable preview of the report.

Report this crash doesn't submit anything on the first click. It expands a textarea holding the complete report body (including the action trail rendered as a markdown table) with the note that the report is redacted and will be publicly visible on GitHub. Edit it, then submit. If the report is too long to fit in a GitHub URL, Daintree says so and copies it to your clipboard for pasting instead.

Crash Watchdog

Daintree runs an external watchdog in a separate Electron utility process. It has no UI of its own. Its job is to notice when the main process has frozen and force-kill it, so the recovery flow can take over on the next launch. The threshold is conservative on purpose: a synchronous garbage collection or a long native-module operation can pause the main process for several seconds, and the watchdog has to avoid mistaking that for a deadlock.

The watchdog expects a heartbeat every 5 seconds. Miss three in a row (roughly 15 seconds of an unresponsive main process) and it writes watchdog-kill.flag synchronously and sends SIGKILL to the main PID. On wake from sleep, a monotonic-clock grace window suppresses queued ticks so a burst of catch-up heartbeats can't trigger a false positive.

On the next launch, Daintree consumes the flag and annotates the crash as watchdog-deadlock with the killed PID, missed-beat count, and kill timestamp. The flag is then unlinked unconditionally, so a stale file from a long-ago kill can't poison later launches.

When the Watchdog Disables Itself

If the watchdog process can't stay running (it crashes repeatedly, or its IPC channel won't come back), Daintree stops restarting it and shows a warning banner:

• Title: "Crash watchdog disabled"
• Description: "Deadlock detection stopped after {N} restart attempts. Restart it to resume monitoring."
• Action: Restart watchdog, which relaunches the process. The banner clears once the restart succeeds.

Banners Queue Rather Than Stack

Daintree can be in more than one of these states at once, and a column of stacked warning banners would bury the one that matters. A coordinator arbitrates a single global banner slot in a fixed order of precedence:

1. Host crash: the terminal backend is unusable right now.
2. Watchdog disabled: the protection layer is down. Ranked above safe mode because the watchdog guards against the next crash, while safe mode is a consequence of the last one.
3. Safe mode: panels weren't restored after a crash loop.
4. Restore confirmation: the "session recovered" notice.
5. Missing prerequisite: a required tool like Git or Node isn't installed.
6. Forge token: a provider's credentials expired.
7. Cloud sync: the project sits in a synced folder.
8. Rosetta: an x64 build being translated on Apple Silicon.

Only the highest-priority applicable banner renders. The ordering runs from "something is broken now" down through "something will bite you later" to "this is environmental and permanent".

Terminal Service Crashes

Daintree's pty-host runs as a separate Electron utility process. Every terminal panel sits on top of it: agent panels and plain shells alike. When the pty-host exits, terminal I/O stops until it comes back. The rest of the app keeps running: review workspaces, browser panels, and settings are unaffected.

The pty-host is health-checked every 5 seconds. Three missed heartbeats and Daintree force-kills it rather than leaving you staring at frozen terminals, roughly 20 seconds worst-case detection. Restarts use full-jitter exponential backoff with a 100 ms floor and a 10-second cap, budgeted against the same 3-crashes-in-30-minutes window the main crash-loop guard uses. Past that budget, automatic restarts stop and the banner switches to manual mode.

Backend Status Banner

Three states:

• connected: no banner, normal operation.
• recovering: a spinner banner titled "Terminal service restarting" with the description "The terminal backend stopped and is restarting automatically." A 400 ms threshold gates it, so a sub-second blip never flashes UI at you.
• disconnected: an error banner with crash-type-specific copy, a Restart service button, and a Send diagnostics action that collects a bundle for the report.

Crash-Type Variants

Once the restart budget is exhausted, the banner copy depends on how the pty-host exited. Classification prefers the authoritative process-gone reason Electron reports, falling back to exit-code heuristics (137 = OOM or SIGKILL, 134 = SIGABRT, and so on).

Workspace Host Crashes

The workspace host is a separate utility process, distinct from the pty-host above. It owns git state, file watching, and worktree operations for an open project. It runs out of the renderer, so a crash there doesn't take down the UI, but worktree updates stall until it's back.

When it exits unexpectedly, Daintree surfaces the crash to the sidebar within about 10 milliseconds. A Reconnecting… label with a spinning icon replaces the Worktrees header while the host restarts in the background. You don't need to do anything. Daintree retries up to three times with exponential backoff (roughly 2, 4, and 8 seconds) and the spinner clears once a fresh snapshot arrives.

If all three retries fail, the sidebar shows a terminal error: "Workspace host crashed and could not recover. Please restart Daintree." Restart the app. If it repeats on every launch, capture a diagnostics bundle and file a bug with it attached.

Crash Files and Session Backups

The session backup captures your panel list, working directories, last agent state per panel, and window layout. That's the data the recovery flow restores from.

Crash logs, backups, and flag files live in the Daintree user data directory:

The files that matter inside it:

• running.lock: the crash-detection marker. Present while Daintree is running. Holds session start time, app version, platform, crash log path, and a heartbeat refreshed on every backup tick.
• crashes/: crash logs in JSON. Up to 10 retained; older ones are pruned automatically.
• backups/session-state.json: the most recent session backup.
• backups/session-state.previous.json: the previous generation, rotated before each new write. The rolling pair keeps a corrupt write from destroying your only snapshot.
• crash-loop-state.json: counter state for the safe-mode guard.
• watchdog-kill.flag: written when the watchdog SIGKILLs the main process. Consumed on the next launch to attribute the crash, then unlinked.
• gpu-disabled.flag: hardware acceleration off. Records whether it was written by a crash or by you, which decides whether it's retried after an update. See Hardware Acceleration.
• gpu-angle-fallback.flag: the Linux + Wayland soft fallback. Present until you toggle hardware acceleration off and back on, or until the full disable takes over and clears it.
• *.corrupted.{timestamp}: quarantined files. When a JSON store fails to parse or carries a future schema version, Daintree moves it aside with this suffix and resets the original. Pruned automatically after 30 days. There's no UI for these; you'd only notice by browsing user data yourself.

For how Daintree persists session state during normal operation, see Session Management.