Skip to content

Emit a JSON error document when --json is active - #8182

Draft
amcaplan wants to merge 2 commits into
mainfrom
render-errors-as-json
Draft

Emit a JSON error document when --json is active#8182
amcaplan wants to merge 2 commits into
mainfrom
render-errors-as-json

Conversation

@amcaplan

Copy link
Copy Markdown
Contributor

WHY are these changes introduced?

--json was never considered by the global error renderer. Because the human error banner is written to stderr, a failing shopify <command> --json | jq previously received nothing on stdout — not malformed output, but no output at all. Scripts had no parseable failure signal.

This routes fatal errors through a JSON serializer when JSON output is active, emitting exactly one document on stdout.

WHAT is this pull request doing?

Shape

{"error":{"type":"abort","message":"Couldn't find an app","tryMessage":"Run shopify app init","nextSteps":["Visit the docs"],"customSections":[{"title":"Extensions","body":[["name","status"],["my-ext","failed"]]}]}}
  • {"error": ...} envelope. Successful --json payloads in this CLI are bare values — theme list and app versions list emit arrays, theme push and store list emit command-shaped objects. A top-level error key is therefore unambiguous, and no existing success payload uses one. It also matches oclif's and npm's convention.
  • type is a closed set: abort, bug, external. Machine-stable by design (see below).
  • Absent fields are omitted, never null — consumers use presence checks, consistent with existing JSON.stringify/conditional-spread practice in the repo.
  • stack only appears for bug, the one type that asks the user to file a report.
  • No exitCode field. errorMapper constructs a fresh AbortError that drops oclif.exit, so any code in the payload could contradict the real status. The process exit status is the contract.
  • Every nested TokenItem is flattened to a plain string with ANSI stripped; no Ink/React props reach the payload.

Why the discriminator is a string literal

Not constructor.name and not error.name. packages/cli/bin/bundle.js builds the published npm bundle with minifyIdentifiers: true and no keepNames, so class AbortError becomes e=class extends s{} — any reflective name is garbage in production. (error.name is worse: no error class in error.ts assigns it, and ES subclasses don't inherit it, so it is "Error" for every type.) Each class carries an explicit literal, backed by a satisfies Record<FatalErrorType, JsonErrorType> map so adding a FatalErrorType member fails the build. A test transforms error.ts with minifyIdentifiers: true and asserts every literal survives.

Known approximations

  • Detection is argv sniffing, not oclif parsing. sniffForJson inspects process.argv before oclif parses, because the error path must work even when parsing itself failed. It respects the -- passthrough boundary and handles clustered short flags. It cannot model per-flag arity, so --path --json (where --json is consumed as another flag's value) is a false positive. Fixing that means reimplementing oclif's parser in the error path; not worth the fragility.
  • theme preview -j is a permanent false positive — that command declares -j for an unrelated flag. Blast radius is limited to printing a JSON error instead of a banner on a failing theme preview.
  • renderConcurrent writes Ink frames to stdout. No command pairs it with --json today (app build/app deploy declare no json flag); only theme-command.ts's multi-environment runner remains, where output is already interleaved across environments. Gating the progress renderers on JSON mode is a sensible hardening follow-up, not a fix this change needs. renderTasks already defaults to stderr.

Review provenance

Three independent reviews on three different models audited this before submission (control flow, public contract, test integrity). They produced one blocker and several fixes, all addressed: the explicit discriminator is now runtime-validated so an unadvertised value from a foreign cli-kit copy cannot reach the wire; FatalErrorType's numeric values are pinned as append-only since they act as a cross-copy wire format; abortSilent can no longer appear in an emitted document's type; and two false-green tests were rewritten to be load-bearing and mutation-verified.

Follow-ups (deliberately out of scope)

  • Both uncaughtException handlers (bootstrap.ts, index.ts) still render human output.
  • Render-and-continue sites in theme utilities and plugin-did-you-mean are ungated.
  • Gate progress renderers (renderConcurrent) on JSON mode.
  • launchCLI double-prints errors (missing skipOclifErrorHandling).
  • error-grouping.ts keys Bugsnag's groupingHash on constructor.name, which minification mangles.
  • theme preview declares json without char: 'j', inconsistent with the global flag.

How to test your changes?

Testing note — an honest limitation

Stream identity is not directly assertable in this harness: shouldOutput() returns false under isUnitTest(), so nothing reaches a real stream, and collectLog keys captured output by log level rather than by stream — so outputResult (stdout) and outputInfo (stderr) both land under info and the captured log cannot tell them apart.

What the suite asserts instead is the logger reference: outputResult delegates to the private output() with consoleLog (process.stdout.write), while outputInfo uses consoleWarn and never calls output() at all. Asserting that call is what catches a switch between the two. That is the closest a unit test here can get to observing stream identity; verifying real file-descriptor separation needs an integration-level test.

This distinction is not academic. The original version of that test asserted only that captured output was non-empty — and mutating outputResult to outputInfo left 27 of 28 tests in the file green. Only the rewritten test catches it.

Testing

  • Four affected test files: 117 passed (117), exit 0
  • Full cli-kit suite: 146 files, 1735 passed | 1 skipped, exit 0
  • Type-check: clean (verified with --skip-nx-cache, since a cached pass would prove nothing)
  • Lint: pnpm eslint src clean, 0 errors 0 warnings
  • The four test-integrity fixes were mutation-verified: each mutation was applied, observed to fail with a recorded message, and reverted. Six mutations total, all six targets confirmed back in their final state.

Checklist

  • I've considered possible cross-platform impacts (Mac, Linux, Windows)
  • I've considered possible documentation changes
  • I've considered analytics changes to measure impact
  • The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

Failing commands rendered only a human error banner, which goes to
stderr, so `shopify <command> --json | jq` received nothing parseable on
stdout. Route fatal errors through a JSON serializer when JSON output is
active, emitting a single `{"error": {...}}` document on stdout.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
@github-actions github-actions Bot added the Area: @shopify/cli @shopify/cli package issues label Jul 28, 2026
The example showed `stack` plus `command`/`args` on an `abort` document,
but `stack` is emitted only for `bug` and `command`/`args` come from
`ExternalError`. Split it into one example per type so no example shows a
combination a real document cannot produce.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Differences in type declarations

We detected differences in the type declarations generated by Typescript for this branch compared to the baseline ('main' branch). Please, review them to ensure they are backward-compatible. Here are some important things to keep in mind:

  • Some seemingly private modules might be re-exported through public modules.
  • If the branch is behind main you might see odd diffs, rebase main into this branch.

New type declarations

packages/cli-kit/dist/private/node/json-error.d.ts
import { TokenItem } from './ui/components/TokenizedText.js';
import type { EmittedJsonErrorType, JsonErrorType } from '../../public/node/error.js';
import type { AlertCustomSection } from '../../public/node/ui.js';
/**
 * A custom section flattened for JSON output. `body` is a plain string for token bodies and
 * a row-major matrix of strings for tabular ones.
 */
interface JsonErrorCustomSection {
    title?: string;
    body: string | string[][];
}
/**
 * The `error` payload of the JSON error document.
 *
 * Absent fields are omitted rather than emitted as `null`, so consumers can test with a
 * simple presence check. `type` and `message` are always present.
 */
interface JsonErrorPayload {
    type: EmittedJsonErrorType;
    message: string;
    tryMessage?: string;
    nextSteps?: string[];
    customSections?: JsonErrorCustomSection[];
    stack?: string;
    command?: string;
    args?: string[];
}
/**
 * The document written to stdout when a command fails with JSON output active.
 *
 * The single `error` key is what lets a script tell failure from success: successful
 * payloads in this CLI are bare values (arrays, or objects shaped by the command), so a
 * bare error object would force consumers to duck-type. This also matches oclif's and
 * npm's own JSON error shape.
 */
export interface JsonErrorDocument {
    error: JsonErrorPayload;
}
/**
 * The subset of `FatalError` this module reads, with every field optional.
 *
 * Deliberately duck-typed rather than typed as `FatalError`: `isFatal` in `error.ts` also
 * duck-types, so errors reaching us may come from a different copy or version of cli-kit
 * and may not carry every field.
 */
interface FatalErrorLike {
    message?: string;
    type?: number;
    jsonErrorType?: JsonErrorType;
    tryMessage?: TokenItem | null;
    nextSteps?: TokenItem[];
    customSections?: AlertCustomSection[];
    stack?: string;
    command?: string;
    args?: string[];
}
/**
 * Flattens a fatal error into the JSON document emitted when `--json` is active.
 *
 * Every nested `TokenItem` is reduced to a plain string with ANSI stripped; raw Ink/React
 * props never reach the payload. No exit code is included: `errorMapper` builds a new
 * `AbortError` that drops `oclif.exit`, so any code we read here could disagree with the
 * status the process actually exits with. The process exit status is the contract.
 *
 * @param error - The fatal error to flatten.
 * @returns The document to write, or `undefined` when the error is intentionally silent.
 */
export declare function fatalErrorToJsonDocument(error: FatalErrorLike): JsonErrorDocument | undefined;
/**
 * Writes the JSON error document for a fatal error to stdout.
 *
 * Goes through `outputResult` rather than `process.stdout.write` for two reasons: it is the
 * only stdout writer in the `output` family, so this matches how successful `--json`
 * payloads are emitted; and it is the only route the `mockAndCaptureOutput` test harness
 * can observe.
 *
 * @param error - The fatal error to write.
 */
export declare function renderFatalErrorAsJson(error: FatalErrorLike): void;
export {};

Existing type declarations

packages/cli-kit/dist/public/node/error.d.ts
@@ -2,11 +2,73 @@ import { OutputMessage } from './output.js';
 import { InlineToken, TokenItem } from '../../private/node/ui/components/TokenizedText.js';
 import type { AlertCustomSection } from './ui.js';
 export { ExtendableError } from 'ts-error';
+/**
+ * How the program should behave when a `FatalError` reaches the top-level handler.
+ *
+ * The values are written out explicitly because they are effectively a cross-version wire
+ * format: `resolveJsonErrorType` reads this number off errors that may have been built by a
+ * different copy of cli-kit (see `bin/bundling/esbuild-plugin-dedup-cli-kit.js`), so a given
+ * number has to keep meaning the same thing across versions. This list is append-only: add
+ * new members with new values, and never reorder or renumber the existing ones.
+ */
 export declare enum FatalErrorType {
     Abort = 0,
     AbortSilent = 1,
     Bug = 2
 }
+/**
+ * Every `JsonErrorType`, as a runtime value.
+ *
+ * `JsonErrorType` is derived from this array rather than declared separately so that the
+ * compile-time union and the runtime allow-list in `isJsonErrorType` cannot drift apart.
+ */
+declare const jsonErrorTypes: readonly ["abort", "abortSilent", "bug", "external"];
+/**
+ * Stable, machine-readable classification of a fatal error, used to derive the `error.type`
+ * field of the JSON error document produced when `--json` is active.
+ *
+ * Not every member reaches the wire: see `EmittedJsonErrorType` for the subset that can
+ * actually appear in a document.
+ *
+ * These are string literals rather than class or enum names on purpose: the published npm
+ * bundle is built with `minifyIdentifiers: true` (see `packages/cli/bin/bundle.js`), which
+ * rewrites `constructor.name` to a single letter that changes between builds. String
+ * literals survive minification.
+ *
+ * `FatalErrorType` on its own is too coarse to use here, because `AbortError` and
+ * `ExternalError` both carry `FatalErrorType.Abort`.
+ */
+export type JsonErrorType = (typeof jsonErrorTypes)[number];
+/**
+ * The `JsonErrorType` values that can appear as `error.type` in an emitted document.
+ *
+ * `abortSilent` is an internal classification only: `AbortSilentError` exists to terminate
+ * the process without printing anything, so no document is emitted for it at all. Excluding
+ * it here keeps the emitted discriminator a closed union that consumers can switch on
+ * exhaustively without handling a value they can never receive.
+ */
+export type EmittedJsonErrorType = Exclude<JsonErrorType, 'abortSilent'>;
+/**
+ * The fields `resolveJsonErrorType` needs, both optional.
+ *
+ * `isFatal` duck-types on the presence of `type` rather than using `instanceof`, so an error
+ * can reach us having been built by a different copy of cli-kit (see
+ * `bin/bundling/esbuild-plugin-dedup-cli-kit.js`) that predates `jsonErrorType`, or carrying
+ * an enum member this version doesn't know about.
+ */
+interface JsonErrorTypeSource {
+    type?: FatalErrorType;
+    jsonErrorType?: JsonErrorType;
+}
+/**
+ * Resolves the JSON classification for a fatal error.
+ *
+ * Anything unrecognised is reported as a bug rather than silently mislabelled.
+ *
+ * @param error - The error to resolve a classification for.
+ * @returns The classification for the error.
+ */
+export declare function resolveJsonErrorType(error: JsonErrorTypeSource): JsonErrorType;
 export declare class CancelExecution extends Error {
 }
 /**
@@ -16,6 +78,7 @@ export declare class CancelExecution extends Error {
 export declare abstract class FatalError extends Error {
     tryMessage: TokenItem | null;
     type: FatalErrorType;
+    jsonErrorType: JsonErrorType;
     nextSteps?: TokenItem<InlineToken>[];
     formattedMessage?: TokenItem;
     customSections?: AlertCustomSection[];
packages/cli-kit/dist/public/node/path.d.ts
@@ -124,6 +124,12 @@ export declare function sniffForPath(argv?: string[]): string | undefined;
 /**
  * Returns whether the `--json` or `-j` flags are present in the arguments.
  *
+ * This is a deliberate approximation of oclif's own `jsonEnabled()`, which can't be called
+ * here because we run before parsing. It errs towards reporting JSON output as enabled: a
+ * `--json` consumed as the *value* of a preceding value-taking flag (`--path --json`) is
+ * counted as a request for JSON, because recognising it as a value would mean knowing every
+ * flag's arity, which is oclif's parser rather than a sniff.
+ *
  * @param argv - The arguments to search for the `--json` and `-j` flags.
  * @returns Whether the `--json` or `-j` flag is present in the arguments.
  */

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Area: @shopify/cli @shopify/cli package issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant