Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .github/workflows/.build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ on:
push:
branches:
- main
tags:
- 'v*'

env:
environment: ${{ inputs.environment || 'release' }}
Expand Down
3 changes: 3 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,9 @@

This file provides guidance to AI agents when working with code in this repository.

> Using the tool rather than working on it? To analyze a Unity build with UnityDataTool, see
> [Using UnityDataTool with an AI Agent](Documentation/agent-guide.md).

## Project Overview

UnityDataTools is a .NET 9.0 command-line tool for analyzing Unity build output (AssetBundles, Player builds, Addressables). It extracts data from Unity's proprietary binary formats into SQLite databases and human-readable text files. The tool showcases the UnityFileSystemApi native library and serves as both a production tool and reference implementation.
Expand Down
176 changes: 176 additions & 0 deletions Documentation/agent-guide.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,176 @@
# Using UnityDataTool with an AI Agent

AI agents (Claude Code, Codex, Cursor, and similar tools) are good at answering questions about a
Unity build when they are pointed at UnityDataTool: the `analyze` command turns the build output into
a SQLite database that an agent can query directly, and the other commands let it drill into
individual files and objects. This page is the recommended workflow, plus the handful of facts that
are not obvious from `--help` and tend to cost the most discovery time. It is written so it can be
pasted (or linked) into an agent's context, and it is just as useful for humans writing scripts.

## The core loop

1. **Analyze the build output into a database.** One build per database (see below).

```
UnityDataTool analyze /path/to/build -o Analysis.db
```

2. **Query the database.** Any SQLite client works. The `sqlite3` command-line shell is the most
convenient; if it is not installed, Python's standard library works without any extra setup:

```
sqlite3 Analysis.db ".mode column" "SELECT * FROM view_breakdown_by_type LIMIT 15;"
```

```
python -c "import sqlite3; [print(*r) for r in sqlite3.connect('Analysis.db').execute(\"SELECT type, count, pretty_size FROM view_breakdown_by_type LIMIT 15\")]"
```

3. **Start from the views, not the raw tables.** The database ships with views that join the
underlying tables into directly useful shapes. `object_view` (one row per object, with resolved
type/archive/file names and `pretty_size`) and `view_breakdown_by_type` (count and total size per
object type) answer most first questions. List everything that is available with:

```
sqlite3 Analysis.db "SELECT name FROM sqlite_master WHERE type = 'view' ORDER BY name;"
```

The full schema is documented in the [Analyzer database reference](analyzer.md); worked example
queries are in [Example usage of Analyze](analyze-examples.md).

4. **Drill down into specific files and objects.** Once a query has identified something
interesting, the other commands show the actual content:
* [`dump`](command-dump.md) prints an object's full serialized properties as text.
* [`serialized-file`](command-serialized-file.md) inspects a SerializedFile's header, object
list, and external references.
* [`archive`](command-archive.md) lists or extracts the contents of an AssetBundle or other
Unity archive.

## Facts that save time

**Two different ids.** `object_view` (and `objects`) has both an `id` and an `object_id` column, and
the two drill-down commands take different ones:

* `id` is a small sequential row number assigned by `analyze`, unique across the whole database.
`find-refs -i` takes this one.
* `object_id` is the object's serialized local file id (the `m_PathID` seen in references), a signed
64-bit value that is only unique within its SerializedFile. `dump -i` takes this one.

**One build per database.** `analyze` refuses input where two archives (or two standalone
SerializedFiles) have the same name, because queries would be ambiguous — this happens when a
directory contains several builds, or the same AssetBundles built for multiple targets. Analyze each
build into its own database and query them separately ([Comparing Builds](comparing-builds.md)
shows patterns for diffing them).

**TypeTrees are required.** Object contents can only be interpreted when the files contain TypeTree
metadata. AssetBundles include it by default; Player builds do not, so analyzing a Player build
typically reports `Files without TypeTrees` and those files contribute nothing to the database (a
run where every file is skipped produces a valid but empty database). To analyze Player data, build
with the `ForceAlwaysWriteTypeTrees` diagnostic switch — see
[Player Build Format](playerbuild-format.md).

**Empty names are normal.** Many object types (Transform, GameObject components in general) have no
`name`; `object_view.game_object` links components back to their named GameObject.

## Tracing why content is in the build

The `refs` table records every object-to-object reference found during analysis: `object` is the
referencing object's `id`, `referenced_object` is the referenced object's `id`, and `refs_view`
adds the property path and type as strings. Querying it directly answers "what references X":

```
sqlite3 Analysis.db "SELECT object, property_path FROM refs_view WHERE referenced_object = 140;"
```

Comment thread
SkowronskiAndrew marked this conversation as resolved.
Repeat the query with the returned `object` ids to walk further up the reference graph. Where such
a walk should stop — the objects that by themselves explain "this is why it is in the build" —
depends on the build type:

* **AssetBundles**: the `assetbundle_assets` table lists the assets explicitly assigned to each
AssetBundle. Reaching one of these explains the inclusion, and an object that is itself listed
there belongs in the build even when nothing references it.
* **Content directory builds**: `content_layout_loadable_objects_view` lists the loadable objects —
the build's entry points. This requires the `ContentLayout.json` in the analyze input; see
[ContentLayout in the Analyze Database](contentlayout-database.md).
* **Player builds**: there is no explicit asset list in the build output. The naming conventions of
the serialized files (`level0`, `sharedassets0.assets`, `resources.assets`, ...) tell you which
scene or shared pool pulled the content in; see [Player Build Format](playerbuild-format.md).

The [`find-refs`](command-find-refs.md) command automates walking these chains, but it is
experimental: it does not yet produce complete or clearly-explained results for all supported build
pipelines ([issue #121](https://github.com/Unity-Technologies/UnityDataTools/issues/121)). Prefer
the direct queries above.

## Worked example: diagnose an AssetBundle build

Given the question "what is in this AssetBundle build and does anything look wasteful?", analyze
the entire build output into one database:

```
UnityDataTool analyze /path/to/build -o Analysis.db
```

Then look at, in order:

```sql
-- What is in it, by type
SELECT * FROM view_breakdown_by_type;

-- The biggest individual objects
SELECT object_id, type, name, archive, pretty_size FROM object_view ORDER BY size DESC LIMIT 20;

-- Texture formats, sizes, and Read/Write flags
SELECT name, format, width, height, mip_count, rw_enabled, pretty_size FROM texture_view;

-- Meshes with Read/Write enabled
SELECT name, vertices, rw_enabled, pretty_size FROM mesh_view;

-- Objects that appear more than once (same name/type/size in different files)
SELECT * FROM view_potential_duplicates;
```

Interpretation notes: `rw_enabled = 1` on textures and meshes doubles their runtime memory cost and
is only needed when scripts access the data on the CPU. `view_potential_duplicates` compares
objects across the whole build (this is why the full analysis matters) and is only expected to have
results for AssetBundle builds: rows that span archives usually mean a shared dependency was not
assigned to a common AssetBundle, so it was duplicated into each AssetBundle that needs it. To see
a suspicious object in full, dump it from the file the query reported:

```
UnityDataTool dump /path/to/build/some.bundle -i <object_id> --stdout
```
Comment thread
SkowronskiAndrew marked this conversation as resolved.

## Worked example: why is this one AssetBundle so large?

When the question is about a single AssetBundle on its own, it can be analyzed in isolation into a
throwaway database (build-wide questions such as duplication still need the full analysis above):

```
UnityDataTool analyze /path/to/assetbundles -o single.db -p my.bundle
```

The same per-object queries apply — `view_breakdown_by_type` for what dominates the AssetBundle, then
`object_view ORDER BY size DESC` and a `dump` of the biggest objects to see what makes them large
(e.g. texture dimensions and format, or mesh vertex counts).

## Providing schema context to a chat-based AI

When using a chat AI that cannot run commands itself, the same information has to be provided as
context. Dump the schema of your database into a text file and attach it before asking for queries:

```
sqlite3 Analysis.db ".schema" > schema_dump.sql.txt
```

Note that the produced database's schema shows bare table definitions; the meaning of the columns is
documented in the [Analyzer database reference](analyzer.md), which is also useful context.

## Related documentation

| Topic | Description |
|-------|-------------|
| [Command-line tool](unitydatatool.md) | All commands and their options |
| [Analyzer database reference](analyzer.md) | Tables, views, and their columns |
| [Example usage of Analyze](analyze-examples.md) | More worked queries |
| [Comparing builds](comparing-builds.md) | Finding what changed between two builds |
| [Overview of Unity Content](unity-content-format.md) | SerializedFiles, Archives, and TypeTrees |
15 changes: 4 additions & 11 deletions Documentation/analyze-examples.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,17 +76,10 @@ Example queries against build report data — build summary, size by type, and o

## Example: Using AI tools to help write queries

This is not a tutorial on using AI tools. However one useful tip:

Many AI tools let you provide context by uploading a file or copying text. They are helpful for crafting SQL statements and creating scripts. However by default they probably do not know what to expect inside a UnityDataTools SQLite database.

To provide this information you could run this command that dumps the current schema into a text file.

```
sqlite3 Analysis.db ".schema" > schema_dump.sql.txt
```

Then provide that file as context, prior to asking it to write queries based on the available tables, views and columns. For example: *Help me write a command line calling sqlite3 for Analysis.db that will print the top 5 shaders by the size column. It will print the name, pretty_size and serialized_file.*
AI tools are a good fit for the analyze database, whether that is an AI agent running `analyze` and
querying the result itself, or a chat AI helping you write SQL. The recommended workflow, the schema
facts worth knowing up front, and tips for providing schema context to chat-based tools are collected
in [Using UnityDataTool with an AI Agent](agent-guide.md).

## Example: Finding AssetBundles containing a certain object type

Expand Down
4 changes: 3 additions & 1 deletion Documentation/command-find-refs.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
# find-refs Command
# find-refs Command (Experimental)

The `find-refs` command traces reference chains leading to specific objects. Use it to understand why an asset was included (and potentially duplicated) in a build.

> **Experimental**: `find-refs` does not yet produce complete or clearly-explained results for all supported build pipelines. For example, a Player build database never produces chains, and a "Found 0 reference chain(s)" result does not distinguish an explicitly-included asset from an unreferenced object. See [issue #121](https://github.com/Unity-Technologies/UnityDataTools/issues/121) for the known problems. Until they are addressed, prefer querying the analyze database's `refs_view` directly, as described in [Tracing why content is in the build](agent-guide.md#tracing-why-content-is-in-the-build).

It walks *up* the reference graph from the target object and **stops at the first asset it reaches**. The reported chains therefore end at the immediate containing asset, not at the ultimate root that transitively depends on the target. For example, if `RootAsset` references `LeafAsset` which references a texture, searching for the texture reports the chain ending at `LeafAsset`; to see that `RootAsset` pulls it in, search for `LeafAsset` instead.

## Quick Reference
Expand Down
8 changes: 4 additions & 4 deletions Documentation/comparing-builds.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ This confirms our understanding that pixel data inside "red.png" is what caused

# Special cases

In some rare cases the binary Serialized File is different between two builds, but the text "dump" is identical.
* This can happen if the change happens in the header of the file, or in some padding bytes. Such cases are rare because the Serialized File format is quite stable, but it has happened when performance or stabilities improvements have been introduced that changed the header or padding.
* Sometimes float or double values might appear to be identical in the text representation, but there could be a difference in the actual binary representation. binary2text has a "-hexfloat" argument that addresses this issue.
In some rare cases the binary Serialized File is different between two builds, but the text "dump" is identical.

* This can happen if the change happens in the header of the file, or in some padding bytes. Such cases are rare because the Serialized File format is quite stable, but is a possibility if comparing files produced by different versions of Unity.
* Sometimes float or double values might appear to be identical in the decimal text representation, but there could be a difference in the actual binary representation. Specify the `--hexfloat` argument to dump to address this issue.
Comment thread
SkowronskiAndrew marked this conversation as resolved.
1 change: 1 addition & 0 deletions Documentation/unitydatatool.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,6 +97,7 @@ If you see a warning about `UnityFileSystemApi.dylib` not being verified, go to
| [TextDumper Output Format](textdumper.md) | Understanding dump output |
| [ReferenceFinder Details](referencefinder.md) | Reference chain output format |
| [Analyze Examples](analyze-examples.md) | Practical database queries |
| [Using UnityDataTool with an AI Agent](agent-guide.md) | Recommended workflow for AI agents analyzing a build |
| [Comparing Builds](comparing-builds.md) | Strategies for build comparison |
| [Unity Content Format](unity-content-format.md) | TypeTrees and file formats |
| [ContentLayout.json](contentlayout.md) | The content layout file produced by content directory builds |
12 changes: 7 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@ New to Unity's data files or to UnityDataTool? These topics are a good place to
| [Command-line tool](./Documentation/unitydatatool.md) | All commands and their options. |
| [Analyzer & database schema](./Documentation/analyzer.md) | The SQLite database that `analyze` produces, including its tables and views. |
| [Example queries](./Documentation/analyze-examples.md) | Worked examples of querying the analyze database. |
| [Using UnityDataTool with an AI agent](./Documentation/agent-guide.md) | The recommended workflow for AI agents (and scripts) analyzing a build. |
| [Comparing builds](./Documentation/comparing-builds.md) | Finding what changed between two builds. |
| [Addressables build reports](./Documentation/addressables-build-reports.md) | Analyzing Addressables JSON build reports. |
| [Build reports](./Documentation/buildreport.md) | Importing a Unity BuildReport to map build output back to source assets. |
Expand Down Expand Up @@ -88,8 +89,9 @@ flowchart TD
database for detailed analysis. It also parses Addressables build reports.
* [TextDumper](Documentation/textdumper.md): dumps SerializedFiles into a human-readable format
(similar to Unity's binary2text).
* [ReferenceFinder](Documentation/referencefinder.md): finds reference chains from one object to
another by querying a database produced by the Analyzer (a data dependency rather than a code one).
* [ReferenceFinder](Documentation/referencefinder.md): experimental; finds reference chains from one
object to another by querying a database produced by the Analyzer (a data dependency rather than a
code one).
* [Archive](Documentation/command-archive.md): inspects and extracts the contents of Unity Archives
(AssetBundles and web platform `.data` files) — the `archive` command.
* [SerializedFile](Documentation/command-serialized-file.md): inspects the header, metadata, object
Expand Down Expand Up @@ -120,13 +122,13 @@ shared test data doubles as convenient sample content for ad hoc use of the tool

## Downloads

Prebuilt Windows and Mac builds are available in the "Actions" tab. Each update to the main branch triggers a new build.
Prebuilt Windows and Mac builds are published on the [Releases page](https://github.com/Unity-Technologies/UnityDataTools/releases). Each release includes a zip per platform containing the `UnityDataTool` executable and the native libraries it needs.

To use:
1. Download and unzip the build for your platform.
2. Run UnityDataTool from the extracted location, or add it to your system PATH.
2. Run UnityDataTool from the extracted location, or add that location to your system PATH.

Refer to the [commit history](https://github.com/Unity-Technologies/UnityDataTools/commits/main/) to see the recent improvements to the tool.
Each release describes what changed; refer to the [commit history](https://github.com/Unity-Technologies/UnityDataTools/commits/main/) for changes since the latest release. To try unreleased changes, [build from source](#how-to-build).

## Getting UnityFileSystemApi

Expand Down
14 changes: 10 additions & 4 deletions UnityDataTool/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -35,8 +35,6 @@ public static async Task<int> Main(string[] args)
return r;
}

const string DocumentationUrl = "https://github.com/Unity-Technologies/UnityDataTools/blob/main/Documentation/unitydatatool.md";

static string BuildRootDescription()
{
var version = Assembly.GetExecutingAssembly()
Expand All @@ -49,13 +47,20 @@ static string BuildRootDescription()
if (plusIndex >= 0)
version = version.Substring(0, plusIndex);

// Release builds have a bare version (e.g. "2.1.0") matching a git tag, so their
// documentation link can be pinned to the matching docs. Dev builds carry a pre-release
// suffix (e.g. "2.2.0-dev") and link to the latest docs on main instead.
var docsRef = version.Contains('-') || version == "unknown" ? "main" : $"v{version}";
var documentationUrl =
$"https://github.com/Unity-Technologies/UnityDataTools/blob/{docsRef}/Documentation/unitydatatool.md";

return
"UnityDataTool inspects and analyzes Unity file formats, for example the content formats for AssetBundles, " +
"Player and content directory builds. It can build a database of the Unity objects and their " +
"references for analysis, dump objects as text, and examine " +
"archive and SerializedFile internals.\n\n" +
"Run 'UnityDataTool [command] --help' for detailed help on a specific command.\n\n" +
$"Documentation: {DocumentationUrl}\n" +
$"Documentation: {documentationUrl}\n" +
$"Version: {version}";
}

Expand Down Expand Up @@ -133,7 +138,8 @@ static Command BuildFindRefsCommand()
var aOpt = new Option<bool>(aliases: new[] { "--find-all", "-a" }, description: "Find all reference chains originating from the same asset (instead of only one), can be very slow");
var stdoutOpt = new Option<bool>(aliases: new[] { "--stdout" }, description: "Write the reference chains to stdout instead of a file.");

var findRefsCommand = new Command("find-refs", "Find reference chains to specified object(s).")
var findRefsCommand = new Command("find-refs",
"Find reference chains to specified object(s) (experimental: results are incomplete for some build types).")
{
pathArg,
oOpt,
Expand Down
Loading
Loading