os: add experimental reproducible Debian/mkosi backend - #816
Merged
Conversation
Collaborator
Author
|
Reproducibility/clean-build follow-up pushed as Verified with the exact full two-clean-build gate: The two Additional gates rerun successfully:
|
Collaborator
Author
|
Post-push QEMU smoke test also passed the intended platform-independent portion on the final reproducible |
Collaborator
Author
|
Added the missing SEV-SNP artifact chain in
Verification:
Rootfs size attribution versus |
Collaborator
Author
|
Added the explicitly opt-in, development-only mkosi component cache in Security/audit boundary:
Cached component outputs: dstack Rust, container stack, Sysbox, nvattest, kernel source/build/stage, NVIDIA, ZFS, and TDX+SNP OVMF. Measured verification (
Usage: DSTACK_DEV_CACHE_DIR="$HOME/.cache/dstack/mkosi-dev" \
./os/mkosi/build.sh dev-image "$PWD/os/mkosi/build-dev" |
Collaborator
Author
|
Follow-up |
Collaborator
Author
|
Refactored the cache implementation in Structure now has three explicit layers:
In particular, Revalidated refactored HEAD:
|
Collaborator
Author
|
Rootfs size parity is now pushed through Measured against the supplied Yocto 0.6.0 artifact: Retained changes:
Verification evidence:
|
Collaborator
Author
|
Final mkosi-native audit and reproducibility verification is complete. Key changes in the final series:
The last item came from a real cross-host failure: local and tdxlab trees had identical paths and file bytes, but privileged GitHub construction differed in one setgid directory bit and backing-filesystem inode order. The canonical stream now removes both inputs. A mechanical rebuild from the formerly different local/GitHub root trees produced identical squashfs bytes. Final verified source revision: Verification:
Custom logic retained is documented in |
The backend pinned Linux 7.1.4 while production Yocto is on 6.18, and that skew alone required two out-of-tree module compatibility patches with no production counterpart: a 937-line OpenZFS backport (ZFS 2.4.0 declares Linux-Maximum 6.18) and an NVIDIA kernel-open patch for the OF GPIO API removed in 7.1. Both exist only to reach 7.1. Pin 6.18.40 (current longterm, matching PREFERRED_VERSION_linux-yocto), delete both patches, and derive the kernel.org download path from the version instead of hardcoding v7.x. acceptance.sh now asserts the series against meta-dstack's distro conf so the skew cannot silently return.
The fragment seeds from x86_64_defconfig, which enables KEXEC, HIBERNATION and PROC_KCORE, and it overrode none of them. os/yocto/tools/aws/audit-aws-ec2 -image-hardening.sh requires KEXEC=n, HIBERNATION=n, DEBUG_FS=n and STRICT_DEVMEM=y; linux-yocto-tiny never had them on, which is why the Yocto fragments do not state them and the skew went unnoticed. kexec inside an attested CVM rebooting into an arbitrary kernel leaves the RTMRs and the dm-verity rootfs describing something that is no longer running, after KMS has released the app keys; hibernation writes private guest memory to the host-visible disk. Also restores the meta-dstack fragment options that were dropped or downgraded: PM/PM_SLEEP/HOTPLUG_CPU=n, XFRM_ESP=n, RXKAD=n, BLK_DEV_THROTTLING_LOW=y, and BRIDGE_NF_EBTABLES/VXLAN back to =y. acceptance.sh now derives the required set from the audit script itself rather than restating it, so the two cannot drift.
Path.is_dir() follows symlinks, so a symlink-to-directory installed by one component was accepted as a mergeable directory by the next: os.walk then descended through it and shutil.copy2 wrote the payload outside the staging root. A component shipping an absolute usr/lib64 -> /usr/lib silently wrote into the build host (the CI job runs as root) while the files it was meant to install were missing from the image, and the merger reported success. Directories were also recreated with the default mkdir() mode instead of the source mode, so a 0700 component-owned state directory shipped 0755. Extends the merge tests to cover the symlinked-directory escape, directory mode preservation and the unsupported-entry branch; the escape test fails against the previous implementation.
…lision DSTACK_GIT_REVISION already means something else: os/yocto/repro-build/ repro-build.sh exports it into the whole Yocto build environment and os/yocto/scripts/export-artifacts.sh consumes it as release metadata, where it carries a bare 40-char SHA. The new compile-time override used the same name with the incompatible "git:<20hex>" display form and validated nothing, so a Yocto repro build started from a shell that had already exported the variable could bake a bare SHA into every binary -- changing Info.rev for anything matching the "git:" prefix, and with it the rootfs hash. Nothing protected against this except BitBake incidentally filtering the variable out of recipe environments. Renames the compile-time override to DSTACK_BUILD_GIT_REVISION, documents the format contract, and adds unit tests covering both expansion arms.
Two defects in the reproducible-ESP change, both on the path shared with the production Yocto backend: No backend entrypoint exports SOURCE_DATE_EPOCH, so the ":-0" fallback was always taken. FAT timestamps start at 1980-01-01 and cannot represent 0, so every directory entry was dated 2107 -- deterministic, but wrong. FAT also stores local time and mtools converts using TZ, so two builders in different time zones produced different disk.raw bytes from identical inputs. Verified with dosfstools 4.2 / mtools 4.0.43: TZ=UTC and TZ=America/Los_Angeles differ at byte 80944. Clamp the epoch to the FAT epoch and pin TZ=UTC for the ESP construction. disk.raw is now genuinely reproducible, so add it to the Yocto repro-build whitelist, where it had been excluded as a known non-reproducible artifact. Also drops the now-stale "command -v mmd" gate: mmd is no longer invoked, so requiring it failed builds on hosts that have mcopy but not mmd.
The cache was mounted through --build-sources, but BuildSourcesEphemeral=yes applies to every BuildSources entry: mkosi gives each one an overlay whose upper layer is a tempfile.TemporaryDirectory removed when the build script exits (mounts.py finalize_source_mounts, called from run_build_scripts). Every component archive and toolchain download was therefore discarded, so dev-image was a permanent 100% cache miss and rebuilt the kernel, NVIDIA, ZFS and OVMF on every run. BuildDirectory is the only mount a build script gets that is both writable and preserved between runs, so the cache and the toolchain downloads move there. The source manifest stays on a source mount: it is a read-only input regenerated on each run. The existing dev-cache test drives dev-cache.sh directly on the host and cannot observe mkosi's mount semantics, so acceptance.sh now pins the storage location instead. Also adds an ERR trap so its bare assertions report which check failed rather than aborting silently.
The dependency loop read lddtree through a process substitution, so its exit status was invisible, and discarded any line that was not an absolute path. lddtree prints a bare soname for anything it cannot resolve inside the root, so both a crashed lddtree and an unresolvable DT_NEEDED entry produced an initramfs missing its libraries. The build succeeded, nothing inspects initramfs contents, measurements and release tarballs were produced, and every CVM then died in early boot with "veritysetup: error while loading shared libraries". The CI workflow already carries a PATH workaround for exactly this fragility. Capture lddtree's output and status, reject empty output, and reject any entry that is not an absolute path inside the image tree. Also replaces the two 'find | head -1' selections. readdir order made the chosen binary filesystem-dependent, and under pipefail a second match would abort the script with exit 141 and no diagnostic.
The prod profile removed only docs, headers and locales, so the production image kept /usr/bin/login, agetty, loginctl, systemd-logind, every getty unit and systemd-getty-generator. The base skeleton masks getty@, serial-getty@ and debug-shell with /dev/null symlinks, so there is no live login prompt -- but those masks live in /etc, which dstack-prepare.sh covers with a writable tmpfs overlay at runtime. meta-dstack's disable_login() instead deletes the binaries and units outright, so in the Yocto image the login stack is structurally absent from the verity-measured rootfs rather than disabled by mutable state. The serial port is a chardev owned by the untrusted host, so mirror that. Suspend/hibernate units go too, since the kernel now drops CONFIG_PM entirely. Verified with mkosi's own parse_ini that the annotated list parses to 40 entries with no comment leakage. parity.json gains the removed paths plus the dev-profile tools that were never gated (tcpdump, vim, gdbserver), and check-parity.py no longer accepts a dangling symlink as a satisfied requirement -- a versioned NVIDIA library whose target was never installed used to pass and then fail at runtime.
mkosi runs 'systemctl --root preset-all', and systemd's fallback policy for a unit matching no rule is enable. 80-dstack.preset had 19 enable lines and no disable, so the effective policy was "start everything with an [Install] section" -- the inverse of Yocto's opt-in SYSTEMD_SERVICE model. Any package a future Packages= line pulls in, directly or as a dependency, would run at boot inside the CVM with no diff to the preset file. Adds a terminal 99-dstack-default.preset with 'disable *' so the enable list is exhaustive and auditable. Also moves 'enable dstack-tee-simulator.service' into a dev-profile skeleton. It shipped unconditionally in the production skeleton, inert only because the unit is installed for dev alone -- a guard independent of the preset. A production image serving synthetic TDX quotes attests successfully while providing no hardware guarantee. normalize-skeleton-modes.sh now covers profile skeletons too, guards the directory loop like the file loop, and skips the skeleton root itself, which it was chmod'ing as a side effect of an empty relative path.
mount --bind of a freshly created empty tmpfs directory hides the lower content rather than layering over it, so everything the measured image ships under /var/cache, /var/lib, /var/log, /var/spool and /var/tmp became invisible and was replaced by empty, writable, unmeasured directories. The image demonstrably has content there: Debian's chrony ships /var/lib/chrony, and the skeleton itself creates /var/lib/docker, /var/lib/containerd and /var/lib/tpm2-tss/system/keystore. A file added under /var/lib/<component>/ in a later change would look correct in the built image, pass the parity check, and be inert on every boot. The bind also took the mode of the empty source directory, turning /var/tmp from 1777 into 0755. Use an overlay, as poky's volatile-binds does for the Yocto image and as dstack-prepare.sh already does for /etc, /usr and /bin. Verified: the bind leaves 0 of 2 entries visible, the overlay leaves both readable with writes landing in the tmpfs upper layer. The unit also becomes RequiredBy=local-fs.target. It was Wanted, so a failed setup left /var/log and /var/lib on the read-only squashfs and the boot continued -- a CVM that attests successfully and then silently misbehaves.
versions.env pinned Go 1.22.2 with a SHA-256, but nvidia-container-toolkit at the pinned revision declares 'go 1.25.0' in go.mod and nothing set GOTOOLCHAIN, so the go command downloaded go1.25.0 from proxy.golang.org during the build. The compiler that actually produced nvidia-ctk and nvidia-container-runtime was therefore neither checksum-verified nor part of any component cache key, and the build could not run at all with GOTOOLCHAIN=local or without proxy access. Bump the pinned toolchain to 1.25.12, which satisfies that requirement and still builds the sysbox modules that declare go 1.22, and set GOTOOLCHAIN=local so a future go.mod bump fails loudly instead of silently fetching a different compiler. Also verify toolchain downloads before publishing them under the cached name. A truncated transfer was cached as-is and the mismatch path did not remove it, so every later build failed identically with no way to tell that the fix was to delete the file.
The gate matched only ^CONFIG_[A-Z0-9_]+=(y|m|n)$ and skipped everything else, so two of the fragment's assertions were never enforced: a dropped CONFIG_LOCALVERSION -- which names the module directory that depmod, modules_install and make-release-artifacts.sh all resolve -- and a clamped CONFIG_NR_CPUS both passed. Verified against the previous implementation: both perturbations are accepted by it and rejected now. An unparsable fragment line is now an error rather than a silent skip, the mismatch message reports the value actually found, and a fragment that yields implausibly few assertions fails instead of trivially passing. A symbol that disappears entirely because its dependencies went away is now accepted for want=n, which previously produced a spurious failure on kernel bumps: absent and 'is not set' both mean not enabled.
…centrally The build applied 2 of the 6 patches meta-dstack's dstack-ovmf_git.bb uses. 0001/0002 are legitimately bitbake-specific, but two were dropped silently: 0003-Debug-prefix-map threads ENV(GCC_PREFIX_MAP)/ENV(NASM_PREFIX_MAP) into the tool definitions. It was replaced by an ad-hoc sed that only reproduced the -Werror overrides and dropped the prefix map entirely. 0005-UefiCpuPkg backports edk2 9ccf8751 for NASM 3.x. The recipe records that stable202502 must be kept for the pre202505 measurement layout and does not assemble with NASM 3.x without this, so the next snapshot bump carrying nasm 3.x would have turned the build into a hard failure with the fix sitting unused in-tree. The encoding is unchanged on NASM 2.x, so applying it also brings this build closer to the production one. NASM_PREFIX_MAP stays empty: Yocto carries a nasm patch adding --debug-prefix-map, and the Debian snapshot ships stock nasm, which rejects the option (confirmed against nasm 2.16.01). The revision also moves to versions.env as OVMF_REVISION -- it was the only component pin hardcoded in a build script -- and acceptance.sh now reads SRCREV out of the recipe instead of restating the hash, which would have happily pinned a stale value after a Yocto bump.
Verified against the installed mkosi 26 rather than from memory. Output= shared the release name, so mkosi tracked assemble.sh's release tarball as one of its own outputs -- something clean and --force are entitled to delete -- and mkosi.postoutput had to delete mkosi's convenience symlink by hand. Give mkosi a name of its own, skip compressing an artifact that is discarded anyway, and drop it explicitly in the staging directory before it is promoted. That also fixes the dev flavor leaving a full unmeasured Debian rootfs tar beside the release, which only the prod name collision had hidden. %i and %v expand in file order, so ImageId/ImageVersion now precede Output=; declaring Output first silently yields "--rootfs.tar". acceptance.sh checks both the ordering and that ImageVersion matches DSTACK_VERSION. mkosi.skeleton/etc/apt/apt.conf.d/99dstack-reproducible was dead: mkosi runs apt from outside the image, so per its own manual a package manager config supplied via SkeletonTrees= never takes effect, and it passes these as -o options which outrank config files anyway. All three settings are already covered natively by Snapshot= and WithRecommends=. SkeletonTrees= and PostInstallationScripts= restated mkosi's auto-discovery and made the skeleton mount twice. Bootloader= is unreachable with Bootable=no, SecureBoot=no is the default, and SplitArtifacts= only gates UKI/kernel paths that Bootable=no already disables. KernelCommandLine= was dead for the same reason and contradicted the command line actually measured into the UKI (panic=30 vs panic=1, a console=tty0 that cannot work with CONFIG_VT=n, and none of the hardening tokens). mkosi clean ran without the build's --output-directory, so mkosi.clean was removing from os/mkosi/ rather than the build tree. Also records why the manual depmod plus mtime normalization is load-bearing for repro-check: it is what keeps mkosi's own nondeterministic depmod from firing.
…staging build.sh recorded git rev-parse HEAD with no dirty detection while BuildSources= mounts and compiles the working tree, so uncommitted changes produced an image whose guest agent and measured metadata.json both claimed a clean upstream revision. Refuse to build from a dirty worktree, with DSTACK_ALLOW_DIRTY=1 to opt in and record -modified instead. sysbox checkouts never ran git clean -qfdx, unlike every other component, so a revision bump in an existing work directory carried the previous vendor/ tree and the copied-in generated protobuf sources into the new binaries. Its subuid/subgid writes also appended to a reused staging tree, accumulating duplicate entries that some userns tooling treats as a parse error. nvattest builds in a fixed path under a world-writable parent; flock serializes but does not establish ownership. Create the directory instead of adopting it, so a leftover or attacker-created tree aborts the build. component_run's bare assertions killed the whole build with no output when a component descriptor was malformed; they now say which file and which symbol. The sshd drop-in restores PermitEmptyPasswords and PubkeyAuthentication from meta-dstack's version. Both match sshd's compiled defaults, but this file is the image's only ssh policy and should not depend on them. write-source-manifest.py uses env python3 like every other script; build.sh invokes it directly, so the absolute path broke on hosts without /usr/bin/python3.
make-release-artifacts.sh baked the command line into the UKI while assemble.sh independently reconstructed the same string for metadata.json. Both feed measurements -- the UKI's PE sections for GCP and AWS, metadata.json for the TDX RTMRs via dstack-mr -- so a token added to one would have produced measurements describing a command line the guest never booted with, and every existing check would still have passed. Move it to os/image/kernel-cmdline.sh and have both call it. Verified the shared definition is byte-identical to both previous strings, so no measurement changes. acceptance.sh also gains the checks that were missing: mkosi.postinst was absent from the bash -n list, so a syntax error there only surfaced part way into a ~30 minute build, and none of the Python helpers were compile-checked at all.
The job checked out a hardcoded BASELINE_REVISION, so it always rebuilt that one commit regardless of the dispatched ref and could never validate the branch it ran on -- and once this branch is squashed or rebased the pinned commit becomes unreachable and the checkout fails outright. It was also workflow_dispatch-only, so no PR or push exercised os/mkosi at all. Add a static job that runs build.sh lint on every PR and master push touching os/. It finishes in seconds and covers the whole static contract, which is where the useful per-change signal is; the ~40 minute image build stays opt-in behind workflow_dispatch. The baseline hashes embed the revision they were produced from, so the comparison now runs only when the built commit is that revision and prints an actionable notice otherwise, instead of comparing against values that cannot match. Also makes the release-member checks assert something: they piped sha256sum.txt and metadata.json to stdout without inspecting them. The DSTACK_DEV_CACHE_DIR check was vacuous since the workflow never sets it, and the run blocks now use set -euo pipefail.
…ensing repro-check computed JOBS once and ran both legs with it, so output that depends on parallelism could never be detected. That failure mode is not hypothetical: the kernel component carries a pahole wrapper solely because its BTF encoder varied with Kbuild's job count, and nothing equivalent guards the OVMF, ZFS or NVIDIA builds. The legs now differ in job count as well as in build path, overridable with REPRO_JOBS_B. The REUSE annotation covering os/mkosi/** used precedence = "override", so any patch backported from an upstream project would be relabelled Apache-2.0 under Phala copyright regardless of its actual license. Drop the override, move the entry next to the Yocto backend it parallels (nothing here is a generated file), and add real SPDX headers to the systemd units, network and sshd drop-ins, kernel fragment and gitignore, which all support comments. Verified with reuse lint: 1051/1051 files, compliant with REUSE 3.3.
Drops the hardcoded toolchain versions that duplicated versions.env, records GOTOOLCHAIN=local, and describes the repro-check job-count variation. Rewrites the acceptance paragraph, which claimed QEMU boot, /proc/config.gz inspection and service-enablement checks as things the backend verifies. None of those are automated; build.sh lint is a source-level static check and only the byte-for-byte comparison runs in CI, behind a manual dispatch. Says so. Also corrects the output description: the dev flavor emits dstack-dev-<version>/, not the same directory name as prod.
The first real build stopped at the config gate, which is what it is for.
Three of the four assertions came from meta-dstack fragments that are
silently ineffective there, because Yocto warns on an unsatisfied fragment
line instead of failing:
HOTPLUG_CPU=n is "def_bool y depends on SMP" on x86, so it cannot be
turned off while SMP is on.
BLK_DEV_THROTTLING_LOW=y no longer exists; the symbol was removed upstream.
BRIDGE_NF_EBTABLES=y depends on BRIDGE, which is a module in both builds, so
only =m is reachable. Restores the original value.
The fourth is a real gap that had to be closed rather than dropped:
DEBUG_FS=n is required by the production hardening audit, but
x86_64_defconfig enables BLK_DEV_IO_TRACE, which selects DEBUG_FS. blktrace
has no role in a production guest, so disable it. Checked every symbol that
selects DEBUG_FS in this kernel; it is the only one defconfig enables.
Verified by generating the real .config from linux-6.18.40 with defconfig plus
this fragment: the gate passes and KEXEC, KEXEC_FILE, HIBERNATION, DEBUG_FS,
PROC_KCORE, PM, SUSPEND and BLK_DEV_IO_TRACE are all off with STRICT_DEVMEM
on. PM_SLEEP disappears entirely, which the gate now tolerates for want=n.
…tArtifacts Both found by the first full build, which produced and verified every release artifact and then failed on two problems introduced by the previous commits. build_image() calls print_output_size() on its own output after the postoutput scripts have run and after staging is promoted, so deleting the rootfs tar in mkosi.postoutput made mkosi die with FileNotFoundError once the release was already complete. Remove it from build.sh after mkosi returns instead. Dropping SplitArtifacts= was wrong. The default is [uki, kernel, initrd] and the kernel split is produced even with Bootable=no -- it copies the kernel out of the rootfs -- so a stray dstack-0.6.0-rootfs.vmlinuz appeared beside the release artifacts. Restore the empty value with the reason recorded. acceptance.sh gains guards for both, checked in each direction.
1.25.12 was chosen as the lowest version satisfying nvidia-container-toolkit's go.mod. oe-core at the revision this repo pins ships go-binary-native_1.26.2, so the production backend compiles the same Go components -- the container stack and sysbox -- with 1.26.2. Use the same toolchain rather than a different one that merely satisfies the constraint. It is also newer than every 'go' directive among the pinned components, which reproduces the property that makes Yocto safe without GOTOOLCHAIN at all: an automatic toolchain switch is never requested. GOTOOLCHAIN=local stays as the explicit backstop, because mkosi builds with WithNetwork=yes throughout and has no equivalent of BitBake's network-free do_compile. The checksum is the one in oe-core's go-binary-native recipe, cross-checked against go.dev.
repro-check's second leg failed with "Your local changes to the following files would be overwritten by checkout" in openssl's gost-engine submodule. The recursive shallow update had partially completed, leaving that submodule dirty, and the retry loop only re-synced URLs -- so every subsequent attempt failed on the leftover state rather than retrying the original transient error, and the build aborted after three identical failures. Pass --force so a submodule whose working tree drifted from the pinned commit is checked out regardless, and reset and clean the submodules before each retry so a retry actually starts from a known state. The first leg had already completed, so this was fragility in the fetch, not a reproducibility difference.
meta-dstack installs tee-simulator.conf as a dstack-prepare.service drop-in. The mkosi backend copied the same file to /etc/tee-simulator.conf, where systemd never looks, so the dev image silently lost the Requires=/After= ordering that makes dstack-prepare wait for the simulator to publish the TEE ABI first. Install it under the drop-in directory and pin both its presence in dev and its absence from prod in the parity manifest.
The dev-only TEE simulator spawns swtpm to back the vTPM it exposes, which is why meta-dstack's dstack-tee-simulator recipe carries it in RDEPENDS. The mkosi dev profile never installed it, so the simulator's TPM backend could only fail at runtime. Add the package and pin it in the parity manifest; the recipe's two other runtime deps, openssl and tpm2-tools, already arrive through ca-certificates and the base package list.
kvinwang
force-pushed
the
feat/experimental-mkosi-os
branch
from
July 27, 2026 02:00
3fac846 to
ef5428d
Compare
The workflow only ever compared a build against three hardcoded baseline hashes that go stale on the next commit, so it verified nothing on any revision but one and published nothing ever. Make it the mkosi backend's release path, mirroring guest-os.yml for the Yocto backend. Publishing is bound to the tag: pushing mkosi-os-v<version> is the only event that creates a GitHub Release, so a workflow_dispatch dry run cannot be mistaken for one. The push trigger deliberately carries no paths filter, because GitHub ANDs path and ref filters and would silently drop a release tag placed on a commit that does not touch os/**. Report the hashes operators actually need, led by os_image_hash: the log gets a ::notice, the job summary a table plus the sha256sum.txt preimage and metadata.json, and image-hashes.txt ships as a release asset. The values are read back out of the published tarball rather than the staging tree, and digest.txt is recomputed from sha256sum.txt and cross-checked against the UKI package, so a corrupt release fails the build instead of being published. The release job rejects a tag whose version disagrees with versions.env or whose measured metadata.json records a different git revision than the tagged commit, which also catches a dirty-worktree build. Replace the rotting baselines with build.sh repro-check, available as an opt-in dispatch input: two legs, differing job counts, byte-for-byte comparison, no hash to keep refreshing.
These files entered the branch through commits that were never pushed, so CI had not seen them until the rebase and the whole prek job failed on the branch. The two formatting fixes are what the hooks produce on their own. The shellcheck findings are noise rather than defects: the prek hook runs without -x, so an existing `source=` directive documents the target but cannot be followed, and the single-quoted greps in acceptance.sh match a literal $BUILDDIR in a generated script rather than expanding a variable. Suppress each one at its site with the reason recorded.
Two workflows now build guest images, and both showed up in the Actions list and the PR check list as "Build Guest Images". Qualify the Yocto one the same way the mkosi one already is, so the run a reader is looking at is identifiable without opening it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an experimental Debian trixie/mkosi implementation of the dstack guest OS,
selectable through the existing backend contract as
./os/build.sh --backend mkosi. The Yocto backend remains the release path andis unchanged.
The acceptance target is functional parity with the Yocto image, not merely
release-archive compatibility.
os/mkosi/parity.jsonis the machine-checkedinventory: required rootfs paths per component, required kernel config symbols,
and the dev-only / prod-forbidden path lists.
6.18.40, NVIDIA 595.58.03 (open modules, userspace, firmware, Fabric Manager,
NSCQ), nvattest 2026.06.09 with the OCSP-freshness patch, OpenZFS 2.4.0,
Sysbox 0.6.7, NVIDIA Container Toolkit, nerdctl, CNI plugins and
stargz-snapshotter 0.18.2.
PREFERRED_VERSION_linux-yoctoin
meta-dstack), whichtests/acceptance.shenforces. That is what keepsthe component set patch-free: ZFS 2.4.0 declares
Linux-Maximum: 6.18and theNVIDIA open modules build against this series unmodified.
from
meta-dstack; patch fuzz is forbidden.the Rust 1.92.0 and Go 1.26.2 archives by SHA-256, mkosi 26, and the OVMF
revision that matches
meta-dstack'sdstack-ovmf_git.bbSRCREV— ageneric OVMF produces TDX measurement material
dstack-mrcannot parse.build overlay from the same snapshot.
TDX/GCP/AWS measurement material through the common
os/image/assemble.shrelease path, so both packages carry one
os_image_hash.Changes outside
os/mkosi/These are the parts that touch shared code and deserve review attention:
os/build.shmkosias a selectable backendos/image/kernel-cmdline.shmetadata.jsonmust describe the same stringos/image/assemble.shdstack/crates/build-infoDSTACK_BUILD_GIT_REVISIONcompile-time override, because the build sandbox has no.git. Deliberately distinct from the Yocto backend'sDSTACK_GIT_REVISION, which carries a bare SHA for release metadata and would otherwise be picked up here in the wrong formatprek.toml,REUSE.tomlos/mkosi/**is intentionally notprecedence = "override", so a backported upstream patch keeps its own licenseCI
.github/workflows/mkosi-build.ymlis the backend's release path, mirroringguest-os.ymlfor Yocto:build.sh linton every change (seconds).workflow_dispatchand on a release tag. Itreports
os_image_hashas a log notice, a job-summary table and animage-hashes.txtrelease asset, alongside the TDXMRTD, GCPuki_auth,AWS
boot_pcr_digest, both tarball digests, thesha256sum.txtpreimage andmetadata.json. Values are read back out of the published tarball, anddigest.txtis recomputed and cross-checked against the UKI package, so acorrupt release fails instead of shipping.
mkosi-os-v<version>is theonly event that creates a GitHub Release, so a dispatch dry run cannot be
mistaken for one. It rejects a tag whose version disagrees with
versions.envor whose measuredmetadata.jsonrecords a revision other thanthe tagged commit, which also catches a dirty-worktree build.
repro_checkdispatch input builds twice with differing job countsand compares the release tarballs byte for byte.
Validation
./os/mkosi/build.sh lint./os/mkosi/build.sh repro-check— two legs, differing job counts,byte-identical
dstack-0.6.0.tar.gzanddstack-0.6.0-uki.tar.gzswitch_root, and Debiansystemd
Notes
This remains experimental, and the release workflow marks its output as a
prerelease to match. Package contents are Debian rather than Yocto, while the
external release contract and the file and partition layout are shared with the
Yocto backend.