Posit Publisher .posit/publish: TOML interop, redeploy, and config-driven bundling - #830
Posit Publisher .posit/publish: TOML interop, redeploy, and config-driven bundling#830mconflitti-pbc wants to merge 14 commits into
Conversation
|
Follow-up work is tracked in #831: applying a |
Read and write Publisher's .posit/publish config + deployment record files alongside the legacy rsconnect-python JSON store, so content can be published with either tool. - New rsconnect/publisher package (schema, serialize, config, record, store) porting Publisher's format: content-type map, $schema-first TOML with multiline arrays, and the random base-32 file-naming methodology. - Dual-write .posit on Connect/SPCS deploys via save_deployed_info (best-effort). - New 'rsconnect redeploy [PATH]' command driven by .posit, with a fallback to manifest.json + legacy rsconnect-python/*.json for pre-.posit content. - 'write-manifest' commands also emit a .posit config. - Connect Cloud files are read/preserved for interop but not deployable here. - Adds tomli-w dependency; tests in tests/test_publisher.py and test_redeploy.py.
save_deployed_info re-derived the config/record filenames independently of what redeploy resolved, so when the resolved record lacked a configuration_name (or its entrypoint differed from the bundle), a fresh <title>-<CODE>.toml config was minted instead of updating the resolved one. redeploy now threads the resolved config_name and record_name through the executor to write_deployment_metadata, which uses them to update those exact files. Adds tests covering the pin and the threading.
Publisher keeps its own credentials in VS Code SecretStorage, which a CLI cannot read, and a .posit record stores only server_url (never the key). So redeploy now matches the record's server_url against rsconnect-python's own saved servers (ServerStore) by normalized URL -- the same join Publisher uses -- and deploys under that nickname when the caller gave no explicit credential. Ambiguous matches (>1 saved server for the same URL) raise, asking for --name. Also fixes normalize_url to strip a trailing slash before the /__api__ suffix so '.../__api__/' compares equal to the base URL.
When a .posit/publish config applies to the content, deploys now bundle exactly the files its `files` patterns select (gitignore-syntax, include by default, `!` excludes), matching Posit Publisher's bundler. Without a config, deploys now honor .gitignore in addition to the built-in ignore list. A config rsconnect writes records the concrete deployed file set so config, manifest, and record all agree. - New rsconnect/publisher/files.py: pathspec-based select_config_files (allowlist) and select_default_files (.gitignore denylist). - bundle.create_file_list gains include_files + a restrict_to_files contextmanager; the executor resolves the selection and wraps builders. - store.resolve_bundle_files chooses config vs. default and force-includes the entrypoint; _config_file_patterns now emits the concrete, root-anchored list plus the .posit config/record paths (mirrors Publisher). - Add pathspec dependency.
A .posit/publish config can declare integration_requests that rsconnect cannot originate itself. Connect reads these from manifest.json (as Posit Publisher's manifestFromConfig does), so when a config applies its integration_requests are now merged into the generated manifest. - bundle.py: overlay_manifest contextmanager + _apply_manifest_overlay, merged in Manifest.__init__ (the single manifest choke point; make_manifest_bundle stays manifest-driven). ManifestData gains integration_requests. - store.py: config_manifest_overlay maps a config's integration_requests to Publisher's manifest shape; resolve_manifest_overlay + _select_applicable_config (extracted from resolve_bundle_files). - api.py: RSConnectExecutor resolves file selection + manifest overlay together and wraps the builder in both contexts.
5fa320b to
3f7218b
Compare
|
A function return annotation used the PEP 585 builtin generic dict[...], which Python 3.8 cannot subscript at definition time, breaking collection of the whole suite on 3.8. Add 'from __future__ import annotations'.
☂️ Python Coverage
Overall Coverage
New Files
Modified Files
|
The integration-test CI job exports CONNECT_SERVER/CONNECT_API_KEY, which leaked through the --server/--api-key env-var options into the redeploy CLI under test and overrode the .posit-record-based server/identity resolution (6 failures, e.g. deploy server resolving to http://localhost:3939 instead of the record's URL). Add an autouse fixture that scrubs the CONNECT_* credential/server env vars so these tests resolve from the record as intended.
Three defects in how a deploy's ``.posit`` metadata is written and read back, all of which made a subsequent ``redeploy`` bundle the wrong file set: - ``write_deployment_metadata`` recomputed ``rname`` after using it to build the config's ``files``. When no record existed yet, ``_new_record_name`` minted a second random name, so the config's ``files`` referenced a deployment record that was never written -- and the record that *was* written never shipped in the next bundle. - ``_config_file_patterns`` anchored the manifest's ``metadata.entrypoint`` even when it is a module reference rather than a path (``deploy shiny`` records ``app`` for ``app.py``), seeding configs with a ``/app`` include that matches nothing. It now only surfaces the entrypoint when it names a deployed file, and still leads the list in that case. - ``resolve_bundle_files`` treated a config with an empty/absent ``files`` list as "no config", falling through to the ``.gitignore``-aware whole-tree default. Publisher's ``collectFiles`` defaults such a list to ``["*"]``; match that, so the config governs and STANDARD_EXCLUSIONS still apply. Adds regression coverage that asserts ``redeploy``'s actual bundle members (the existing redeploy tests stub out ``make_bundle``, so they never saw the file list): only the config's files are bundled, ``redeploy`` and an equivalent ``deploy`` agree, and repeated redeploys stay stable.
I'm not sure that's what we want, the reason IIRC that there is a separate tracking of included/excluded files is that for rendered content, you might gitignore the build output (the html that a quarto project produces, for example) because you don't want to check it in, but might want to deploy exactly that. That's at least why .gitignore wasn't just used before. It's possible that that's not the right tradeoff, and that there's a better way to solve for that. But I wouldn't just casually do that here, I think that's a more significant behavior change. |
Yeah that is fair! Will revert that |
Config-less deploys now bundle exactly as they always have: the whole-tree walk with only the built-in ignore list. .gitignore is the wrong signal for bundling. Rendered content is routinely gitignored precisely because it shouldn't be committed --- a Quarto project's HTML output, for example --- yet that output is exactly what needs to deploy. That is why bundling tracked included/excluded files separately in the first place. Narrowing the default file set may still be worth doing, but it is a larger behavior change than .posit interop and belongs in its own change. The config-driven path is unaffected: when a .posit/publish config applies, its `files` list still decides the selection. Drops `select_default_files` and `_gitignore_spec`; `resolve_bundle_files` now returns None when no config applies, which `restrict_to_files` already treats as a no-op.
A project that never had .posit was still changing behavior on its *second* deploy. Deploy #1 writes a config recording the concrete set of files it deployed; deploy #2 finds that config and treats the snapshot as user curation. A module added in between, or output rendered in between, silently stopped being bundled --- no warning, no diagnostic. rsconnect can't know which files a user *meant* to exclude, so it no longer guesses: a config it mints records files = ["*"], and resolve_bundle_files treats an absent/empty/["*"] list (ignoring the .posit paths Publisher adds so they ship) as "no restriction", falling through to the unchanged whole-tree walk. Hand-curated files lists are still honored --- that's the actual feature. Verified a first deploy of a plain project selects a byte-identical file list on this branch and on main, and that a second deploy adds only the .posit files themselves.
Covers the interop workflow end to end: rsconnect deploys (writing
files = ["*"]), the user narrows the list to three files in Publisher,
then rsconnect redeploys. Asserts the curated list is honored by both
`redeploy` and a plain `deploy`, and that rsconnect does not overwrite
the curation with its own default.
Also pins two `files` shapes that could plausibly regress:
- curated patterns alongside the .posit paths Publisher appends --- the
.posit entries must not make the list read as unrestricted
- ["*", "!secrets.txt"], which is curation by exclusion and must not be
flattened away by the "* means everything" check
CHANGELOG now states the round-trip contract explicitly.
test_no_config_bundles_gitignored_files asserted a hardcoded forward-slash path, but create_file_list's whole-tree walk returns os.path.relpath results, which use the native separator. Failed CI on windows-latest (py3.13) with backslash paths; passed everywhere else.
|
I'm a bit confused about the interactions with |
|
Why do we need |
Publisher's Go backend rejects a redeploy with "the account provided is for a different server; it must match the server for this deployment" whenever a record's server_url isn't byte-identical to the selected account's URL (internal/state/state.go: `target.ServerURL != account.URL`, a plain string comparison -- no normalization at compare time). Publisher's own writers always store the account's URL, which is normalized via purell (FlagsSafe | RemoveTrailingSlash | RemoveDotSegments | RemoveDuplicateSlashes) whenever a credential is created or loaded. rsconnect-python was writing server_url verbatim -- whatever string the user typed for --server or saved via `rsconnect add`, e.g. with a trailing slash, mixed-case host, or an /__api__ suffix. A project deployed first with rsconnect-python and then opened in Publisher could therefore fail with a server-URL mismatch even though Publisher correctly resolved the intended account by name (visible in Publisher's access log as req.account=<name>) -- the account was right, but the stored URL didn't match its normalized form byte-for-byte. Fix: write normalize_url(server_url) into the record instead of the raw value. Also brought normalize_url's own normalization up to match purell's rules exactly -- it was already stripping /__api__ and lowercasing scheme+host, but didn't strip an explicit default port (:443 for https, :80 for http) or collapse duplicate slashes in the path, both of which purell does. Verified: test_written_server_url_matches_publisher_normalization pins Publisher's exact set of cosmetic variations (case, trailing slash, default port, duplicate slashes, /__api__ with and without a trailing slash) against the values rsconnect-python now writes.
rsconnect-python's Python deploys record metadata.entrypoint as a bare
importable module reference ("app" for app.py, sometimes "module:object")
rather than a path. That value flowed unmodified into a written
.posit/publish config's entrypoint field, even though Publisher's schema
documents entrypoint as "Name of the primary file containing the
content", and Publisher's own detectors (pyshiny.ts, pythonApp.ts) always
record the literal filename for Shiny/Flask/FastAPI/Dash -- only Shiny
Express writes a module:object reference, and that module never names a
real file.
_recover_entrypoint_path (record.py) strips a trailing ":object" and
checks whether "<module>.py" is one of the manifest's actual files; if
so that's the real, verified path to record. Left unchanged when no
matching file exists, so a genuine module:object reference (Shiny
Express, or an explicit --entrypoint override with no corresponding
file) is untouched. Verified against Publisher's own detector source
(posit-dev/publisher) that Flask/FastAPI/Dash/plain-Shiny all expect the
literal filename and only Shiny Express uses shiny.express.app:<var>.
Sits in details_from_manifest, the single choke point both the deploy
write path and write-manifest go through.
to avoid breaking changes/behavior with existing deploy commands. Could try to trigger redeploy behavior with empty deploy command but figured we could avoid that. |
That is fair. Was punting on UX improvements of existing commands if I can help it. If you have ideas, I am open to them! |
I think because when users say "I want to deploy",
and now there would be
that's why I was wondering how we can ease reasoning about this for users. Especially seems it looks like that this PR would always introduce the |
Summary
Adds interoperability with Posit Publisher's
.posit/publishproject files, so content can be published with either rsconnect-python or the Publisher VS Code extension. Addresses #829.rsconnect-python now reads and writes Publisher's TOML configuration (
.posit/publish/<name>.toml) and deployment record (.posit/publish/deployments/<name>.toml) alongside the existing legacyrsconnect-python/*.jsonstore, and gains aredeploycommand that redeploys straight from a.positproject.Bundling becomes config-driven only when a configuration curates a subset: if a
.positconfig'sfileslist names real content patterns, exactly those files are bundled (Publisher's.gitignore-syntax include/exclude semantics — a matching pattern includes,!excludes). Otherwise — no config, or a config declaringfiles = ["*"]— file selection is unchanged from today: the whole-tree walk with only the built-in ignore list. A config'sintegration_requestsare propagated into the generatedmanifest.jsonexactly as Publisher emits them, so integrations authored in Publisher are honored on deploy.Usage examples
1. Deploy as usual — now also writes
.posit2. Zero-arg redeploy from an existing
.positproject3. Redeploy a project by path
$ rsconnect redeploy ./my-app4. Interop: redeploy a project that was published from the Publisher VS Code extension
5. Credentials are matched by the record's server URL
6. Disambiguate when a project has multiple configs or servers
7. First-ever deploy from a Publisher config (config exists, no record yet)
8.
write-manifestnow also emits a.positconfig9. Redeploy pre-
.positcontent (manifest + legacy JSON fallback)10. Config-driven bundling — only the config's files are bundled
A config rsconnect-python writes itself records
files = ["*"], not a snapshot ofthe files that happened to deploy — so a project that never had
.positkeepsbundling identically on every subsequent deploy, and newly added modules or freshly
rendered output are still picked up.
What's included
rsconnect/publisher/package porting Publisher's format (posit-dev/publisher):schema.py— schema URLs, theapp_mode ↔ content-typemap (frombundler/appMode.ts),.positpath helpers.serialize.py—tomli_wwriter matching Publisher (quoted$schemafirst, multiline arrays, tables last); reader via stdlibtomllib/tomlbackport.config.py/record.py— dataclasses with read‑merge‑write that preserves fields rsconnect doesn't own (incl. first-classconnect_cloud); recoversfiles/requirements/entrypoint from the built bundle manifest.store.py— dual-write on deploy +resolve_publisher_deploy_targetfor reads; Publisher-compatible random file naming (<title>-<CODE>/deployment-<CODE>, 4-char base‑32) with find-or-reuse so redeploys update in place.RSConnectExecutor.save_deployed_info(best-effort; never fails the deploy).rsconnect redeploy [PATH]with the behaviors shown above.write-manifestnow also emits a.positconfig.rsconnect/publisher/files.py(apathspec/gitwildmatch engine reproducing Publisher'sbundler/collect.ts:select_config_filesallowlist +STANDARD_EXCLUSIONS). Wired throughbundle.create_file_listvia arestrict_to_filescontext set by the executor, so all deploy commands honor it.store.resolve_bundle_filesreturns the config's selection (force-including the entrypoint), orNone— impose no restriction, leaving today's whole-tree walk untouched — when no config applies or when the config declares no real restriction (filesabsent, empty, or["*"], discounting the.positpaths Publisher adds so they ship). Configs rsconnect mints recordfiles = ["*"]; the concrete deployed set lives on the deployment record, where it can't be mistaken for curation.integration_requests→manifest.json—bundle.overlay_manifestmerges config-authored manifest fields rsconnect can't derive (currentlyintegration_requests) inManifest.__init__;store.config_manifest_overlay/resolve_manifest_overlaymap them to Publisher's manifest shape._plan_deploy_bundlehelper extracted fromdeploy pyproject(pure refactor, reused byredeploy).tomli-wandpathspecdependencies.Scope notes
connect.posit.cloud).positfiles are read/preserved for interop, but deploying to Connect Cloud is not supported by this tool (no client) — only Posit Connect and Snowflake/SPCS write.posit. shinyapps.io / Posit Cloud stay legacy-JSON-only (as Publisher also writes no.positfor them).fileslist. Two revisions of this PR got this wrong and were fixed:.gitignore. Reverted —.gitignoreis the wrong signal for bundling: a Quarto project's rendered HTML is routinely gitignored precisely because it shouldn't be committed, yet it's exactly what needs to deploy. Narrowing the default file set is a larger behavior change than.positinterop and belongs in its own PR.files = ["*"], and an unrestricted list imposes no restriction.Verified by capturing the tarball member list for a first deploy of a plain project on this branch and on
main: byte-identical. A second deploy adds only the two.positfiles.rsconnect sync#831): a config's Connect-side settings that Publisher applies via the Connect API rather than the manifest —description,connect.runtime.*,connect.access.*,connect.kubernetes.*, andenvironment/secrets— are preserved but not yet applied on deploy. Tracked for a post-deploy hook + arsconnect synccommand in Apply .posit/publish config Connect-side settings: post-deploy hook +rsconnect sync#831.uv.lockdiff is large but only addstomli-w/pathspec(a reserialization; nothing removed or version-bumped).Testing
tests/test_publisher.py— type-map coverage (incl. TensorFlow →unknown), round-trips, idempotent redeploy, URL normalization, a Publisher-authored interop fixture,connect_cloudround-trip, naming methodology, config/record pinning, and that a written config'sfilesreference the record actually written.tests/test_redeploy.py— command registration, identity reuse, config-name selection, legacy fallback, credential matching by URL (incl. ambiguous → error), and tarball-level assertions thatredeploybundles exactly the config's files, matchesdeploy's bundle, stays stable across repeated redeploys, and — for a project with no.posit— that a second deploy still bundles a later-added module and gitignored rendered output.tests/test_publisher_files.py— config allowlist matching (rooted vs. basename,!exclusion, dir includes,**,STANDARD_EXCLUSIONS, venv/renv skipping), thecreate_file_listrestriction + builder-exclude interaction,resolve_bundle_filesreturningNonewithout a config (and a gitignored build artifact still bundling) and for an unrestricted config (absent /["*"]/["*"]+.positpaths), restriction for a genuinely curated config, entrypoint force-include, andintegration_requestspropagation into the manifest (incl. an end-to-end executor test).ruff format/ruff checkclean. A session-scoped conftest fixture keeps deploy/write-manifest tests from leaving.positartifacts intests/testdata.Try it
uvx --from "git+https://github.com/posit-dev/rsconnect-python.git@feat/publisher-toml-support" rsconnect redeploy --help🤖 Generated with Claude Code