Skip to content

feat: add an AI page#821

Merged
henryiii merged 16 commits into
mainfrom
henryiii/feat/aipage
Jul 24, 2026
Merged

feat: add an AI page#821
henryiii merged 16 commits into
mainfrom
henryiii/feat/aipage

Conversation

@henryiii

@henryiii henryiii commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator

Fix #807.

This adds an AI page. Fitting to the theme, it was started by pointing Claude Opus at the issue and then reworked and added to (original output available in a comment on that issue).

There still are some things left to do:

  • AI_POLICY.md probably needs a proper human-written version for all three examples.
  • We should add some setup to the cookie, either ignores or an AGENTS.md. I'd be biased toward not providing an AGENTS.md, but looking at the generated AGENTS.md and making sure it covers everything; anything it's missing we could add to the exisiting contribution guide, which should then show up in the AGENTS.md. But open to playing with this and other options. (Followup, and only if anything is needed)
  • I think we should not add an AI_POLICY.md, at least yet, and let projects select and modify their own.

I'll also leave this open for a bit for reviews and suggestions.


📚 Documentation preview 📚: https://scientific-python-cookie--821.org.readthedocs.build/

@henryiii
henryiii marked this pull request as draft June 25, 2026 15:47
Comment thread docs/guides/ai.md Outdated
@guenp

guenp commented Jul 16, 2026

Copy link
Copy Markdown

@ericmjl ericmjl left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, really loving the policy! So many things resonate. I will be doing a series of blog posts on the weekend about scipy, including commentating on my thoughts on the use of AI in open source, and I think referencing this PR is going to be part of the story!

Comment thread docs/guides/ai.md Outdated
Comment thread docs/guides/ai.md Outdated
Comment thread docs/guides/ai.md
Comment thread docs/guides/ai.md Outdated
@lundybernard

lundybernard commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Something for the tips section, or a separate Patterns article we could reference from this one.

I would recommend people add Architecture Design Records (ADRs) to projects where they use or support the use of Ai. They make explicit, design decisions that are often invisible or obscure in the code its self, and are beneficial to human contributors even in a world without Ai. Recommending from personal experience, I've seen major improvements in agent behavior using this strategy.

I had Fable dig up some papers, review my own setup, and write this argument for their use with citations:


🤖 AI text below 🤖

There's early quantitative support that decision-record-style context is the kind that actually helps agents. A controlled study found agents complied with project decisions 46% of the time from code alone vs. 95% when decision records were retrievable — with the gap concentrated in decisions that aren't visible in the code arXiv:2605.08112. On SWE-bench Verified, tuned repository guidance lifted resolve rates from 25.5% to 33.0% (p<0.001), and the useful content was largely structural/architectural knowledge arXiv:2606.20512. Notably, a null-result study on AGENTS.md files found generic repo overviews don't help — but prescriptive instructions are followed arXiv:2602.11988. ADRs land on the right side of that split: decisions and constraints, not prose summaries.


@henryiii

henryiii commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

I've had Fable look at other AI_POLICY.md's, review below.

🤖 Details 🤖

All three research agents are done. Here's the gap analysis against our three samples.

What the samples already cover well

Disclosure with tool naming, "review and understand every line," the quality bar, the no-AI-co-author rule with the Assisted-by: trailer, and (in Moderate) prior maintainer buy-in. These match the mainstream of real policies — the arXiv survey of 118 GitHub repo policies found 78% permissive, ~51% requiring disclosure, 74% requiring human oversight, so the three-level framing holds up.

Gaps — things nearly every real policy has that our samples don't

  1. No autonomous-agent line. This is the most consistent clause in the wild: NumPy, SciPy, JAX, LLVM, matplotlib, PyTorch, scikit-learn, and Home Assistant all prohibit fully autonomous agents submitting PRs, even in otherwise welcoming policies. Our page's intro makes exactly this distinction (power tool vs. unattended slop factory), but none of the three samples state it. Even "All in" should say something like "a human must drive the tool and personally submit the result."

  2. Samples only cover PRs. Real policies almost all scope to issues, comments, and review replies too. pandas, scikit-learn, and Home Assistant explicitly cover issues; PyTorch bans AI solution write-ups in issues; Kubernetes closes PRs if you answer review comments through an AI ("don't be a message courier" — Astropy). Our page prose has "keep human review human-to-human" but the samples don't.

  3. No translation/grammar exception. Almost universal — scikit-learn, NumPy, SciPy, JAX, matplotlib, curl, Fedora, Home Assistant all carve out grammar fixing and translation for non-native speakers. Without it, a strict reading of our Moderate/Minimal samples forbids something every real project permits. This is also an inclusivity point the page's intro note already makes.

  4. No copyright/licensing responsibility clause. The SymPy-lineage policies (NumPy, SciPy) say plainly: "Code generated by AI may infringe on copyright, and it is the submitter's responsibility to avoid infringement." It's the entire basis of the ban-side policies (QEMU, Gentoo, NetBSD, ASF/LF legal guidance). Our samples touch licensing only via the co-author sentence.

  5. Thin enforcement language. Real policies get specific: close without review (scikit-learn), two-strikes ban (pandas), block the account and report to GitHub (matplotlib), public denouncement list (Ghostty). The "All in" sample has no consequence at all; even a welcoming policy usually ends with "non-compliant contributions will be closed."

  6. Extent, not just tool name. NumPy/SciPy/Ghostty require stating which parts are AI-generated and to what extent, not just naming the model. Ghostty and Fedora also add the opposite bound: trivial help (tab-completion, grammar) needs no disclosure. Our samples have neither the upper nor the lower bound.

Angles no sample covers (candidates for surrounding prose rather than the samples)

  • Security reports. curl requires disclosure of AI in vulnerability reports and killed its bug bounty in Jan 2026 over slop reports (restored, unpaid, in March). Seth Larson documented the same for CPython/pip. Worth at least a sentence in the guide.
  • AI on the reviewer side. Fedora: reviewers may use AI to assist but not to wholly automate review or make the accept/reject call. Django bars automated AI reviews outright.
  • Good-first-issue carve-out. LLVM and Mozilla both forbid using AI to snipe "good first issues" — it squanders the learning opportunity they exist for.
  • Rate/size limits. Homebrew allows one open AI-assisted PR at a time; JAX asks for "review at a human pace"; Kubernetes bans large AI-generated PRs. A concrete, enforceable anti-slop lever the samples lack.
  • The kernel policy is now official — merged as Documentation/process/coding-assistants.rst. Exact format is Assisted-by: AGENT_NAME:MODEL_VERSION [TOOL1] [TOOL2], and it adds a rule worth quoting: AI agents must never add Signed-off-by: — DCO certification is a human legal act. Our "Linux kernel trailer" reference can now cite the real doc.
  • The trailer isn't universal. Kubernetes explicitly forbids Assisted-by:-style trailers and wants disclosure in the PR description instead; ASF uses Generated-by:. Our samples present the trailer as the way; a caveat ("or however the project requests disclosure") may be fair.
  • LLVM's "golden rule" is the best one-line summary in any policy and matches our "Don't submit slop" section: "a contribution should be worth more to the project than the time it takes to review it" (curl uses the same sentence). Worth quoting/linking.
  • Good link candidates: melissawm's list of ~130 project policies, LLVM's AIToolPolicy, the kernel doc, and real AI_POLICY.md examples (PyTorch, Ghostty, Home Assistant) as "see also" references — also validates that AI_POLICY.md is the de facto filename, as the page claims.

What I'd change, concretely

To each sample, add three short lines: an autonomous-agent clause, scope covering issues/comments/review replies ("respond to reviewers yourself"), and a translation/grammar exception. Add a copyright-responsibility sentence to Moderate and Minimal, and a closure consequence to "All in". Then a short paragraph in the surrounding prose noting the security-report and reviewer-side angles with links to curl/Fedora/LLVM, and update the kernel trailer paragraph to cite the merged kernel doc plus the no-Signed-off-by rule.

henryiii and others added 9 commits July 23, 2026 23:47
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Added tips for using AgentsView and Claude Code.
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Co-authored-by: Eric Ma <ericmjl@users.noreply.github.com>
Co-authored-by: Eric Ma <ericmjl@users.noreply.github.com>
Co-authored-by: Lundy Bernard <lundybernard@users.noreply.github.com>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
@henryiii
henryiii force-pushed the henryiii/feat/aipage branch from c328c0b to 39d368e Compare July 24, 2026 03:48
henryiii added 3 commits July 24, 2026 00:02
Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Assisted-by: ClaudeCode:claude-fable-5
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
@henryiii

Copy link
Copy Markdown
Collaborator Author

I'll let you (@lundybernard) add an ADR tip paragraph in a followup (or put it here and I'll paste it in). We probably can split this up and have more detailed tips in second page, but let's go with this for a first attempt (and keep tips to a paragraph each for now).

I've worked on the page a bit, tightened it up with a few drop-downs, added some things other people put in their policies, and expanded the tips a bit. I'm pretty happy with it, if others are we can put it in.

@henryiii
henryiii marked this pull request as ready for review July 24, 2026 04:39
@henryiii

Copy link
Copy Markdown
Collaborator Author

Oh, and I'm leaving any cookie updates (if there are any) for later.

@henryiii

Copy link
Copy Markdown
Collaborator Author

https://scientific-python-cookie--821.org.readthedocs.build/en/821/guides/ai/ is the preview

Assisted-by: ClaudeCode:claude-opus-5
@henryiii

Copy link
Copy Markdown
Collaborator Author

Missed one of the links in #821 (comment), added it now.

henryiii added 2 commits July 24, 2026 17:11
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
Name a real project for each AI_POLICY.md level: the Linux kernel and
CPython for all in, NumPy for moderate, and nvim-tree for minimal. Also
add CPython to the other examples list.

Assisted-by: ClaudeCode:claude-opus-5
Signed-off-by: Henry Schreiner <henryfs@princeton.edu>
@henryiii
henryiii merged commit 1cbbed0 into main Jul 24, 2026
7 checks passed
@henryiii
henryiii deleted the henryiii/feat/aipage branch July 24, 2026 21:55
@henryiii

Copy link
Copy Markdown
Collaborator Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

New page idea: Agentic AI

4 participants