Skip to content

Tests: Test exception type in error tests#22799

Draft
NickSdot wants to merge 60 commits into
php:masterfrom
NickSdot:test-hardening
Draft

Tests: Test exception type in error tests#22799
NickSdot wants to merge 60 commits into
php:masterfrom
NickSdot:test-hardening

Conversation

@NickSdot

@NickSdot NickSdot commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Test files sweep to make exception assertions more robust, and slightly unify style.

Specifically:

  • adds thrown class to all exception assertions to protected against regressions
  • further established the already recommended single exception formatting
  • adds missing last line whitespaces were missing

Target format:

echo $e::class, ': ', $e->getMessage(), PHP_EOL; // + variations like line(), code(), etc.

Approach:
Rewrites are handled by this deterministic helper for conservative and safe, automated replacements. All exception assertion "flavours" from php-src were extracted, and the tests where they were found promoted to source fixtures. Rules were then ran against the fixtures, the results manually reviewed, and promoted to target fixtures. All replacement rules are tested against the source and target fixtures. The actual "fixing" works as follows:

  1. Scan PHPT files for exception-message output inside catch blocks.
  2. Run the original PHPT through run-tests.php; skipped or already failing tests are not rewritten.
  3. Rewrite the PHPT to the normalised exception-output shape.
  4. Run the rewritten PHPT through run-tests.php.
  5. If output changed as expected, update the expected section from the actual run output.
  6. Run the updated PHPT once more.
  7. Keep only rewrites that pass verification.

Modules & Commits

Target Commit Files
ext/bcmath 159e86c 49
ext/bz2 0549fcf 7
ext/calendar 0326a84 15
ext/curl 33ea095 24
ext/date 64d2f63 75
ext/dba 9c2e40c 6
ext/dom bc3b809 194
ext/exif 515cefd 4
ext/ffi fad8341 29
ext/fileinfo e915ce1 12
ext/filter c101e9c 13
ext/ftp 5c08cb3 13
ext/gd f731248 40
ext/gettext e4a6966 8
ext/gmp d7da7fb 62
ext/hash d7f05e7 19
ext/iconv 7eb14e6 3
ext/intl d0dda28 66
ext/json 46db245 8
ext/ldap 440d8b6 9
ext/libxml da846f1 5
ext/mbstring a2b9874 78
ext/mysqli 6aacba6 6
ext/odbc 5d0b82e 1
ext/opcache 8a43524 114
ext/openssl b73e007 40
ext/pcntl 91b2944 15
ext/pcre 98c7dea 10
ext/pdo 292e224 3
ext/pdo_sqlite eaf2a88 21
ext/pgsql 726b2c0 2
ext/phar cc62522 183
ext/posix 5b5304d 8
ext/random a5ad2fa 21
ext/readline 4469fdd 2
ext/reflection e35b23f 107
ext/session 8fe5b85 18
ext/shmop ad29684 1
ext/simplexml 14f21a6 10
ext/snmp e08369c 2
ext/soap 3759798 17
ext/sockets 3668650 26
ext/sodium 98303d9 17
ext/spl 8144355 175
ext/sqlite3 0a000c1 19
ext/standard eac7e9b 512
ext/sysvmsg 5b4d5a4 3
ext/sysvshm 72cc773 5
ext/tidy eba9e5f 10
ext/tokenizer 803ce00 5
ext/uri 9d68fd0 111
ext/xml f35e569 5
ext/xmlreader a83aeb3 23
ext/xmlwriter 4426f57 5
ext/zend_test 767a49e 13
ext/zip 42c2e7d 17
ext/zlib 2802421 18
Zend 7213ed7 632
tests f89fe1f 55
sapi/cli 059e2af 1

Changelog:

  • 2026-07-20: added ext/intl and ext/phar
  • 2026-07-21: target format \PHP_EOL -> PHP_EOL

PoC, marked as draft for now, and in coordination with Tim for now only applied to ext/random and ext/uri.

@NickSdot

Copy link
Copy Markdown
Contributor Author

@LamentXU123 I noticed your like and since you own ext/intl and ext/phar I applied the fixers to it and pushed two new commits. Would love your feedback!

Aside: since I didn't built with all extensions ext/intl serves as a good example for non-provable rewrites. In my run listformatter_with_parameters_error.phpt was skipped.

@thg2k

thg2k commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

Is the \PHP_EOF really necessary? I think it just is silly in the tests. Just PHP_EOF should be fine.

@NickSdot

Copy link
Copy Markdown
Contributor Author

Is the \PHP_EOF really necessary? I think it just is silly in the tests. Just PHP_EOF should be fine.

Nah. Picked it out of habit. No strong opinion; can adjust to whatever we decide.

@LamentXU123

Copy link
Copy Markdown
Member

I actually think this change is trivial. I don't have strong opinions on this (I am on the fence of this huge code churn) so might need to wait for other's opinion.

@LamentXU123

Copy link
Copy Markdown
Member

nit: any reasons for targeting 8.5?

@NickSdot

Copy link
Copy Markdown
Contributor Author

I actually think this change is trivial. I don't have strong opinions on this (I am on the fence of this huge code churn) so might need to wait for other's opinion.

Sometimes not avoidable to move things forward. :) Personally, I think it adds enough value to be justified. Flipping \PHP_EOL to PHP_EOL would decrease churn by a bit. Do you also think it's silly? If yes could pull the trigger on making it PHP_EOL.

nit: any reasons for targeting 8.5?

Nope. Could be anything. Which one you want?

@LamentXU123

Copy link
Copy Markdown
Member

Nope. Could be anything. Which one you want?

Should be targeting master. (hint: convert this to draft when you are changing the base to avoid accidentally request a review from everyone)

@NickSdot

Copy link
Copy Markdown
Contributor Author

Nope. Could be anything. Which one you want?

Should be targeting master. (hint: convert this to draft when you are changing the base to avoid accidentally request a review from everyone)

Given that you mentioned code churn, I am wondering if targeting master is the right call? Like, when someone diffs 8.6 against 8.5 this would all be extra in the diff. Why not 8.5 and then upmerge to master? Madee a full run over all tests, merging it up would have only 37 merge conflicts on a total of 2424 affected files. After the upmerge a separate master targeted run would only affect 83 new files. Moderate, no?

@NickSdot

Copy link
Copy Markdown
Contributor Author

Changed to PHP_EOL; 37 files less affected; still targets PHP 8.5.

@TimWolla

Copy link
Copy Markdown
Member

I am wondering if targeting master is the right call?

Yes. This is a refactoring, which is not acceptable in released branches.


To reduce churn, can the formatter also keep the type of quotes for the colon in cases where the general format is already as expected? The changes in ext/uri provide no value to improve the strength of the actual assertion. Phrased differently: If the change to the test code does not result in a change in the EXPECT section, then drop the change to the test code.

@NickSdot

Copy link
Copy Markdown
Contributor Author

I am wondering if targeting master is the right call?

Yes. This is a refactoring, which is not acceptable in released branches.

Will target master! (surprised that this applies to tests)

To reduce churn, can the formatter also keep the type of quotes for the colon in cases where the general format is already as expected? The changes in ext/uri provide no value to improve the strength of the actual assertion. Phrased differently: If the change to the test code does not result in a change in the EXPECT section, then drop the change to the test code.

I evaluated this before. Not worth the extra complexity and inconsistency, IMO. Why: your modules are already so clean that ext/uri is in fact the clear outlier -- 73/104 files are in your ext/uri, only 31 more files that would be affected by what you propose are in all other tests (as in: all phpt, not only in the four modules committed here). Would not start to make things inconsistent or introduce extra rules for that, personally. However, if changing these 104 files (out of 2,369 total) would be an absolute dealbreaker I of course can make one manual commit to revert them. :)

Though, there are other cases like where output is fixed in try/catch that not actually throw. These are risk free to fix, I did so, and I think we should. Why: contributor comes to an existing test file to add test, copies existing test, style is wrong, you review and flag it. You wasted time, contributor wasted time. Or it's missed and merged. Both avoidable. Having one consistency sweep will decrease this "risk". Even code owners have recent PRs that do not follow the regression-safe pattern. When -- after this here would be merged -- everyone sees everywhere the exact same pattern it will naturally be be better adapted. The same would of course be possible if we would make this part of the CI; but you didn't seem very bullish on that?

Assumed we would do what you say, here some stats: 545 total files less; 243 are one lines changes, 334 out of it are <= 2 line diffs, 411 are <= 5 line diffs.

@LamentXU123

Copy link
Copy Markdown
Member

The document CI failure is fixed in 8334703.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants