A dict watcher that follows the documented error protocol makes CPython re-enter Python mid-update

[devdanzin]

Crash report

What happened?

Doc/c-api/dict.rst tells a PyDict_WatchCallback two things:

The callback may inspect but must not modify dict; doing so could have unpredictable effects, including infinite recursion. Do not trigger Python code execution in the callback, as it could modify the dict as a side effect.

— dict.rst:582-584

If the callback sets an exception, it must return -1; this exception will be printed as an unraisable exception using PyErr_WriteUnraisable.

— dict.rst:595-596

A callback that does exactly the second thing causes CPython to do exactly the first thing on its behalf. _PyDict_SendEvent (Objects/dictobject.c:8309-8317):

if (cb && (cb(event, (PyObject*)mp, key, value) < 0)) {
    PyErr_FormatUnraisable(
        "Exception ignored in %s watcher callback for <dict at %p>",
        dict_event_name(event), mp);
}

PyErr_FormatUnraisable runs sys.unraisablehook, which is settable from pure Python. So the "do not trigger Python code execution" requirement is discharged by the callback and then violated by the runtime — inside the notify window, while the dict is mid-update.

The docs do say the exception becomes an unraisable. What they don't say is that this happens synchronously, inside the operation, at a point where the calling code has already captured an entry index, an entry pointer, a bound, or a borrowed reference that it will use afterwards.

Thirteen of the fourteen notify sites hold stale state across that window

I drove all fourteen. Thirteen hold state across the notify and all thirteen produce an observable failure; twelve of those are memory corruption or a memory-invariant assertion. One site is safe.

site	enclosing function	stale value	worst observed
:1917	insert_combined_dict	dk_usable bound	ASan heap-buffer-overflow WRITE, "0 bytes after" a 1520-byte region; release: silent corruption, exit 0
:1997	_PyDict_InsertSplitValue	ix, split-table-ness	SIGSEGV at :1998, ma_values == NULL — 5/5 on each of four builds
:2003	_PyDict_InsertSplitValue	ix, old_value	SIGSEGV at :2004 5/5 on four builds; embedded variant asserts in clear_freelist
:2060	insertdict	ix, old_value	ASan heap-UAF READ at :2076; heap-buffer-overflow WRITE at :2068; _Py_NegativeRefcount 5/5
:2103	insert_to_emptydict	unpublished newkeys	ma_used desync puts a NULL in a list → SIGSEGV in list_sort_impl
:3038	_PyDict_DelItem_KnownHash_LockHeld	ix	debug assert(hashpos >= 0); release Py_DECREF(NULL)
:3083	delitemif_lock_held	ix, old_value, hash	ASan SEGV Py_DECREF(NULL); len() → NULL 5/5 release
:3142	clear_lock_held	keys pointer	ASan use-after-free at dictkeys_decref:496
:3307	_PyDict_Pop_KnownHash	ix, old_value	ASan heap-UAF READ at :3308; SIGSEGV 5/5 release
:4234	dict_dict_merge	the whole guard set	debug assert in clone_combined_dict_keys; silent data loss 5/5 — no memory corruption reached
:5051	dict_popitem_impl (unicode)	ep0	ASan UAF R+W; popitem() returns a tuple whose [1] is a raw C NULL
:5066	dict_popitem_impl (general)	ep0, i	ASan heap-UAF READ at the site, :5067
:7510	store_instance_attr_lock_held	values, dict, ix, old_value	assert(i < size); ASan overflow WRITE + UAF READ; len(), list() and vars() all raise
:3652	dict_dealloc	(none)	safe — reads ma_values/ma_keys at :3656-3657, after the notify

Also reached from these paths: delete_index_from_values:2943, whose search loop is bounded only by assert(i < size) — debug SIGABRT, and on plain release it silently loses a dict entry.

Four that are worth calling out individually

:1997/:2003 are in _PyDict_InsertSplitValue, which is PyAPI_FUNC and carries the comment // Exported for external JIT support. The hook that kills it is d.clear() after del obj has detached the dict, which makes clear_lock_held take its set_values(mp, NULL) branch; the split-table store then dereferences NULL. gdb confirms ma_values == 0x0, ma_keys == &empty_keys_struct, ix == 2.

:3083 breaks a promise the file makes in a comment. dictobject.c:3090-3094 tells callers the predicate → deletion sequence is atomic provided the predicate does not re-enter. is_dead_weakref does not re-enter; the notify does. Reachable as _weakref._remove_dead_weakref(dct, key).

:3307 hands freed memory back to Python. *result at :3312 is the dict's own reference, returned straight to the dict.pop() caller after the hook dropped it.

:7510 gives the sharpest Python-visible invariant break. delete_index_from_values does size-- on a size the hook zeroed, and values->size is a uint8_t, so it wraps to 255 while :7527 drives ma_used to -1. After a plain del o.attr, on release-gil-nojit, 5/5:

len(d)   -> SystemError: <built-in function len> returned NULL without setting an exception
list(d)  -> ValueError: length must be positive
vars(o)  -> ValueError: length must be positive

The one I could not corrupt

:4234 (dict_dict_merge, PyDict_EVENT_CLONED) holds every guard established at :4219 and :4228-4232 across the notify, and those guards are genuinely unchecked afterwards — but I could not reach memory corruption in roughly nine hook variants (0/3 under ASan). The two guards whose falsification would give a type-confused dict are out of reach from Python: mp is a plain dict so a hook cannot make mp->ma_values non-NULL, and a combined other cannot be turned back into a split table. What it does give is a 5/5 debug assertion violation in clone_combined_dict_keys and 5/5 silent data loss on all four builds — d.update(src) discards everything the hook wrote to d, with no error. On release, that path also leaves d owning a heap keys object carrying _Py_DICT_IMMORTAL_INITIAL_REFCNT (memcpy'd from empty_keys_struct), which can never be freed and permanently falsifies clear_lock_held's assert(oldkeys->dk_refcnt == 1) at :3148.

Reproducer

The callback is CPython's own — dict_watch_callback_error in Modules/_testcapi/watchers.c:90-98, in its entirety:

static int
dict_watch_callback_error(PyDict_WatchEvent event, PyObject *dict,
                          PyObject *key, PyObject *new_value)
{
    PyErr_SetString(PyExc_RuntimeError, "boom!");
    return -1;
}

That is the documented protocol and nothing else: it sets an exception, returns -1, and runs no Python. The re-entry comes entirely from CPython's response to it.

import sys, _testcapi

d = {}
for i in range(200):
    d["k%d" % i] = i

fired = []
def hook(unraisable):
    if fired:
        return
    fired.append(1)
    d.clear()                       # re-enter the dict being mutated

sys.unraisablehook = hook
wid = _testcapi.add_dict_watcher(1) # installs dict_watch_callback_error
_testcapi.watch_dict(wid, d)

del d["k100"]                       # notify -> -1 -> unraisable -> hook -> clear()

print("len now %d" % len(d))

# debug build -- exit 134 (SIGABRT)
python: Objects/dictobject.c:2963: void delitem_common(...): Assertion `hashpos >= 0' failed.

# release build -- exit 1
SystemError: <built-in function len> returned NULL without setting an exception

One per-site reproducer for each of the thirteen is attached, along with negative controls — for :2060, hooks that store immortal small ints or force a resize do not crash it, so the claim is not "any hook corrupts anything". That site needs heap values specifically: with small ints the stale Py_XDECREF(old_value) at :2076 is a no-op because they are immortal, which is why an earlier attempt at this site came back clean.

Registering a watcher requires the C API, so this is not reachable from pure Python alone — but sys.unraisablehook is pure Python, and any extension that installs a conforming watcher hands that reach to arbitrary Python code.

The one safe site, and why

dict_dealloc (:3650-3658) is the only notify site that survives this. It brackets the notify in _PyObject_ResurrectStart/End, but that bracket defends against resurrection — the comment at _PyDict_SendEvent:8309-8312 names that threat and only that threat. Its mutation-safety is incidental: it re-reads ma_keys/ma_values after the notify rather than before.

So the thing to propagate is the ordering, not the bracket.

Notes toward a fix

Three directions, not mutually exclusive:

1. Re-validate after the notify at each of the thirteen sites — reload the index, entry pointer, or bound. Correct but repetitive, and easy to regress: this is thirteen independent chances to get it wrong again.
2. Defer the unraisable until the dict operation completes, so the callback's error is still reported but not from inside the window.
3. Guard the re-entry centrally in format_unraisable_v (Python/errors.c:1737), which is the single choke point every unraisable site funnels through.

Given that thirteen of fourteen sites are affected, the central options look more attractive than the per-site one.

Objects/typeobject.c:1219 already carries the observation, added in gh-127266:

// Note that PyErr_FormatUnraisable is potentially re-entrant and the watcher
// callback might be too

dictobject.c has no equivalent, and its notify sites were never audited against it.

Related

• CPython's own reference watcher, dict_watch_callback (watchers.c:31-70), violates both halves of the documented contract: PyUnicode_FromFormat("new:%S:%S", ...) calls PyObject_Str (forbidden by dict.rst:583-584), and it calls PyUnicode_FromFormat + PyList_Append with no save/restore (required by dict.rst:599-603). If the example cannot follow the contract, it is worth asking whether the contract is stateable.
• PyType_WatchCallback (Doc/c-api/type.rst:138-155) documents no error protocol at all, yet typeobject.c:1222 treats < 0 as "exception set" and reports it the same way.

Found with [cpython-review-toolkit]](https://github.com/devdanzin/cpython-review-toolkit); reduced, reproduced and drafted with AI assistance (Claude Code)

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

Python 3.16.0a0 (heads/main:a1d580430c8, Jul 18 2026, 20:21:17) [Clang 21.1.8 (6ubuntu1)]

[BHUVANSH855]

@devdanzin
I investigated the existing watcher implementations before working further on a patch.

Type, code, function, and context watchers all call PyErr_FormatUnraisable() immediately when a callback fails. In particular, typeobject.c explicitly documents that PyErr_FormatUnraisable() and watcher callbacks are potentially re-entrant, so immediate reporting appears to be the intended design.

I also prototyped revalidating the lookup in _PyDict_DelItem_KnownHash_LockHeld() after _PyDict_NotifyEvent(). That eliminates the crash from the reproducer, confirming that the stale lookup state is the immediate cause, but it also changes semantics (the reproducer raises KeyError instead).

Based on that investigation, it seems the issue is specific to dict notification occurring before the mutation has completed, rather than to the generic watcher error-reporting mechanism.

Before extending the prototype to the other notification sites, is the preferred direction to rework the dict notification/mutation sequence, or is there another approach you'd like the fix to follow?

[devdanzin]

@BHUVANSH855 Thank you! This is a really good piece of investigation, and I want to say why before I get to the details: you checked how the sibling watcher implementations behave before writing a patch, and you built a prototype to test whether your explanation of the cause was actually true. That order (understand, then verify, then patch) is the habit that matters most in this codebase, and plenty of experienced contributors skip it. You also found something I had not stated clearly, which I'll come to below.

Let me go through your findings, then answer your question directly.

Your two factual points both check out

Immediate PyErr_FormatUnraisable() is intended. You're right, and the code says so in as many words — Objects/typeobject.c:1219-1220:

// Note that PyErr_FormatUnraisable is potentially re-entrant
// and the watcher callback might be too.

Your localization of the immediate cause is sharper than my original report. I framed this around the watcher error-reporting mechanism; you've narrowed it to a specific stale-state consumption, which is more useful. In _PyDict_DelItem_KnownHash_LockHeld:

ix = _Py_dict_lookup(mp, key, hash, &old_value);           /* :3030  compute  */
...
_PyDict_NotifyEvent(PyDict_EVENT_DELETED, mp, key, NULL);  /* :3038  runs user Python */
delitem_common(mp, hash, ix, old_value);                   /* :3039  consume the stale pair */

That's the defect in three lines, and it's a shape worth learning to recognize: read some state, call into user Python, then keep using the state as if nothing happened. It's the same shape as gh-154709, and I've been finding it repeatedly elsewhere in the C code. Once you've seen it a few times you start spotting it by eye.

One nuance I'd add to your conclusion

You concluded the issue is dict-specific "rather than the generic watcher error-reporting mechanism." I'd put it slightly differently: the other four watchers get away with immediate reporting because none of them are holding a stale index across it. Dict is. So it isn't quite that the error mechanism is unrelated, it's that dict is the one caller for which that mechanism's (documented, intended) re-entrancy is unsafe.

That distinction matters because it means there are two places you could fix this, and both work.

The documented constraint that answers half your question

You asked whether the preferred direction is to rework the dict notification/mutation sequence. I think that one is actually off the table, and it's worth knowing why — Doc/c-api/dict.rst makes it a promise:

Callbacks occur before the notified modification to dict takes place, so the prior state of dict can be inspected.

Moving the notify after the mutation would break that guarantee, so it can't be done as a bugfix. Good news for you: that's one of three options eliminated for free.

While you're in that file, it's worth reading the whole callback contract, because there's a real tension in it that I think is the root of this issue. The same document says callbacks

may not modify dict or otherwise cause code execution in the callback, as it could modify the dict as a side effect

and also says that if the callback sets an exception,

it must return -1; this exception will be printed as an unraisable exception

So a callback that does exactly what it's told, i.e. returns -1 on failure, causes CPython itself to execute arbitrary Python through sys.unraisablehook, at precisely the moment the contract says code must not run. A conforming callback can't avoid it. That's why I don't think this is only a dict-ordering bug: the contract as written can't be satisfied by anyone.

Two suggestions on your prototype

The KeyError may not be forced. If the key has already been removed by the time you revalidate, then the postcondition of del d[k] is satisfied, the key is gone. Returning 0 rather than raising is arguably more defensible and avoids the visible behaviour change you were rightly worried about. (The case where the watcher replaced the value instead of deleting it is genuinely murkier, and probably does need to raise or re-delete.)

There's a third option worth considering. Rather than revalidating, you could defer the report: save the exception, let delitem_common finish, then call PyErr_FormatUnraisable. That preserves both documented guarantees and changes no Python-visible behavior at all, which makes it the narrowest possible fix. The typeobject.c comment you found documents that PyErr_FormatUnraisable is re-entrant. I'd read that as a warning to callers to be careful around it, rather than evidence that every existing caller is already safe.

I'm not certain that's the right answer either. It's just the option I'd want on the table.

Why I'm not going to pick a direction

You asked which direction is preferred, and the answer is that I can't authorize one. All of the remaining options change either a documented C-API guarantee or Python-visible behaviour, and the real question underneath is which of two documented guarantees should yield. That's a call for the people who own the watcher API, not for me as the reporter, and not for either of us to settle in an issue thread. Both of those doc sentences trace back to the original dict watcher work (a4b77948879, gh-91052) and its follow-up (1e703a47334, gh-102381), by Carl Meyer.

So I'd suggest we get core-dev input before more code gets written. I'm happy to open a post on discuss.python.org in the C API category laying out the three options and their tradeoffs (the doc contradiction, the ordering guarantee, the revalidate-vs-defer choice, and the KeyError-vs-0 question) so the API owners and other interested folks can weigh in somewhere more visible than this thread. I'll link it here when I do, and I'd like to credit your investigation in it, since the localization and the prototype result are both yours.

Concretely, I'd hold off on extending the prototype to the other notification sites for now. Not because the work isn't good — because that's exactly the effort that gets thrown away if the chosen direction turns out to be deferring the report rather than revalidating. Better to spend a week waiting than a week rewriting.

If you want something to dig into while that settles: the same read-then-call-user-code-then-use pattern is worth grepping for around the other 13 _PyDict_NotifyEvent call sites. Knowing which of them hold state across the notify — and which don't — would be genuinely useful input to the discussion, whichever direction wins, and it's analysis rather than code, so none of it is wasted.

Thanks again for picking this up and for asking before charging ahead. That instinct is the right one.

Investigation and draft assisted by Claude Code.

[devdanzin]

DPO thread: https://discuss.python.org/t/dict-watchers-the-documented-callback-contract-cannot-be-satisfied-gh-154710/108282.