From 931a870ec7c3fdbdbaaa317401c71a8c287b405f Mon Sep 17 00:00:00 2001 From: igerber Date: Sun, 26 Jul 2026 15:54:58 -0400 Subject: [PATCH] fix(v4): ETWFE reference-period normalization (#724) + Stata jwdid/csdid parity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes issue #724 and anchors both staggered estimators to their canonical Stata implementations. Eleven local review rounds turned up further identification defects in the same code, including a P0; those are fixed here too. ## The #724 fix With `control_group="never_treated"`, the saturated ETWFE design emitted every cohort x time indicator INCLUDING the reference period -- collinear with the absorbed unit fixed effects -- and left generic QR rank detection to remove whichever column it picked. On the real `mpdta` panel that silently removed ATT(2004, 2004) and ATT(2006, 2007), returning 5 of 7 post-treatment cells. The reference is not a free choice: W2025 Eq. 6.1 excludes `dg_i·f(g-1)_t` "so that s = g-1 is the reference period", and Stata `jwdid ... never` omits the same cell. The design now omits `ref(g)` explicitly -- per-cohort and weight-aware, since a panel-wide rule omits an all-zero column when a cohort is unobserved there and leaves the collinearity intact. The surviving pre-fix estimates were not merely incomplete, they were measured against the wrong baseline. ATT(2007,2007) by hand against references 2006/2005/2004/2003 gives -0.0261 / -0.0571 / -0.0599 / -0.0294; the library returned -0.0571, i.e. a correct 2x2 DiD against a silently chosen 2005. Not confined to `never_treated`: any cohort whose cell block spans its own cohort dummy hits it, which `anticipation > 0` makes reachable on the default `not_yet_treated` path. ## Stata parity arm `benchmarks/stata/generate_etwfe_cs_golden.do` anchors ETWFE and CallawaySantAnna against `jwdid` / `csdid` on the committed `mpdta` panel (pinned SHA-256, so CI needs neither Stata nor network). This REPLACES a test that validated nothing: `test_wooldridge.py` asserted ETWFE == CS within 5e-3, which is false on real data (0.0171 at (2007,2007)) and passed only because `load_mpdta()` was silently serving a synthetic effect-homogeneous DGP when its source 404'd (issue #722). Stata's own two implementations reproduce the same disagreement, establishing it as an estimator property rather than a bug. The arm also surfaced a real deviation: ETWFE `hc1` SEs are uniformly BELOW `jwdid`'s (anti-conservative), ~0.1% at G=500 and ~2.8% at G=20. The mechanism is NOT identified -- it tracks sqrt(G/(G-1)) but sits above it, and `solve_ols` already applies the full CR1 -- so it is recorded as measured, not diagnosed, and pinned rather than "corrected" with an unverified factor. The never-treated arm shows the SAME factor over a different cell count, which rules out explanations keyed to the comparison set. ## Identification defects found while reviewing the above - P0: partial rank loss returned MISLABELED contrasts as ATTs. When some treatment columns were dropped, the survivors were re-identified as whatever contrast the reduced design supports and kept their ATT(g,t) labels. Measured with true effects 1.0 and 3.0: ATT(3,5) came back as -1.9393 -- wrong sign -- inside overall_att = +0.7703 against a true ~1.8, with NO warning under `rank_deficient_action="silent"`. Pre-existing and worse at base. A fit now fails closed unless every requested cell survives. - Within-cohort support CONNECTIVITY: per-period support is necessary but not sufficient; identification runs through the unit FE. A cohort whose units split into disjoint observation windows had a genuine post cell dropped (`g4_t5`) with the overall ATT computed from the survivor. OLS-only -- the nonlinear paths use explicit dummies and are unaffected. - A fit no longer returns a NaN overall ATT as success (point estimate only; NaN SE stays legitimate where only the variance is unidentifiable). - Invalid survey weights (NaN / negative / Inf, fractional fweight) and invalid strata/PSU/FPC could hide in rows that cohort exclusion deleted, yielding a confident finite estimate: 0.021416714008590323, with a warning blaming a missing pre-treatment period. The FULL design is now validated pre-exclusion. - `survey_design=` + unidentified-cohort exclusion is REFUSED: deleting rows drops their PSUs/strata from the TSL variance and from `df_survey = n_PSU - n_strata` (measured 22 -> 14, finite SE, no diagnostic). Domain estimation via `SurveyDesign.subpopulation()` is the correct answer and is tracked. - `rank_deficient_action="error"` was silently downgraded to a warning for skipped cells; all three modes now dispatch. ## Deliberate capability regression All-eventually-treated panels (no never-treated group) now RAISE. They previously returned finite, silently wrong ATTs -- on a true-1.5 DGP: ATT(3,8) = -0.0096, ATT(5,9) = -0.2393, overall_att = 1.0023. W2025 Section 5.4 defines the fix (last cohort as control, its columns dropped deliberately rather than by QR); the library already does this for the cohort-TREND column but not for the cells. Tracked at High priority; landing it restores these fits. Four existing tests asserted only that such values were FINITE, which is what let a P0 live in a tested path; they now assert the refusal with the measured numbers in their docstrings. ## Tracking Every refusal added here is classified terminal vs stopgap in TODO.md, with the stopgaps carrying what "done" means: survey domain estimation, anticipation-only estimand semantics, same-period comparison support, component-aware estimation, W2025 5.4 cell normalization, the SE-derivation subsample ladder, and the duplicate survey resolution. M-123/M-124 and the CHANGELOG record the user-visible behavior changes. Docs: REGISTRY normalization + connectivity + invariant notes, tutorial 16 (lead-and-lag vs lag-only, demonstrated rather than asserted), API page, constructor docstring, and the local paper review all corrected to match runtime behavior. Verification: 6992 passed across every WooldridgeDiD/SurveyDesign-touching suite; notebook re-validated with --nbmake; mypy 2.3.0 clean; black/ruff clean. Stata golden regenerated with local Stata (StataSE 19), verified zero numeric or key-set change -- only `meta.ssc_versions` moved. --- CHANGELOG.md | 107 +- DEFERRED.md | 2 +- TODO.md | 9 + benchmarks/data/etwfe_cs_stata_golden.json | 70 + benchmarks/data/mpdta_stata_panel.csv | 2501 +++++++++++++++++ benchmarks/stata/README.md | 50 + benchmarks/stata/generate_etwfe_cs_golden.do | 278 ++ benchmarks/stata/requirements.do | 7 +- diff_diff/survey.py | 69 +- diff_diff/wooldridge.py | 788 +++++- diff_diff/wooldridge_results.py | 7 +- docs/api/wooldridge_etwfe.rst | 16 +- docs/methodology/REGISTRY.md | 17 +- .../papers/wooldridge-2025-review.md | 2 +- docs/tutorials/16_wooldridge_etwfe.ipynb | 26 +- docs/v4-deprecations.yaml | 26 + tests/test_etwfe_cs_stata_parity.py | 358 +++ tests/test_methodology_wooldridge.py | 61 +- tests/test_v4_matrix.py | 11 +- tests/test_wooldridge.py | 1361 ++++++++- 20 files changed, 5516 insertions(+), 250 deletions(-) create mode 100644 benchmarks/data/etwfe_cs_stata_golden.json create mode 100644 benchmarks/data/mpdta_stata_panel.csv create mode 100644 benchmarks/stata/generate_etwfe_cs_golden.do create mode 100644 tests/test_etwfe_cs_stata_parity.py diff --git a/CHANGELOG.md b/CHANGELOG.md index 72e8962a0..1bce6ae09 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,7 +7,112 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] -### Added +### Fixed +- **`WooldridgeDiD` silently dropped genuine post-treatment effects + (issue #724).** With `control_group="never_treated"`, the ETWFE design emitted + every cohort×time indicator *including* the reference period — which is + collinear with the absorbed unit fixed effects — and left generic QR rank + detection to remove whichever column it happened to pick. On the real + Callaway-Sant'Anna `mpdta` panel that removed `ATT(2004, 2004)` and + `ATT(2006, 2007)`, returning **5 of 7** post-treatment cells with only a + rank-deficiency warning to explain it. + The reference is not a free choice: Wooldridge (2025) Eq. 6.1 excludes + `dg_i·f(g-1)_t` "so that `s = g − 1` is the reference period", and Stata + `jwdid ... never` omits the same cell. The design now omits + `ref(g) = max{t : t < g − anticipation AND cohort g is observed at t}` + explicitly — per-cohort, since a panel-wide rule omits an all-zero column + whenever a cohort is unobserved there and leaves the collinearity intact. + All 7 cells now return with **no rank warning**, matching `jwdid`'s cell set + exactly and its ATTs to ~1e-14. + Two related cases are handled rather than left to QR: `(g, t)` pairs a cohort + never observes are skipped (and named in a warning, gated on + `rank_deficient_action`), and a cohort with no pre-treatment period at all is + **excluded from the estimation sample** with a warning naming it — dropping + only its columns would leave its rows in the omitted baseline, loading its + treatment effect onto the time fixed effects and biasing the cohorts that are + identified. Where that exclusion removes the last comparison group, `fit()` + now raises instead of returning an all-NaN result. + The bug was **not** confined to `never_treated`: any cohort whose cell block + spans its own cohort dummy hits it, which `anticipation > 0` makes reachable + on the default `not_yet_treated` path. +- **`WooldridgeDiD` no longer returns a NaN overall ATT as a successful fit.** + Several designs previously produced a result object with + `overall_att = overall_se = NaN` and no error, which reads as a completed + estimate: every treatment cell removed by rank reduction, every cell falling + inside the anticipation window (`overall_att` averages `t >= g`, while cells + from `t >= g − anticipation` are estimated), and post-treatment coefficients + all rank-dropped. `fit()` now raises with a cause-specific message. Verified + pre-existing rather than introduced by the reference-period work. The + invariant covers the **point estimate only** — `overall_se` and inference may + still be NaN where the variance alone is unidentifiable (`hc2_bm` without + Bell-McCaffrey DOF, `aggregate(weights="cohort_share")`), both fail-closed by + design. Note a design with **two** surviving treated cohorts can still be + degenerate when neither post cell has a same-period comparison, so cohort + count is not a proxy for comparison support. +- **`rank_deficient_action="error"` is honored for unobserved cohort×time + cells.** Those cells are now skipped before the solve rather than left to QR. + The first version of that change treated `"error"` and `"warn"` identically, + which silently downgraded an explicit fail-closed request to a warning; all + three modes now dispatch, and `"error"` raises naming the skipped pairs. + +### Changed +- **`WooldridgeDiD` now raises when rank reduction removes any requested + cohort×time cell**, instead of reporting the survivors under their original + `ATT(g,t)` labels. Once a cell is dropped, the remaining coefficients + identify whatever contrast the reduced design supports — typically a + difference between two treated cohorts — so the labels no longer describe the + estimand. Measured on an all-eventually-treated panel with a true ATT of 1.5 + in every post cell: `ATT(3,8) = -0.0096`, `ATT(5,9) = -0.2393`, and + `overall_att = 1.0023`, with **no warning** under + `rank_deficient_action="silent"`. + **This is a capability regression, taken deliberately.** Panels with no + never-treated group — where periods in which every unit is treated have no + eligible comparison — previously produced those wrong numbers and now fail + closed, including under `survey_design=` and `cohort_trends=True`. Wooldridge + (2025) Section 5.4 defines the correct treatment (the last cohort serves as + the control, its columns dropped deliberately rather than by QR); the library + already applies that to the cohort-trend column but not to the cells, and + implementing the cell half is tracked to restore these fits. +- **`WooldridgeDiD` refuses `survey_design=` combined with an unidentified + cohort** (`NotImplementedError`, all three methods). Excluding a cohort under + a complex survey design is domain estimation, which this path does not + implement: deleting the rows also removes their PSUs and strata from the + Taylor-linearized variance and from `df_survey = n_PSU − n_strata`, so + surviving ATTs would carry variance computed on a design the user never + specified (measured on a two-stratum panel: `df_survey` 22 → 14, finite SE, no + diagnostic). The correct behavior is zero-padding per REGISTRY *Subpopulation + Analysis* / Lumley (2004) §3.4; until that lands the combination is refused + rather than approximated. Restrict the frame to identified cohorts yourself, + reduce `anticipation`, or fit without `survey_design=`. Tracked in `TODO.md`. +- **Survey designs are validated before any row is dropped.** The full + `SurveyDesign` (weights, `weight_type`, strata, PSU, FPC, nesting) is now + resolved against the pre-exclusion sample. Previously validation ran after + cohort exclusion, so invalid metadata confined to an excluded cohort — NaN / + negative / infinite weights, fractional `fweight`, missing PSU or strata — + vanished with those rows and the fit returned finite estimates instead of + rejecting the design. + +### Added +- **Stata parity arm for ETWFE and Callaway-Sant'Anna ATT(g,t) (`jwdid` / `csdid`).** + `benchmarks/stata/generate_etwfe_cs_golden.do` anchors both staggered + estimators against their canonical Stata implementations on the genuine + `mpdta` panel (committed at its pinned SHA-256, so the arm never needs + network). Both match their reference to ~3e-8 on every post-treatment cell. + This **replaces a test that was validating nothing**: `test_wooldridge.py` + asserted ETWFE ATT(g,t) *equals* CS ATT(g,t) within `5e-3`, which is false on + real data — at `(2007, 2007)` they differ by `0.0171`. It passed only because + `load_mpdta()` was silently substituting a synthetic, effect-homogeneous DGP + when its source 404'd (issue #722). Stata's `jwdid` and `csdid` reproduce the + same disagreement, establishing it as a property of the estimators rather + than a defect in either. + The arm also **surfaced a real deviation**: `WooldridgeDiD`'s default `hc1` + SEs are uniformly smaller than `jwdid`'s, making them anti-conservative + (~0.1% at G=500, ~2.8% at G=20 — the gap grows as clusters shrink). + `CallawaySantAnna` shows no such gap. The mechanism is **not yet + identified** — it tracks `sqrt(G/(G-1))` but sits above it, and `solve_ols` + already applies the full CR1 — so the deviation is recorded and pinned as + measured rather than diagnosed; see the REGISTRY `## WooldridgeDiD (ETWFE)` + note and the `TODO.md` row. - **Post-fit aggregation: `results.aggregate(type=...)` on CallawaySantAnna (4.0 program Phase 2b, ledger rows M-020 / M-117 / M-122).** Estimate once, aggregate as a post-fit step — the ecosystem norm (`did::aggte`, diff --git a/DEFERRED.md b/DEFERRED.md index a184b5d13..78f6ed757 100644 --- a/DEFERRED.md +++ b/DEFERRED.md @@ -46,7 +46,7 @@ exists but parity can't be verified without a local toolchain. | `StaggeredTripleDifference` R cross-validation: CSV fixtures not committed (gitignored); tests skip without local R + `triplediff`. Commit fixtures or generate deterministically. | `tests/test_methodology_staggered_triple_diff.py` | #245 | Medium | | `StaggeredTripleDifference` R parity: benchmark only tests the no-covariate path (`xformla=~1`). Add covariate-adjusted scenarios + aggregation-SE parity assertions. | `benchmarks/R/benchmark_staggered_triplediff.R` | #245 | Medium | | `StaggeredTripleDifference` per-cohort group-effect SEs include WIF (conservative vs R's `wif=NULL`); documented in REGISTRY. Could override the mixin for an exact R match (verification needs R `triplediff`). | `staggered_triple_diff.py` | #245 | Low | -| **WooldridgeDiD follow-up cluster** (PR-B Stage D/E fail-closed surfaces; re-enable after R/Stata validation):
• QMLE sandwich uses `aweight` cluster adjustment `(G/(G-1))·(n-1)/(n-k)` vs Stata's `G/(G-1)` (conservative); add a `qmle` weight type if Stata goldens confirm a material difference (`wooldridge.py`, `linalg.py`).
• response-scale APE / log-link coefficient bridge for R `etwfe(family=poisson|logit)` cell-level parity — needs `emfx()` APE extraction or link-inversion with baseline-mean adjustment (`generate_wooldridge_golden.R`, `test_methodology_wooldridge.py`).
• `aggregate(weights="cohort_share")` on survey-weighted fits: `_n_g_per_cohort` uses raw `unit.nunique()`; implement design-weighted unit totals per cohort (paper W2025 §7) and lift the `ValueError` gate (`wooldridge.py`, `wooldridge_results.py`).
• unconditional inference for `cohort_share` accounting for ω̂_g sampling uncertainty (W2025 §7.5); currently NaN-closed (`wooldridge_results.py`).
• `cohort_trends=True × survey_design` and `× control_group="never_treated"` raise `NotImplementedError` (unvalidated TSL variance / trend columns spanned by placebo cell-dummies) (`wooldridge.py`).
• Stata `jwdid` golden-value `TestReferenceValues` (`tests/test_wooldridge.py`). | `wooldridge.py`, `wooldridge_results.py`, `linalg.py`, benchmarks | #216 · PR-B | Med-Low | +| **WooldridgeDiD follow-up cluster** (PR-B Stage D/E fail-closed surfaces; re-enable after R/Stata validation):
• QMLE sandwich uses `aweight` cluster adjustment `(G/(G-1))·(n-1)/(n-k)` vs Stata's `G/(G-1)` (conservative); add a `qmle` weight type if Stata goldens confirm a material difference (`wooldridge.py`, `linalg.py`).
• response-scale APE / log-link coefficient bridge for R `etwfe(family=poisson|logit)` cell-level parity — needs `emfx()` APE extraction or link-inversion with baseline-mean adjustment (`generate_wooldridge_golden.R`, `test_methodology_wooldridge.py`).
• `aggregate(weights="cohort_share")` on survey-weighted fits: `_n_g_per_cohort` uses raw `unit.nunique()`; implement design-weighted unit totals per cohort (paper W2025 §7) and lift the `ValueError` gate (`wooldridge.py`, `wooldridge_results.py`).
• unconditional inference for `cohort_share` accounting for ω̂_g sampling uncertainty (W2025 §7.5); currently NaN-closed (`wooldridge_results.py`).
• `cohort_trends=True × survey_design` and `× control_group="never_treated"` raise `NotImplementedError` (unvalidated TSL variance / trend columns spanned jointly by the placebo cells and the unit FE, which absorb the cohort indicator and recover the omitted reference) (`wooldridge.py`).
• Stata `jwdid` golden-value `TestReferenceValues` (`tests/test_wooldridge.py`). | `wooldridge.py`, `wooldridge_results.py`, `linalg.py`, benchmarks | #216 · PR-B | Med-Low | | Extend `WooldridgeDiD` `method ∈ {logit, poisson}` with `vcov_type ∈ {classical, hc2, hc2_bm}`: composing HC2 leverage + Bell-McCaffrey DOF with the QMLE pseudo-residual sandwich needs derivation + R parity vs `clubSandwich::vcovCR(glm, type="CR2")`. Rejected at `__init__`. | `wooldridge.py` | follow-up | Medium | | Multi-constraint CR2 parallel-trends test (AHT/HTZ) for `hc2_bm` fits: DiagnosticReport's PT check routes `vcov_type="hc2_bm"` sources to Bonferroni over the BM-adjusted per-row p-values because the generic chi-square joint Wald would discard the CR2 small-sample correction (see REPORTING.md "hc2_bm parallel-trends policy"). The proper joint test is the AHT/HTZ Wald with a Satterthwaite-style denominator df over the pre-period contrast block; needs derivation for the stacked/pooled WLS-CR2 layout + parity vs `clubSandwich::Wald_test(..., test="HTZ")`. | `diagnostic_report.py`, `linalg.py` | vcov/df round-trip PR | Low | | `PreTrendsPower` CS/SA `anticipation=1` R-parity fixture: R `pretrends` has no anticipation parameter, so the Python `_extract_pre_period_params` anticipation filter isn't R-parity-locked. Build a synthetic CS/SA result with `anticipation=1` and assert γ_p matches R's `slope_for_power()`. (Mechanism already covered by MC + full-VCV tests.) | `tests/test_methodology_pretrends.py`, `generate_pretrends_golden.R` | PR-C | Low | diff --git a/TODO.md b/TODO.md index dd8347993..caf3cd2b4 100644 --- a/TODO.md +++ b/TODO.md @@ -23,6 +23,15 @@ Related tracking surfaces: |-------|----------|--------|--------|----------| | `SyntheticControl` conformal (CWZ 2021) AR / innovation-permutation path (Lemmas 5-7) for time-series proxies — the residual-permutation shortcut is only valid for time-permutation-invariant proxies (SC/Lasso/DiD); an AR proxy needs innovation permutation. | `diff_diff/conformal.py`, `diff_diff/synthetic_control_results.py` | CWZ-2021 | Heavy | Low | | Make the post-fit `results.aggregate("event_study")` container consumable downstream. `EventStudyResults` is rejected by all THREE consumers that read a CS event study — `compute_honest_did` (`honest_did.py`, dispatches on `CallawaySantAnnaResults` and raises `TypeError`), `compute_pretrends_power` (`pretrends.py`, same), and `plot_event_study` (`visualization`, same) — so `fit(aggregate="event_study")` is still the only route for them and their error messages say so explicitly. Needs an `EventStudyResults` branch in each extraction path (consuming `event_time` / `is_reference` / `vcov` / `vcov_index` / per-row `df`) PLUS `base_period` and `anticipation` provenance, which the unified container does not carry and HonestDiD needs for its universal-base-period warning and pre-period classification. Gate with end-to-end tests: `compute_honest_did(res.aggregate("event_study"))` at `base_period="universal"`, and `compute_pretrends_power(...)` at `anticipation=1`. | `diff_diff/honest_did.py`, `diff_diff/pretrends.py`, `diff_diff/results_base.py` | #726 | Mid | Medium | +| Derive and fix the `WooldridgeDiD` `hc1` SE gap vs Stata `jwdid`. Measured: every SE is uniformly SMALLER than `jwdid`'s, by 1.0280 at G=20 / 1.0132 at G=40 / 1.0010 at G=500 (ATT(g,t) points match exactly). The gap tracks `sqrt(G/(G-1))` but sits consistently above it, and `solve_ols` already applies the full CR1 `(G/(G-1))*((n-1)/(n-k))`, so a missing cluster factor is ruled out -- the likely source is the within-transform `k` accounting vs `hdfe`/`reghdfe`'s. Derive the exact factor FIRST; a `sqrt(G/(G-1))` patch would match only approximately and would force a loose tolerance on the parity test, defeating its purpose. Move the REGISTRY note and the pinned ratio in `tests/test_etwfe_cs_stata_parity.py` together with the fix. **Required artifact:** a committed subsample LADDER in the Stata golden (rosters = first N units per `first_treat` by sorted `countyreal`; rungs spanning G≈20..500, storing Stata `G`/`n`/`df_a`/`rank`/`df_r` and per-cell `att`/`se`), with parameterized ratio assertions. Only the G=500 ratio is pinned today, so the few-cluster behavior — where the gap is materially largest (~2.8%) — is ungated, and the ladder is also the instrument for comparing Stata's `df_a`/`rank` against the library's within-transform `k`, which is the leading hypothesis for the factor. | `diff_diff/wooldridge.py`, `diff_diff/linalg.py` | #723-followup | Mid | Medium | +| `SunAbraham`: a cohort not observed at its own reference relative period (`e = -1 - anticipation`) makes that cohort's block collinear, so QR drops an unnamed column (`dropping 1 of 12 columns (column 9)`) and `overall_att` comes back **NaN**. Found by auditing the sibling estimator while fixing the ETWFE analogue (#724); PRE-EXISTING, not introduced there. Lower severity than #724 — that returned a silently WRONG finite number, this returns NaN with a rank warning — but the event-study surface still looks complete, so a user may not notice the loss. SA already omits its reference explicitly and tracks `_reference_observed`, so the fix is per-cohort support for that flag rather than the ETWFE-style redesign. | `diff_diff/sun_abraham.py` | #724-audit | Mid | Low | +| `WooldridgeDiD` + `survey_design=` does not support DOMAIN ESTIMATION, so unidentified-cohort exclusion is currently REFUSED (`NotImplementedError`, all three methods) rather than performed. Implementing it properly means zero-padding the excluded rows' weights while retaining strata/PSU/FPC, per REGISTRY *Subpopulation Analysis (Phase 6)* / Lumley (2004) 3.4, so TSL variance and `df_survey = n_PSU - n_strata` use the full design (naive deletion measured 22 -> 14 on a two-stratum panel). `SurveyDesign.subpopulation()` already implements the contract and SpilloverDiD Wave E.3 is the in-repo precedent; the blocker is that the weighted within-transform rejects zero-weight units, shared machinery behind 7 estimators. Landing it would turn the refusal back into a supported fit. Gate with a `SurveyDesign.subpopulation()` parity test on ATT, TSL SE and survey df where the excluded cohort exhausts a PSU. | `diff_diff/wooldridge.py`, `diff_diff/utils.py` | #724-codex-R4/R5 | Heavy | Medium | +| `WooldridgeDiD` REFUSES a fit whose only treatment cells fall inside the anticipation window, discarding estimates it successfully computed. `_require_estimable_overall_att` ([M-124]) raises when no cell has `t >= g`, because the overall ATT averages only `t >= g` (W2025 excludes anticipation leads) while cells from `t >= g - anticipation` are ESTIMATED. For a cohort never observed at or after its own treatment date, those anticipation-window ATT(g, t) are real, identified estimates and are thrown away with the fit. **The refusal is a stopgap for the missing estimand semantics, not the intended end state.** Real fix: decide what such a fit should return — most likely the per-cell ATT(g, t) plus an overall that is explicitly undefined with a stated reason (not a bare NaN, per the no-silent-NaN convention) — then relax the guard to that. Needs a REGISTRY note defining the estimand and an `aggregate()` story for the anticipation-only case. | `diff_diff/wooldridge.py` | #724-codex-R2 | Mid | Medium | +| `WooldridgeDiD` has no same-period COMPARISON-SUPPORT accounting; the `has_untreated` guard is a proxy that counts a treated cohort's OWN pre-treatment rows as comparison observations. So a design with no usable control at any post period passes the upfront check and fails late and opaquely — as rank reduction, then the blanket `_require_estimable_overall_att` refusal ([M-124]). Measured: cohort count is NOT a valid proxy either — two surviving treated cohorts can still be fully degenerate when neither post cell has a same-period control (`TestOverallAttFailsClosed::test_two_cohorts_without_same_period_controls_fail_closed`). **The post-hoc refusal is a stopgap for a missing upfront diagnostic.** Real fix: compute per-`(g, t)` eligible-control support (never-treated, or not-yet-treated at that period), report exactly which cells are unidentified and why BEFORE solving, and only refuse when none survives. This also subsumes the reviewer's R2 item (b). | `diff_diff/wooldridge.py` | #724-codex-R2 | Mid | Medium | +| `WooldridgeDiD` DROPS the observations of a cohort with no supported pre-period before `g - anticipation` ([M-123]) rather than identifying it. Excluding the rows is correct given `g-1` normalization — leaving them in silently loads the cohort's effect onto the time FE — but dropping a cohort a user supplied is a lossy last resort, and it is the trigger for the survey refusal in the row above. **Evaluate whether such cohorts are identifiable at all before accepting exclusion as permanent.** Two candidate routes, both needing a methodology decision recorded in REGISTRY: (a) an explicit user-supplied reference period per cohort — W2025 Section 6.1 says any pre-treatment period may serve as reference and the pre-trend `t`-test is invariant to the choice, so a cohort with ANY supported pre-period is a candidate even when `g-1` is missing; (b) the paper's no-never-treated last-cohort normalization (`wooldridge-2025-review.md:461-463`), which applies ONLY when there is no never-treated group and so does not cover `control_group='never_treated'`. If neither identifies the cohort, convert this row into a REGISTRY Note recording exclusion as the deliberate final answer. | `diff_diff/wooldridge.py`, `docs/methodology/REGISTRY.md` | #724 | Heavy | Medium | +| `WooldridgeDiD` REFUSES a panel whose units split into disconnected support groups within a cohort, rather than estimating what IS identified. The connectivity guard (REGISTRY *within-cohort support connectivity*) correctly detects that a closed component's cells are collinear with the unit FE — previously QR dropped one silently and the overall ATT averaged an incomplete set (issue #724's failure mode via unit support). **Refusing is the safe answer, not the complete one.** The connected component containing the reference is still fully identified, so the estimable resolution is either (a) estimate the connected component and report the disconnected units as excluded, with the estimand restated (a sub-population of units, so it needs a REGISTRY definition and interacts with the survey-domain row above), or (b) per-component references, if a component with its own pre-period can carry its own normalization — needs a methodology decision, since components then are not comparable on one baseline. Gate with the split-support fixture in `TestWithinCohortSupportConnectivity`. | `diff_diff/wooldridge.py` | #724-codex-R7 | Heavy | Medium | +| `WooldridgeDiD` REFUSES all-eventually-treated panels (no never-treated group) because at fully-treated periods the cohort cells are jointly collinear with the time FE; since the completeness gate ([M-124]) a lost cell fails closed. **This is a capability regression traded for correctness** — those fits previously returned finite, silently WRONG ATTs (measured on a true-1.5 DGP: `ATT(3,8) = -0.0096`, `ATT(5,9) = -0.2393`, `overall_att = 1.0023`), because the survivors identify contrasts BETWEEN treated cohorts while keeping their `ATT(g,t)` labels. **W2025 Section 5.4 defines the fix**: with no never-treated group the LAST COHORT serves as the control, so its columns are dropped deliberately rather than by QR, leaving the remaining cells identified (relative to the last cohort's effect — an estimand change that needs a REGISTRY definition). The library ALREADY implements this for the cohort-TREND column (`cohort_trend_coefs` drops the last cohort) but not for the cells, so the two halves of one paper normalization are inconsistent and the trend half is now unreachable end-to-end. Landing it turns `TestAllEventuallyTreated` and `test_cohort_trends_true_all_treated_panel_is_refused` back into fits. | `diff_diff/wooldridge.py`, `docs/methodology/REGISTRY.md` | #724-codex-R8 | Heavy | High | +| `WooldridgeDiD` fully resolves the `SurveyDesign` TWICE on every supported survey fit. The pre-exclusion validation pass (added so invalid metadata cannot hide in rows that cohort exclusion deletes) calls `survey_design.resolve(sample)`, and each fitter then calls `_resolve_survey_for_wooldridge` -> `_resolve_survey_for_fit` on the same frame, repeating weight normalization, strata/PSU/FPC validation and design-array construction. Any fit that REACHES the second resolve has an unchanged sample (survey + unidentified-cohort exclusion raises first), so the first result is reusable: capture the `_resolve_survey_for_fit` 4-tuple early and thread it into the three fitters as an optional `pre_resolved`. **Caveat that makes this non-trivial:** `sample = sample.reset_index(drop=True)` runs BETWEEN the two calls, so the reused object must be verified index-independent (resolution extracts positional numpy arrays, but `_inject_cluster_as_psu` and the metadata recompute need checking), and the early call must stop suppressing warnings or the user loses the weight-normalization notice. Gate with a survey fit asserting one normalization warning and byte-identical SEs. | `diff_diff/wooldridge.py` | #724-codex-R10 | Mid | Low | | Bootstrap re-aggregation for `CallawaySantAnnaResults.aggregate()` — a bootstrapped fit currently RAISES rather than substituting analytical inference for percentile-bootstrap statistics. The value-bound `BootstrapReplaySpec` (bit-identical replay, picklable, immune to post-fit `set_params`) is already in-tree and spike-verified; wiring it needs the per-`(g,t)` / per-event-time draw retention plus `assert_allclose` parity tests against the fit-time bootstrap numbers (NOT bit-identity — the fused GEMM's column count differs post-fit, ~1 ULP reassociation). | `diff_diff/aggregation.py`, `diff_diff/staggered_results.py` | #726 | Mid | Medium | | Consolidate the inference-df precedence duplicated across `honest_did.py` (3 copies at ~L655/L836/L1004) onto the shared `resolve_inference_df()` helper added in `diff_diff/aggregation.py`. The copies are correct today; the risk is drift if the survey/replicate precedence changes in one place only. | `diff_diff/honest_did.py` | #726 | Quick | Low | | `ContinuousDiD` CGBS-2024 remaining extensions (earlier phases — `covariates=` reg/dr, `treatment_type="discrete"`, single-cohort `control_group="lowest_dose"` with estimand `ATT(d)−ATT(d_L)` — are already supported; see REGISTRY Note #7). Remaining (all deferred `NotImplementedError`, documented): `estimation_method="ipw"` on the dose curve (scalar-adjustment / degenerate); `covariates=` × `survey_design=` (weighted OR + weighted nuisance IF); multi-cohort **heterogeneous-support** discrete aggregation (support-aware: average each dose only over the cohorts that observe it); **multi-cohort `lowest_dose`** (within-cohort `d_L` reference + support-aware cross-cohort aggregation); and **`covariates=` × `lowest_dose`** (conditional-PT-relative-to-`d_L` estimand). Single-cohort / 2-period / shared-support multi-cohort are supported. | `continuous_did.py` | CGBS-2024 | Heavy | Low | diff --git a/benchmarks/data/etwfe_cs_stata_golden.json b/benchmarks/data/etwfe_cs_stata_golden.json new file mode 100644 index 000000000..033efa83a --- /dev/null +++ b/benchmarks/data/etwfe_cs_stata_golden.json @@ -0,0 +1,70 @@ +{ + "meta": { + "generator": "benchmarks/stata/generate_etwfe_cs_golden.do", + "source_panel": "benchmarks/data/mpdta_stata_panel.csv", + "source_sha256": "2283bea1221a152420f98dfa20f633c5d054ea51d881115c8cd702a97bcd3167", + "ssc_versions": { + "drdid": "checksum:3018598276 len:91307", + "csdid": "v1.81 by pedro Sant'Anna. Compatibility checks", + "jwdid": "v2.21 Bug with trgvar", + "hdfe": "reghdfe 3.2.9 21feb2016" + }, + "stata_edition": "SE", + "csdid_cmd": "csdid lemp, ivar(countyreal) time(year) gvar(first_treat) notyet method(reg)", + "jwdid_cmd": "jwdid lemp, ivar(countyreal) tvar(year) gvar(first_treat)", + "jwdid_never_cmd": "jwdid lemp, ivar(countyreal) tvar(year) gvar(first_treat) never", + "note": "No covariates, so csdid method is immaterial. ETWFE and CS genuinely DISAGREE at (2007,2007).", + "stata_version": 19 + }, + "csdid": { + "g2004:t_2003_2004": {"att": -0.019372363675922127, "se": 0.022310112883680573}, + "g2004:t_2003_2005": {"att": -0.078319099062060693, "se": 0.030390228543397232}, + "g2004:t_2003_2006": {"att": -0.13627434632867791, "se": 0.035403384968910059}, + "g2004:t_2003_2007": {"att": -0.10081136308540416, "se": 0.034359225834673233}, + "g2006:t_2003_2004": {"att": -0.0025625509426109808, "se": 0.022530235145338813}, + "g2006:t_2004_2005": {"att": -0.0019392460957887456, "se": 0.019042158605818947}, + "g2006:t_2005_2006": {"att": 0.0046608763199761519, "se": 0.01633558424682359}, + "g2006:t_2005_2007": {"att": -0.041224471546217473, "se": 0.020229180704106838}, + "g2007:t_2003_2004": {"att": 0.030506655583292804, "se": 0.015033560280130042}, + "g2007:t_2004_2005": {"att": -0.0027258928861162608, "se": 0.016395832895534423}, + "g2007:t_2005_2006": {"att": -0.031087119389688882, "se": 0.01787751131334352}, + "g2007:t_2006_2007": {"att": -0.026054410719196626, "se": 0.016655435349252175}, + "wgt:w2004_2004": {"att": 0.026178010471204188, "se": 0.0057353147382739906}, + "wgt:w2004_2005": {"att": 0.026178010471204188, "se": 0.0057353147382739906}, + "wgt:w2004_2006": {"att": 0.026178010471204188, "se": 0.0057353147382739906}, + "wgt:w2004_2007": {"att": 0.026178010471204188, "se": 0.0057353147382739906}, + "wgt:w2006_2004": {"att": 0.052356020942408377, "se": 0.0079401837110487442}, + "wgt:w2006_2005": {"att": 0.052356020942408377, "se": 0.0079401837110487442}, + "wgt:w2006_2006": {"att": 0.052356020942408377, "se": 0.0079401837110487442}, + "wgt:w2006_2007": {"att": 0.052356020942408377, "se": 0.0079401837110487442}, + "wgt:w2007_2004": {"att": 0.17146596858638741, "se": 0.012869760653910581}, + "wgt:w2007_2005": {"att": 0.17146596858638741, "se": 0.012869760653910581}, + "wgt:w2007_2006": {"att": 0.17146596858638741, "se": 0.012869760653910581}, + "wgt:w2007_2007": {"att": 0.17146596858638741, "se": 0.012869760653910581} + }, + "jwdid": { + "2004bn.first_treat#2004bn.year#c.__tr__": {"att": -0.019372363675909551, "se": 0.022381770443295975}, + "2004bn.first_treat#2005.year#c.__tr__": {"att": -0.078319099062053255, "se": 0.030487838520761604}, + "2004bn.first_treat#2006.year#c.__tr__": {"att": -0.13607811444031598, "se": 0.035455486551175253}, + "2004bn.first_treat#2007.year#c.__tr__": {"att": -0.10470747157659592, "se": 0.033874305460323405}, + "2006.first_treat#2006.year#c.__tr__": {"att": 0.0025138619419130628, "se": 0.019932816904001841}, + "2006.first_treat#2007.year#c.__tr__": {"att": -0.039192735591725474, "se": 0.024008748273466048}, + "2007.first_treat#2007.year#c.__tr__": {"att": -0.043106032808698021, "se": 0.018431147223806212}, + "_cons": {"att": 5.7780697609148426, "se": 0.001544041086555641} + }, + "jwdid_never": { + "2004bn.first_treat#2004.year#c.__tr__": {"att": -0.01050324622095701, "se": 0.023349189673294385}, + "2004bn.first_treat#2005.year#c.__tr__": {"att": -0.070423158103148337, "se": 0.031115567690950852}, + "2004bn.first_treat#2006.year#c.__tr__": {"att": -0.13725873888941312, "se": 0.036589475963765966}, + "2004bn.first_treat#2007.year#c.__tr__": {"att": -0.100811363085407, "se": 0.034504271910207822}, + "2006.first_treat#2003b.year#c.__tr__": {"att": -0.0037692936737138269, "se": 0.031474336697626552}, + "2006.first_treat#2004.year#c.__tr__": {"att": 0.0027508187505188874, "se": 0.019641126706453155}, + "2006.first_treat#2006.year#c.__tr__": {"att": -0.0045946069528630213, "se": 0.017830149500418953}, + "2006.first_treat#2007.year#c.__tr__": {"att": -0.041224471546218548, "se": 0.020314577368354389}, + "2007.first_treat#2003b.year#c.__tr__": {"att": 0.0033063566925126296, "se": 0.024555095531863514}, + "2007.first_treat#2004.year#c.__tr__": {"att": 0.033813012275805458, "se": 0.02121837087758589}, + "2007.first_treat#2005.year#c.__tr__": {"att": 0.031087119389688365, "se": 0.017952980500827595}, + "2007.first_treat#2007.year#c.__tr__": {"att": -0.026054410719196439, "se": 0.016725745592717861}, + "_cons": {"att": 5.7736089304641052, "se": 0.0033784078861242291} + } +} diff --git a/benchmarks/data/mpdta_stata_panel.csv b/benchmarks/data/mpdta_stata_panel.csv new file mode 100644 index 000000000..cd77209b1 --- /dev/null +++ b/benchmarks/data/mpdta_stata_panel.csv @@ -0,0 +1,2501 @@ +"year","countyreal","lpop","lemp","first.treat","treat" +2003,8001,5.8967609333053,8.46146904264388,2007,1 +2004,8001,5.8967609333053,8.33686963728496,2007,1 +2005,8001,5.8967609333053,8.34021732094704,2007,1 +2006,8001,5.8967609333053,8.37816098272068,2007,1 +2007,8001,5.8967609333053,8.48735234940522,2007,1 +2003,8019,2.23237719795055,4.99721227376411,2007,1 +2004,8019,2.23237719795055,5.08140436498446,2007,1 +2005,8019,2.23237719795055,4.78749174278205,2007,1 +2006,8019,2.23237719795055,4.99043258677874,2007,1 +2007,8019,2.23237719795055,5.03695260241363,2007,1 +2003,8023,1.29828248379668,2.19722457733622,2007,1 +2004,8023,1.29828248379668,2.30258509299405,2007,1 +2005,8023,1.29828248379668,2.77258872223978,2007,1 +2006,8023,1.29828248379668,1.94591014905531,2007,1 +2007,8023,1.29828248379668,2.99573227355399,2007,1 +2003,8029,3.32625829499766,5.88332238848828,2007,1 +2004,8029,3.32625829499766,5.78996017089725,2007,1 +2005,8029,3.32625829499766,5.85793315448346,2007,1 +2006,8029,3.32625829499766,5.99893656194668,2007,1 +2007,8029,3.32625829499766,6.10479323241498,2007,1 +2003,8041,6.24790553432335,8.96865090337487,2007,1 +2004,8041,6.24790553432335,8.9157008189569,2007,1 +2005,8041,6.24790553432335,8.90095787254506,2007,1 +2006,8041,6.24790553432335,8.90272766403552,2007,1 +2007,8041,6.24790553432335,8.88100262425557,2007,1 +2003,8063,2.08081559723298,4.9416424226093,2007,1 +2004,8063,2.08081559723298,4.86753445045558,2007,1 +2005,8063,2.08081559723298,4.66343909411207,2007,1 +2006,8063,2.08081559723298,4.81218435537242,2007,1 +2007,8063,2.08081559723298,4.91998092582813,2007,1 +2003,8075,3.02061998906177,5.6970934865054,2007,1 +2004,8075,3.02061998906177,5.63121178182137,2007,1 +2005,8075,3.02061998906177,5.72031177660741,2007,1 +2006,8075,3.02061998906177,5.79605775076537,2007,1 +2007,8075,3.02061998906177,5.56068163101553,2007,1 +2003,8103,1.78942340943051,4.45434729625351,2007,1 +2004,8103,1.78942340943051,4.58496747867057,2007,1 +2005,8103,1.78942340943051,4.75359019110636,2007,1 +2006,8103,1.78942340943051,4.90527477843843,2007,1 +2007,8103,1.78942340943051,4.71849887129509,2007,1 +2003,8117,3.15904089116601,6.37842618365159,2007,1 +2004,8117,3.15904089116601,6.27476202124194,2007,1 +2005,8117,3.15904089116601,6.2841341610708,2007,1 +2006,8117,3.15904089116601,6.3818160174061,2007,1 +2007,8117,3.15904089116601,6.43133108193348,2007,1 +2003,8123,5.19814337757751,7.7553388128465,2007,1 +2004,8123,5.19814337757751,7.68799716639302,2007,1 +2005,8123,5.19814337757751,7.75405263903576,2007,1 +2006,8123,5.19814337757751,7.78945456608667,2007,1 +2007,8123,5.19814337757751,7.80302664363222,2007,1 +2003,12007,3.26147543848702,5.50125821054473,2006,1 +2004,12007,3.26147543848702,5.59471137960184,2006,1 +2005,12007,3.26147543848702,5.24702407216049,2006,1 +2006,12007,3.26147543848702,5.43372200355424,2006,1 +2007,12007,3.26147543848702,5.44241771052179,2006,1 +2003,12019,4.94743987059911,7.77275271646874,2006,1 +2004,12019,4.94743987059911,7.81116338502528,2006,1 +2005,12019,4.94743987059911,7.80139132029149,2006,1 +2006,12019,4.94743987059911,7.86326672400957,2006,1 +2007,12019,4.94743987059911,7.87016594646984,2006,1 +2003,12023,4.03447070018167,6.99117688712121,2006,1 +2004,12023,4.03447070018167,6.53813982376767,2006,1 +2005,12023,4.03447070018167,6.59030104819669,2006,1 +2006,12023,4.03447070018167,6.5510803350434,2006,1 +2007,12023,4.03447070018167,6.71780469502369,2006,1 +2003,12029,2.62662320240648,3.85014760171006,2006,1 +2004,12029,2.62662320240648,4.07753744390572,2006,1 +2005,12029,2.62662320240648,4.35670882668959,2006,1 +2006,12029,2.62662320240648,4.31748811353631,2006,1 +2007,12029,2.62662320240648,4.33073334028633,2006,1 +2003,12051,3.58933532377755,6.42648845745769,2006,1 +2004,12051,3.58933532377755,6.42324696353352,2006,1 +2005,12051,3.58933532377755,6.37331978957701,2006,1 +2006,12051,3.58933532377755,6.4118182677099,2006,1 +2007,12051,3.58933532377755,6.37331978957701,2006,1 +2003,12071,6.08879087493748,8.75872660784204,2006,1 +2004,12071,6.08879087493748,8.79845469606509,2006,1 +2005,12071,6.08879087493748,8.99118884193151,2006,1 +2006,12071,6.08879087493748,9.1516513756275,2006,1 +2007,12071,6.08879087493748,9.06843112579349,2006,1 +2003,12087,4.37687589234631,6.72383244082121,2006,1 +2004,12087,4.37687589234631,6.79234442747081,2006,1 +2005,12087,4.37687589234631,6.75809450442773,2006,1 +2006,12087,4.37687589234631,6.74641212857337,2006,1 +2007,12087,4.37687589234631,6.74758652682932,2006,1 +2003,12093,3.58101580823799,5.72358510195238,2006,1 +2004,12093,3.58101580823799,6.01126717440416,2006,1 +2005,12093,3.58101580823799,6.10255859461357,2006,1 +2006,12093,3.58101580823799,6.0913098820777,2006,1 +2007,12093,3.58101580823799,6.14418563412565,2006,1 +2003,12097,5.15035665593789,7.59085212368858,2006,1 +2004,12097,5.15035665593789,7.59387784460512,2006,1 +2005,12097,5.15035665593789,7.76556908109732,2006,1 +2006,12097,5.15035665593789,7.87321705486274,2006,1 +2007,12097,5.15035665593789,7.84737183615979,2006,1 +2003,12099,7.03102015077071,9.76864100976636,2006,1 +2004,12099,7.03102015077071,9.62918227712599,2006,1 +2005,12099,7.03102015077071,9.72202562615406,2006,1 +2006,12099,7.03102015077071,9.81623978397185,2006,1 +2007,12099,7.03102015077071,9.89363921648131,2006,1 +2003,12109,4.81328131446834,7.55066124310534,2006,1 +2004,12109,4.81328131446834,7.62705741701893,2006,1 +2005,12109,4.81328131446834,7.71199650704767,2006,1 +2006,12109,4.81328131446834,7.86288203464149,2006,1 +2007,12109,4.81328131446834,7.8811822022271,2006,1 +2003,12113,4.76850428314641,7.09090982207998,2006,1 +2004,12113,4.76850428314641,7.16394668434255,2006,1 +2005,12113,4.76850428314641,7.36073990305828,2006,1 +2006,12113,4.76850428314641,7.40306109109009,2006,1 +2007,12113,4.76850428314641,7.33367639565768,2006,1 +2003,12121,3.55088095595665,5.67332326717149,2006,1 +2004,12121,3.55088095595665,5.6970934865054,2006,1 +2005,12121,3.55088095595665,5.70044357339069,2006,1 +2006,12121,3.55088095595665,5.77765232322266,2006,1 +2007,12121,3.55088095595665,5.78382518232974,2006,1 +2003,13011,2.66875481849457,5.27299955856375,0,0 +2004,13011,2.66875481849457,5.31320597904179,0,0 +2005,13011,2.66875481849457,5.49306144334055,0,0 +2006,13011,2.66875481849457,5.59842195899838,0,0 +2007,13011,2.66875481849457,5.57215403217776,0,0 +2003,13013,3.83176694166248,6.18826412308259,0,0 +2004,13013,3.83176694166248,6.18208490671663,0,0 +2005,13013,3.83176694166248,6.25575004175337,0,0 +2006,13013,3.83176694166248,6.4377516497364,0,0 +2007,13013,3.83176694166248,6.40191719672719,0,0 +2003,13019,2.78716940555481,5.56452040732269,0,0 +2004,13019,2.78716940555481,5.55682806169954,0,0 +2005,13019,2.78716940555481,5.65599181081985,0,0 +2006,13019,2.78716940555481,5.62762111369064,0,0 +2007,13019,2.78716940555481,4.99721227376411,0,0 +2003,13021,5.03621856684166,7.73892375743946,0,0 +2004,13021,5.03621856684166,7.72797554210556,0,0 +2005,13021,5.03621856684166,7.73149202924568,0,0 +2006,13021,5.03621856684166,7.70975686445416,0,0 +2007,13021,5.03621856684166,7.60738142563979,0,0 +2003,13037,1.84371920815877,2.77258872223978,0,0 +2004,13037,1.84371920815877,3.04452243772342,0,0 +2005,13037,1.84371920815877,2.70805020110221,0,0 +2006,13037,1.84371920815877,3.17805383034795,0,0 +2007,13037,1.84371920815877,3.46573590279973,0,0 +2003,13039,3.77652396396598,6.22455842927536,0,0 +2004,13039,3.77652396396598,6.29156913955832,0,0 +2005,13039,3.77652396396598,6.35610766069589,0,0 +2006,13039,3.77652396396598,6.34212141872115,0,0 +2007,13039,3.77652396396598,6.29341927884648,0,0 +2003,13049,2.33039479363331,4.62497281328427,0,0 +2004,13049,2.33039479363331,4.46590811865458,0,0 +2005,13049,2.33039479363331,4.66343909411207,0,0 +2006,13049,2.33039479363331,4.70048036579242,0,0 +2007,13049,2.33039479363331,4.75359019110636,0,0 +2003,13057,4.95514372559156,7.42237370098682,0,0 +2004,13057,4.95514372559156,7.49997654095212,0,0 +2005,13057,4.95514372559156,7.62608275807238,0,0 +2006,13057,4.95514372559156,7.71378461659875,0,0 +2007,13057,4.95514372559156,7.80343505695217,0,0 +2003,13077,4.49104918688339,7.11314210870709,0,0 +2004,13077,4.49104918688339,7.07834157955767,0,0 +2005,13077,4.49104918688339,7.10414409298753,0,0 +2006,13077,4.49104918688339,7.16162200293919,0,0 +2007,13077,4.49104918688339,7.26332961747684,0,0 +2003,13079,2.52532856428692,4.02535169073515,0,0 +2004,13079,2.52532856428692,3.66356164612965,0,0 +2005,13079,2.52532856428692,3.43398720448515,0,0 +2006,13079,2.52532856428692,3.40119738166216,0,0 +2007,13079,2.52532856428692,3.13549421592915,0,0 +2003,13093,2.44451858888271,3.76120011569356,0,0 +2004,13093,2.44451858888271,3.73766961828337,0,0 +2005,13093,2.44451858888271,3.63758615972639,0,0 +2006,13093,2.44451858888271,4.0943445622221,0,0 +2007,13093,2.44451858888271,4.34380542185368,0,0 +2003,13097,4.52367809513269,7.42714413340862,0,0 +2004,13097,4.52367809513269,7.46794233228585,0,0 +2005,13097,4.52367809513269,7.57507169950756,0,0 +2006,13097,4.52367809513269,7.71512360363211,0,0 +2007,13097,4.52367809513269,7.82763954636642,0,0 +2003,13109,2.3508989532726,5.19849703126583,0,0 +2004,13109,2.3508989532726,5.15905529921453,0,0 +2005,13109,2.3508989532726,5.21493575760899,0,0 +2006,13109,2.3508989532726,5.13579843705026,0,0 +2007,13109,2.3508989532726,5.20948615284142,0,0 +2003,13113,4.51374544806874,7.42535788702715,0,0 +2004,13113,4.51374544806874,7.44366368311559,0,0 +2005,13113,4.51374544806874,7.45182223652793,0,0 +2006,13113,4.51374544806874,7.51752085060303,0,0 +2007,13113,4.51374544806874,7.63723438878947,0,0 +2003,13117,4.58911193973939,7.33106030521863,0,0 +2004,13117,4.58911193973939,7.38894609761844,0,0 +2005,13117,4.58911193973939,7.52779398772144,0,0 +2006,13117,4.58911193973939,7.64730883235624,0,0 +2007,13117,4.58911193973939,7.63819824428578,0,0 +2003,13129,3.78655048128738,6.44571981938558,0,0 +2004,13129,3.78655048128738,6.37672694789863,0,0 +2005,13129,3.78655048128738,6.361302477573,0,0 +2006,13129,3.78655048128738,6.35437004079735,0,0 +2007,13129,3.78655048128738,6.39859493453521,0,0 +2003,13135,6.37748856255844,9.19482056031815,0,0 +2004,13135,6.37748856255844,9.14590851181679,0,0 +2005,13135,6.37748856255844,9.19482056031815,0,0 +2006,13135,6.37748856255844,9.29504909990314,0,0 +2007,13135,6.37748856255844,9.29569203907569,0,0 +2003,13147,3.13536377263918,5.5834963087817,0,0 +2004,13147,3.13536377263918,5.44241771052179,0,0 +2005,13147,3.13536377263918,5.5834963087817,0,0 +2006,13147,3.13536377263918,5.51342874616498,0,0 +2007,13147,3.13536377263918,5.57215403217776,0,0 +2003,13155,2.29566117792122,4.06044301054642,0,0 +2004,13155,2.29566117792122,3.8286413964891,0,0 +2005,13155,2.29566117792122,4.12713438504509,0,0 +2006,13155,2.29566117792122,3.8286413964891,0,0 +2007,13155,2.29566117792122,3.87120101090789,0,0 +2003,13161,2.5403413566761,5.01063529409626,0,0 +2004,13161,2.5403413566761,4.91265488573605,0,0 +2005,13161,2.5403413566761,5.07517381523383,0,0 +2006,13161,2.5403413566761,5.19849703126583,0,0 +2007,13161,2.5403413566761,5.23644196282995,0,0 +2003,13179,4.12082419502648,6.02344759296103,0,0 +2004,13179,4.12082419502648,5.91620206260743,0,0 +2005,13179,4.12082419502648,6.17586727010576,0,0 +2006,13179,4.12082419502648,6.13556489108174,0,0 +2007,13179,4.12082419502648,6.15060276844628,0,0 +2003,13195,3.24765762629365,4.75359019110636,0,0 +2004,13195,3.24765762629365,4.77912349311153,0,0 +2005,13195,3.24765762629365,4.66343909411207,0,0 +2006,13195,3.24765762629365,4.87519732320115,0,0 +2007,13195,3.24765762629365,5.00394630594546,0,0 +2003,13197,1.9662728435742,3.73766961828337,0,0 +2004,13197,1.9662728435742,3.63758615972639,0,0 +2005,13197,1.9662728435742,3.52636052461616,0,0 +2006,13197,1.9662728435742,3.89182029811063,0,0 +2007,13197,1.9662728435742,3.95124371858143,0,0 +2003,13209,2.1126345090356,3.8286413964891,0,0 +2004,13209,2.1126345090356,3.40119738166216,0,0 +2005,13209,2.1126345090356,3.55534806148941,0,0 +2006,13209,2.1126345090356,3.66356164612965,0,0 +2007,13209,2.1126345090356,3.58351893845611,0,0 +2003,13211,2.73806197517018,5.19295685089021,0,0 +2004,13211,2.73806197517018,5.18738580584075,0,0 +2005,13211,2.73806197517018,5.28320372873799,0,0 +2006,13211,2.73806197517018,5.605802066296,0,0 +2007,13211,2.73806197517018,5.59471137960184,0,0 +2003,13213,3.59747663064059,5.71042701737487,0,0 +2004,13213,3.59747663064059,5.68357976733868,0,0 +2005,13213,3.59747663064059,5.38449506278909,0,0 +2006,13213,3.59747663064059,5.72031177660741,0,0 +2007,13213,3.59747663064059,5.65248918026865,0,0 +2003,13217,4.12715051394728,6.45204895443723,0,0 +2004,13217,4.12715051394728,6.49677499018586,0,0 +2005,13217,4.12715051394728,6.51619307604296,0,0 +2006,13217,4.12715051394728,6.62804137617953,0,0 +2007,13217,4.12715051394728,6.62539236800796,0,0 +2003,13223,4.40278468775839,6.38519439899773,0,0 +2004,13223,4.40278468775839,6.48463523563525,0,0 +2005,13223,4.40278468775839,6.71295620067707,0,0 +2006,13223,4.40278468775839,6.80682936039218,0,0 +2007,13223,4.40278468775839,6.98471632011827,0,0 +2003,13227,3.13475481220279,5.32300997913841,0,0 +2004,13227,3.13475481220279,5.4380793089232,0,0 +2005,13227,3.13475481220279,5.37527840768417,0,0 +2006,13227,3.13475481220279,5.40717177146012,0,0 +2007,13227,3.13475481220279,5.4510384535657,0,0 +2003,13233,3.64092269256804,5.96614673912369,0,0 +2004,13233,3.64092269256804,5.92692602597041,0,0 +2005,13233,3.64092269256804,5.94803498918065,0,0 +2006,13233,3.64092269256804,5.90808293816893,0,0 +2007,13233,3.64092269256804,5.91079664404053,0,0 +2003,13237,2.9344949640828,5.00394630594546,0,0 +2004,13237,2.9344949640828,5.03695260241363,0,0 +2005,13237,2.9344949640828,5.08759633523238,0,0 +2006,13237,2.9344949640828,5.52545293913178,0,0 +2007,13237,2.9344949640828,5.4249500174814,0,0 +2003,13241,2.71137799119488,5.4510384535657,0,0 +2004,13241,2.71137799119488,5.26269018890489,0,0 +2005,13241,2.71137799119488,5.4249500174814,0,0 +2006,13241,2.71137799119488,5.37063802812766,0,0 +2007,13241,2.71137799119488,5.40717177146012,0,0 +2003,13263,1.87149443724691,2.39789527279837,0,0 +2004,13263,1.87149443724691,2.77258872223978,0,0 +2005,13263,1.87149443724691,1.94591014905531,0,0 +2006,13263,1.87149443724691,1.79175946922805,0,0 +2007,13263,1.87149443724691,2.30258509299405,0,0 +2003,13271,2.46759092757831,4.68213122712422,0,0 +2004,13271,2.46759092757831,4.86753445045558,0,0 +2005,13271,2.46759092757831,5.16478597392351,0,0 +2006,13271,2.46759092757831,5.28320372873799,0,0 +2007,13271,2.46759092757831,5.19295685089021,0,0 +2003,13273,2.39516427428714,3.98898404656427,0,0 +2004,13273,2.39516427428714,4.0943445622221,0,0 +2005,13273,2.39516427428714,4.17438726989564,0,0 +2006,13273,2.39516427428714,4.18965474202643,0,0 +2007,13273,2.39516427428714,4.0943445622221,0,0 +2003,13297,4.10572950708866,6.53233429222235,0,0 +2004,13297,4.10572950708866,6.4085287910595,0,0 +2005,13297,4.10572950708866,6.35437004079735,0,0 +2006,13297,4.10572950708866,6.52649485957079,0,0 +2007,13297,4.10572950708866,6.62936325343745,0,0 +2003,13303,3.0528684649248,5.11799381241676,0,0 +2004,13303,3.0528684649248,4.99043258677874,0,0 +2005,13303,3.0528684649248,4.969813299576,0,0 +2006,13303,3.0528684649248,4.93447393313069,0,0 +2007,13303,3.0528684649248,5.04342511691925,0,0 +2003,13313,4.42514598824282,7.15148546390474,0,0 +2004,13313,4.42514598824282,7.15539630189673,0,0 +2005,13313,4.42514598824282,7.20042489294496,0,0 +2006,13313,4.42514598824282,7.27586460054653,0,0 +2007,13313,4.42514598824282,7.18841273649695,0,0 +2003,13317,2.36902804954276,4.92725368515721,0,0 +2004,13317,2.36902804954276,4.74493212836325,0,0 +2005,13317,2.36902804954276,4.64439089914137,0,0 +2006,13317,2.36902804954276,4.59511985013459,0,0 +2007,13317,2.36902804954276,4.54329478227,0,0 +2003,13321,3.08954132723205,4.96284463025991,0,0 +2004,13321,3.08954132723205,4.97673374242057,0,0 +2005,13321,3.08954132723205,5.08759633523238,0,0 +2006,13321,3.08954132723205,5.15329159449778,0,0 +2007,13321,3.08954132723205,4.89034912822175,0,0 +2003,16003,1.24588220740315,3.25809653802148,0,0 +2004,16003,1.24588220740315,3.61091791264422,0,0 +2005,16003,1.24588220740315,3.98898404656427,0,0 +2006,16003,1.24588220740315,3.98898404656427,0,0 +2007,16003,1.24588220740315,3.89182029811063,0,0 +2003,16015,1.89761985992753,3.89182029811063,0,0 +2004,16015,1.89761985992753,3.91202300542815,0,0 +2005,16015,1.89761985992753,4.17438726989564,0,0 +2006,16015,1.89761985992753,4.02535169073515,0,0 +2007,16015,1.89761985992753,4.14313472639153,0,0 +2003,16017,3.60644848013478,6.29710931993394,0,0 +2004,16017,3.60644848013478,6.29894924685594,0,0 +2005,16017,3.60644848013478,6.4281052726846,0,0 +2006,16017,3.60644848013478,6.42971947803914,0,0 +2007,16017,3.60644848013478,6.40522845803084,0,0 +2003,16021,2.28960116543575,4.89783979995091,0,0 +2004,16021,2.28960116543575,4.83628190695148,0,0 +2005,16021,2.28960116543575,4.96284463025991,0,0 +2006,16021,2.28960116543575,4.99721227376411,0,0 +2007,16021,2.28960116543575,4.82028156560504,0,0 +2003,16029,1.98842214329267,4.9416424226093,0,0 +2004,16029,1.98842214329267,4.79579054559674,0,0 +2005,16029,1.98842214329267,4.98360662170834,0,0 +2006,16029,1.98842214329267,4.78749174278205,0,0 +2007,16029,1.98842214329267,5.03695260241363,0,0 +2003,16041,2.42736580989135,5.63835466933375,0,0 +2004,16041,2.42736580989135,5.53733426701854,0,0 +2005,16041,2.42736580989135,5.62762111369064,0,0 +2006,16041,2.42736580989135,5.6937321388027,0,0 +2007,16041,2.42736580989135,5.74939298590825,0,0 +2003,16045,2.72004464595058,5.01727983681492,0,0 +2004,16045,2.72004464595058,4.92725368515721,0,0 +2005,16045,2.72004464595058,5.12989871492307,0,0 +2006,16045,2.72004464595058,5.08759633523238,0,0 +2007,16045,2.72004464595058,5.19849703126583,0,0 +2003,16067,3.00439464663264,5.73334127689775,0,0 +2004,16067,3.00439464663264,5.73979291217923,0,0 +2005,16067,3.00439464663264,5.73334127689775,0,0 +2006,16067,3.00439464663264,5.77455154554441,0,0 +2007,16067,3.00439464663264,5.80814248998044,0,0 +2003,16079,2.62256493174499,5.09375020080676,0,0 +2004,16079,2.62256493174499,5.17614973257383,0,0 +2005,16079,2.62256493174499,5.14749447681345,0,0 +2006,16079,2.62256493174499,5.24702407216049,0,0 +2007,16079,2.62256493174499,5.38449506278909,0,0 +2003,16083,4.16331076668682,7.20489251020467,0,0 +2004,16083,4.16331076668682,7.18311170174328,0,0 +2005,16083,4.16331076668682,7.14045304310116,0,0 +2006,16083,4.16331076668682,7.23489842031483,0,0 +2007,16083,4.16331076668682,7.28961052145117,0,0 +2003,17005,2.86977214642579,5.18738580584075,2004,1 +2004,17005,2.86977214642579,5.06890420222023,2004,1 +2005,17005,2.86977214642579,4.93447393313069,2004,1 +2006,17005,2.86977214642579,4.89034912822175,2004,1 +2007,17005,2.86977214642579,4.99043258677874,2004,1 +2003,17015,2.81385061998842,5.42934562895444,2004,1 +2004,17015,2.81385061998842,5.42053499927229,2004,1 +2005,17015,2.81385061998842,5.50125821054473,2004,1 +2006,17015,2.81385061998842,5.46806014113513,2004,1 +2007,17015,2.81385061998842,5.4553211153577,2004,1 +2003,17025,2.67827804276854,5.08759633523238,2004,1 +2004,17025,2.67827804276854,5.00394630594546,2004,1 +2005,17025,2.67827804276854,4.85203026391962,2004,1 +2006,17025,2.67827804276854,4.98360662170834,2004,1 +2007,17025,2.67827804276854,4.97673374242057,2004,1 +2003,17035,2.42063475976786,4.92725368515721,2004,1 +2004,17035,2.42063475976786,4.59511985013459,2004,1 +2005,17035,2.42063475976786,4.44265125649032,2004,1 +2006,17035,2.42063475976786,4.30406509320417,2004,1 +2007,17035,2.42063475976786,4.46590811865458,2004,1 +2003,17047,1.94175868650426,4.35670882668959,2004,1 +2004,17047,1.94175868650426,4.39444915467244,2004,1 +2005,17047,1.94175868650426,4.20469261939097,2004,1 +2006,17047,1.94175868650426,4.00733318523247,2004,1 +2007,17047,1.94175868650426,4.12713438504509,2004,1 +2003,17049,3.53409524031307,6.99025650049388,2004,1 +2004,17049,3.53409524031307,6.90975328164481,2004,1 +2005,17049,3.53409524031307,6.87523208727658,2004,1 +2006,17049,3.53409524031307,6.8330317327862,2004,1 +2007,17049,3.53409524031307,6.87626461189077,2004,1 +2003,17053,2.65612512823729,5.61677109766657,2004,1 +2004,17053,2.65612512823729,5.48893772615669,2004,1 +2005,17053,2.65612512823729,5.48893772615669,2004,1 +2006,17053,2.65612512823729,5.51342874616498,2004,1 +2007,17053,2.65612512823729,5.31320597904179,2004,1 +2003,17063,3.62527383102497,6.58063913728495,2004,1 +2004,17063,3.62527383102497,6.47697236288968,2004,1 +2005,17063,3.62527383102497,6.49375383985169,2004,1 +2006,17063,3.62527383102497,6.4831073514572,2004,1 +2007,17063,3.62527383102497,6.5694814204143,2004,1 +2003,17089,6.00170938904395,9.0067543198775,2004,1 +2004,17089,6.00170938904395,8.96763166516713,2004,1 +2005,17089,6.00170938904395,8.99118884193151,2004,1 +2006,17089,6.00170938904395,8.97802993283716,2004,1 +2007,17089,6.00170938904395,9.02509454366498,2004,1 +2003,17097,6.46825136840052,9.27369068747694,2004,1 +2004,17097,6.46825136840052,9.22147811638664,2004,1 +2005,17097,6.46825136840052,9.2072355570227,2004,1 +2006,17097,6.46825136840052,9.24599704918422,2004,1 +2007,17097,6.46825136840052,9.23824732522919,2004,1 +2003,17113,5.01351780234143,7.89095671613892,2004,1 +2004,17113,5.01351780234143,7.77653502818524,2004,1 +2005,17113,5.01351780234143,7.755767170103,2004,1 +2006,17113,5.01351780234143,7.74022952476318,2004,1 +2007,17113,5.01351780234143,7.8188324438034,2004,1 +2003,17125,2.77496090638484,5.24702407216049,2004,1 +2004,17125,2.77496090638484,5.25227342804663,2004,1 +2005,17125,2.77496090638484,4.97673374242057,2004,1 +2006,17125,2.77496090638484,4.99043258677874,2004,1 +2007,17125,2.77496090638484,5.05624580534831,2004,1 +2003,17135,3.42269791310816,6.07764224334903,2004,1 +2004,17135,3.42269791310816,6.10255859461357,2004,1 +2005,17135,3.42269791310816,6.15485809401642,2004,1 +2006,17135,3.42269791310816,6.0137151560428,2004,1 +2007,17135,3.42269791310816,6.01615715969835,2004,1 +2003,17155,1.80599105147478,4.02535169073515,2004,1 +2004,17155,1.80599105147478,3.73766961828337,2004,1 +2005,17155,1.80599105147478,3.91202300542815,2004,1 +2006,17155,1.80599105147478,3.78418963391826,2004,1 +2007,17155,1.80599105147478,4.24849524204936,2004,1 +2003,17161,5.0064532281027,7.87663846097546,2004,1 +2004,17161,5.0064532281027,7.82404601085629,2004,1 +2005,17161,5.0064532281027,7.86557175768479,2004,1 +2006,17161,5.0064532281027,7.85166117788927,2004,1 +2007,17161,5.0064532281027,7.82923253754359,2004,1 +2003,17169,1.97255208000173,4.52178857704904,2004,1 +2004,17169,1.97255208000173,4.53259949315326,2004,1 +2005,17169,1.97255208000173,4.31748811353631,2004,1 +2006,17169,1.97255208000173,4.38202663467388,2004,1 +2007,17169,1.97255208000173,4.43081679884331,2004,1 +2003,17193,2.7324826172428,5.4510384535657,2004,1 +2004,17193,2.7324826172428,5.3890717298165,2004,1 +2005,17193,2.7324826172428,5.25227342804663,2004,1 +2006,17193,2.7324826172428,5.17614973257383,2004,1 +2007,17193,2.7324826172428,5.27299955856375,2004,1 +2003,17195,4.10516909830525,6.71780469502369,2004,1 +2004,17195,4.10516909830525,6.6682282484174,2004,1 +2005,17195,4.10516909830525,6.76849321164863,2004,1 +2006,17195,4.10516909830525,6.68959926917897,2004,1 +2007,17195,4.10516909830525,6.73221070646721,2004,1 +2003,17199,4.11571458795829,6.81014245011514,2004,1 +2004,17199,4.11571458795829,6.8596149036542,2004,1 +2005,17199,4.11571458795829,6.77764659363512,2004,1 +2006,17199,4.11571458795829,6.72982407048948,2004,1 +2007,17199,4.11571458795829,6.77878489768518,2004,1 +2003,17203,3.56865907554909,6.5191472879404,2004,1 +2004,17203,3.56865907554909,6.44094654063292,2004,1 +2005,17203,3.56865907554909,6.41673228251233,2004,1 +2006,17203,3.56865907554909,6.46925031679577,2004,1 +2007,17203,3.56865907554909,6.48616078894409,2004,1 +2003,18005,4.26878794531712,7.15305163493748,0,0 +2004,18005,4.26878794531712,7.11720550316434,0,0 +2005,18005,4.26878794531712,7.16626597413364,0,0 +2006,18005,4.26878794531712,7.10002716662926,0,0 +2007,18005,4.26878794531712,7.08170858610557,0,0 +2003,18017,3.71186329050431,6.29156913955832,0,0 +2004,18017,3.71186329050431,6.20455776256869,0,0 +2005,18017,3.71186329050431,6.10255859461357,0,0 +2006,18017,3.71186329050431,6.0913098820777,0,0 +2007,18017,3.71186329050431,6.06145691892802,0,0 +2003,18019,4.56925281080129,7.40610338123702,0,0 +2004,18019,4.56925281080129,7.31388683163346,0,0 +2005,18019,4.56925281080129,7.27931883541462,0,0 +2006,18019,4.56925281080129,7.40974195408092,0,0 +2007,18019,4.56925281080129,7.36897040219479,0,0 +2003,18021,3.27925571085691,5.47227067367148,0,0 +2004,18021,3.27925571085691,5.73657229747919,0,0 +2005,18021,3.27925571085691,5.75574221358691,0,0 +2006,18021,3.27925571085691,5.60947179518496,0,0 +2007,18021,3.27925571085691,5.56068163101553,0,0 +2003,18023,3.52241156124315,6.12029741895095,0,0 +2004,18023,3.52241156124315,6.07073772800249,0,0 +2005,18023,3.52241156124315,6.04263283368238,0,0 +2006,18023,3.52241156124315,6.00141487796115,0,0 +2007,18023,3.52241156124315,6.02344759296103,0,0 +2003,18029,3.83100815871387,6.80239476332431,0,0 +2004,18029,3.83100815871387,6.74405918631135,0,0 +2005,18029,3.83100815871387,6.75693238924755,0,0 +2006,18029,3.83100815871387,6.64898455002478,0,0 +2007,18029,3.83100815871387,6.81014245011514,0,0 +2003,18033,3.69597919122916,6.50727771238501,0,0 +2004,18033,3.69597919122916,6.42162226780652,0,0 +2005,18033,3.69597919122916,6.43615036836943,0,0 +2006,18033,3.69597919122916,6.51174532964473,0,0 +2007,18033,3.69597919122916,6.37842618365159,0,0 +2003,18035,4.77718043011612,7.34729970074316,0,0 +2004,18035,4.77718043011612,7.22766249872865,0,0 +2005,18035,4.77718043011612,7.1929342212158,0,0 +2006,18035,4.77718043011612,7.18992217074581,0,0 +2007,18035,4.77718043011612,7.17472430983638,0,0 +2003,18053,4.29596480671984,6.87109129461055,0,0 +2004,18053,4.29596480671984,6.78105762593618,0,0 +2005,18053,4.29596480671984,6.75809450442773,0,0 +2006,18053,4.29596480671984,6.67076632084587,0,0 +2007,18053,4.29596480671984,6.66568371778241,0,0 +2003,18063,4.64528473032447,7.56060116276856,0,0 +2004,18063,4.64528473032447,7.59789795052178,0,0 +2005,18063,4.64528473032447,7.64444076155657,0,0 +2006,18063,4.64528473032447,7.70616297019958,0,0 +2007,18063,4.64528473032447,7.74066440191724,0,0 +2003,18081,4.74674787021019,7.74975340627444,0,0 +2004,18081,4.74674787021019,7.73805229768932,0,0 +2005,18081,4.74674787021019,7.71868549519847,0,0 +2006,18081,4.74674787021019,7.7106533235012,0,0 +2007,18081,4.74674787021019,7.71646080017636,0,0 +2003,18083,3.6701042987876,6.33150184989369,0,0 +2004,18083,3.6701042987876,6.24997524225948,0,0 +2005,18083,3.6701042987876,6.24804287450843,0,0 +2006,18083,3.6701042987876,6.29341927884648,0,0 +2007,18083,3.6701042987876,6.18208490671663,0,0 +2003,18091,4.70144353815659,7.43543801981455,0,0 +2004,18091,4.70144353815659,7.37337430991005,0,0 +2005,18091,4.70144353815659,7.39510754656249,0,0 +2006,18091,4.70144353815659,7.3790081276283,0,0 +2007,18091,4.70144353815659,7.37148929521428,0,0 +2003,18093,3.82694430506983,6.21060007702465,0,0 +2004,18093,3.82694430506983,6.15485809401642,0,0 +2005,18093,3.82694430506983,6.15697898558556,0,0 +2006,18093,3.82694430506983,6.15909538849193,0,0 +2007,18093,3.82694430506983,6.20253551718792,0,0 +2003,18099,3.80950289643767,6.57368016696065,0,0 +2004,18099,3.80950289643767,6.51471269087253,0,0 +2005,18099,3.80950289643767,6.48004456192665,0,0 +2006,18099,3.80950289643767,6.52356230614951,0,0 +2007,18099,3.80950289643767,6.45833828334479,0,0 +2003,18109,4.20004002177996,6.68085467879022,0,0 +2004,18109,4.20004002177996,6.63068338564237,0,0 +2005,18109,4.20004002177996,6.65929391968364,0,0 +2006,18109,4.20004002177996,6.61606518513282,0,0 +2007,18109,4.20004002177996,6.59714570188665,0,0 +2003,18113,3.83460185845956,6.49072353450251,0,0 +2004,18113,3.83460185845956,6.45676965557216,0,0 +2005,18113,3.83460185845956,6.3919171133926,0,0 +2006,18113,3.83460185845956,6.41999492814714,0,0 +2007,18113,3.83460185845956,6.39692965521615,0,0 +2003,18115,1.72686532931006,4.29045944114839,0,0 +2004,18115,1.72686532931006,4.26267987704132,0,0 +2005,18115,1.72686532931006,4.17438726989564,0,0 +2006,18115,1.72686532931006,4.04305126783455,0,0 +2007,18115,1.72686532931006,4.07753744390572,0,0 +2003,18119,3.08126756166025,5.28320372873799,0,0 +2004,18119,3.08126756166025,5.30330490805908,0,0 +2005,18119,3.08126756166025,5.35185813347607,0,0 +2006,18119,3.08126756166025,5.30826769740121,0,0 +2007,18119,3.08126756166025,5.27299955856375,0,0 +2003,18133,3.58404657700818,6.4425401664682,0,0 +2004,18133,3.58404657700818,6.47697236288968,0,0 +2005,18133,3.58404657700818,6.43294009273918,0,0 +2006,18133,3.58404657700818,6.28785856016178,0,0 +2007,18133,3.58404657700818,6.24610676548156,0,0 +2003,18145,3.77149577029117,6.48768401848461,0,0 +2004,18145,3.77149577029117,6.4723462945009,0,0 +2005,18145,3.77149577029117,6.31173480915291,0,0 +2006,18145,3.77149577029117,6.18826412308259,0,0 +2007,18145,3.77149577029117,6.13556489108174,0,0 +2003,18157,5.0036442469074,7.70616297019958,0,0 +2004,18157,5.0036442469074,7.6425241342329,0,0 +2005,18157,5.0036442469074,7.70751219460034,0,0 +2006,18157,5.0036442469074,7.73587031995257,0,0 +2007,18157,5.0036442469074,7.74196789982069,0,0 +2003,18175,3.30406220422892,5.35185813347607,0,0 +2004,18175,3.30406220422892,5.31811999384422,0,0 +2005,18175,3.30406220422892,5.22035582507832,0,0 +2006,18175,3.30406220422892,5.24702407216049,0,0 +2007,18175,3.30406220422892,5.27299955856375,0,0 +2003,18179,3.3178157727231,6.08449941307517,0,0 +2004,18179,3.3178157727231,5.94803498918065,0,0 +2005,18179,3.3178157727231,5.92958914338989,0,0 +2006,18179,3.3178157727231,6.05208916892442,0,0 +2007,18179,3.3178157727231,5.83188247728352,0,0 +2003,18181,3.22949919650533,5.53338948872752,0,0 +2004,18181,3.22949919650533,5.32300997913841,0,0 +2005,18181,3.22949919650533,5.28826703069454,0,0 +2006,18181,3.22949919650533,5.25749537202778,0,0 +2007,18181,3.22949919650533,5.23110861685459,0,0 +2003,19013,4.85212400952536,7.9043348420851,0,0 +2004,19013,4.85212400952536,7.83794891602528,0,0 +2005,19013,4.85212400952536,7.8407064517494,0,0 +2006,19013,4.85212400952536,7.85476918349913,0,0 +2007,19013,4.85212400952536,7.87929148508227,0,0 +2003,19025,2.40829554741616,5.46806014113513,0,0 +2004,19025,2.40829554741616,5.2257466737132,0,0 +2005,19025,2.40829554741616,5.18738580584075,0,0 +2006,19025,2.40829554741616,5.22035582507832,0,0 +2007,19025,2.40829554741616,5.15329159449778,0,0 +2003,19069,2.37061750338587,5.28320372873799,0,0 +2004,19069,2.37061750338587,5.21493575760899,0,0 +2005,19069,2.37061750338587,5.31320597904179,0,0 +2006,19069,2.37061750338587,5.19849703126583,0,0 +2007,19069,2.37061750338587,5.2040066870768,0,0 +2003,19077,2.4294820261824,5.05624580534831,0,0 +2004,19077,2.4294820261824,4.97673374242057,0,0 +2005,19077,2.4294820261824,4.9416424226093,0,0 +2006,19077,2.4294820261824,4.84418708645859,0,0 +2007,19077,2.4294820261824,4.87519732320115,0,0 +2003,19083,2.9344949640828,5.87493073085203,0,0 +2004,19083,2.9344949640828,5.78689738136671,0,0 +2005,19083,2.9344949640828,5.75257263882563,0,0 +2006,19083,2.9344949640828,5.77144112313002,0,0 +2007,19083,2.9344949640828,5.75257263882563,0,0 +2003,19087,3.0123927144471,6.20253551718792,0,0 +2004,19087,3.0123927144471,6.01126717440416,0,0 +2005,19087,3.0123927144471,6.0137151560428,0,0 +2006,19087,3.0123927144471,5.98896141688986,0,0 +2007,19087,3.0123927144471,5.92692602597041,0,0 +2003,19111,3.63895364534421,6.40687998606931,0,0 +2004,19111,3.63895364534421,6.32076829425058,0,0 +2005,19111,3.63895364534421,6.39359075395063,0,0 +2006,19111,3.63895364534421,6.43615036836943,0,0 +2007,19111,3.63895364534421,6.43294009273918,0,0 +2003,19113,5.25593686652208,8.30992298925832,0,0 +2004,19113,5.25593686652208,8.28702502516506,0,0 +2005,19113,5.25593686652208,8.25400859056484,0,0 +2006,19113,5.25593686652208,8.28526113406895,0,0 +2007,19113,5.25593686652208,8.36985260351753,0,0 +2003,19123,3.10615495480408,5.89989735358249,0,0 +2004,19123,3.10615495480408,5.8406416573734,0,0 +2005,19123,3.10615495480408,5.73657229747919,0,0 +2006,19123,3.10615495480408,5.80211837537706,0,0 +2007,19123,3.10615495480408,5.77455154554441,0,0 +2003,19131,2.38637461872373,5.26785815906333,0,0 +2004,19131,2.38637461872373,5.18738580584075,0,0 +2005,19131,2.38637461872373,5.27811465923052,0,0 +2006,19131,2.38637461872373,5.25749537202778,0,0 +2007,19131,2.38637461872373,5.31811999384422,0,0 +2003,19137,2.46563887943034,5.37527840768417,0,0 +2004,19137,2.46563887943034,5.44673737166631,0,0 +2005,19137,2.46563887943034,5.37989735354046,0,0 +2006,19137,2.46563887943034,5.4510384535657,0,0 +2007,19137,2.46563887943034,5.4553211153577,0,0 +2003,19141,2.71482718538123,6.08904487544685,0,0 +2004,19141,2.71482718538123,6.00141487796115,0,0 +2005,19141,2.71482718538123,6.0330862217988,0,0 +2006,19141,2.71482718538123,6.05208916892442,0,0 +2007,19141,2.71482718538123,5.88610403145016,0,0 +2003,19153,5.92586145952057,9.19136140537838,0,0 +2004,19153,5.92586145952057,9.12565356380899,0,0 +2005,19153,5.92586145952057,9.12956406194889,0,0 +2006,19153,5.92586145952057,9.14729407025855,0,0 +2007,19153,5.92586145952057,9.1557785839835,0,0 +2003,19157,2.9346544240454,6.3456363608286,0,0 +2004,19157,2.9346544240454,6.09582456243222,0,0 +2005,19157,2.9346544240454,6.06378520868761,0,0 +2006,19157,2.9346544240454,6.1463292576689,0,0 +2007,19157,2.9346544240454,6.15485809401642,0,0 +2003,19159,1.69909578436464,4.64439089914137,0,0 +2004,19159,1.69909578436464,4.53259949315326,0,0 +2005,19159,1.69909578436464,4.33073334028633,0,0 +2006,19159,1.69909578436464,4.27666611901606,0,0 +2007,19159,1.69909578436464,4.36944785246702,0,0 +2003,19183,3.02868337369368,6.09582456243222,0,0 +2004,19183,3.02868337369368,5.92692602597041,0,0 +2005,19183,3.02868337369368,5.88887795833288,0,0 +2006,19183,3.02868337369368,5.97888576490112,0,0 +2007,19183,3.02868337369368,6.12468339089421,0,0 +2003,19185,1.90657514365664,4.24849524204936,0,0 +2004,19185,1.90657514365664,3.93182563272433,0,0 +2005,19185,1.90657514365664,4.02535169073515,0,0 +2006,19185,1.90657514365664,4.06044301054642,0,0 +2007,19185,1.90657514365664,4.4188406077966,0,0 +2003,19187,3.6947372635981,6.73340189183736,0,0 +2004,19187,3.6947372635981,6.68336094576627,0,0 +2005,19187,3.6947372635981,6.64898455002478,0,0 +2006,19187,3.6947372635981,6.68336094576627,0,0 +2007,19187,3.6947372635981,6.67329796776765,0,0 +2003,20013,2.37248422037371,5.42934562895444,0,0 +2004,20013,2.37248422037371,5.45958551414416,0,0 +2005,20013,2.37248422037371,5.50125821054473,0,0 +2006,20013,2.37248422037371,5.31811999384422,0,0 +2007,20013,2.37248422037371,5.28320372873799,0,0 +2003,20021,3.11817112074657,5.59098698051086,0,0 +2004,20021,3.11817112074657,5.35658627467201,0,0 +2005,20021,3.11817112074657,5.54517744447956,0,0 +2006,20021,3.11817112074657,5.40267738187228,0,0 +2007,20021,3.11817112074657,5.31811999384422,0,0 +2003,20027,2.17724860168275,5.14166355650266,0,0 +2004,20027,2.17724860168275,5.23644196282995,0,0 +2005,20027,2.17724860168275,5.37063802812766,0,0 +2006,20027,2.17724860168275,5.39362754635236,0,0 +2007,20027,2.17724860168275,5.40267738187228,0,0 +2003,20031,2.18211093952617,5.28826703069454,0,0 +2004,20031,2.18211093952617,5.19849703126583,0,0 +2005,20031,2.18211093952617,4.94875989037817,0,0 +2006,20031,2.18211093952617,5.37527840768417,0,0 +2007,20031,2.18211093952617,5.39362754635236,0,0 +2003,20045,4.6047901137698,7.52240023138712,0,0 +2004,20045,4.6047901137698,7.3664451483276,0,0 +2005,20045,4.6047901137698,7.37525577800975,0,0 +2006,20045,4.6047901137698,7.40306109109009,0,0 +2007,20045,4.6047901137698,7.4265490723973,0,0 +2003,20047,1.2380843339547,3.66356164612965,0,0 +2004,20047,1.2380843339547,3.80666248977032,0,0 +2005,20047,1.2380843339547,3.63758615972639,0,0 +2006,20047,1.2380843339547,2.94443897916644,0,0 +2007,20047,1.2380843339547,3.8286413964891,0,0 +2003,20063,1.12102588350501,4.26267987704132,0,0 +2004,20063,1.12102588350501,4.18965474202643,0,0 +2005,20063,1.12102588350501,4.00733318523247,0,0 +2006,20063,1.12102588350501,4.04305126783455,0,0 +2007,20063,1.12102588350501,3.97029191355212,0,0 +2003,20085,2.53821042181031,5.4380793089232,0,0 +2004,20085,2.53821042181031,5.37527840768417,0,0 +2005,20085,2.53821042181031,5.3890717298165,0,0 +2006,20085,2.53821042181031,5.35185813347607,0,0 +2007,20085,2.53821042181031,5.40717177146012,0,0 +2003,20095,2.16021475170232,4.92725368515721,0,0 +2004,20095,2.16021475170232,4.91265488573605,0,0 +2005,20095,2.16021475170232,5.03695260241363,0,0 +2006,20095,2.16021475170232,4.969813299576,0,0 +2007,20095,2.16021475170232,5.05624580534831,0,0 +2003,20103,4.22961818627967,6.59578051396131,0,0 +2004,20103,4.22961818627967,6.58892647753352,0,0 +2005,20103,4.22961818627967,6.46458830368996,0,0 +2006,20103,4.22961818627967,6.49828214947643,0,0 +2007,20103,4.22961818627967,6.45676965557216,0,0 +2003,20133,2.83303685789521,6.11146733950268,0,0 +2004,20133,2.83303685789521,5.92157841964382,0,0 +2005,20133,2.83303685789521,5.82008293035236,0,0 +2006,20133,2.83303685789521,5.84354441703136,0,0 +2007,20133,2.83303685789521,5.72358510195238,0,0 +2003,20137,1.78389529417125,4.7361984483945,0,0 +2004,20137,1.78389529417125,4.70953020131233,0,0 +2005,20137,1.78389529417125,4.85203026391962,0,0 +2006,20137,1.78389529417125,4.79579054559674,0,0 +2007,20137,1.78389529417125,4.64439089914137,0,0 +2003,20143,1.81856367198011,3.95124371858143,0,0 +2004,20143,1.81856367198011,3.98898404656427,0,0 +2005,20143,1.81856367198011,3.98898404656427,0,0 +2006,20143,1.81856367198011,4.18965474202643,0,0 +2007,20143,1.81856367198011,4.06044301054642,0,0 +2003,20145,1.97865388786714,4.49980967033027,0,0 +2004,20145,1.97865388786714,4.48863636973214,0,0 +2005,20145,1.97865388786714,4.63472898822964,0,0 +2006,20145,1.97865388786714,4.80402104473326,0,0 +2007,20145,1.97865388786714,4.70048036579242,0,0 +2003,20171,1.63315443905142,5.03043792139244,0,0 +2004,20171,1.63315443905142,4.91265488573605,0,0 +2005,20171,1.63315443905142,4.81218435537242,0,0 +2006,20171,1.63315443905142,4.8283137373023,0,0 +2007,20171,1.63315443905142,4.79579054559674,0,0 +2003,20183,1.51204556642545,3.97029191355212,0,0 +2004,20183,1.51204556642545,3.93182563272433,0,0 +2005,20183,1.51204556642545,3.63758615972639,0,0 +2006,20183,1.51204556642545,4.17438726989564,0,0 +2007,20183,1.51204556642545,4.38202663467388,0,0 +2003,20205,2.33524587523359,4.79579054559674,0,0 +2004,20205,2.33524587523359,4.79579054559674,0,0 +2005,20205,2.33524587523359,4.81218435537242,0,0 +2006,20205,2.33524587523359,4.94875989037817,0,0 +2007,20205,2.33524587523359,5.16478597392351,0,0 +2003,22007,3.15222307031342,4.77912349311153,0,0 +2004,22007,3.15222307031342,4.83628190695148,0,0 +2005,22007,3.15222307031342,4.57471097850338,0,0 +2006,22007,3.15222307031342,4.91998092582813,0,0 +2007,22007,3.15222307031342,5.07517381523383,0,0 +2003,22033,6.02308917522617,8.74957394808293,0,0 +2004,22033,6.02308917522617,8.68016201969438,0,0 +2005,22033,6.02308917522617,8.68440111040014,0,0 +2006,22033,6.02308917522617,8.78094111357239,0,0 +2007,22033,6.02308917522617,8.65782432115598,0,0 +2003,22045,4.29409665407894,6.65415252018322,0,0 +2004,22045,4.29409665407894,6.58063913728495,0,0 +2005,22045,4.29409665407894,6.58202513889283,0,0 +2006,22045,4.29409665407894,6.76272950693188,0,0 +2007,22045,4.29409665407894,6.77992190747225,0,0 +2003,22069,3.66561082717458,5.64190707093811,0,0 +2004,22069,3.66561082717458,5.53733426701854,0,0 +2005,22069,3.66561082717458,5.40717177146012,0,0 +2006,22069,3.66561082717458,5.53733426701854,0,0 +2007,22069,3.66561082717458,5.78689738136671,0,0 +2003,22071,6.18347649998489,8.31164394850298,0,0 +2004,22071,6.18347649998489,8.1371033896393,0,0 +2005,22071,6.18347649998489,8.14206328310415,0,0 +2006,22071,6.18347649998489,7.1785454837637,0,0 +2007,22071,6.18347649998489,7.75147531802146,0,0 +2003,22073,4.9921318225317,7.51043055637801,0,0 +2004,22073,4.9921318225317,7.35115822643069,0,0 +2005,22073,4.9921318225317,7.40367029001237,0,0 +2006,22073,4.9921318225317,7.45240245122364,0,0 +2007,22073,4.9921318225317,7.50988306115491,0,0 +2003,22081,2.26405214327726,4.02535169073515,0,0 +2004,22081,2.26405214327726,3.78418963391826,0,0 +2005,22081,2.26405214327726,3.36729582998647,0,0 +2006,22081,2.26405214327726,3.3322045101752,0,0 +2007,22081,2.26405214327726,3.3322045101752,0,0 +2003,22087,4.20810470205826,6.74051935960622,0,0 +2004,22087,4.20810470205826,6.58892647753352,0,0 +2005,22087,4.20810470205826,6.5792512120101,0,0 +2006,22087,4.20810470205826,4.45434729625351,0,0 +2007,22087,4.20810470205826,5.21493575760899,0,0 +2003,22091,2.35375337956845,3.2188758248682,0,0 +2004,22091,2.35375337956845,3.55534806148941,0,0 +2005,22091,2.35375337956845,3.17805383034795,0,0 +2006,22091,2.35375337956845,3.3322045101752,0,0 +2007,22091,2.35375337956845,3.36729582998647,0,0 +2003,22111,3.12689210625201,5.05624580534831,0,0 +2004,22111,3.12689210625201,4.69134788222914,0,0 +2005,22111,3.12689210625201,5.05624580534831,0,0 +2006,22111,3.12689210625201,4.82028156560504,0,0 +2007,22111,3.12689210625201,4.88280192258637,0,0 +2003,22115,3.96140347152558,5.605802066296,0,0 +2004,22115,3.96140347152558,5.64544689764324,0,0 +2005,22115,3.96140347152558,5.65599181081985,0,0 +2006,22115,3.96140347152558,5.74620319054015,0,0 +2007,22115,3.96140347152558,5.78382518232974,0,0 +2003,22119,3.73363769145571,5.6970934865054,0,0 +2004,22119,3.73363769145571,5.46806014113513,0,0 +2005,22119,3.73363769145571,5.61677109766657,0,0 +2006,22119,3.73363769145571,5.57594910314632,0,0 +2007,22119,3.73363769145571,5.81413053182507,0,0 +2003,24009,4.31164440561939,7.02553831463852,2007,1 +2004,24009,4.31164440561939,7.14677217945264,2007,1 +2005,24009,4.31164440561939,7.16162200293919,2007,1 +2006,24009,4.31164440561939,7.1066061377273,2007,1 +2007,24009,4.31164440561939,7.1929342212158,2007,1 +2003,24015,4.4537773664325,7.04053639021596,2007,1 +2004,24015,4.4537773664325,7.15305163493748,2007,1 +2005,24015,4.4537773664325,7.14440718032114,2007,1 +2006,24015,4.4537773664325,7.18841273649695,2007,1 +2007,24015,4.4537773664325,7.11151211649616,2007,1 +2003,24017,4.79203142282408,7.68017564043659,2007,1 +2004,24017,4.79203142282408,7.69256964806791,2007,1 +2005,24017,4.79203142282408,7.69074316354187,2007,1 +2006,24017,4.79203142282408,7.75491027202143,2007,1 +2007,24017,4.79203142282408,7.75619534394812,2007,1 +2003,24021,5.27441906341037,8.31016902198191,2007,1 +2004,24021,5.27441906341037,8.36264243156764,2007,1 +2005,24021,5.27441906341037,8.30819906320645,2007,1 +2006,24021,5.27441906341037,8.32748441618826,2007,1 +2007,24021,5.27441906341037,8.28803156777646,2007,1 +2003,24027,5.5127914463584,8.76826314537129,2007,1 +2004,24027,5.5127914463584,8.74273386733017,2007,1 +2005,24027,5.5127914463584,8.67077227934454,2007,1 +2006,24027,5.5127914463584,8.71719093322313,2007,1 +2007,24027,5.5127914463584,8.75131624677346,2007,1 +2003,24033,6.68650368678403,9.10830764641103,2007,1 +2004,24033,6.68650368678403,9.14034700015565,2007,1 +2005,24033,6.68650368678403,9.17232692977797,2007,1 +2006,24033,6.68650368678403,9.17926241640532,2007,1 +2007,24033,6.68650368678403,9.15567297063387,2007,1 +2003,26005,4.66027371221412,7.1693500166706,2007,1 +2004,26005,4.66027371221412,7.15773548424991,2007,1 +2005,26005,4.66027371221412,7.10578612948127,2007,1 +2006,26005,4.66027371221412,7.05185562295589,2007,1 +2007,26005,4.66027371221412,6.98749024700099,2007,1 +2003,26015,4.0387437582073,6.34388043412633,2007,1 +2004,26015,4.0387437582073,6.25190388316589,2007,1 +2005,26015,4.0387437582073,6.28599809450886,2007,1 +2006,26015,4.0387437582073,6.18001665365257,2007,1 +2007,26015,4.0387437582073,6.19440539110467,2007,1 +2003,26027,3.9338627720326,5.42053499927229,2007,1 +2004,26027,3.9338627720326,5.62401750618734,2007,1 +2005,26027,3.9338627720326,5.72358510195238,2007,1 +2006,26027,3.9338627720326,5.58724865840025,2007,1 +2007,26027,3.9338627720326,5.27299955856375,2007,1 +2003,26031,3.27518054107867,5.27811465923052,2007,1 +2004,26031,3.27518054107867,5.12989871492307,2007,1 +2005,26031,3.27518054107867,5.12989871492307,2007,1 +2006,26031,3.27518054107867,5.11198778835654,2007,1 +2007,26031,3.27518054107867,4.99721227376411,2007,1 +2003,26041,3.65117758692992,6.31173480915291,2007,1 +2004,26041,3.65117758692992,6.32076829425058,2007,1 +2005,26041,3.65117758692992,6.25766758788264,2007,1 +2006,26041,3.65117758692992,6.25382881157547,2007,1 +2007,26041,3.65117758692992,6.13988455222626,2007,1 +2003,26057,3.74443241323419,6.13988455222626,2007,1 +2004,26057,3.74443241323419,6.09582456243222,2007,1 +2005,26057,3.74443241323419,6.25190388316589,2007,1 +2006,26057,3.74443241323419,6.12249280951439,2007,1 +2007,26057,3.74443241323419,5.92692602597041,2007,1 +2003,26061,3.58396328416438,6.42648845745769,2007,1 +2004,26061,3.58396328416438,6.40025744530882,2007,1 +2005,26061,3.58396328416438,6.33859407820318,2007,1 +2006,26061,3.58396328416438,6.22851100359118,2007,1 +2007,26061,3.58396328416438,6.13556489108174,2007,1 +2003,26065,5.63235807797783,8.4142741374084,2007,1 +2004,26065,5.63235807797783,8.33254893925264,2007,1 +2005,26065,5.63235807797783,8.24643378616036,2007,1 +2006,26065,5.63235807797783,8.21283958467648,2007,1 +2007,26065,5.63235807797783,8.16365617616843,2007,1 +2003,26073,4.14869069196221,6.73933662735717,2007,1 +2004,26073,4.14869069196221,6.59850902861452,2007,1 +2005,26073,4.14869069196221,6.68586094706836,2007,1 +2006,26073,4.14869069196221,6.53087762772588,2007,1 +2007,26073,4.14869069196221,6.51025834052315,2007,1 +2003,26085,2.42771882375081,3.66356164612965,2007,1 +2004,26085,2.42771882375081,3.40119738166216,2007,1 +2005,26085,2.42771882375081,3.63758615972639,2007,1 +2006,26085,2.42771882375081,3.2188758248682,2007,1 +2007,26085,2.42771882375081,3.36729582998647,2007,1 +2003,26089,3.05017310923224,5.64190707093811,2007,1 +2004,26089,3.05017310923224,5.52942908751142,2007,1 +2005,26089,3.05017310923224,5.47227067367148,2007,1 +2006,26089,3.05017310923224,5.28320372873799,2007,1 +2007,26089,3.05017310923224,5.34233425196481,2007,1 +2003,26105,3.34194265444484,5.75890177387728,2007,1 +2004,26105,3.34194265444484,5.59842195899838,2007,1 +2005,26105,3.34194265444484,5.75574221358691,2007,1 +2006,26105,3.34194265444484,5.61312810638807,2007,1 +2007,26105,3.34194265444484,5.52942908751142,2007,1 +2003,26127,3.29112206507397,5.52942908751142,2007,1 +2004,26127,3.29112206507397,5.51745289646471,2007,1 +2005,26127,3.29112206507397,5.54907608489522,2007,1 +2006,26127,3.29112206507397,5.39362754635236,2007,1 +2007,26127,3.29112206507397,5.37527840768417,2007,1 +2003,26129,3.07477448089394,5.74300318780948,2007,1 +2004,26129,3.07477448089394,5.75574221358691,2007,1 +2005,26129,3.07477448089394,5.75890177387728,2007,1 +2006,26129,3.07477448089394,5.5683445037611,2007,1 +2007,26129,3.07477448089394,5.52942908751142,2007,1 +2003,26137,3.14849627810562,6.2363695902037,2007,1 +2004,26137,3.14849627810562,6.22851100359118,2007,1 +2005,26137,3.14849627810562,6.18620862390049,2007,1 +2006,26137,3.14849627810562,6.10479323241498,2007,1 +2007,26137,3.14849627810562,5.94542060860658,2007,1 +2003,26149,4.13391777734183,6.62273632394984,2007,1 +2004,26149,4.13391777734183,6.45204895443723,2007,1 +2005,26149,4.13391777734183,6.50727771238501,2007,1 +2006,26149,4.13391777734183,6.41836493593621,2007,1 +2007,26149,4.13391777734183,6.36990098282823,2007,1 +2003,27005,3.40119738166216,6.84374994900622,2006,1 +2004,27005,3.40119738166216,6.76157276880406,2006,1 +2005,27005,3.40119738166216,6.78671695060508,2006,1 +2006,27005,3.40119738166216,6.96790920180188,2006,1 +2007,27005,3.40119738166216,6.91075078796194,2006,1 +2003,27009,3.53298558915526,6.52356230614951,2006,1 +2004,27009,3.53298558915526,6.45362499889269,2006,1 +2005,27009,3.53298558915526,6.47697236288968,2006,1 +2006,27009,3.53298558915526,6.51767127291227,2006,1 +2007,27009,3.53298558915526,6.41673228251233,2006,1 +2003,27023,2.57169577986039,5.79301360838414,2006,1 +2004,27023,2.57169577986039,5.65599181081985,2006,1 +2005,27023,2.57169577986039,5.79909265446053,2006,1 +2006,27023,2.57169577986039,5.6970934865054,2006,1 +2007,27023,2.57169577986039,5.72358510195238,2006,1 +2003,27035,4.00913156717385,7.4899708988348,2006,1 +2004,27035,4.00913156717385,7.22983877815125,2006,1 +2005,27035,4.00913156717385,7.30114780585603,2006,1 +2006,27035,4.00913156717385,7.39510754656249,2006,1 +2007,27035,4.00913156717385,7.22329567956231,2006,1 +2003,27043,2.78383771441767,5.57594910314632,2006,1 +2004,27043,2.78383771441767,5.39362754635236,2006,1 +2005,27043,2.78383771441767,5.54907608489522,2006,1 +2006,27043,2.78383771441767,5.46383180502561,2006,1 +2007,27043,2.78383771441767,5.41164605185504,2006,1 +2003,27049,3.78707184000532,7.2991214627108,2006,1 +2004,27049,3.78707184000532,7.03174125876313,2006,1 +2005,27049,3.78707184000532,6.93731408122368,2006,1 +2006,27049,3.78707184000532,6.96979066990159,2006,1 +2007,27049,3.78707184000532,6.95844839329766,2006,1 +2003,27051,1.83880207556137,4.54329478227,2006,1 +2004,27051,1.83880207556137,4.49980967033027,2006,1 +2005,27051,1.83880207556137,4.33073334028633,2006,1 +2006,27051,1.83880207556137,4.44265125649032,2006,1 +2007,27051,1.83880207556137,4.06044301054642,2006,1 +2003,27055,2.98153192415285,5.77765232322266,2006,1 +2004,27055,2.98153192415285,5.74300318780948,2006,1 +2005,27055,2.98153192415285,5.73979291217923,2006,1 +2006,27055,2.98153192415285,5.71373280550937,2006,1 +2007,27055,2.98153192415285,5.66296048013595,2006,1 +2003,27091,3.08200170870617,6.30991827822652,2006,1 +2004,27091,3.08200170870617,6.28785856016178,2006,1 +2005,27091,3.08200170870617,6.3578422665081,2006,1 +2006,27091,3.08200170870617,6.40522845803084,2006,1 +2007,27091,3.08200170870617,6.3456363608286,2006,1 +2003,27115,3.27827616814965,6.12686918411419,2006,1 +2004,27115,3.27827616814965,6.12468339089421,2006,1 +2005,27115,3.27827616814965,6.09582456243222,2006,1 +2006,27115,3.27827616814965,5.96614673912369,2006,1 +2007,27115,3.27827616814965,6.00388706710654,2006,1 +2003,27123,6.23643808100884,9.2617936535651,2006,1 +2004,27123,6.23643808100884,9.18450961194166,2006,1 +2005,27123,6.23643808100884,9.13841463240459,2006,1 +2006,27123,6.23643808100884,9.19146331620649,2006,1 +2007,27123,6.23643808100884,9.11294802596753,2006,1 +2003,27129,2.84223138256823,5.77455154554441,2006,1 +2004,27129,2.84223138256823,5.72358510195238,2006,1 +2005,27129,2.84223138256823,5.75257263882563,2006,1 +2006,27129,2.84223138256823,5.64190707093811,2006,1 +2007,27129,2.84223138256823,5.87773578177964,2006,1 +2003,27149,2.30787109742328,5.35185813347607,2006,1 +2004,27149,2.30787109742328,5.14166355650266,2006,1 +2005,27149,2.30787109742328,5.2040066870768,2006,1 +2006,27149,2.30787109742328,5.17614973257383,2006,1 +2007,27149,2.30787109742328,5.35185813347607,2006,1 +2003,27157,3.07315617051879,6.04737217904628,2006,1 +2004,27157,3.07315617051879,5.95324333428778,2006,1 +2005,27157,3.07315617051879,5.89164421182577,2006,1 +2006,27157,3.07315617051879,5.92157841964382,2006,1 +2007,27157,3.07315617051879,5.91620206260743,2006,1 +2003,27163,5.30395146516513,8.27231514795602,2006,1 +2004,27163,5.30395146516513,8.21419441485256,2006,1 +2005,27163,5.30395146516513,8.17075142375753,2006,1 +2006,27163,5.30395146516513,8.25035895147729,2006,1 +2007,27163,5.30395146516513,8.20303024171486,2006,1 +2003,27165,2.47451955690151,5.48479693349065,2006,1 +2004,27165,2.47451955690151,5.40267738187228,2006,1 +2005,27165,2.47451955690151,5.30330490805908,2006,1 +2006,27165,2.47451955690151,5.41610040220442,2006,1 +2007,27165,2.47451955690151,5.38449506278909,2006,1 +2003,29003,2.80287541484465,4.83628190695148,2007,1 +2004,29003,2.80287541484465,4.85981240436167,2007,1 +2005,29003,2.80287541484465,4.87519732320115,2007,1 +2006,29003,2.80287541484465,4.88280192258637,2007,1 +2007,29003,2.80287541484465,5.01063529409626,2007,1 +2003,29011,2.52900327684191,5.51342874616498,2007,1 +2004,29011,2.52900327684191,5.51745289646471,2007,1 +2005,29011,2.52900327684191,5.40267738187228,2007,1 +2006,29011,2.52900327684191,5.30330490805908,2007,1 +2007,29011,2.52900327684191,5.25749537202778,2007,1 +2003,29013,2.81259038037613,5.36129216570943,2007,1 +2004,29013,2.81259038037613,5.25227342804663,2007,1 +2005,29013,2.81259038037613,5.32787616878958,2007,1 +2006,29013,2.81259038037613,5.18738580584075,2007,1 +2007,29013,2.81259038037613,5.12396397940326,2007,1 +2003,29023,3.71032289140759,6.40025744530882,2007,1 +2004,29023,3.71032289140759,6.51174532964473,2007,1 +2005,29023,3.71032289140759,6.53813982376767,2007,1 +2006,29023,3.71032289140759,6.41345895716736,2007,1 +2007,29023,3.71032289140759,6.29341927884648,2007,1 +2003,29033,2.33068652310492,4.82028156560504,2007,1 +2004,29033,2.33068652310492,4.80402104473326,2007,1 +2005,29033,2.33068652310492,4.65396035015752,2007,1 +2006,29033,2.33068652310492,4.69134788222914,2007,1 +2007,29033,2.33068652310492,4.8283137373023,2007,1 +2003,29041,2.13274531370215,4.71849887129509,2007,1 +2004,29041,2.13274531370215,4.65396035015752,2007,1 +2005,29041,2.13274531370215,4.64439089914137,2007,1 +2006,29041,2.13274531370215,4.36944785246702,2007,1 +2007,29041,2.13274531370215,4.4188406077966,2007,1 +2003,29065,2.7031716536511,5.07517381523383,2007,1 +2004,29065,2.7031716536511,5.13579843705026,2007,1 +2005,29065,2.7031716536511,5.11799381241676,2007,1 +2006,29065,2.7031716536511,5.09375020080676,2007,1 +2007,29065,2.7031716536511,5.02388052084628,2007,1 +2003,29073,2.7305941655371,5.8111409929767,2007,1 +2004,29073,2.7305941655371,5.76205138278018,2007,1 +2005,29073,2.7305941655371,5.87211778947542,2007,1 +2006,29073,2.7305941655371,5.94542060860658,2007,1 +2007,29073,2.7305941655371,5.8805329864007,2007,1 +2003,29077,5.48226676435625,8.44827174594982,2007,1 +2004,29077,5.48226676435625,8.39117635083275,2007,1 +2005,29077,5.48226676435625,8.39976009452414,2007,1 +2006,29077,5.48226676435625,8.46779284112111,2007,1 +2007,29077,5.48226676435625,8.44891435066294,2007,1 +2003,29079,2.3448780051843,4.94875989037817,2007,1 +2004,29079,2.3448780051843,4.84418708645859,2007,1 +2005,29079,2.3448780051843,4.969813299576,2007,1 +2006,29079,2.3448780051843,4.8283137373023,2007,1 +2007,29079,2.3448780051843,4.8283137373023,2007,1 +2003,29081,2.18041745901984,5.11198778835654,2007,1 +2004,29081,2.18041745901984,5.06259503302697,2007,1 +2005,29081,2.18041745901984,5.17048399503815,2007,1 +2006,29081,2.18041745901984,4.94875989037817,2007,1 +2007,29081,2.18041745901984,4.90527477843843,2007,1 +2003,29085,2.19053558918542,3.46573590279973,2007,1 +2004,29085,2.19053558918542,3.25809653802148,2007,1 +2005,29085,2.19053558918542,3.3322045101752,2007,1 +2006,29085,2.19053558918542,3.40119738166216,2007,1 +2007,29085,2.19053558918542,3.49650756146648,2007,1 +2003,29089,2.32356349937924,4.70048036579242,2007,1 +2004,29089,2.32356349937924,4.7361984483945,2007,1 +2005,29089,2.32356349937924,4.70953020131233,2007,1 +2006,29089,2.32356349937924,4.69134788222914,2007,1 +2007,29089,2.32356349937924,4.62497281328427,2007,1 +2003,29099,5.28876690573619,7.80628928926703,2007,1 +2004,29099,5.28876690573619,7.78572089653462,2007,1 +2005,29099,5.28876690573619,7.74283595543075,2007,1 +2006,29099,5.28876690573619,7.73848812249465,2007,1 +2007,29099,5.28876690573619,7.7367436824535,2007,1 +2003,29101,3.87656161714999,6.07993319509559,2007,1 +2004,29101,3.87656161714999,6.14203740558736,2007,1 +2005,29101,3.87656161714999,6.20050917404269,2007,1 +2006,29101,3.87656161714999,6.16120732169508,2007,1 +2007,29101,3.87656161714999,6.18414889093748,2007,1 +2003,29105,3.48164000935702,6.13556489108174,2007,1 +2004,29105,3.48164000935702,6.0591231955818,2007,1 +2005,29105,3.48164000935702,6.1463292576689,2007,1 +2006,29105,3.48164000935702,6.28226674689601,2007,1 +2007,29105,3.48164000935702,6.27476202124194,2007,1 +2003,29107,3.49529470504127,6.1527326947041,2007,1 +2004,29107,3.49529470504127,6.19236248947487,2007,1 +2005,29107,3.49529470504127,6.1463292576689,2007,1 +2006,29107,3.49529470504127,6.09582456243222,2007,1 +2007,29107,3.49529470504127,6.12029741895095,2007,1 +2003,29117,2.67814067069607,5.7268477475872,2007,1 +2004,29117,2.67814067069607,5.75574221358691,2007,1 +2005,29117,2.67814067069607,5.64544689764324,2007,1 +2006,29117,2.67814067069607,5.67675380226828,2007,1 +2007,29117,2.67814067069607,5.54517744447956,2007,1 +2003,29119,3.07643630096895,5.35658627467201,2007,1 +2004,29119,3.07643630096895,5.23110861685459,2007,1 +2005,29119,3.07643630096895,5.16478597392351,2007,1 +2006,29119,3.07643630096895,5.28320372873799,2007,1 +2007,29119,3.07643630096895,5.33753807970132,2007,1 +2003,29127,3.34247303650915,6.39359075395063,2007,1 +2004,29127,3.34247303650915,6.33150184989369,2007,1 +2005,29127,3.34247303650915,6.42324696353352,2007,1 +2006,29127,3.34247303650915,6.39692965521615,2007,1 +2007,29127,3.34247303650915,6.36818718635049,2007,1 +2003,29131,3.15972012365944,5.52942908751142,2007,1 +2004,29131,3.15972012365944,5.34233425196481,2007,1 +2005,29131,3.15972012365944,5.34710753071747,2007,1 +2006,29131,3.15972012365944,5.41610040220442,2007,1 +2007,29131,3.15972012365944,5.38449506278909,2007,1 +2003,29149,2.33640664146956,4.21950770517611,2007,1 +2004,29149,2.33640664146956,4.18965474202643,2007,1 +2005,29149,2.33640664146956,4.35670882668959,2007,1 +2006,29149,2.33640664146956,4.35670882668959,2007,1 +2007,29149,2.33640664146956,4.51085950651685,2007,1 +2003,29155,2.99807951662234,5.4971682252932,2007,1 +2004,29155,2.99807951662234,5.34233425196481,2007,1 +2005,29155,2.99807951662234,5.26269018890489,2007,1 +2006,29155,2.99807951662234,5.45958551414416,2007,1 +2007,29155,2.99807951662234,5.55295958492162,2007,1 +2003,29163,2.90968406892854,5.605802066296,2007,1 +2004,29163,2.90968406892854,5.69035945432406,2007,1 +2005,29163,2.90968406892854,5.72031177660741,2007,1 +2006,29163,2.90968406892854,5.7037824746562,2007,1 +2007,29163,2.90968406892854,5.63835466933375,2007,1 +2003,29177,3.15076827565352,5.48063892334199,2007,1 +2004,29177,3.15076827565352,5.26785815906333,2007,1 +2005,29177,3.15076827565352,5.28320372873799,2007,1 +2006,29177,3.15076827565352,5.18738580584075,2007,1 +2007,29177,3.15076827565352,5.22035582507832,2007,1 +2003,29201,3.69937419120658,6.08904487544685,2007,1 +2004,29201,3.69937419120658,6.19031540585315,2007,1 +2005,29201,3.69937419120658,6.15060276844628,2007,1 +2006,29201,3.69937419120658,6.09807428216624,2007,1 +2007,29201,3.69937419120658,6.00635315960173,2007,1 +2003,29209,3.35543263629441,5.00394630594546,2007,1 +2004,29209,3.35543263629441,5.19849703126583,2007,1 +2005,29209,3.35543263629441,5.19849703126583,2007,1 +2006,29209,3.35543263629441,5.23644196282995,2007,1 +2007,29209,3.35543263629441,5.32300997913841,2007,1 +2003,29211,1.97671643915702,4.26267987704132,2007,1 +2004,29211,1.97671643915702,4.24849524204936,2007,1 +2005,29211,1.97671643915702,3.93182563272433,2007,1 +2006,29211,1.97671643915702,4.02535169073515,2007,1 +2007,29211,1.97671643915702,4.12713438504509,2007,1 +2003,29215,3.13562464220588,5.37989735354046,2007,1 +2004,29215,3.13562464220588,5.29831736654804,2007,1 +2005,29215,3.13562464220588,5.43372200355424,2007,1 +2006,29215,3.13562464220588,5.41610040220442,2007,1 +2007,29215,3.13562464220588,5.40267738187228,2007,1 +2003,29229,2.88786862767805,5.52942908751142,2007,1 +2004,29229,2.88786862767805,5.54517744447956,2007,1 +2005,29229,2.88786862767805,5.51745289646471,2007,1 +2006,29229,2.88786862767805,5.42053499927229,2007,1 +2007,29229,2.88786862767805,5.26269018890489,2007,1 +2003,29510,5.85274543579545,8.40200678160712,2007,1 +2004,29510,5.85274543579545,8.34616759436413,2007,1 +2005,29510,5.85274543579545,8.29704514908183,2007,1 +2006,29510,5.85274543579545,8.29679586577005,2007,1 +2007,29510,5.85274543579545,8.25816336153762,2007,1 +2003,30009,2.25675055665025,5.08140436498446,2007,1 +2004,30009,2.25675055665025,4.92725368515721,2007,1 +2005,30009,2.25675055665025,4.77912349311153,2007,1 +2006,30009,2.25675055665025,4.87519732320115,2007,1 +2007,30009,2.25675055665025,4.97673374242057,2007,1 +2003,30011,0.307484699747961,2.484906649788,2007,1 +2004,30011,0.307484699747961,1.94591014905531,2007,1 +2005,30011,0.307484699747961,1.6094379124341,2007,1 +2006,30011,0.307484699747961,1.94591014905531,2007,1 +2007,30011,0.307484699747961,2.07944154167984,2007,1 +2003,30013,4.3864792072439,7.27793857294566,2007,1 +2004,30013,4.3864792072439,7.16626597413364,2007,1 +2005,30013,4.3864792072439,7.19142933003638,2007,1 +2006,30013,4.3864792072439,7.25981961036319,2007,1 +2007,30013,4.3864792072439,7.28138566357028,2007,1 +2003,30021,2.20375873868715,5.37063802812766,2007,1 +2004,30021,2.20375873868715,5.30826769740121,2007,1 +2005,30021,2.20375873868715,5.16478597392351,2007,1 +2006,30021,2.20375873868715,5.44241771052179,2007,1 +2007,30021,2.20375873868715,5.25227342804663,2007,1 +2003,30033,0.246078522596706,2.39789527279837,2007,1 +2004,30033,0.246078522596706,2.89037175789616,2007,1 +2005,30033,0.246078522596706,2.484906649788,2007,1 +2006,30033,0.246078522596706,2.83321334405622,2007,1 +2007,30033,0.246078522596706,2.89037175789616,2007,1 +2003,30049,4.02026735882578,7.12929754892937,2007,1 +2004,30049,4.02026735882578,7.0604763659998,2007,1 +2005,30049,4.02026735882578,7.00397413672268,2007,1 +2006,30049,4.02026735882578,7.09506437728713,2007,1 +2007,30049,4.02026735882578,7.09589322109753,2007,1 +2003,30081,3.58546149491545,6.21260609575152,2007,1 +2004,30081,3.58546149491545,6.21860011969173,2007,1 +2005,30081,3.58546149491545,6.18208490671663,2007,1 +2006,30081,3.58546149491545,6.26909628370626,2007,1 +2007,30081,3.58546149491545,6.27852142416584,2007,1 +2003,30087,2.23889954130791,5.35185813347607,2007,1 +2004,30087,2.23889954130791,5.32300997913841,2007,1 +2005,30087,2.23889954130791,5.11198778835654,2007,1 +2006,30087,2.23889954130791,5.0998664278242,2007,1 +2007,30087,2.23889954130791,4.969813299576,2007,1 +2003,30095,2.10352421219579,4.56434819146784,2007,1 +2004,30095,2.10352421219579,4.38202663467388,2007,1 +2005,30095,2.10352421219579,4.65396035015752,2007,1 +2006,30095,2.10352421219579,4.65396035015752,2007,1 +2007,30095,2.10352421219579,4.63472898822964,2007,1 +2003,30101,1.66146094050125,4.24849524204936,2007,1 +2004,30101,1.66146094050125,4.00733318523247,2007,1 +2005,30101,1.66146094050125,4.26267987704132,2007,1 +2006,30101,1.66146094050125,4.12713438504509,2007,1 +2007,30101,1.66146094050125,4.29045944114839,2007,1 +2003,30109,0.0657877405380032,2.70805020110221,2007,1 +2004,30109,0.0657877405380032,2.63905732961526,2007,1 +2005,30109,0.0657877405380032,2.19722457733622,2007,1 +2006,30109,0.0657877405380032,2.19722457733622,2007,1 +2007,30109,0.0657877405380032,2.30258509299405,2007,1 +2003,31003,2.00848245273934,4.62497281328427,0,0 +2004,31003,2.00848245273934,4.44265125649032,0,0 +2005,31003,2.00848245273934,4.4188406077966,0,0 +2006,31003,2.00848245273934,4.55387689160054,0,0 +2007,31003,2.00848245273934,4.51085950651685,0,0 +2003,31023,2.17099467260645,4.76217393479776,0,0 +2004,31023,2.17099467260645,4.51085950651685,0,0 +2005,31023,2.17099467260645,4.70953020131233,0,0 +2006,31023,2.17099467260645,4.62497281328427,0,0 +2007,31023,2.17099467260645,4.77068462446567,0,0 +2003,31031,1.81612682567635,4.88280192258637,0,0 +2004,31031,1.81612682567635,5.01063529409626,0,0 +2005,31031,1.81612682567635,5.07517381523383,0,0 +2006,31031,1.81612682567635,4.96284463025991,0,0 +2007,31031,1.81612682567635,4.87519732320115,0,0 +2003,31033,2.28543893415908,5.49306144334055,0,0 +2004,31033,2.28543893415908,5.5683445037611,0,0 +2005,31033,2.28543893415908,5.5834963087817,0,0 +2006,31033,2.28543893415908,5.54126354515843,0,0 +2007,31033,2.28543893415908,5.59842195899838,0,0 +2003,31041,2.46750613510783,5.11198778835654,0,0 +2004,31041,2.46750613510783,5.04342511691925,0,0 +2005,31041,2.46750613510783,5.06890420222023,0,0 +2006,31041,2.46750613510783,5.21493575760899,0,0 +2007,31041,2.46750613510783,5.28826703069454,0,0 +2003,31055,6.13898975546329,9.36546200884348,0,0 +2004,31055,6.13898975546329,9.3321154289021,0,0 +2005,31055,6.13898975546329,9.31118068690225,0,0 +2006,31055,6.13898975546329,9.32197118814737,0,0 +2007,31055,6.13898975546329,9.32027043134838,0,0 +2003,31059,1.89220794052486,4.97673374242057,0,0 +2004,31059,1.89220794052486,4.77912349311153,0,0 +2005,31059,1.89220794052486,4.74493212836325,0,0 +2006,31059,1.89220794052486,4.70048036579242,0,0 +2007,31059,1.89220794052486,4.64439089914137,0,0 +2003,31061,1.27368541673724,3.71357206670431,0,0 +2004,31061,1.27368541673724,3.61091791264422,0,0 +2005,31061,1.27368541673724,3.40119738166216,0,0 +2006,31061,1.27368541673724,3.3322045101752,0,0 +2007,31061,1.27368541673724,3.61091791264422,0,0 +2003,31073,0.76220671649144,2.70805020110221,0,0 +2004,31073,0.76220671649144,2.77258872223978,0,0 +2005,31073,0.76220671649144,2.83321334405622,0,0 +2006,31073,0.76220671649144,2.70805020110221,0,0 +2007,31073,0.76220671649144,2.63905732961526,0,0 +2003,31079,3.98031696606713,7.1708884785125,0,0 +2004,31079,3.98031696606713,7.09920174355309,0,0 +2005,31079,3.98031696606713,7.06646697013696,0,0 +2006,31079,3.98031696606713,7.16549347506085,0,0 +2007,31079,3.98031696606713,7.17548971362422,0,0 +2003,31081,2.24102878729494,5.21493575760899,0,0 +2004,31081,2.24102878729494,5.24174701505964,0,0 +2005,31081,2.24102878729494,5.20948615284142,0,0 +2006,31081,2.24102878729494,5.15329159449778,0,0 +2007,31081,2.24102878729494,5.14749447681345,0,0 +2003,31083,1.33131005278713,3.85014760171006,0,0 +2004,31083,1.33131005278713,3.73766961828337,0,0 +2005,31083,1.33131005278713,3.89182029811063,0,0 +2006,31083,1.33131005278713,3.98898404656427,0,0 +2007,31083,1.33131005278713,3.85014760171006,0,0 +2003,31101,2.18323833536148,5.4249500174814,0,0 +2004,31101,2.18323833536148,5.30826769740121,0,0 +2005,31101,2.18323833536148,5.30330490805908,0,0 +2006,31101,2.18323833536148,5.23110861685459,0,0 +2007,31101,2.18323833536148,5.26269018890489,0,0 +2003,31107,2.23793989950051,5.04985600724954,0,0 +2004,31107,2.23793989950051,5.19295685089021,0,0 +2005,31107,2.23793989950051,5.08140436498446,0,0 +2006,31107,2.23793989950051,5.03695260241363,0,0 +2007,31107,2.23793989950051,5.14166355650266,0,0 +2003,31109,5.52262424093949,8.57828829077605,0,0 +2004,31109,5.52262424093949,8.52694548285892,0,0 +2005,31109,5.52262424093949,8.53168763756669,0,0 +2006,31109,5.52262424093949,8.52038808231276,0,0 +2007,31109,5.52262424093949,8.5571828396324,0,0 +2003,31123,1.69377906086785,4.46590811865458,0,0 +2004,31123,1.69377906086785,4.40671924726425,0,0 +2005,31123,1.69377906086785,4.20469261939097,0,0 +2006,31123,1.69377906086785,4.24849524204936,0,0 +2007,31123,1.69377906086785,4.04305126783455,0,0 +2003,31127,2.02498535588378,4.92725368515721,0,0 +2004,31127,2.02498535588378,4.95582705760126,0,0 +2005,31127,2.02498535588378,4.71849887129509,0,0 +2006,31127,2.02498535588378,4.64439089914137,0,0 +2007,31127,2.02498535588378,4.67282883446191,0,0 +2003,31131,2.7341077354215,5.94542060860658,0,0 +2004,31131,2.7341077354215,5.70711026474888,0,0 +2005,31131,2.7341077354215,5.69035945432406,0,0 +2006,31131,2.7341077354215,5.74939298590825,0,0 +2007,31131,2.7341077354215,5.80814248998044,0,0 +2003,31133,1.12719974552002,3.13549421592915,0,0 +2004,31133,1.12719974552002,3.09104245335832,0,0 +2005,31133,1.12719974552002,3.13549421592915,0,0 +2006,31133,1.12719974552002,3.49650756146648,0,0 +2007,31133,1.12719974552002,3.55534806148941,0,0 +2003,31147,2.25454964395549,4.83628190695148,0,0 +2004,31147,2.25454964395549,4.92725368515721,0,0 +2005,31147,2.25454964395549,4.93447393313069,0,0 +2006,31147,2.25454964395549,4.72738781871234,0,0 +2007,31147,2.25454964395549,4.74493212836325,0,0 +2003,31151,2.6277796897056,5.66296048013595,0,0 +2004,31151,2.6277796897056,5.44241771052179,0,0 +2005,31151,2.6277796897056,5.50125821054473,0,0 +2006,31151,2.6277796897056,5.4971682252932,0,0 +2007,31151,2.6277796897056,5.40267738187228,0,0 +2003,31153,4.8088862396362,7.53315880745556,0,0 +2004,31153,4.8088862396362,7.49665243816828,0,0 +2005,31153,4.8088862396362,7.53529670244409,0,0 +2006,31153,4.8088862396362,7.53743003658651,0,0 +2007,31153,4.8088862396362,7.61283103040736,0,0 +2003,31155,2.9871959425317,5.60947179518496,0,0 +2004,31155,2.9871959425317,5.6021188208797,0,0 +2005,31155,2.9871959425317,5.46806014113513,0,0 +2006,31155,2.9871959425317,5.50533153593236,0,0 +2007,31155,2.9871959425317,5.63835466933375,0,0 +2003,31161,1.82422665936556,4.45434729625351,0,0 +2004,31161,1.82422665936556,4.31748811353631,0,0 +2005,31161,1.82422665936556,4.26267987704132,0,0 +2006,31161,1.82422665936556,4.27666611901606,0,0 +2007,31161,1.82422665936556,4.11087386417331,0,0 +2003,31177,2.93279247378012,5.84932477994686,0,0 +2004,31177,2.93279247378012,5.83773044716594,0,0 +2005,31177,2.93279247378012,5.73009978297357,0,0 +2006,31177,2.93279247378012,5.67332326717149,0,0 +2007,31177,2.93279247378012,5.59842195899838,0,0 +2003,32007,3.81310833725371,6.51323011091231,2007,1 +2004,32007,3.81310833725371,6.67203294546107,2007,1 +2005,32007,3.81310833725371,6.75110146893676,2007,1 +2006,32007,3.81310833725371,6.81563999007433,2007,1 +2007,32007,3.81310833725371,6.88653164253051,2007,1 +2003,32011,0.501381164937991,2.56494935746154,2007,1 +2004,32011,0.501381164937991,3.13549421592915,2007,1 +2005,32011,0.501381164937991,2.83321334405622,2007,1 +2006,32011,0.501381164937991,3.25809653802148,2007,1 +2007,32011,0.501381164937991,3.3322045101752,2007,1 +2003,32033,2.21713613116156,4.98360662170834,2007,1 +2004,32033,2.21713613116156,4.91265488573605,2007,1 +2005,32033,2.21713613116156,5.07517381523383,2007,1 +2006,32033,2.21713613116156,5.18738580584075,2007,1 +2007,32033,2.21713613116156,5.18178355029209,2007,1 +2003,35006,3.24239701990955,5.44241771052179,0,0 +2004,35006,3.24239701990955,5.13579843705026,0,0 +2005,35006,3.24239701990955,5.26269018890489,0,0 +2006,35006,3.24239701990955,5.47227067367148,0,0 +2007,35006,3.24239701990955,5.44241771052179,0,0 +2003,35013,5.16296717805949,7.57865685059476,0,0 +2004,35013,5.16296717805949,7.56060116276856,0,0 +2005,35013,5.16296717805949,7.62851762657506,0,0 +2006,35013,5.16296717805949,7.59689443814454,0,0 +2007,35013,5.16296717805949,7.65681009148038,0,0 +2003,35025,4.01658119931192,6.71901315438526,0,0 +2004,35025,4.01658119931192,6.70563909486,0,0 +2005,35025,4.01658119931192,6.79122146272619,0,0 +2006,35025,4.01658119931192,6.89162589705225,0,0 +2007,35025,4.01658119931192,7.03262426102801,0,0 +2003,35033,1.64480505627139,3.04452243772342,0,0 +2004,35033,1.64480505627139,2.19722457733622,0,0 +2005,35033,1.64480505627139,2.484906649788,0,0 +2006,35033,1.64480505627139,3.04452243772342,0,0 +2007,35033,1.64480505627139,2.83321334405622,0,0 +2003,35045,4.73445130899984,7.44658509915773,0,0 +2004,35045,4.73445130899984,7.37713371283395,0,0 +2005,35045,4.73445130899984,7.35308192051543,0,0 +2006,35045,4.73445130899984,7.36770857237437,0,0 +2007,35045,4.73445130899984,7.44775128004791,0,0 +2003,37005,2.36809189519617,4.62497281328427,2007,1 +2004,37005,2.36809189519617,4.58496747867057,2007,1 +2005,37005,2.36809189519617,4.67282883446191,2007,1 +2006,37005,2.36809189519617,4.63472898822964,2007,1 +2007,37005,2.36809189519617,4.62497281328427,2007,1 +2003,37021,5.32947682879714,8.0487882835342,2007,1 +2004,37021,5.32947682879714,8.06589354696427,2007,1 +2005,37021,5.32947682879714,8.14206328310415,2007,1 +2006,37021,5.32947682879714,8.18395571730495,2007,1 +2007,37021,5.32947682879714,8.24800570160062,2007,1 +2003,37025,4.87567812362863,7.6231530684769,2007,1 +2004,37025,4.87567812362863,7.68570306123455,2007,1 +2005,37025,4.87567812362863,7.70571282389443,2007,1 +2006,37025,4.87567812362863,7.8087293067444,2007,1 +2007,37025,4.87567812362863,7.900636613018,2007,1 +2003,37029,1.92934513218062,3.29583686600433,2007,1 +2004,37029,1.92934513218062,2.94443897916644,2007,1 +2005,37029,1.92934513218062,3.46573590279973,2007,1 +2006,37029,1.92934513218062,3.36729582998647,2007,1 +2007,37029,1.92934513218062,3.25809653802148,2007,1 +2003,37041,2.67594014718398,5.06890420222023,2007,1 +2004,37041,2.67594014718398,5.18738580584075,2007,1 +2005,37041,2.67594014718398,4.93447393313069,2007,1 +2006,37041,2.67594014718398,5.08140436498446,2007,1 +2007,37041,2.67594014718398,5.08140436498446,2007,1 +2003,37051,5.71361068584184,8.03624994213212,2007,1 +2004,37051,5.71361068584184,8.04269949689764,2007,1 +2005,37051,5.71361068584184,8.08948247436075,2007,1 +2006,37051,5.71361068584184,8.12622252945853,2007,1 +2007,37051,5.71361068584184,8.13973227971767,2007,1 +2003,37057,4.99210465747682,7.2841348061952,2007,1 +2004,37057,4.99210465747682,7.2283884515736,2007,1 +2005,37057,4.99210465747682,7.22983877815125,2007,1 +2006,37057,4.99210465747682,7.24208235925696,2007,1 +2007,37057,4.99210465747682,7.24064969425547,2007,1 +2003,37059,3.55062262848208,5.95842469302978,2007,1 +2004,37059,3.55062262848208,5.9427993751267,2007,1 +2005,37059,3.55062262848208,6.13988455222626,2007,1 +2006,37059,3.55062262848208,6.13556489108174,2007,1 +2007,37059,3.55062262848208,6.10924758276437,2007,1 +2003,37063,5.40857885280557,7.96797317966293,2007,1 +2004,37063,5.40857885280557,7.96485088744731,2007,1 +2005,37063,5.40857885280557,7.95296679092313,2007,1 +2006,37063,5.40857885280557,8.02059914989697,2007,1 +2007,37063,5.40857885280557,8.02845516411425,2007,1 +2003,37075,2.07856615864388,4.02535169073515,2007,1 +2004,37075,2.07856615864388,4.24849524204936,2007,1 +2005,37075,2.07856615864388,4.17438726989564,2007,1 +2006,37075,2.07856615864388,4.47733681447821,2007,1 +2007,37075,2.07856615864388,4.75359019110636,2007,1 +2003,37079,2.94306962097069,4.61512051684126,2007,1 +2004,37079,2.94306962097069,4.70953020131233,2007,1 +2005,37079,2.94306962097069,4.51085950651685,2007,1 +2006,37079,2.94306962097069,4.51085950651685,2007,1 +2007,37079,2.94306962097069,4.51085950651685,2007,1 +2003,37085,4.5111341940615,6.84374994900622,2007,1 +2004,37085,4.5111341940615,6.85751406254539,2007,1 +2005,37085,4.5111341940615,6.93731408122368,2007,1 +2006,37085,4.5111341940615,7.0352685992811,2007,1 +2007,37085,4.5111341940615,6.99301512293296,2007,1 +2003,37093,3.51589417829697,5.23110861685459,2007,1 +2004,37093,3.51589417829697,5.14166355650266,2007,1 +2005,37093,3.51589417829697,5.28826703069454,2007,1 +2006,37093,3.51589417829697,5.25227342804663,2007,1 +2007,37093,3.51589417829697,5.30826769740121,2007,1 +2003,37099,3.50016752229797,5.79301360838414,2007,1 +2004,37099,3.50016752229797,5.75574221358691,2007,1 +2005,37099,3.50016752229797,5.82600010738045,2007,1 +2006,37099,3.50016752229797,5.84932477994686,2007,1 +2007,37099,3.50016752229797,5.97380961186926,2007,1 +2003,37103,2.33997721221111,3.8286413964891,2007,1 +2004,37103,2.33997721221111,3.66356164612965,2007,1 +2005,37103,2.33997721221111,3.76120011569356,2007,1 +2006,37103,2.33997721221111,4.11087386417331,2007,1 +2007,37103,2.33997721221111,3.13549421592915,2007,1 +2003,37105,3.89263629162796,6.59850902861452,2007,1 +2004,37105,3.89263629162796,6.58617165485467,2007,1 +2005,37105,3.89263629162796,6.67582322163485,2007,1 +2006,37105,3.89263629162796,6.59304453414244,2007,1 +2007,37105,3.89263629162796,6.65027904858742,2007,1 +2003,37117,3.24231887659467,5.35658627467201,2007,1 +2004,37117,3.24231887659467,5.40267738187228,2007,1 +2005,37117,3.24231887659467,5.48893772615669,2007,1 +2006,37117,3.24231887659467,5.44241771052179,2007,1 +2007,37117,3.24231887659467,5.41164605185504,2007,1 +2003,37123,3.28922244628875,5.53338948872752,2007,1 +2004,37123,3.28922244628875,5.4380793089232,2007,1 +2005,37123,3.28922244628875,5.55682806169954,2007,1 +2006,37123,3.28922244628875,5.605802066296,2007,1 +2007,37123,3.28922244628875,5.57972982598622,2007,1 +2003,37127,4.47072408943517,7.1708884785125,2007,1 +2004,37127,4.47072408943517,7.11558212618445,2007,1 +2005,37127,4.47072408943517,7.11639414409346,2007,1 +2006,37127,4.47072408943517,7.22548147278229,2007,1 +2007,37127,4.47072408943517,7.35627987655075,2007,1 +2003,37155,4.81493666186062,6.85751406254539,2007,1 +2004,37155,4.81493666186062,6.85329909318608,2007,1 +2005,37155,4.81493666186062,7.05444965813294,2007,1 +2006,37155,4.81493666186062,6.99393297522319,2007,1 +2007,37155,4.81493666186062,7.03790596344718,2007,1 +2003,37161,4.14153026532882,6.30078579466324,2007,1 +2004,37161,4.14153026532882,6.24610676548156,2007,1 +2005,37161,4.14153026532882,6.25575004175337,2007,1 +2006,37161,4.14153026532882,6.29710931993394,2007,1 +2007,37161,4.14153026532882,6.4281052726846,2007,1 +2003,37163,4.09702430184386,6.31716468674728,2007,1 +2004,37163,4.09702430184386,6.38519439899773,2007,1 +2005,37163,4.09702430184386,6.32435896238131,2007,1 +2006,37163,4.09702430184386,6.34212141872115,2007,1 +2007,37163,4.09702430184386,6.44730586254121,2007,1 +2003,37165,3.58346338135729,6.15485809401642,2007,1 +2004,37165,3.58346338135729,6.20657592672493,2007,1 +2005,37165,3.58346338135729,6.14846829591765,2007,1 +2006,37165,3.58346338135729,6.18208490671663,2007,1 +2007,37165,3.58346338135729,6.16331480403464,2007,1 +2003,37167,4.06216566385787,6.62539236800796,2007,1 +2004,37167,4.06216566385787,6.54391184556479,2007,1 +2005,37167,4.06216566385787,6.61338421837956,2007,1 +2006,37167,4.06216566385787,6.62936325343745,2007,1 +2007,37167,4.06216566385787,6.61740297797448,2007,1 +2003,37175,3.37874725282456,5.67675380226828,2007,1 +2004,37175,3.37874725282456,5.70711026474888,2007,1 +2005,37175,3.37874725282456,5.75890177387728,2007,1 +2006,37175,3.37874725282456,5.78074351579233,2007,1 +2007,37175,3.37874725282456,5.71373280550937,2007,1 +2003,37189,3.75408181735957,6.3261494731551,2007,1 +2004,37189,3.75408181735957,6.35610766069589,2007,1 +2005,37189,3.75408181735957,6.35088571671474,2007,1 +2006,37189,3.75408181735957,6.44413125670044,2007,1 +2007,37189,3.75408181735957,6.43294009273918,2007,1 +2003,37191,4.73029509291699,7.24708058458576,2007,1 +2004,37191,4.73029509291699,7.2283884515736,2007,1 +2005,37191,4.73029509291699,7.21744343169653,2007,1 +2006,37191,4.73029509291699,7.30988148582479,2007,1 +2007,37191,4.73029509291699,7.30451594646016,2007,1 +2003,38001,0.952815506556527,4.11087386417331,0,0 +2004,38001,0.952815506556527,4.0943445622221,0,0 +2005,38001,0.952815506556527,4.06044301054642,0,0 +2006,38001,0.952815506556527,3.89182029811063,0,0 +2007,38001,0.952815506556527,4.17438726989564,0,0 +2003,38021,1.75041650569367,4.68213122712422,0,0 +2004,38021,1.75041650569367,4.29045944114839,0,0 +2005,38021,1.75041650569367,4.51085950651685,0,0 +2006,38021,1.75041650569367,4.74493212836325,0,0 +2007,38021,1.75041650569367,4.62497281328427,0,0 +2003,38023,0.825490367547658,4.11087386417331,0,0 +2004,38023,0.825490367547658,4.11087386417331,0,0 +2005,38023,0.825490367547658,3.87120101090789,0,0 +2006,38023,0.825490367547658,3.89182029811063,0,0 +2007,38023,0.825490367547658,3.85014760171006,0,0 +2003,38035,4.19130489492644,7.28138566357028,0,0 +2004,38035,4.19130489492644,7.20637729147225,0,0 +2005,38035,4.19130489492644,7.29369772060144,0,0 +2006,38035,4.19130489492644,7.28207365809346,0,0 +2007,38035,4.19130489492644,7.28207365809346,0,0 +2003,38037,1.04415610287205,3.46573590279973,0,0 +2004,38037,1.04415610287205,3.2188758248682,0,0 +2005,38037,1.04415610287205,3.13549421592915,0,0 +2006,38037,1.04415610287205,3.29583686600433,0,0 +2007,38037,1.04415610287205,3.46573590279973,0,0 +2003,38049,1.78959045194322,3.98898404656427,0,0 +2004,38049,1.78959045194322,3.63758615972639,0,0 +2005,38049,1.78959045194322,3.89182029811063,0,0 +2006,38049,1.78959045194322,3.8286413964891,0,0 +2007,38049,1.78959045194322,4.02535169073515,0,0 +2003,38061,1.89175562238673,4.9416424226093,0,0 +2004,38061,1.89175562238673,4.70953020131233,0,0 +2005,38061,1.89175562238673,4.64439089914137,0,0 +2006,38061,1.89175562238673,4.59511985013459,0,0 +2007,38061,1.89175562238673,4.7361984483945,0,0 +2003,38063,1.31237867816972,3.91202300542815,0,0 +2004,38063,1.31237867816972,4.04305126783455,0,0 +2005,38063,1.31237867816972,4.04305126783455,0,0 +2006,38063,1.31237867816972,3.80666248977032,0,0 +2007,38063,1.31237867816972,3.71357206670431,0,0 +2003,38073,1.7732559976635,4.83628190695148,0,0 +2004,38073,1.7732559976635,4.87519732320115,0,0 +2005,38073,1.7732559976635,4.75359019110636,0,0 +2006,38073,1.7732559976635,4.76217393479776,0,0 +2007,38073,1.7732559976635,4.79579054559674,0,0 +2003,38097,2.13735661362627,5.16478597392351,0,0 +2004,38097,2.13735661362627,5.14166355650266,0,0 +2005,38097,2.13735661362627,5.01727983681492,0,0 +2006,38097,2.13735661362627,5.08759633523238,0,0 +2007,38097,2.13735661362627,5.05624580534831,0,0 +2003,38099,2.5168089821336,5.84643877505772,0,0 +2004,38099,2.5168089821336,5.77144112313002,0,0 +2005,38099,2.5168089821336,5.6970934865054,0,0 +2006,38099,2.5168089821336,5.69035945432406,0,0 +2007,38099,2.5168089821336,5.67332326717149,0,0 +2003,38105,2.98371029832667,6.16751649088834,0,0 +2004,38105,2.98371029832667,6.07073772800249,0,0 +2005,38105,2.98371029832667,6.12468339089421,0,0 +2006,38105,2.98371029832667,6.16961073249146,0,0 +2007,38105,2.98371029832667,6.18826412308259,0,0 +2003,39019,3.36162460654233,5.46383180502561,2007,1 +2004,39019,3.36162460654233,5.44241771052179,2007,1 +2005,39019,3.36162460654233,5.40267738187228,2007,1 +2006,39019,3.36162460654233,5.27811465923052,2007,1 +2007,39019,3.36162460654233,5.35185813347607,2007,1 +2003,39021,3.66073714816766,5.89989735358249,2007,1 +2004,39021,3.66073714816766,5.77455154554441,2007,1 +2005,39021,3.66073714816766,5.80211837537706,2007,1 +2006,39021,3.66073714816766,5.8636311755981,2007,1 +2007,39021,3.66073714816766,5.78382518232974,2007,1 +2003,39035,7.23991680927341,10.0020198086004,2007,1 +2004,39035,7.23991680927341,9.94755228369823,2007,1 +2005,39035,7.23991680927341,9.91467474192669,2007,1 +2006,39035,7.23991680927341,9.91595945403145,2007,1 +2007,39035,7.23991680927341,9.89374020108125,2007,1 +2003,39047,3.34755044243138,6.34212141872115,2007,1 +2004,39047,3.34755044243138,6.3297209055227,2007,1 +2005,39047,3.34755044243138,6.33859407820318,2007,1 +2006,39047,3.34755044243138,6.23244801655052,2007,1 +2007,39047,3.34755044243138,6.24416690066374,2007,1 +2003,39055,4.50970499417002,7.29437729928882,2007,1 +2004,39055,4.50970499417002,7.24208235925696,2007,1 +2005,39055,4.50970499417002,7.22037383672395,2007,1 +2006,39055,4.50970499417002,7.30921236569276,2007,1 +2007,39055,4.50970499417002,7.24279792279376,2007,1 +2003,39063,4.26682619873408,7.15461535691366,2007,1 +2004,39063,4.26682619873408,7.06219163228656,2007,1 +2005,39063,4.26682619873408,7.12929754892937,2007,1 +2006,39063,4.26682619873408,7.11395610956603,2007,1 +2007,39063,4.26682619873408,7.14677217945264,2007,1 +2003,39071,3.71051862921742,6.11368217983223,2007,1 +2004,39071,3.71051862921742,6.14418563412565,2007,1 +2005,39071,3.71051862921742,6.21060007702465,2007,1 +2006,39071,3.71051862921742,6.19236248947487,2007,1 +2007,39071,3.71051862921742,6.10702288774225,2007,1 +2003,39079,3.4855691668815,5.8348107370626,2007,1 +2004,39079,3.4855691668815,5.75574221358691,2007,1 +2005,39079,3.4855691668815,5.86929691313377,2007,1 +2006,39079,3.4855691668815,5.82600010738045,2007,1 +2007,39079,3.4855691668815,5.78996017089725,2007,1 +2003,39095,6.12041609322756,9.01054706927019,2007,1 +2004,39095,6.12041609322756,8.92718138825199,2007,1 +2005,39095,6.12041609322756,8.84202652949881,2007,1 +2006,39095,6.12041609322756,8.84029066908897,2007,1 +2007,39095,6.12041609322756,8.82173238093444,2007,1 +2003,39097,3.69419032643251,5.97635090929793,2007,1 +2004,39097,3.69419032643251,5.94017125272043,2007,1 +2005,39097,3.69419032643251,6.13339804299665,2007,1 +2006,39097,3.69419032643251,5.95064255258773,2007,1 +2007,39097,3.69419032643251,5.85507192220243,2007,1 +2003,39107,3.71171668801669,6.77650699237218,2007,1 +2004,39107,3.71171668801669,6.7696419768525,2007,1 +2005,39107,3.71171668801669,6.73459165997295,2007,1 +2006,39107,3.71171668801669,6.80572255341699,2007,1 +2007,39107,3.71171668801669,6.71901315438526,2007,1 +2003,39111,2.71997877196748,3.3322045101752,2007,1 +2004,39111,2.71997877196748,3.25809653802148,2007,1 +2005,39111,2.71997877196748,3.52636052461616,2007,1 +2006,39111,2.71997877196748,3.49650756146648,2007,1 +2007,39111,2.71997877196748,3.40119738166216,2007,1 +2003,39119,4.43775694592726,7.12044437239249,2007,1 +2004,39119,4.43775694592726,7.02553831463852,2007,1 +2005,39119,4.43775694592726,6.94601399109923,2007,1 +2006,39119,4.43775694592726,6.8596149036542,2007,1 +2007,39119,4.43775694592726,6.89568269774787,2007,1 +2003,39143,4.1237739062461,6.83195356556585,2007,1 +2004,39143,4.1237739062461,6.82871207164168,2007,1 +2005,39143,4.1237739062461,6.74523634948436,2007,1 +2006,39143,4.1237739062461,6.73340189183736,2007,1 +2007,39143,4.1237739062461,6.76503897678054,2007,1 +2003,39163,2.54991381109661,4.23410650459726,2007,1 +2004,39163,2.54991381109661,4.24849524204936,2007,1 +2005,39163,2.54991381109661,4.0943445622221,2007,1 +2006,39163,2.54991381109661,3.87120101090789,2007,1 +2007,39163,2.54991381109661,4.00733318523247,2007,1 +2003,39173,4.79632759144445,7.55799495853081,2007,1 +2004,39173,4.79632759144445,7.52131798019924,2007,1 +2005,39173,4.79632759144445,7.46508273639955,2007,1 +2006,39173,4.79632759144445,7.47363710849621,2007,1 +2007,39173,4.79632759144445,7.51261754467451,2007,1 +2003,40003,1.80910810756267,4.02535169073515,0,0 +2004,40003,1.80910810756267,3.76120011569356,0,0 +2005,40003,1.80910810756267,3.91202300542815,0,0 +2006,40003,1.80910810756267,3.85014760171006,0,0 +2007,40003,1.80910810756267,3.98898404656427,0,0 +2003,40013,3.59824333385449,6.1779441140506,0,0 +2004,40013,3.59824333385449,6.24222326545517,0,0 +2005,40013,3.59824333385449,6.2803958389602,0,0 +2006,40013,3.59824333385449,6.13556489108174,0,0 +2007,40013,3.59824333385449,6.19236248947487,0,0 +2003,40015,3.40618492317319,5.45958551414416,0,0 +2004,40015,3.40618492317319,5.27811465923052,0,0 +2005,40015,3.40618492317319,5.32787616878958,0,0 +2006,40015,3.40618492317319,5.41164605185504,0,0 +2007,40015,3.40618492317319,5.3890717298165,0,0 +2003,40017,4.47388769126739,7.12125245324454,0,0 +2004,40017,4.47388769126739,7.08590146436561,0,0 +2005,40017,4.47388769126739,7.00033446027523,0,0 +2006,40017,4.47388769126739,7.07834157955767,0,0 +2007,40017,4.47388769126739,7.09090982207998,0,0 +2003,40025,1.14676733055516,3.80666248977032,0,0 +2004,40025,1.14676733055516,3.58351893845611,0,0 +2005,40025,1.14676733055516,3.55534806148941,0,0 +2006,40025,1.14676733055516,3.93182563272433,0,0 +2007,40025,1.14676733055516,3.55534806148941,0,0 +2003,40039,3.26354321636336,6.04500531403601,0,0 +2004,40039,3.26354321636336,5.98645200528444,0,0 +2005,40039,3.26354321636336,5.89989735358249,0,0 +2006,40039,3.26354321636336,5.98393628068719,0,0 +2007,40039,3.26354321636336,6.02344759296103,0,0 +2003,40063,2.64999726965359,4.62497281328427,0,0 +2004,40063,2.64999726965359,4.45434729625351,0,0 +2005,40063,2.64999726965359,4.27666611901606,0,0 +2006,40063,2.64999726965359,4.11087386417331,0,0 +2007,40063,2.64999726965359,4.52178857704904,0,0 +2003,40069,2.35261258659369,4.49980967033027,0,0 +2004,40069,2.35261258659369,4.38202663467388,0,0 +2005,40069,2.35261258659369,4.18965474202643,0,0 +2006,40069,2.35261258659369,4.18965474202643,0,0 +2007,40069,2.35261258659369,4.66343909411207,0,0 +2003,40075,2.32503128182388,4.60517018598809,0,0 +2004,40075,2.32503128182388,4.38202663467388,0,0 +2005,40075,2.32503128182388,4.26267987704132,0,0 +2006,40075,2.32503128182388,4.26267987704132,0,0 +2007,40075,2.32503128182388,4.11087386417331,0,0 +2003,40083,3.52412272849945,5.65599181081985,0,0 +2004,40083,3.52412272849945,5.38449506278909,0,0 +2005,40083,3.52412272849945,5.42934562895444,0,0 +2006,40083,3.52412272849945,5.4380793089232,0,0 +2007,40083,3.52412272849945,5.47227067367148,0,0 +2003,40101,4.24062146794478,7.02642680869964,0,0 +2004,40101,4.24062146794478,6.7945865808765,0,0 +2005,40101,4.24062146794478,6.91075078796194,0,0 +2006,40101,4.24062146794478,6.95177216439891,0,0 +2007,40101,4.24062146794478,6.78332520060396,0,0 +2003,40119,4.22229792653256,6.96034772910131,0,0 +2004,40119,4.22229792653256,6.87729607149743,0,0 +2005,40119,4.22229792653256,6.7202201551353,0,0 +2006,40119,4.22229792653256,6.81014245011514,0,0 +2007,40119,4.22229792653256,6.83625927727707,0,0 +2003,40137,3.76542374175372,6.20657592672493,0,0 +2004,40137,3.76542374175372,6.17170059741091,0,0 +2005,40137,3.76542374175372,6.19440539110467,0,0 +2006,40137,3.76542374175372,6.30261897574491,0,0 +2007,40137,3.76542374175372,6.30809844150953,0,0 +2003,40143,6.33381057064612,9.27556615689389,0,0 +2004,40143,6.33381057064612,9.18502256399189,0,0 +2005,40143,6.33381057064612,9.21303673352392,0,0 +2006,40143,6.33381057064612,9.31775876874345,0,0 +2007,40143,6.33381057064612,9.36108525866861,0,0 +2003,40145,4.05162841381337,5.89164421182577,0,0 +2004,40145,4.05162841381337,5.85220247977447,0,0 +2005,40145,4.05162841381337,5.92425579741453,0,0 +2006,40145,4.05162841381337,6.00635315960173,0,0 +2007,40145,4.05162841381337,6.02102334934953,0,0 +2003,40153,2.91701368884259,5.73334127689775,0,0 +2004,40153,2.91701368884259,5.64190707093811,0,0 +2005,40153,2.91701368884259,5.7268477475872,0,0 +2006,40153,2.91701368884259,5.86929691313377,0,0 +2007,40153,2.91701368884259,5.95064255258773,0,0 +2003,45007,5.1104202954209,7.6586995582683,0,0 +2004,45007,5.1104202954209,7.61184239958042,0,0 +2005,45007,5.1104202954209,7.62217459481762,0,0 +2006,45007,5.1104202954209,7.65822752616135,0,0 +2007,45007,5.1104202954209,7.63964228785801,0,0 +2003,45009,2.81289058151315,5.03043792139244,0,0 +2004,45009,2.81289058151315,4.93447393313069,0,0 +2005,45009,2.81289058151315,5.12989871492307,0,0 +2006,45009,2.81289058151315,5.05624580534831,0,0 +2007,45009,2.81289058151315,5.14166355650266,0,0 +2003,45013,4.79526974884863,7.58222919427646,0,0 +2004,45013,4.79526974884863,7.57044325205737,0,0 +2005,45013,4.79526974884863,7.6511201757027,0,0 +2006,45013,4.79526974884863,7.74586822979227,0,0 +2007,45013,4.79526974884863,7.71779621101358,0,0 +2003,45017,2.72030809849771,4.21950770517611,0,0 +2004,45017,2.72030809849771,4.12713438504509,0,0 +2005,45017,2.72030809849771,4.14313472639153,0,0 +2006,45017,2.72030809849771,4.17438726989564,0,0 +2007,45017,2.72030809849771,4.23410650459726,0,0 +2003,45019,5.73647229247886,8.52237971810354,0,0 +2004,45019,5.73647229247886,8.50936261230105,0,0 +2005,45019,5.73647229247886,8.60575336839572,0,0 +2006,45019,5.73647229247886,8.70018119275335,0,0 +2007,45019,5.73647229247886,8.71292443512011,0,0 +2003,45023,3.52835852727883,5.34710753071747,0,0 +2004,45023,3.52835852727883,5.35185813347607,0,0 +2005,45023,3.52835852727883,5.35658627467201,0,0 +2006,45023,3.52835852727883,5.47646355193151,0,0 +2007,45023,3.52835852727883,5.52545293913178,0,0 +2003,45025,3.75579015939656,5.80513496891649,0,0 +2004,45025,3.75579015939656,5.73334127689775,0,0 +2005,45025,3.75579015939656,5.85220247977447,0,0 +2006,45025,3.75579015939656,5.78382518232974,0,0 +2007,45025,3.75579015939656,5.89989735358249,0,0 +2003,45037,3.2025431702473,5.0998664278242,0,0 +2004,45037,3.2025431702473,4.969813299576,0,0 +2005,45037,3.2025431702473,5.03043792139244,0,0 +2006,45037,3.2025431702473,4.99721227376411,0,0 +2007,45037,3.2025431702473,5.27299955856375,0,0 +2003,45051,5.28131870446459,7.98786409608569,0,0 +2004,45051,5.28131870446459,7.97453284413023,0,0 +2005,45051,5.28131870446459,8.11282747875137,0,0 +2006,45051,5.28131870446459,8.19973896063079,0,0 +2007,45051,5.28131870446459,8.28652137368124,0,0 +2003,45055,3.96360925689958,6.1779441140506,0,0 +2004,45055,3.96360925689958,6.30261897574491,0,0 +2005,45055,3.96360925689958,6.4085287910595,0,0 +2006,45055,3.96360925689958,6.28599809450886,0,0 +2007,45055,3.96360925689958,6.3818160174061,0,0 +2003,45059,4.24229031697041,6.42162226780652,0,0 +2004,45059,4.24229031697041,6.25766758788264,0,0 +2005,45059,4.24229031697041,6.32435896238131,0,0 +2006,45059,4.24229031697041,6.35610766069589,0,0 +2007,45059,4.24229031697041,6.34212141872115,0,0 +2003,45067,3.56857449107029,5.69035945432406,0,0 +2004,45067,3.56857449107029,5.605802066296,0,0 +2005,45067,3.56857449107029,5.61312810638807,0,0 +2006,45067,3.56857449107029,5.64544689764324,0,0 +2007,45067,3.56857449107029,5.63835466933375,0,0 +2003,45073,4.19290702337893,6.56526497003536,0,0 +2004,45073,4.19290702337893,6.43294009273918,0,0 +2005,45073,4.19290702337893,6.39692965521615,0,0 +2006,45073,4.19290702337893,6.41836493593621,0,0 +2007,45073,4.19290702337893,6.44413125670044,0,0 +2003,45081,2.95392020573955,5.04985600724954,0,0 +2004,45081,2.95392020573955,5.10594547390058,0,0 +2005,45081,2.95392020573955,5.15329159449778,0,0 +2006,45081,2.95392020573955,5.16478597392351,0,0 +2007,45081,2.95392020573955,5.11198778835654,0,0 +2003,45087,3.39722282690674,5.2257466737132,0,0 +2004,45087,3.39722282690674,5.01727983681492,0,0 +2005,45087,3.39722282690674,5.09375020080676,0,0 +2006,45087,3.39722282690674,5.32787616878958,0,0 +2007,45087,3.39722282690674,5.40717177146012,0,0 +2003,46013,3.56840530064606,6.74993119378857,0,0 +2004,46013,3.56840530064606,6.68586094706836,0,0 +2005,46013,3.56840530064606,6.70196036600254,0,0 +2006,46013,3.56840530064606,6.78219205600679,0,0 +2007,46013,3.56840530064606,6.78671695060508,0,0 +2003,46021,0.577736329048618,3.04452243772342,0,0 +2004,46021,0.577736329048618,2.99573227355399,0,0 +2005,46021,0.577736329048618,2.70805020110221,0,0 +2006,46021,0.577736329048618,2.70805020110221,0,0 +2007,46021,0.577736329048618,2.56494935746154,0,0 +2003,46029,3.25412713188084,6.61606518513282,0,0 +2004,46029,3.25412713188084,6.56244409369372,0,0 +2005,46029,3.25412713188084,6.62406522779989,0,0 +2006,46029,3.25412713188084,6.59304453414244,0,0 +2007,46029,3.25412713188084,6.57228254269401,0,0 +2003,46033,1.98444381305756,4.23410650459726,0,0 +2004,46033,1.98444381305756,4.18965474202643,0,0 +2005,46033,1.98444381305756,4.4188406077966,0,0 +2006,46033,1.98444381305756,4.47733681447821,0,0 +2007,46033,1.98444381305756,4.39444915467244,0,0 +2003,46047,2.00861663589956,4.60517018598809,0,0 +2004,46047,2.00861663589956,4.69134788222914,0,0 +2005,46047,2.00861663589956,4.53259949315326,0,0 +2006,46047,2.00861663589956,4.8283137373023,0,0 +2007,46047,2.00861663589956,4.83628190695148,0,0 +2003,46065,2.80220820225204,6.13339804299665,0,0 +2004,46065,2.80220820225204,6.12686918411419,0,0 +2005,46065,2.80220820225204,6.16751649088834,0,0 +2006,46065,2.80220820225204,6.13556489108174,0,0 +2007,46065,2.80220820225204,6.10479323241498,0,0 +2003,46075,0.176471143115779,3.49650756146648,0,0 +2004,46075,0.176471143115779,3.49650756146648,0,0 +2005,46075,0.176471143115779,3.25809653802148,0,0 +2006,46075,0.176471143115779,2.94443897916644,0,0 +2007,46075,0.176471143115779,3.17805383034795,0,0 +2003,46079,2.42267657325187,5.45958551414416,0,0 +2004,46079,2.42267657325187,5.31320597904179,0,0 +2005,46079,2.42267657325187,5.4249500174814,0,0 +2006,46079,2.42267657325187,5.30330490805908,0,0 +2007,46079,2.42267657325187,5.32300997913841,0,0 +2003,46095,0.733809162278835,2.39789527279837,0,0 +2004,46095,0.733809162278835,2.70805020110221,0,0 +2005,46095,0.733809162278835,2.83321334405622,0,0 +2006,46095,0.733809162278835,2.94443897916644,0,0 +2007,46095,0.733809162278835,3.36729582998647,0,0 +2003,46105,1.21283343275813,4.04305126783455,0,0 +2004,46105,1.21283343275813,4.02535169073515,0,0 +2005,46105,1.21283343275813,4.20469261939097,0,0 +2006,46105,1.21283343275813,4.17438726989564,0,0 +2007,46105,1.21283343275813,4.07753744390572,0,0 +2003,46109,2.30418381435774,5.14749447681345,0,0 +2004,46109,2.30418381435774,5.15905529921453,0,0 +2005,46109,2.30418381435774,5.24702407216049,0,0 +2006,46109,2.30418381435774,5.21493575760899,0,0 +2007,46109,2.30418381435774,5.12989871492307,0,0 +2003,46115,2.00875080105707,4.81218435537242,0,0 +2004,46115,2.00875080105707,4.91265488573605,0,0 +2005,46115,2.00875080105707,5.02388052084628,0,0 +2006,46115,2.00875080105707,5.08759633523238,0,0 +2007,46115,2.00875080105707,5.05624580534831,0,0 +2003,46119,0.4421184257562,2.83321334405622,0,0 +2004,46119,0.4421184257562,2.99573227355399,0,0 +2005,46119,0.4421184257562,2.70805020110221,0,0 +2006,46119,0.4421184257562,2.99573227355399,0,0 +2007,46119,0.4421184257562,2.89037175789616,0,0 +2003,46121,2.20276475771183,3.49650756146648,0,0 +2004,46121,2.20276475771183,3.76120011569356,0,0 +2005,46121,2.20276475771183,3.46573590279973,0,0 +2006,46121,2.20276475771183,3.58351893845611,0,0 +2007,46121,2.20276475771183,3.17805383034795,0,0 +2003,47005,2.80560029466806,5.06890420222023,0,0 +2004,47005,2.80560029466806,4.95582705760126,0,0 +2005,47005,2.80560029466806,4.99043258677874,0,0 +2006,47005,2.80560029466806,5.00394630594546,0,0 +2007,47005,2.80560029466806,4.9416424226093,0,0 +2003,47007,2.51503163475992,3.97029191355212,0,0 +2004,47007,2.51503163475992,3.85014760171006,0,0 +2005,47007,2.51503163475992,4.07753744390572,0,0 +2006,47007,2.51503163475992,3.93182563272433,0,0 +2007,47007,2.51503163475992,4.15888308335967,0,0 +2003,47009,4.66176788710248,6.94312242281943,0,0 +2004,47009,4.66176788710248,7.00215595440362,0,0 +2005,47009,4.66176788710248,7.06817200038804,0,0 +2006,47009,4.66176788710248,7.12044437239249,0,0 +2007,47009,4.66176788710248,7.09672137849476,0,0 +2003,47013,3.68522277661039,5.53733426701854,0,0 +2004,47013,3.68522277661039,5.54126354515843,0,0 +2005,47013,3.68522277661039,5.68357976733868,0,0 +2006,47013,3.68522277661039,5.61312810638807,0,0 +2007,47013,3.68522277661039,5.65599181081985,0,0 +2003,47029,3.51348385739071,5.36129216570943,0,0 +2004,47029,3.51348385739071,5.23644196282995,0,0 +2005,47029,3.51348385739071,5.34710753071747,0,0 +2006,47029,3.51348385739071,5.32300997913841,0,0 +2007,47029,3.51348385739071,5.4380793089232,0,0 +2003,47047,3.36058369877686,5.06890420222023,0,0 +2004,47047,3.36058369877686,5.07517381523383,0,0 +2005,47047,3.36058369877686,5.14749447681345,0,0 +2006,47047,3.36058369877686,5.40717177146012,0,0 +2007,47047,3.36058369877686,5.48063892334199,0,0 +2003,47065,5.72976206361515,8.57319538153152,0,0 +2004,47065,5.72976206361515,8.60300384782935,0,0 +2005,47065,5.72976206361515,8.63194942871443,0,0 +2006,47065,5.72976206361515,8.5577591531629,0,0 +2007,47065,5.72976206361515,8.56369502506766,0,0 +2003,47073,3.98085853120358,6.12686918411419,0,0 +2004,47073,3.98085853120358,6.00881318544259,0,0 +2005,47073,3.98085853120358,5.93489419561959,0,0 +2006,47073,3.98085853120358,6.04263283368238,0,0 +2007,47073,3.98085853120358,5.93753620508243,0,0 +2003,47081,3.10436243807944,4.98360662170834,0,0 +2004,47081,3.10436243807944,4.85203026391962,0,0 +2005,47081,3.10436243807944,4.8283137373023,0,0 +2006,47081,3.10436243807944,4.89034912822175,0,0 +2007,47081,3.10436243807944,5.01063529409626,0,0 +2003,47097,3.29957062757386,4.7361984483945,0,0 +2004,47097,3.29957062757386,4.72738781871234,0,0 +2005,47097,3.29957062757386,4.52178857704904,0,0 +2006,47097,3.29957062757386,4.78749174278205,0,0 +2007,47097,3.29957062757386,4.88280192258637,0,0 +2003,47099,3.68702774075046,5.89715386763674,0,0 +2004,47099,3.68702774075046,5.91889385427315,0,0 +2005,47099,3.68702774075046,5.96614673912369,0,0 +2006,47099,3.68702774075046,5.96614673912369,0,0 +2007,47099,3.68702774075046,6.05208916892442,0,0 +2003,47125,4.90355478157301,7.31654817718298,0,0 +2004,47125,4.90355478157301,7.33498187887181,0,0 +2005,47125,4.90355478157301,7.38894609761844,0,0 +2006,47125,4.90355478157301,7.42595365707754,0,0 +2007,47125,4.90355478157301,7.51479976048867,0,0 +2003,47145,3.94951144983914,6.02344759296103,0,0 +2004,47145,3.94951144983914,6.10255859461357,0,0 +2005,47145,3.94951144983914,6.19236248947487,0,0 +2006,47145,3.94951144983914,6.07993319509559,0,0 +2007,47145,3.94951144983914,6.04973345523196,0,0 +2003,47157,6.79958192210419,9.48607637340956,0,0 +2004,47157,6.79958192210419,9.44770236052795,0,0 +2005,47157,6.79958192210419,9.40640045488065,0,0 +2006,47157,6.79958192210419,9.47031717333574,0,0 +2007,47157,6.79958192210419,9.52493210973482,0,0 +2003,47171,2.87169849263054,4.99721227376411,0,0 +2004,47171,2.87169849263054,4.69134788222914,0,0 +2005,47171,2.87169849263054,4.85981240436167,0,0 +2006,47171,2.87169849263054,4.89034912822175,0,0 +2007,47171,2.87169849263054,4.85981240436167,0,0 +2003,47173,2.87964779453319,4.51085950651685,0,0 +2004,47173,2.87964779453319,4.34380542185368,0,0 +2005,47173,2.87964779453319,4.33073334028633,0,0 +2006,47173,2.87964779453319,4.63472898822964,0,0 +2007,47173,2.87964779453319,4.45434729625351,0,0 +2003,48015,3.16082289341958,5.73979291217923,0,0 +2004,48015,3.16082289341958,5.8664680569333,0,0 +2005,48015,3.16082289341958,5.8664680569333,0,0 +2006,48015,3.16082289341958,5.89989735358249,0,0 +2007,48015,3.16082289341958,5.99893656194668,0,0 +2003,48019,2.87045245712723,4.75359019110636,0,0 +2004,48019,2.87045245712723,4.67282883446191,0,0 +2005,48019,2.87045245712723,4.83628190695148,0,0 +2006,48019,2.87045245712723,4.79579054559674,0,0 +2007,48019,2.87045245712723,4.79579054559674,0,0 +2003,48025,3.47689218936423,5.22035582507832,0,0 +2004,48025,3.47689218936423,5.24174701505964,0,0 +2005,48025,3.47689218936423,5.19295685089021,0,0 +2006,48025,3.47689218936423,5.36597601502185,0,0 +2007,48025,3.47689218936423,5.33271879326537,0,0 +2003,48031,2.13037227034838,4.57471097850338,0,0 +2004,48031,2.13037227034838,4.60517018598809,0,0 +2005,48031,2.13037227034838,4.4188406077966,0,0 +2006,48031,2.13037227034838,4.45434729625351,0,0 +2007,48031,2.13037227034838,4.59511985013459,0,0 +2003,48043,2.18222373632286,4.83628190695148,0,0 +2004,48043,2.18222373632286,4.89034912822175,0,0 +2005,48043,2.18222373632286,5.16478597392351,0,0 +2006,48043,2.18222373632286,5.32787616878958,0,0 +2007,48043,2.18222373632286,5.4971682252932,0,0 +2003,48055,3.47178009978404,5.35185813347607,0,0 +2004,48055,3.47178009978404,5.4971682252932,0,0 +2005,48055,3.47178009978404,5.37989735354046,0,0 +2006,48055,3.47178009978404,5.44673737166631,0,0 +2007,48055,3.47178009978404,5.43372200355424,0,0 +2003,48061,5.81480791429005,7.84149292446001,0,0 +2004,48061,5.81480791429005,7.8164169836918,0,0 +2005,48061,5.81480791429005,7.74240202181578,0,0 +2006,48061,5.81480791429005,7.81843027207066,0,0 +2007,48061,5.81480791429005,7.92768504561578,0,0 +2003,48073,3.84286583473053,6.35262939631957,0,0 +2004,48073,3.84286583473053,6.07764224334903,0,0 +2005,48073,3.84286583473053,6.19440539110467,0,0 +2006,48073,3.84286583473053,6.21060007702465,0,0 +2007,48073,3.84286583473053,6.22653666928747,0,0 +2003,48077,2.39844057863757,4.34380542185368,0,0 +2004,48077,2.39844057863757,4.11087386417331,0,0 +2005,48077,2.39844057863757,4.15888308335967,0,0 +2006,48077,2.39844057863757,3.91202300542815,0,0 +2007,48077,2.39844057863757,3.80666248977032,0,0 +2003,48113,7.70476640589851,10.4426965306939,0,0 +2004,48113,7.70476640589851,10.3391261606518,0,0 +2005,48113,7.70476640589851,10.3371189079706,0,0 +2006,48113,7.70476640589851,10.3784787232929,0,0 +2007,48113,7.70476640589851,10.4352622050525,0,0 +2003,48117,2.92106260521696,5.4249500174814,0,0 +2004,48117,2.92106260521696,5.28320372873799,0,0 +2005,48117,2.92106260521696,5.26785815906333,0,0 +2006,48117,2.92106260521696,5.46383180502561,0,0 +2007,48117,2.92106260521696,5.4971682252932,0,0 +2003,48121,6.07068229921461,8.62353322718756,0,0 +2004,48121,6.07068229921461,8.63034328934889,0,0 +2005,48121,6.07068229921461,8.68253812400308,0,0 +2006,48121,6.07068229921461,8.75731184693641,0,0 +2007,48121,6.07068229921461,8.80956425335415,0,0 +2003,48151,1.46879558263163,2.94443897916644,0,0 +2004,48151,1.46879558263163,2.99573227355399,0,0 +2005,48151,1.46879558263163,2.56494935746154,0,0 +2006,48151,1.46879558263163,2.99573227355399,0,0 +2007,48151,1.46879558263163,2.484906649788,0,0 +2003,48173,0.340748793388473,2.39789527279837,0,0 +2004,48173,0.340748793388473,1.79175946922805,0,0 +2005,48173,0.340748793388473,1.94591014905531,0,0 +2006,48173,0.340748793388473,2.07944154167984,0,0 +2007,48173,0.340748793388473,1.38629436111989,0,0 +2003,48213,4.29424678066323,6.57088296233958,0,0 +2004,48213,4.29424678066323,6.47389069635227,0,0 +2005,48213,4.29424678066323,6.39692965521615,0,0 +2006,48213,4.29424678066323,6.49526555593701,0,0 +2007,48213,4.29424678066323,6.46458830368996,0,0 +2003,48221,3.71600812150219,6.2363695902037,0,0 +2004,48221,3.71600812150219,6.13772705408623,0,0 +2005,48221,3.71600812150219,6.05678401322862,0,0 +2006,48221,3.71600812150219,6.25382881157547,0,0 +2007,48221,3.71600812150219,6.36302810354046,0,0 +2003,48223,3.46448512089807,5.95583736946483,0,0 +2004,48223,3.46448512089807,5.91889385427315,0,0 +2005,48223,3.46448512089807,5.85793315448346,0,0 +2006,48223,3.46448512089807,5.95064255258773,0,0 +2007,48223,3.46448512089807,5.90536184805457,0,0 +2003,48245,5.52963144798754,8.01763715990848,0,0 +2004,48245,5.52963144798754,7.92334821193015,0,0 +2005,48245,5.52963144798754,8.09162741160107,0,0 +2006,48245,5.52963144798754,8.18618599422608,0,0 +2007,48245,5.52963144798754,8.17301131172497,0,0 +2003,48249,3.67188587773737,5.62040086571715,0,0 +2004,48249,3.67188587773737,5.76205138278018,0,0 +2005,48249,3.67188587773737,5.73979291217923,0,0 +2006,48249,3.67188587773737,5.78689738136671,0,0 +2007,48249,3.67188587773737,5.86929691313377,0,0 +2003,48271,1.21757980773215,2.63905732961526,0,0 +2004,48271,1.21757980773215,1.6094379124341,0,0 +2005,48271,1.21757980773215,1.6094379124341,0,0 +2006,48271,1.21757980773215,2.19722457733622,0,0 +2007,48271,1.21757980773215,2.30258509299405,0,0 +2003,48277,3.88154317917417,6.32256523992728,0,0 +2004,48277,3.88154317917417,6.24027584517077,0,0 +2005,48277,3.88154317917417,6.15060276844628,0,0 +2006,48277,3.88154317917417,6.37672694789863,0,0 +2007,48277,3.88154317917417,6.45676965557216,0,0 +2003,48291,4.25069282559285,6.26530121273771,0,0 +2004,48291,4.25069282559285,6.24610676548156,0,0 +2005,48291,4.25069282559285,6.19644412779452,0,0 +2006,48291,4.25069282559285,6.27664348934164,0,0 +2007,48291,4.25069282559285,6.289715570909,0,0 +2003,48295,1.1174340429087,3.58351893845611,0,0 +2004,48295,1.1174340429087,3.91202300542815,0,0 +2005,48295,1.1174340429087,4.04305126783455,0,0 +2006,48295,1.1174340429087,3.98898404656427,0,0 +2007,48295,1.1174340429087,4.20469261939097,0,0 +2003,48305,1.87946504964716,3.78418963391826,0,0 +2004,48305,1.87946504964716,3.71357206670431,0,0 +2005,48305,1.87946504964716,3.80666248977032,0,0 +2006,48305,1.87946504964716,3.87120101090789,0,0 +2007,48305,1.87946504964716,2.99573227355399,0,0 +2003,48317,1.55730215801357,2.56494935746154,0,0 +2004,48317,1.55730215801357,1.09861228866811,0,0 +2005,48317,1.55730215801357,2.07944154167984,0,0 +2006,48317,1.55730215801357,1.38629436111989,0,0 +2007,48317,1.55730215801357,1.94591014905531,0,0 +2003,48323,3.85644686853929,5.78996017089725,0,0 +2004,48323,3.85644686853929,5.4553211153577,0,0 +2005,48323,3.85644686853929,5.50125821054473,0,0 +2006,48323,3.85644686853929,5.71702770140622,0,0 +2007,48323,3.85644686853929,5.91620206260743,0,0 +2003,48341,3.00176404578571,5.74939298590825,0,0 +2004,48341,3.00176404578571,5.59471137960184,0,0 +2005,48341,3.00176404578571,5.58724865840025,0,0 +2006,48341,3.00176404578571,5.6937321388027,0,0 +2007,48341,3.00176404578571,5.6970934865054,0,0 +2003,48351,2.71283871783401,3.3322045101752,0,0 +2004,48351,2.71283871783401,3.46573590279973,0,0 +2005,48351,2.71283871783401,3.80666248977032,0,0 +2006,48351,2.71283871783401,3.61091791264422,0,0 +2007,48351,2.71283871783401,3.8286413964891,0,0 +2003,48357,2.19789102187938,5.30330490805908,0,0 +2004,48357,2.19789102187938,5.23644196282995,0,0 +2005,48357,2.19789102187938,5.18178355029209,0,0 +2006,48357,2.19789102187938,5.25227342804663,0,0 +2007,48357,2.19789102187938,5.25749537202778,0,0 +2003,48363,3.29679936561589,5.61312810638807,0,0 +2004,48363,3.29679936561589,5.62762111369064,0,0 +2005,48363,3.29679936561589,5.67332326717149,0,0 +2006,48363,3.29679936561589,5.78689738136671,0,0 +2007,48363,3.29679936561589,5.78996017089725,0,0 +2003,48369,2.30418381435774,4.76217393479776,0,0 +2004,48369,2.30418381435774,4.88280192258637,0,0 +2005,48369,2.30418381435774,4.87519732320115,0,0 +2006,48369,2.30418381435774,4.77912349311153,0,0 +2007,48369,2.30418381435774,4.79579054559674,0,0 +2003,48383,1.20177038077074,3.43398720448515,0,0 +2004,48383,1.20177038077074,3.61091791264422,0,0 +2005,48383,1.20177038077074,3.29583686600433,0,0 +2006,48383,1.20177038077074,3.61091791264422,0,0 +2007,48383,1.20177038077074,3.76120011569356,0,0 +2003,48385,1.11415750000357,3.09104245335832,0,0 +2004,48385,1.11415750000357,2.70805020110221,0,0 +2005,48385,1.11415750000357,2.39789527279837,0,0 +2006,48385,1.11415750000357,2.70805020110221,0,0 +2007,48385,1.11415750000357,2.94443897916644,0,0 +2003,48397,3.76305885228819,6.55535689181067,0,0 +2004,48397,3.76305885228819,6.6052979209482,0,0 +2005,48397,3.76305885228819,6.68835471394676,0,0 +2006,48397,3.76305885228819,6.78332520060396,0,0 +2007,48397,3.76305885228819,6.93828448401696,0,0 +2003,48399,2.44191215821514,4.86753445045558,0,0 +2004,48399,2.44191215821514,4.75359019110636,0,0 +2005,48399,2.44191215821514,4.65396035015752,0,0 +2006,48399,2.44191215821514,4.62497281328427,0,0 +2007,48399,2.44191215821514,4.62497281328427,0,0 +2003,48409,4.20675020260656,6.12249280951439,0,0 +2004,48409,4.20675020260656,6.13988455222626,0,0 +2005,48409,4.20675020260656,6.19644412779452,0,0 +2006,48409,4.20675020260656,6.30627528694802,0,0 +2007,48409,4.20675020260656,6.38856140554563,0,0 +2003,48415,2.79490045400094,5.19849703126583,0,0 +2004,48415,2.79490045400094,5.03043792139244,0,0 +2005,48415,2.79490045400094,4.969813299576,0,0 +2006,48415,2.79490045400094,5.07517381523383,0,0 +2007,48415,2.79490045400094,5.19849703126583,0,0 +2003,48417,1.19452834549794,3.58351893845611,0,0 +2004,48417,1.19452834549794,3.55534806148941,0,0 +2005,48417,1.19452834549794,3.66356164612965,0,0 +2006,48417,1.19452834549794,4.15888308335967,0,0 +2007,48417,1.19452834549794,3.93182563272433,0,0 +2003,48447,0.615185639090233,2.56494935746154,0,0 +2004,48447,0.615185639090233,2.19722457733622,0,0 +2005,48447,0.615185639090233,2.484906649788,0,0 +2006,48447,0.615185639090233,2.484906649788,0,0 +2007,48447,0.615185639090233,2.484906649788,0,0 +2003,48461,1.22495121071113,2.77258872223978,0,0 +2004,48461,1.22495121071113,2.30258509299405,0,0 +2005,48461,1.22495121071113,2.77258872223978,0,0 +2006,48461,1.22495121071113,2.83321334405622,0,0 +2007,48461,1.22495121071113,2.56494935746154,0,0 +2003,48465,3.80345735882137,6.01126717440416,0,0 +2004,48465,3.80345735882137,5.82304589548302,0,0 +2005,48465,3.80345735882137,5.78382518232974,0,0 +2006,48465,3.80345735882137,5.80513496891649,0,0 +2007,48465,3.80345735882137,5.92692602597041,0,0 +2003,48477,3.41355405587153,6.32435896238131,0,0 +2004,48477,3.41355405587153,6.32435896238131,0,0 +2005,48477,3.41355405587153,6.23244801655052,0,0 +2006,48477,3.41355405587153,6.22059017009974,0,0 +2007,48477,3.41355405587153,6.39692965521615,0,0 +2003,48493,3.47840530590039,5.55295958492162,0,0 +2004,48493,3.47840530590039,5.50938833662798,0,0 +2005,48493,3.47840530590039,5.46806014113513,0,0 +2006,48493,3.47840530590039,5.73334127689775,0,0 +2007,48493,3.47840530590039,5.69035945432406,0,0 +2003,48503,2.88720006673053,5.48063892334199,0,0 +2004,48503,2.88720006673053,5.4971682252932,0,0 +2005,48503,2.88720006673053,5.51342874616498,0,0 +2006,48503,2.88720006673053,5.54517744447956,0,0 +2007,48503,2.88720006673053,5.51342874616498,0,0 +2003,48505,2.49995945241429,3.87120101090789,0,0 +2004,48505,2.49995945241429,4.12713438504509,0,0 +2005,48505,2.49995945241429,4.26267987704132,0,0 +2006,48505,2.49995945241429,4.39444915467244,0,0 +2007,48505,2.49995945241429,4.33073334028633,0,0 +2003,48507,2.45100509811232,3.40119738166216,0,0 +2004,48507,2.45100509811232,3.43398720448515,0,0 +2005,48507,2.45100509811232,3.3322045101752,0,0 +2006,48507,2.45100509811232,3.40119738166216,0,0 +2007,48507,2.45100509811232,3.55534806148941,0,0 +2003,49003,3.7552522294782,6.84481547920826,0,0 +2004,49003,3.7552522294782,6.79234442747081,0,0 +2005,49003,3.7552522294782,6.79682371827486,0,0 +2006,49003,3.7552522294782,6.82654522355659,0,0 +2007,49003,3.7552522294782,6.93049476595163,0,0 +2003,49011,5.47643844701387,8.52257866369258,0,0 +2004,49011,5.47643844701387,8.49412925181769,0,0 +2005,49011,5.47643844701387,8.51959031601596,0,0 +2006,49011,5.47643844701387,8.59192953753026,0,0 +2007,49011,5.47643844701387,8.6909784171879,0,0 +2003,49023,2.10875759601389,5.29330482472449,0,0 +2004,49023,2.10875759601389,5.22035582507832,0,0 +2005,49023,2.10875759601389,5.27811465923052,0,0 +2006,49023,2.10875759601389,5.46383180502561,0,0 +2007,49023,2.10875759601389,5.48893772615669,0,0 +2003,49037,2.66813057710097,5.01727983681492,0,0 +2004,49037,2.66813057710097,5.12989871492307,0,0 +2005,49037,2.66813057710097,5.07517381523383,0,0 +2006,49037,2.66813057710097,5.23644196282995,0,0 +2007,49037,2.66813057710097,5.01063529409626,0,0 +2003,49053,4.50373528833274,7.84109976542212,0,0 +2004,49053,4.50373528833274,7.82763954636642,0,0 +2005,49053,4.50373528833274,7.92840602618053,0,0 +2006,49053,4.50373528833274,8.06180227453835,0,0 +2007,49053,4.50373528833274,8.08978917578932,0,0 +2003,51003,4.37243074100043,7.08086789669078,0,0 +2004,51003,4.37243074100043,7.12044437239249,0,0 +2005,51003,4.37243074100043,7.06987412845857,0,0 +2006,51003,4.37243074100043,7.18916773842032,0,0 +2007,51003,4.37243074100043,7.27931883541462,0,0 +2003,51015,4.18380432834825,6.46925031679577,0,0 +2004,51015,4.18380432834825,6.39359075395063,0,0 +2005,51015,4.18380432834825,6.46458830368996,0,0 +2006,51015,4.18380432834825,6.49677499018586,0,0 +2007,51015,4.18380432834825,6.48463523563525,0,0 +2003,51017,1.61899212523891,2.89037175789616,0,0 +2004,51017,1.61899212523891,2.39789527279837,0,0 +2005,51017,1.61899212523891,2.77258872223978,0,0 +2006,51017,1.61899212523891,2.56494935746154,0,0 +2007,51017,1.61899212523891,2.30258509299405,0,0 +2003,51019,4.10050885718995,5.80513496891649,0,0 +2004,51019,4.10050885718995,5.82304589548302,0,0 +2005,51019,4.10050885718995,5.89715386763674,0,0 +2006,51019,4.10050885718995,5.94542060860658,0,0 +2007,51019,4.10050885718995,5.88332238848828,0,0 +2003,51023,3.41759552747179,5.56452040732269,0,0 +2004,51023,3.41759552747179,5.50938833662798,0,0 +2005,51023,3.41759552747179,5.65599181081985,0,0 +2006,51023,3.41759552747179,5.89440283426485,0,0 +2007,51023,3.41759552747179,5.98393628068719,0,0 +2003,51025,2.91338274053697,4.64439089914137,0,0 +2004,51025,2.91338274053697,4.54329478227,0,0 +2005,51025,2.91338274053697,4.56434819146784,0,0 +2006,51025,2.91338274053697,4.54329478227,0,0 +2007,51025,2.91338274053697,4.61512051684126,0,0 +2003,51041,5.56030848448194,8.49801037199946,0,0 +2004,51041,5.56030848448194,8.50875771259514,0,0 +2005,51041,5.56030848448194,8.53129331579502,0,0 +2006,51041,5.56030848448194,8.57395152523485,0,0 +2007,51041,5.56030848448194,8.56254889313703,0,0 +2003,51047,3.53403686830827,6.04500531403601,0,0 +2004,51047,3.53403686830827,6.13772705408623,0,0 +2005,51047,3.53403686830827,6.1779441140506,0,0 +2006,51047,3.53403686830827,6.24027584517077,0,0 +2007,51047,3.53403686830827,6.28226674689601,0,0 +2003,51053,3.2000191500179,5.45958551414416,0,0 +2004,51053,3.2000191500179,5.4249500174814,0,0 +2005,51053,3.2000191500179,5.26269018890489,0,0 +2006,51053,3.2000191500179,5.39816270151775,0,0 +2007,51053,3.2000191500179,5.46806014113513,0,0 +2003,51059,6.87703727512594,9.39499254410842,0,0 +2004,51059,6.87703727512594,9.38681151509196,0,0 +2005,51059,6.87703727512594,9.45297277587424,0,0 +2006,51059,6.87703727512594,9.4922055591064,0,0 +2007,51059,6.87703727512594,9.4753933263676,0,0 +2003,51061,4.0098572697765,6.71174039505618,0,0 +2004,51061,4.0098572697765,6.77078942390898,0,0 +2005,51061,4.0098572697765,6.74758652682932,0,0 +2006,51061,4.0098572697765,6.76041469108343,0,0 +2007,51061,4.0098572697765,6.86693328446188,0,0 +2003,51077,2.88574998281776,4.06044301054642,0,0 +2004,51077,2.88574998281776,4.26267987704132,0,0 +2005,51077,2.88574998281776,4.07753744390572,0,0 +2006,51077,2.88574998281776,4.26267987704132,0,0 +2007,51077,2.88574998281776,4.14313472639153,0,0 +2003,51079,2.72418598301161,5.30826769740121,0,0 +2004,51079,2.72418598301161,4.90527477843843,0,0 +2005,51079,2.72418598301161,4.8283137373023,0,0 +2006,51079,2.72418598301161,4.78749174278205,0,0 +2007,51079,2.72418598301161,4.70953020131233,0,0 +2003,51089,4.05923538510853,6.15909538849193,0,0 +2004,51089,4.05923538510853,5.96357934361845,0,0 +2005,51089,4.05923538510853,6.06610809010375,0,0 +2006,51089,4.05923538510853,5.98141421125448,0,0 +2007,51089,4.05923538510853,6.0591231955818,0,0 +2003,51109,3.24364648319341,5.45958551414416,0,0 +2004,51109,3.24364648319341,5.40267738187228,0,0 +2005,51109,3.24364648319341,5.55682806169954,0,0 +2006,51109,3.24364648319341,5.52942908751142,0,0 +2007,51109,3.24364648319341,5.71042701737487,0,0 +2003,51111,2.57611752984138,4.39444915467244,0,0 +2004,51111,2.57611752984138,4.35670882668959,0,0 +2005,51111,2.57611752984138,4.38202663467388,0,0 +2006,51111,2.57611752984138,4.38202663467388,0,0 +2007,51111,2.57611752984138,4.20469261939097,0,0 +2003,51121,4.4263903498904,6.96885037834195,0,0 +2004,51121,4.4263903498904,6.89972310728487,0,0 +2005,51121,4.4263903498904,6.95749737087695,0,0 +2006,51121,4.4263903498904,7.0057890192535,0,0 +2007,51121,4.4263903498904,7.05703698169789,0,0 +2003,51149,3.49793079061946,5.97888576490112,0,0 +2004,51149,3.49793079061946,5.8664680569333,0,0 +2005,51149,3.49793079061946,5.78074351579233,0,0 +2006,51149,3.49793079061946,5.71373280550937,0,0 +2007,51149,3.49793079061946,5.73009978297357,0,0 +2003,51185,3.79768901497121,6.21460809842219,0,0 +2004,51185,3.79768901497121,6.02586597382531,0,0 +2005,51185,3.79768901497121,6.19440539110467,0,0 +2006,51185,3.79768901497121,6.26530121273771,0,0 +2007,51185,3.79768901497121,6.14846829591765,0,0 +2003,51515,1.84039089063979,5.07517381523383,0,0 +2004,51515,1.84039089063979,5.11799381241676,0,0 +2005,51515,1.84039089063979,5.03695260241363,0,0 +2006,51515,1.84039089063979,5.20948615284142,0,0 +2007,51515,1.84039089063979,5.21493575760899,0,0 +2003,51520,2.85457185376498,6.10255859461357,0,0 +2004,51520,2.85457185376498,6.16331480403464,0,0 +2005,51520,2.85457185376498,6.20253551718792,0,0 +2006,51520,2.85457185376498,6.22455842927536,0,0 +2007,51520,2.85457185376498,6.289715570909,0,0 +2003,51630,2.95901642061733,6.81892406527552,0,0 +2004,51630,2.95901642061733,6.92067150424868,0,0 +2005,51630,2.95901642061733,7.02197642307216,0,0 +2006,51630,2.95901642061733,6.99301512293296,0,0 +2007,51630,2.95901642061733,7.00306545878646,0,0 +2003,51650,4.98659530183624,7.62657020629066,0,0 +2004,51650,4.98659530183624,7.6275443904885,0,0 +2005,51650,4.98659530183624,7.70616297019958,0,0 +2006,51650,4.98659530183624,7.65207074611648,0,0 +2007,51650,4.98659530183624,7.57814547241947,0,0 +2003,51660,3.70051153834364,6.86380339145295,0,0 +2004,51660,3.70051153834364,6.85118492749374,0,0 +2005,51660,3.70051153834364,6.90975328164481,0,0 +2006,51660,3.70051153834364,6.9555926083963,0,0 +2007,51660,3.70051153834364,7.01571242048723,0,0 +2003,51680,4.17851719155535,7.21744343169653,0,0 +2004,51680,4.17851719155535,7.1785454837637,0,0 +2005,51680,4.17851719155535,7.2211050981825,0,0 +2006,51680,4.17851719155535,7.27031288607902,0,0 +2007,51680,4.17851719155535,7.37713371283395,0,0 +2003,51700,5.1937898371941,7.6685611080159,0,0 +2004,51700,5.1937898371941,7.73236922228439,0,0 +2005,51700,5.1937898371941,7.67229245562876,0,0 +2006,51700,5.1937898371941,7.73061406606374,0,0 +2007,51700,5.1937898371941,7.78113850984502,0,0 +2003,51710,5.45704185625577,7.98309894071089,0,0 +2004,51710,5.45704185625577,7.95962530509811,0,0 +2005,51710,5.45704185625577,7.95331834656043,0,0 +2006,51710,5.45704185625577,8.00869818298853,0,0 +2007,51710,5.45704185625577,8.0149968943483,0,0 +2003,51735,2.44806975973603,4.85981240436167,0,0 +2004,51735,2.44806975973603,4.91998092582813,0,0 +2005,51735,2.44806975973603,4.96284463025991,0,0 +2006,51735,2.44806975973603,4.99721227376411,0,0 +2007,51735,2.44806975973603,4.98360662170834,0,0 +2003,51770,4.55293961038444,7.6685611080159,0,0 +2004,51770,4.55293961038444,7.59438124255182,0,0 +2005,51770,4.55293961038444,7.60837447438078,0,0 +2006,51770,4.55293961038444,7.66152708135852,0,0 +2007,51770,4.55293961038444,7.73543335249969,0,0 +2003,51790,3.17190999558746,6.21260609575152,0,0 +2004,51790,3.17190999558746,6.19236248947487,0,0 +2005,51790,3.17190999558746,6.1779441140506,0,0 +2006,51790,3.17190999558746,6.19847871649231,0,0 +2007,51790,3.17190999558746,6.31173480915291,0,0 +2003,51800,4.15382342987363,6.60800062529609,0,0 +2004,51800,4.15382342987363,6.70441435496411,0,0 +2005,51800,4.15382342987363,6.71659477352098,0,0 +2006,51800,4.15382342987363,6.77992190747225,0,0 +2007,51800,4.15382342987363,6.83195356556585,0,0 +2003,51830,2.4847399692309,6.45833828334479,0,0 +2004,51830,2.4847399692309,6.4281052726846,0,0 +2005,51830,2.4847399692309,6.35437004079735,0,0 +2006,51830,2.4847399692309,6.42648845745769,0,0 +2007,51830,2.4847399692309,6.35610766069589,0,0 +2003,51840,3.16061091673622,6.96508034560141,0,0 +2004,51840,3.16061091673622,7.02908756414966,0,0 +2005,51840,3.16061091673622,7.00669522683704,0,0 +2006,51840,3.16061091673622,6.98933526597456,0,0 +2007,51840,3.16061091673622,7.02286808608264,0,0 +2003,54005,3.24005006009961,5.04342511691925,2007,1 +2004,54005,3.24005006009961,4.79579054559674,2007,1 +2005,54005,3.24005006009961,4.72738781871234,2007,1 +2006,54005,3.24005006009961,4.36944785246702,2007,1 +2007,54005,3.24005006009961,4.4188406077966,2007,1 +2003,54017,2.00188532346096,3.36729582998647,2007,1 +2004,54017,2.00188532346096,3.17805383034795,2007,1 +2005,54017,2.00188532346096,2.484906649788,2007,1 +2006,54017,2.00188532346096,2.19722457733622,2007,1 +2007,54017,2.00188532346096,2.07944154167984,2007,1 +2003,54021,1.96850998097255,2.77258872223978,2007,1 +2004,54021,1.96850998097255,2.94443897916644,2007,1 +2005,54021,1.96850998097255,3.52636052461616,2007,1 +2006,54021,1.96850998097255,3.46573590279973,2007,1 +2007,54021,1.96850998097255,3.09104245335832,2007,1 +2003,54027,3.00583110823141,4.89783979995091,2007,1 +2004,54027,3.00583110823141,4.72738781871234,2007,1 +2005,54027,3.00583110823141,4.80402104473326,2007,1 +2006,54027,3.00583110823141,4.96284463025991,2007,1 +2007,54027,3.00583110823141,4.85981240436167,2007,1 +2003,54029,3.48636539403203,5.78382518232974,2007,1 +2004,54029,3.48636539403203,5.73334127689775,2007,1 +2005,54029,3.48636539403203,5.67332326717149,2007,1 +2006,54029,3.48636539403203,5.65948221575962,2007,1 +2007,54029,3.48636539403203,5.60947179518496,2007,1 +2003,54037,3.74218322613625,6.17170059741091,2007,1 +2004,54037,3.74218322613625,6.32435896238131,2007,1 +2005,54037,3.74218322613625,6.36647044773144,2007,1 +2006,54037,3.74218322613625,6.30444880242198,2007,1 +2007,54037,3.74218322613625,6.27852142416584,2007,1 +2003,54047,3.30794840944252,4.07753744390572,2007,1 +2004,54047,3.30794840944252,3.73766961828337,2007,1 +2005,54047,3.30794840944252,3.80666248977032,2007,1 +2006,54047,3.30794840944252,4.18965474202643,2007,1 +2007,54047,3.30794840944252,4.06044301054642,2007,1 +2003,54051,3.57006776457451,5.21493575760899,2007,1 +2004,54051,3.57006776457451,5.18738580584075,2007,1 +2005,54051,3.57006776457451,5.37527840768417,2007,1 +2006,54051,3.57006776457451,5.33753807970132,2007,1 +2007,54051,3.57006776457451,5.4249500174814,2007,1 +2003,54085,2.33630996239567,3.71357206670431,2007,1 +2004,54085,2.33630996239567,3.52636052461616,2007,1 +2005,54085,2.33630996239567,3.66356164612965,2007,1 +2006,54085,2.33630996239567,4.20469261939097,2007,1 +2007,54085,2.33630996239567,4.26267987704132,2007,1 +2003,54087,2.73735007014707,3.93182563272433,2007,1 +2004,54087,2.73735007014707,3.8286413964891,2007,1 +2005,54087,2.73735007014707,4.02535169073515,2007,1 +2006,54087,2.73735007014707,4.14313472639153,2007,1 +2007,54087,2.73735007014707,4.38202663467388,2007,1 +2003,54099,3.75894175355891,5.23110861685459,2007,1 +2004,54099,3.75894175355891,5.27811465923052,2007,1 +2005,54099,3.75894175355891,5.15329159449778,2007,1 +2006,54099,3.75894175355891,5.05624580534831,2007,1 +2007,54099,3.75894175355891,5.14166355650266,2007,1 +2003,55001,2.92547074051332,5.08140436498446,2006,1 +2004,55001,2.92547074051332,4.7361984483945,2006,1 +2005,55001,2.92547074051332,4.59511985013459,2006,1 +2006,55001,2.92547074051332,4.8283137373023,2006,1 +2007,55001,2.92547074051332,4.79579054559674,2006,1 +2003,55051,1.92585320370875,4.76217393479776,2006,1 +2004,55051,1.92585320370875,4.70048036579242,2006,1 +2005,55051,1.92585320370875,4.64439089914137,2006,1 +2006,55051,1.92585320370875,4.63472898822964,2006,1 +2007,55051,1.92585320370875,4.4188406077966,2006,1 +2003,55059,5.00781131040515,8.00235954625271,2006,1 +2004,55059,5.00781131040515,7.91644286012226,2006,1 +2005,55059,5.00781131040515,7.95050243480885,2006,1 +2006,55059,5.00781131040515,7.9208096792886,2006,1 +2007,55059,5.00781131040515,7.91205688817901,2006,1 +2003,55069,3.3891585380504,6.59578051396131,2006,1 +2004,55069,3.3891585380504,6.55535689181067,2006,1 +2005,55069,3.3891585380504,6.5366915975913,2006,1 +2006,55069,3.3891585380504,6.4150969591716,2006,1 +2007,55069,3.3891585380504,6.45204895443723,2006,1 +2003,55081,3.7111056128721,6.65801104587075,2006,1 +2004,55081,3.7111056128721,6.68710860786651,2006,1 +2005,55081,3.7111056128721,6.62406522779989,2006,1 +2006,55081,3.7111056128721,6.68210859744981,2006,1 +2007,55081,3.7111056128721,6.71901315438526,2006,1 +2003,55085,3.60484545850393,6.72503364216684,2006,1 +2004,55085,3.60484545850393,6.59850902861452,2006,1 +2005,55085,3.60484545850393,6.63463335786169,2006,1 +2006,55085,3.60484545850393,6.58479139238572,2006,1 +2007,55085,3.60484545850393,6.56526497003536,2006,1 +2003,55103,2.8861405969239,5.6937321388027,2006,1 +2004,55103,2.8861405969239,5.54907608489522,2006,1 +2005,55103,2.8861405969239,5.48063892334199,2006,1 +2006,55103,2.8861405969239,5.53338948872752,2006,1 +2007,55103,2.8861405969239,5.55682806169954,2006,1 +2003,55109,4.14559202222594,7.45760928971561,2006,1 +2004,55109,4.14559202222594,7.37713371283395,2006,1 +2005,55109,4.14559202222594,7.35755620091035,2006,1 +2006,55109,4.14559202222594,7.35627987655075,2006,1 +2007,55109,4.14559202222594,7.31121838441963,2006,1 +2003,55123,3.33420251283788,6.04263283368238,2006,1 +2004,55123,3.33420251283788,6.01859321449623,2006,1 +2005,55123,3.33420251283788,5.91350300563827,2006,1 +2006,55123,3.33420251283788,5.89440283426485,2006,1 +2007,55123,3.33420251283788,5.91079664404053,2006,1 +2003,55131,4.7663787573415,8.0870254706677,2006,1 +2004,55131,4.7663787573415,8.0925452638913,2006,1 +2005,55131,4.7663787573415,8.12118324207883,2006,1 +2006,55131,4.7663787573415,8.07340296898641,2006,1 +2007,55131,4.7663787573415,8.11820704940578,2006,1 +2003,55137,3.14216755178354,5.95583736946483,2006,1 +2004,55137,3.14216755178354,5.92425579741453,2006,1 +2005,55137,3.14216755178354,5.9427993751267,2006,1 +2006,55137,3.14216755178354,5.93753620508243,2006,1 +2007,55137,3.14216755178354,5.96357934361845,2006,1 diff --git a/benchmarks/stata/README.md b/benchmarks/stata/README.md index 6a5a6828d..1c36f51ca 100644 --- a/benchmarks/stata/README.md +++ b/benchmarks/stata/README.md @@ -185,6 +185,56 @@ grep -E '^r\([0-9]+\);' generate_imputation_loo_golden.log # must print nothin } ``` +--- + +# `jwdid` / `csdid` parity for the ETWFE and Callaway-Sant'Anna ATT(g,t) + +`benchmarks/stata/generate_etwfe_cs_golden.do` produces +`benchmarks/data/etwfe_cs_stata_golden.json`, consumed by +`tests/test_etwfe_cs_stata_parity.py`. It anchors **both** staggered estimators +on the genuine `mpdta` panel: `WooldridgeDiD` against `jwdid` (Rios-Avila's +Wooldridge ETWFE) and `CallawaySantAnna` against `csdid`. + +## Why this arm + +`tests/test_wooldridge.py` asserted that ETWFE ATT(g,t) **equals** CS ATT(g,t) +within `5e-3`. That is false on real data — at `(g=2007, t=2007)` they differ by +`0.0171` (`-0.0431` vs `-0.0261`). The assertion only ever passed because +`load_mpdta()` was silently substituting a synthetic, effect-homogeneous DGP +when its source URL 404'd (issue #722), and on that DGP the two estimators do +coincide. + +Stata's `jwdid` and `csdid` reproduce the **same** disagreement, which is what +establishes it as a property of the estimators rather than a bug in either. So +one self-referential cross-check that validated nothing was replaced by two +external anchors, with the ETWFE-vs-CS gap recorded rather than asserted away. + +## What it measures + +- **Point estimates** — both estimators match their reference to `atol=1e-6` + (observed ~3e-8, i.e. Stata's log-output rounding) on all 7 post-treatment cells. +- **CS SEs** — match `csdid` outright (`rtol=1e-5`). +- **ETWFE SEs** — do **not** match `jwdid`. Every cell is uniformly SMALLER + than Stata's, by a factor that shrinks as the cluster count grows: 1.0280 at + G=20, 1.0132 at G=40, 1.0010 at G=500. **The mechanism is not identified.** + The gap tracks `sqrt(G/(G-1))` but sits consistently ABOVE it, and the CR1 + factor in `linalg.py` already applies `(G/(G-1)) * ((n-1)/(n-k))` — so a + missing cluster term is ruled out. The test pins the observed ratio and its + within-fit uniformity rather than loosening a tolerance, so a change in + magnitude *or* uniformity fails loudly; it records the gap as MEASURED, not + diagnosed. See the REGISTRY `## WooldridgeDiD (ETWFE)` note and the `TODO.md` + row. + +## Input panel + +Reads (does **not** regenerate) `benchmarks/data/mpdta_stata_panel.csv` — the +upstream `mpdta.csv` at SHA-256 `2283bea1…3167`, the same digest pinned in +`diff_diff/datasets.py`. The panel is **committed rather than fetched** so this +arm never depends on network availability; network dependence is precisely the +failure mode that produced the false assertion. Both the generator and the +Python test assert the digest, so a swapped panel cannot silently retarget the +parity. + ## Known constraints - **Batch mode always exits 0**, even on a hard error (`r(NNN);`). Never trust the diff --git a/benchmarks/stata/generate_etwfe_cs_golden.do b/benchmarks/stata/generate_etwfe_cs_golden.do new file mode 100644 index 000000000..437ac9840 --- /dev/null +++ b/benchmarks/stata/generate_etwfe_cs_golden.do @@ -0,0 +1,278 @@ +*! Golden generator: WooldridgeDiD (ETWFE) and CallawaySantAnna ATT(g,t) vs the +*! canonical Stata implementations `jwdid` and `csdid`. +*! +*! Purpose +*! Anchor BOTH staggered estimators against their reference implementations on +*! the genuine Callaway-Sant'Anna `mpdta` panel, and MEASURE the ETWFE-vs-CS +*! relationship instead of asserting it. +*! +*! This arm exists because `tests/test_wooldridge.py` previously asserted that +*! ETWFE ATT(g,t) EQUALS CallawaySantAnna ATT(g,t) within 5e-3. That claim is +*! false on real data: at (g=2007, t=2007) the two estimators differ by 0.0171 +*! (-0.0431 vs -0.0261). The assertion only ever passed because the loader it +*! read from was silently returning a synthetic, effect-homogeneous DGP (see +*! issue #722 / PR #723) on which the two estimators do coincide. +*! +*! Stata `jwdid` and `csdid` reproduce that SAME disagreement, which is what +*! establishes the divergence as a property of the estimators rather than a bug +*! in either implementation. This generator records both reference vectors so +*! the Python side can pin each estimator to its own anchor. +*! +*! Consuming test +*! tests/test_etwfe_cs_stata_parity.py +*! +*! Outputs (checked into the repo) +*! benchmarks/data/etwfe_cs_stata_golden.json +*! +*! Usage (run from the repo root) +*! /Applications/Stata/StataSE.app/Contents/MacOS/stata-se -b do \ +*! benchmarks/stata/generate_etwfe_cs_golden.do +*! Then confirm the log is clean: grep -E '^r\([0-9]+\);' generate_etwfe_cs_golden.log +*! (Stata batch mode ALWAYS exits 0, even on error - never trust the exit code.) +*! +*! SSC dependencies (install via benchmarks/stata/requirements.do) +*! drdid, csdid - Callaway-Sant'Anna (Rios-Avila / Sant'Anna) +*! jwdid, hdfe - Wooldridge ETWFE (Rios-Avila); hdfe is a jwdid dependency +*! +*! Notes +*! - Reads (does NOT regenerate) benchmarks/data/mpdta_stata_panel.csv, the +*! SHA-256-verified upstream `mpdta.csv` +*! (2283bea1221a152420f98dfa20f633c5d054ea51d881115c8cd702a97bcd3167). The +*! panel is committed rather than fetched so this arm never depends on +*! network availability - which is the exact failure mode that produced the +*! false assertion in the first place. +*! - Both commands are run WITHOUT covariates. `csdid`'s method is therefore +*! immaterial (reg/ipw/dr coincide absent covariates); `method(reg)` is +*! passed only to make the no-covariate path explicit. +*! - SEs are emitted alongside the point estimates. The library's ETWFE `hc1` +*! SEs are uniformly SMALLER than `jwdid`'s, by a factor that shrinks as the +*! cluster count grows. The MECHANISM is not identified: the gap tracks +*! sqrt(G/(G-1)) but sits consistently above it, and the CR1 factor at +*! linalg.py already applies (G/(G-1))*((n-1)/(n-k)), so a missing cluster +*! term is ruled out. The consuming test pins the POINT estimates tightly and +*! records the SE ratio as MEASURED, not diagnosed. +*! - The `never` arm is the external anchor for issue #724: `jwdid ... never` +*! omits the `g-1` reference cell per cohort (W2025 Eq. 6.1/6.4), which the +*! library previously left to QR rank detection. + +version 19 +clear all +set more off +set type double + +* Format a scalar as a JSON number at round-trip-exact precision. Stata's %21.17g +* renders |x|<1 as ".455"/"-.119" (leading dot); JSON requires a leading 0, so we +* patch ".x" -> "0.x" and "-.x" -> "-0.x". Returns r(s). +capture program drop _jnum +program define _jnum, rclass + args x fmt + if "`fmt'" == "" local fmt "%21.17g" + local s = strtrim(string(`x', "`fmt'")) + if substr("`s'", 1, 1) == "." local s = "0" + "`s'" + else if substr("`s'", 1, 2) == "-." local s = "-0" + substr("`s'", 2, .) + return local s "`s'" +end + +capture program drop _adover +program define _adover, rclass + args pkg + capture findfile `pkg'.ado + if _rc { + return local v "MISSING" + exit + } + local fn = r(fn) + tempname vh + local ver "" + local first "" + local n = 0 + file open `vh' using "`fn'", read text + file read `vh' line + while r(eof) == 0 & `n' < 15 { + if substr(`"`macval(line)'"', 1, 2) == "*!" { + local body = strtrim(substr(`"`macval(line)'"', 3, .)) + if `"`first'"' == "" local first `"`body'"' + if strpos(lower(`"`body'"'), "version") > 0 { + local ver `"`body'"' + continue, break + } + } + local ++n + file read `vh' line + } + file close `vh' + if `"`ver'"' == "" local ver `"`first'"' + * No parseable "*!" version header (drdid is one such package). Fall back to + * a checksum of the .ado itself: an opaque identifier is still a usable DRIFT + * signal, whereas "unknown" silently disables drift detection for that + * dependency -- the SSC packages are unpinned, so this metadata is the only + * thing standing between a silent upstream change and an unexplained golden + * diff (codex R6 DT-2). + if `"`ver'"' == "" { + capture checksum "`fn'" + if _rc == 0 { + local ver "checksum:`r(checksum)' len:`r(filelen)'" + } + else { + local ver "unknown" + } + } + * JSON-sanitize: drop double-quotes/backslashes/tabs, truncate. + local ver = subinstr(`"`ver'"', `"""', "'", .) + local ver = subinstr(`"`ver'"', "\", "/", .) + local ver = subinstr(`"`ver'"', char(9), " ", .) + if length(`"`ver'"') > 100 local ver = substr(`"`ver'"', 1, 100) + return local v `"`ver'"' +end + + +* ------------------------------------------------------------------------------ +* Fail closed if an SSC dependency is missing: a silent skip here would emit a +* truncated golden, which is the same class of failure this arm exists to kill. +* ------------------------------------------------------------------------------ +foreach p in drdid csdid jwdid hdfe { + capture which `p' + if _rc { + display as error "Missing SSC package `p'. Run benchmarks/stata/requirements.do first." + exit 111 + } +} + +* Record installed SSC versions: SSC has no version history, so this is the only +* drift signal if a regenerated golden ever moves (README "Known constraints"). +foreach p in drdid csdid jwdid hdfe { + _adover `p' + local v_`p' = r(v) +} + +* ------------------------------------------------------------------------------ +* Load the committed panel and confirm the expected schema. +* ------------------------------------------------------------------------------ +import delimited using "benchmarks/data/mpdta_stata_panel.csv", clear varnames(1) +rename firsttreat first_treat +confirm numeric variable year countyreal lpop lemp first_treat treat +quietly count +assert r(N) == 2500 +quietly levelsof countyreal, local(_units) +assert `: word count `_units'' == 500 + +xtset countyreal year + +* ------------------------------------------------------------------------------ +* CSDID: Callaway-Sant'Anna, not-yet-treated control, no covariates. +* Cell coefficients land in e(b) named g:t__. +* ------------------------------------------------------------------------------ +csdid lemp, ivar(countyreal) time(year) gvar(first_treat) notyet method(reg) +matrix cs_b = e(b) +matrix cs_V = e(V) +local cs_names : colnames cs_b +local cs_eqs : coleq cs_b + +* ------------------------------------------------------------------------------ +* JWDID: Wooldridge ETWFE. Cell coefficients are named `_` in +* e(b) after the ATT(g,t) table is formed. +* ------------------------------------------------------------------------------ +jwdid lemp, ivar(countyreal) tvar(year) gvar(first_treat) +matrix jw_b = e(b) +matrix jw_V = e(V) +local jw_names : colnames jw_b + +* ------------------------------------------------------------------------------ +* JWDID with `never`: never-treated controls only. This is the external anchor +* for issue #724 -- jwdid omits the g-1 reference cell per cohort (W2025 +* Eq. 6.1/6.4), which is exactly what the library failed to do. +* ------------------------------------------------------------------------------ +jwdid lemp, ivar(countyreal) tvar(year) gvar(first_treat) never +matrix jn_b = e(b) +matrix jn_V = e(V) +local jn_names : colnames jn_b + +* ------------------------------------------------------------------------------ +* Emit the golden. +* ------------------------------------------------------------------------------ +local sver = c(stata_version) +local sedition = c(edition_real) + +tempname fh +file open `fh' using "benchmarks/data/etwfe_cs_stata_golden.json", write replace text +file write `fh' "{" _n +file write `fh' `" "meta": {"' _n +file write `fh' `" "generator": "benchmarks/stata/generate_etwfe_cs_golden.do","' _n +file write `fh' `" "source_panel": "benchmarks/data/mpdta_stata_panel.csv","' _n +file write `fh' `" "source_sha256": "2283bea1221a152420f98dfa20f633c5d054ea51d881115c8cd702a97bcd3167","' _n +file write `fh' `" "ssc_versions": {"' _n +file write `fh' `" "drdid": "`v_drdid'","' _n +file write `fh' `" "csdid": "`v_csdid'","' _n +file write `fh' `" "jwdid": "`v_jwdid'","' _n +file write `fh' `" "hdfe": "`v_hdfe'""' _n +file write `fh' `" },"' _n +file write `fh' `" "stata_edition": "`sedition'","' _n +file write `fh' `" "csdid_cmd": "csdid lemp, ivar(countyreal) time(year) gvar(first_treat) notyet method(reg)","' _n +file write `fh' `" "jwdid_cmd": "jwdid lemp, ivar(countyreal) tvar(year) gvar(first_treat)","' _n +file write `fh' `" "jwdid_never_cmd": "jwdid lemp, ivar(countyreal) tvar(year) gvar(first_treat) never","' _n +file write `fh' `" "note": "No covariates, so csdid method is immaterial. ETWFE and CS genuinely DISAGREE at (2007,2007).","' _n +file write `fh' `" "stata_version": `sver'"' _n +file write `fh' " }," _n + +* --- csdid cells ------------------------------------------------------------- +file write `fh' `" "csdid": {"' +local sep "" +local k = colsof(cs_b) +forvalues i = 1/`k' { + local nm : word `i' of `cs_names' + local eq : word `i' of `cs_eqs' + local nm "`eq':`nm'" + scalar bval = cs_b[1, `i'] + scalar seval = sqrt(cs_V[`i', `i']) + if !missing(bval) { + _jnum bval + local a = r(s) + _jnum seval + local s = r(s) + file write `fh' "`sep'" _n `" "`nm'": {"att": `a', "se": `s'}"' + local sep "," + } +} +file write `fh' _n " }," _n + +* --- jwdid cells ------------------------------------------------------------- +file write `fh' `" "jwdid": {"' +local sep "" +local k = colsof(jw_b) +forvalues i = 1/`k' { + local nm : word `i' of `jw_names' + scalar bval = jw_b[1, `i'] + scalar seval = sqrt(jw_V[`i', `i']) + if !missing(bval) { + _jnum bval + local a = r(s) + _jnum seval + local s = r(s) + file write `fh' "`sep'" _n `" "`nm'": {"att": `a', "se": `s'}"' + local sep "," + } +} +file write `fh' _n " }," _n + +* --- jwdid never-treated cells (issue #724 anchor) --------------------------- +file write `fh' `" "jwdid_never": {"' +local sep "" +local k = colsof(jn_b) +forvalues i = 1/`k' { + local nm : word `i' of `jn_names' + scalar bval = jn_b[1, `i'] + scalar seval = sqrt(jn_V[`i', `i']) + if !missing(bval) { + _jnum bval + local a = r(s) + _jnum seval + local s2 = r(s) + file write `fh' "`sep'" _n `" "`nm'": {"att": `a', "se": `s2'}"' + local sep "," + } +} +file write `fh' _n " }" _n +file write `fh' "}" _n +file close `fh' + +display "Wrote benchmarks/data/etwfe_cs_stata_golden.json" diff --git a/benchmarks/stata/requirements.do b/benchmarks/stata/requirements.do index 4359c35b8..8fd64d7aa 100644 --- a/benchmarks/stata/requirements.do +++ b/benchmarks/stata/requirements.do @@ -18,9 +18,14 @@ *! did_imputation - Borusyak-Jaravel-Spiess (2024) imputation DiD; the only *! implementation of the App. A.9 leave-one-out variance *! (consumed by generate_imputation_loo_golden.do) +*! drdid, csdid - Callaway-Sant'Anna reference implementation +*! hdfe, jwdid - Wooldridge ETWFE reference implementation (hdfe is a +*! jwdid dependency; jwdid errors with "You need to install +*! hdfe from SSC" without it) +*! (both consumed by generate_etwfe_cs_golden.do) version 19 -foreach p in ftools require reghdfe did_imputation { +foreach p in ftools require reghdfe did_imputation drdid csdid hdfe jwdid { capture which `p' if _rc { di as txt "Installing `p' from SSC ..." diff --git a/diff_diff/survey.py b/diff_diff/survey.py index 04f3ef33c..0f97bc77a 100644 --- a/diff_diff/survey.py +++ b/diff_diff/survey.py @@ -24,6 +24,52 @@ from diff_diff.linalg import _factorize_cluster_ids +def validate_raw_weights(raw_weights: np.ndarray, weight_type: Optional[str] = None) -> None: + """Reject a raw (pre-normalization) weight vector that cannot be used. + + Shared by ``SurveyDesign.resolve`` and by callers that must read the raw + weight column BEFORE resolution, so both paths raise the same errors in the + same order rather than drifting apart. + + ``WooldridgeDiD`` is such a caller: it decides cohort-time cell SUPPORT from + raw weights (a cell whose rows all carry zero weight is absent from the + effective regression), and an unsupported reference period EXCLUDES that + cohort's rows. An invalid weight sitting only in the excluded rows would + therefore never reach ``resolve()`` -- the fit would drop a cohort, report a + confident finite estimate, and warn that the cohort "has no pre-treatment + period" when in fact its periods were merely poisoned. Validating the raw + vector up front closes that path. + + Raises + ------ + ValueError + If any weight is NaN, infinite, or negative, or if every weight is zero. + """ + if np.any(np.isnan(raw_weights)): + raise ValueError("Weights contain NaN values") + if np.any(~np.isfinite(raw_weights)): + raise ValueError("Weights contain Inf values") + if np.any(raw_weights < 0): + raise ValueError("Weights must be non-negative") + if np.any(raw_weights == 0) and np.all(raw_weights == 0): + raise ValueError( + "All weights are zero. At least one observation must " "have a positive weight." + ) + # fweight validation: must be non-negative integers. Lives here, not only in + # resolve(), because callers that read raw weights early must enforce EVERY + # documented weight-type rule -- a fractional fweight in a cohort that + # support-based exclusion later removes would otherwise never be seen. + if weight_type == "fweight": + pos_mask = raw_weights > 0 + if np.any(pos_mask): + fractional = raw_weights[pos_mask] - np.round(raw_weights[pos_mask]) + if np.any(np.abs(fractional) > 1e-10): + raise ValueError( + "Frequency weights (fweight) must be non-negative integers. " + "Fractional values detected. Use pweight for non-integer weights." + ) + + @dataclass class SurveyDesign: """ @@ -163,28 +209,7 @@ def resolve(self, data: pd.DataFrame) -> "ResolvedSurveyDesign": raise ValueError(f"Weight column '{self.weights}' not found in data") raw_weights = data[self.weights].values.astype(np.float64) - # Validate weights - if np.any(np.isnan(raw_weights)): - raise ValueError("Weights contain NaN values") - if np.any(~np.isfinite(raw_weights)): - raise ValueError("Weights contain Inf values") - if np.any(raw_weights < 0): - raise ValueError("Weights must be non-negative") - if np.any(raw_weights == 0) and np.all(raw_weights == 0): - raise ValueError( - "All weights are zero. At least one observation must " "have a positive weight." - ) - - # fweight validation: must be non-negative integers - if self.weight_type == "fweight": - pos_mask = raw_weights > 0 - if np.any(pos_mask): - fractional = raw_weights[pos_mask] - np.round(raw_weights[pos_mask]) - if np.any(np.abs(fractional) > 1e-10): - raise ValueError( - "Frequency weights (fweight) must be non-negative integers. " - "Fractional values detected. Use pweight for non-integer weights." - ) + validate_raw_weights(raw_weights, self.weight_type) # Normalize: pweights/aweights to sum=n (mean=1); fweights unchanged # Skip normalization for replicate designs — the IF path uses diff --git a/diff_diff/wooldridge.py b/diff_diff/wooldridge.py index caf03ba44..9e8445309 100644 --- a/diff_diff/wooldridge.py +++ b/diff_diff/wooldridge.py @@ -14,6 +14,7 @@ from __future__ import annotations import warnings +from dataclasses import dataclass from typing import Any, Dict, List, Optional, Tuple import numpy as np @@ -74,6 +75,189 @@ def _compute_weighted_agg( return {"att": att, "se": se, "t_stat": t_stat, "p_value": p_value, "conf_int": conf_int} +def _reject_zero_weight_groups( + weights: Optional[np.ndarray], + frame: pd.DataFrame, + unit: str, + time: str, +) -> None: + """Reject a design where some unit or period carries zero total weight. + + A zero-weight group makes the weighted within-transformation compute 0/0. + The library's contract is to REJECT it rather than estimate around it + (pinned by ``test_ols_survey_zero_weight_unit_rejected``); genuine domain + estimation is `SurveyDesign.subpopulation`, which the ETWFE within-transform + path does not implement. + + Called TWICE on the OLS path by design: once on the PRE-exclusion frame and + once just before the within-transform. Only the second existed originally, + and unidentified cohort exclusion could delete the offending rows before it + ran -- so a subpopulation-style fit whose zero-weight units happened to make + up a whole unidentified cohort silently degraded to naive subsetting, + returning a fit whose survey df was computed from a design missing entire + PSUs (measured: ``df`` 22 -> 14) instead of raising. Checking pre-exclusion + restores the contract. Normalization is multiplicative and positive, so a + raw zero-sum group is a normalized zero-sum group -- the raw vector is a + valid input. + + OLS ONLY. This is the weighted within-transform's 0/0 constraint; the + logit/Poisson paths use explicit cohort + time dummies and fit zero-weight + cells cleanly by design, so the caller gates the pre-exclusion call on + ``method == "ols"``. Those paths therefore still express cohort exclusion as + row deletion under a survey design -- the general domain-estimation gap + (`SurveyDesign.subpopulation` semantics inside ETWFE) is tracked in TODO.md. + """ + if weights is None or not np.any(weights == 0): + return + w_series = pd.Series(weights, index=frame.index) + for grp_col, grp_label in [(unit, "unit"), (time, "time period")]: + grp_sums = w_series.groupby(frame[grp_col]).sum() + zero_grps = grp_sums[grp_sums == 0].index.tolist() + if zero_grps: + raise ValueError( + f"Survey weights sum to zero for {grp_label}(s) " + f"{zero_grps[:3]}. Cannot compute weighted " + f"within-transformation. Remove zero-weight " + f"{grp_label}s or use non-zero weights." + ) + + +def _require_complete_cell_set( + requested_keys: List[Tuple[Any, Any]], + gt_effects: Dict, +) -> None: + """Every cell the design ASKED for must come back identified. + + A finite ``overall_att`` does NOT mean the reported ATT set is complete. + When rank reduction removes some but not all treatment columns, the + survivors are silently re-identified as whatever contrast the remaining + design supports -- and they keep their ``ATT(g, t)`` labels. + + Measured: never-treated units observed through t=4, cohorts 3 and 4 through + t=5. At t=5 every unit is treated, so ``g3_t5 + g4_t5`` equals the t=5 + indicator and the pair is collinear with the time FE. QR drops one; the + survivor identifies ``-(effect_4 - effect_3)`` and was reported as + ``ATT(3, 5) = -1.9393`` when the true ATT(3, 5) is ``+1.0`` -- wrong SIGN -- + and folded into ``overall_att = +0.7703`` against a true ~1.8. Under + ``rank_deficient_action="silent"`` there was no warning at all. + + So this compares the cells the builder emitted against the coefficients that + actually survived, and fails closed on any unexplained loss. NOT gated on + ``rank_deficient_action``: that setting controls how rank warnings surface, + not whether a mislabeled contrast may be returned as an ATT. Cells the + builder deliberately never emitted (the reference, unobserved pairs) are not + in ``requested_keys`` and so are not counted as losses. + + The same-period comparison-support diagnostic that would catch this BEFORE + the solve, with a better message, is tracked in TODO.md; this is the + correctness backstop that must hold regardless. + """ + missing = [ + k + for k in requested_keys + if k not in gt_effects or not np.isfinite(gt_effects[k].get("att", float("nan"))) + ] + if not missing: + return + shown = ", ".join(f"({g}, {t})" for g, t in missing[:8]) + more = f" (+{len(missing) - 8} more)" if len(missing) > 8 else "" + raise ValueError( + f"{len(missing)} requested cohort-time cell(s) were not identified and " + f"were removed by rank reduction: {shown}{more}. The remaining " + "coefficients are NOT the requested ATT(g, t) -- once a cell is dropped, " + "the survivors identify whatever contrast the reduced design supports " + "(e.g. a difference between two treated cohorts), so reporting them " + "under their original labels, or averaging them into the overall ATT, " + "would be silently wrong. This usually means those periods have no " + "eligible comparison group: every unit is already treated, or the " + "never-treated units are not observed then. Restrict the panel to " + "periods with an available comparison group, or add never-treated " + "observations covering them." + ) + + +def _require_estimable_overall_att( + overall: Dict, + gt_effects: Dict, + anticipation: int, + control_group: str, +) -> None: + """Fail closed when a fit cannot produce a finite overall ATT. + + THE INVARIANT: a successful ``WooldridgeDiD.fit()`` either reports a finite + overall ATT or raises explaining why it cannot. It must never hand back a + result object whose headline number is NaN, which reads as a completed + estimate and silently defeats any downstream aggregation or plot. + + This is the LAST line of defence, deliberately placed after the solve so it + sees rank reduction. `fit()` also guards at BUILD time (no supported + post-treatment cell), which fires earlier and gives a sharper message; that + check cannot see coefficients dropped by QR, and the build-time notion of a + treatment cell (``t >= g - anticipation``, every emitted cell) is WIDER than + the aggregation's (``t >= g``, anticipation-window leads excluded per + W2025), so neither check subsumes the other. Three distinct causes reach + here, and each gets its own message rather than a generic "undefined": + + 1. no cell survived the solve at all (collinear design), + 2. cells survived but all are pre-treatment / inside the anticipation + window, so none contributes to the overall ATT, + 3. post-treatment cells survived but their coefficients are NaN. + + Raising is correct rather than warning: every one of these means the + requested estimand does not exist in the fitted design (per the library's + no-silent-NaN convention, ledger row ``M-124``). + + NOT every branch is a final answer. Cause (2) discards anticipation-window + ATT(g, t) that ARE identified, and cause (3) is a post-hoc stand-in for the + same-period comparison-support diagnostic this estimator does not yet + compute. Both are STOPGAPS with follow-up rows in ``TODO.md``; only cause + (1) and the zero-weight case are terminal. + """ + if np.isfinite(overall.get("att", float("nan"))): + return + + post = [(g, t) for (g, t) in gt_effects if t >= g] + finite_post = [k for k in post if np.isfinite(gt_effects[k].get("att", float("nan")))] + hint = ( + "Add a never-treated group, lengthen the post-treatment window, or " + "reduce `anticipation`." + ) + + if not gt_effects: + raise ValueError( + "No treatment effect is identified: every cohort-time cell was " + "removed from the design, so the overall ATT is undefined. This " + "usually means the treatment cells are collinear with the absorbed " + f"fixed effects -- with control_group={control_group!r}, a single " + "treated cohort and no never-treated units leaves every post-period " + f"cell equal to a time indicator. {hint}" + ) + if not post: + raise ValueError( + f"No cohort-time cell contributes to the overall ATT: all " + f"{len(gt_effects)} estimated cell(s) fall before their cohort's " + f"treatment period (t < g). With anticipation={anticipation}, cells " + f"from t >= g - {anticipation} are ESTIMATED, but the overall ATT " + "averages only t >= g (anticipation-window leads are excluded per " + f"Wooldridge 2025), so no cohort is observed once treated. {hint} " + "Per-cell ATT(g, t) for the anticipation window are available on a " + "fit that also observes at least one t >= g cell." + ) + if not finite_post: + raise ValueError( + f"The overall ATT is undefined: all {len(post)} post-treatment " + "cell(s) have NaN coefficients, so nothing finite remains to " + "average. Their columns were dropped as rank-deficient -- they are " + "collinear with the absorbed fixed effects or with each other. " + f"{hint}" + ) + raise ValueError( + "The overall ATT is undefined: the post-treatment cells carry zero " + "total weight, so the weighted average has no denominator. Check for " + "zero or missing survey weights on the treated observations." + ) + + def _resolve_survey_for_wooldridge(survey_design, sample, cluster_ids, cluster_name): """Resolve survey design, inject cluster as PSU, recompute metadata. @@ -204,6 +388,169 @@ def _filter_sample( return df[treated_mask | control_mask].copy() +def _find_disconnected_cells( + unit_vals: np.ndarray, + time_vals: np.ndarray, + supported: np.ndarray, + emitted_times: List[Any], +) -> List[Any]: + """Return the emitted periods that are NOT identified against the reference. + + Per-period support says a cohort has *some* positive-weight observation at + ``t``. That is NECESSARY but not SUFFICIENT: identification runs through the + unit fixed effects, so what matters is whether the cell is CONNECTED to an + unemitted period in the bipartite graph whose nodes are this cohort's units + and periods (edge = a supported observation). + + Within any connected component, if EVERY period is emitted as a cell, those + columns sum to the component's unit indicators over its own rows -- which the + unit FE absorb -- so the block is rank-deficient and QR drops one of them, + possibly a genuine post-treatment effect. That is issue #724's failure mode + reached by a different route: the reference omission fixes the whole-cohort + case, this covers the sub-cohort case. + + Concretely, on ``never_treated`` + OLS every period except the reference is + emitted, so a component that does not contain the reference is doomed. Units + observed at ``{2, 4}`` and others at ``{1, 5}`` give exactly that: ``ref=2`` + sits in the first component, and ``{1, 5}`` is a closed component whose two + cells are collinear. Measured before this check: ``g4_t5`` silently dropped + and ``overall_att`` computed from the single surviving post cell. + + On ``not_yet_treated`` (and the nonlinear paths) pre-treatment periods are + not emitted at all, so a component holding any pre-period is fine -- which is + why the condition is "contains an unemitted period", not "contains the + reference". + + Mirrors the ``spillover.py`` Omega_0 connectivity guard; scipy is already a + hard dependency. + """ + from scipy.sparse import csr_matrix + from scipy.sparse.csgraph import connected_components + + if not emitted_times: + return [] + + uniq_units, unit_idx = np.unique(unit_vals[supported], return_inverse=True) + uniq_times, time_idx = np.unique(time_vals[supported], return_inverse=True) + n_u, n_t = len(uniq_units), len(uniq_times) + if n_u == 0 or n_t == 0: + return [] + + # Bipartite adjacency: nodes 0..n_u-1 are units, n_u.. are periods. + rows = np.concatenate([unit_idx, n_u + time_idx]) + cols = np.concatenate([n_u + time_idx, unit_idx]) + adj = csr_matrix( + (np.ones(len(rows), dtype=np.int8), (rows, cols)), + shape=(n_u + n_t, n_u + n_t), + ) + _, labels = connected_components(adj, directed=False) + time_labels = labels[n_u:] + + emitted_set = set(emitted_times) + # A component is SAFE if it holds at least one period that is not emitted. + safe = {int(lbl) for lbl, t in zip(time_labels, uniq_times) if t not in emitted_set} + return [ + t for lbl, t in zip(time_labels, uniq_times) if t in emitted_set and int(lbl) not in safe + ] + + +def _exclude_unidentified_cohorts( + sample: pd.DataFrame, + cohort: str, + time: str, + unidentified: List[Any], + control_group: str, +) -> pd.DataFrame: + """Drop cohorts with no ETWFE reference period, then re-check identification. + + A cohort with no observed period before ``g - anticipation`` has no + reference cell, so none of its ATTs are identified. Its ROWS must go, not + just its columns: ``_filter_sample`` retains every treated row, so a cohort + left in the frame without cells joins the omitted baseline and its treatment + effect loads onto the time fixed effects, biasing the cohorts that ARE + identified. + + Removing rows can cascade — those observations may have been the + not-yet-treated comparison for another cohort. On a ``{1, 3}`` panel over + ``t=1..4`` with no never-treated group, excluding cohort 1 leaves cohort 3 + alone and every one of its cells collinear with the time FE, which produces + an all-NaN fit rather than an error. So the surviving frame is re-checked + here and fails closed, rather than deferring to a rank warning. + """ + kept = sample[~sample[cohort].isin(unidentified)].copy() + + labels = ", ".join(str(g) for g in sorted(unidentified)) + warnings.warn( + f"Cohort(s) {labels} have no pre-treatment period before treatment begins " + f"(accounting for anticipation), so their ATT(g, t) are unidentified under " + f"the ETWFE normalization (Wooldridge 2025, Eq. 6.1/6.4 use g-1 as the " + f"omitted reference). Their observations are EXCLUDED from the estimation " + f"sample -- retaining them would make a treated cohort part of the " + f"comparison baseline. The surviving cohorts are therefore estimated on " + f"this REDUCED sample, and their ATT(g, t) will differ from a fit that " + f"retained the excluded rows (that difference is the bias being removed).", + UserWarning, + stacklevel=3, + ) + + surviving = sorted(g for g in kept[cohort].unique() if g > 0) + has_never_treated = bool((kept[cohort] == 0).any()) + + if not surviving: + raise ValueError( + f"Excluding unidentified cohort(s) {labels} leaves no treated cohort. " + "Every treated cohort lacks a pre-treatment period, so no ATT is " + "identified. Provide a panel with pre-treatment observations, or " + "reduce `anticipation`." + ) + if control_group == "not_yet_treated" and len(surviving) < 2 and not has_never_treated: + raise ValueError( + f"Excluding unidentified cohort(s) {labels} leaves a single treated " + f"cohort ({surviving[0]}) with no never-treated group, so no " + "comparison observations remain and its post-treatment cells are " + "collinear with the time fixed effects. Add a never-treated group, " + "or reduce `anticipation` so the excluded cohort(s) retain a " + "reference period." + ) + return kept + + +@dataclass +class InteractionDiagnostics: + """What the interaction builder decided, for the caller to act on. + + The builder cannot drop rows — it returns a design matrix — so an + unidentified cohort is reported here and ``fit()`` removes its observations. + Retaining them would silently move a treated cohort into the comparison + baseline (measured on ``mpdta`` at ``anticipation=1``: ATT(2006, 2006) moves + 0.0077, well past the 5e-3 this library treats as material). + + Attributes + ---------- + references : dict + ``{cohort: reference period}`` — the omitted ETWFE reference cell + (W2025 Eq. 6.1/6.4), or ``None`` where the cohort has no pre-treatment + support and is therefore unidentified. + unidentified : list + Cohorts with no reference period, in sorted order. ``fit()`` excludes + their rows and warns by cohort. + skipped_unobserved : list + ``(g, t)`` pairs omitted because that cohort has no observation there. + Emitting them would add identically-zero columns for QR to drop. + disconnected : list + ``(g, t)`` pairs that ARE observed but sit in a connected component of + the cohort's (unit x period) support graph that contains no unemitted + period -- so the component's cells sum to its unit indicators, which the + unit fixed effects absorb. Aggregate per-period support does not imply + identification; see ``_find_disconnected_cells``. + """ + + references: Dict[Any, Any] + unidentified: List[Any] + skipped_unobserved: List[Tuple[Any, Any]] + disconnected: List[Tuple[Any, Any]] + + def _build_interaction_matrix( data: pd.DataFrame, cohort: str, @@ -211,18 +558,21 @@ def _build_interaction_matrix( anticipation: int, control_group: str = "not_yet_treated", method: str = "ols", -) -> Tuple[np.ndarray, List[str], List[Tuple[Any, Any]]]: + weights: Optional[np.ndarray] = None, + unit: Optional[str] = None, +) -> Tuple[np.ndarray, List[str], List[Tuple[Any, Any]], "InteractionDiagnostics"]: """Build the saturated cohort×time interaction design matrix. For ``not_yet_treated``: only post-treatment cells (t >= g - anticipation). Pre-treatment obs from treated units sit in the regression baseline alongside not-yet-treated controls. - For ``never_treated`` + OLS: ALL (g, t) pairs for each treated cohort. This - "absorbs" pre-treatment obs from treated units into their own indicators so - they do not serve as implicit controls in the baseline. Only never-treated - observations remain in the omitted category. Pre-treatment coefficients - (t < g) serve as placebo/pre-trend tests. + For ``never_treated`` + OLS: every (g, t) pair for each treated cohort + EXCEPT its reference cell ``ref(g)`` (see below). This "absorbs" + pre-treatment obs from treated units into their own indicators so they do + not serve as implicit controls in the baseline. Only never-treated + observations and the omitted reference cells remain in the baseline. + Pre-treatment coefficients (t < g) serve as placebo/pre-trend tests. For ``never_treated`` + nonlinear (logit/Poisson): post-treatment cells only. Nonlinear paths use explicit cohort + time dummies (not within-transformation), @@ -245,21 +595,126 @@ def _build_interaction_matrix( # not_yet_treated: post-treatment only (always) include_pre = control_group == "never_treated" and method == "ols" + # ETWFE reference period, per cohort (W2025 Eq. 6.1/6.4: the `g-1` cell is + # EXCLUDED so it serves as the reference). Computed over each cohort's OWN + # observed support, never the panel-wide `times`: a cohort unobserved at the + # globally-latest pre-period would otherwise have an all-zero column omitted, + # leaving its cell block spanning the unit-FE-absorbed cohort dummy and + # letting QR drop an arbitrary — possibly post-treatment — cell instead + # (issue #724). ``None`` marks a cohort with no pre-treatment support at all, + # whose ATTs are unidentified; the caller excludes it. + # A cell "exists" only if it carries POSITIVE total weight. Survey weights + # scale the design by sqrt(w) (`solve_ols`), so a cell whose rows all carry + # zero pweight is absent from the effective regression even though its rows + # are present in the frame - and a reference chosen from raw presence would + # then omit nothing, leaving the cohort's block spanning the absorbed cohort + # dummy and QR free to drop a real post-treatment cell (issue #724 again, + # this time on the survey path). Weight NORMALIZATION is multiplicative, so + # a raw zero stays zero and reading the pre-resolution column here is + # equivalent FOR ZEROS -- which is the whole of what that argument covers. + # It says nothing about NaN / negative / Inf, and `nansum` would read those + # cells as unsupported, silently reshaping the sample. `fit()` therefore + # runs `survey.validate_raw_weights` on the raw column before it reaches + # this function, so by here the vector is finite and non-negative. + # Computed ONCE by grouped accumulation rather than per (g, t). The naive + # form allocated an n-row boolean mask for every cohort-period pair during + # reference selection and again while building columns -- O(n*G*T) on top of + # the matrix construction, material on large staggered panels (codex R7). + # `nan_to_num` reproduces the previous `np.nansum` semantics: a NaN weight + # contributes 0 rather than poisoning the cell's total. (`fit()` rejects NaN + # weights upstream; direct callers of this function may still pass them.) + _cell_totals = ( + pd.Series( + np.ones(len(data)) + if weights is None + else np.nan_to_num(np.asarray(weights, dtype=float), nan=0.0) + ) + .groupby([pd.Series(cohort_vals), pd.Series(time_vals)], sort=False) + .sum() + ) + _supported_pairs = {key for key, total in _cell_totals.items() if total > 0} + + def _cell_supported(g: Any, t: Any) -> bool: + return (g, t) in _supported_pairs + + references: Dict[Any, Any] = {} + for g in groups: + observed = np.unique(time_vals[cohort_vals == g]) + # latest SUPPORTED pre-period, not merely the latest observed one + pre = [t for t in observed if t < g - anticipation and _cell_supported(g, t)] + references[g] = max(pre) if pre else None + cols = [] col_names = [] gt_keys = [] + skipped_unobserved: List[Tuple[Any, Any]] = [] for g in groups: + if references[g] is None: + # Unidentified cohort: emit nothing. The caller drops its rows too — + # retaining them would silently make a treated cohort part of the + # comparison baseline. + continue for t in times: - if include_pre or t >= g - anticipation: - indicator = ((cohort_vals == g) & (time_vals == t)).astype(float) - cols.append(indicator) - col_names.append(f"g{g}_t{t}") - gt_keys.append((g, t)) + if not (include_pre or t >= g - anticipation): + continue + if t == references[g]: + continue # the omitted reference cell + if not _cell_supported(g, t): + # An identically-zero column: QR would drop it with a generic + # rank warning. Skip it and report the pair instead. + skipped_unobserved.append((g, t)) + continue + indicator = ((cohort_vals == g) & (time_vals == t)).astype(float) + cols.append(indicator) + col_names.append(f"g{g}_t{t}") + gt_keys.append((g, t)) + + unidentified = [g for g in groups if references[g] is None] + + # Per-period support is necessary but NOT sufficient: identification runs + # through the unit FE, so each emitted cell must be connected to an + # unemitted period within its cohort's support graph. Requires unit + # identity, which only `fit()` supplies -- direct callers that omit `unit` + # (tests exercising column layout) skip the check. + disconnected: List[Tuple[Any, Any]] = [] + # OLS ONLY. The condition is collinearity with ABSORBED UNIT FIXED EFFECTS, + # and only the OLS path within-transforms them away. logit/Poisson use + # explicit cohort + time dummies with no unit FE, so nothing absorbs a + # component's cell block and such designs are full rank -- applying the + # check there rejects valid fits (codex R9). Verified: a cohort whose + # pre-period units are disjoint from its post-period units is correctly + # refused under OLS and fits cleanly under logit and Poisson. + if unit is not None and method == "ols": + unit_vals = data[unit].values + for g in groups: + if references[g] is None: + continue + emitted_g = [t for (gg, t) in gt_keys if gg == g] + if not emitted_g: + continue + in_cohort = cohort_vals == g + if weights is None: + supported_rows = in_cohort + else: + w = np.nan_to_num(np.asarray(weights, dtype=float), nan=0.0) + supported_rows = in_cohort & (w > 0) + bad_times = _find_disconnected_cells(unit_vals, time_vals, supported_rows, emitted_g) + disconnected.extend((g, t) for t in bad_times) if not cols: - return np.empty((len(data), 0)), [], [] - return np.column_stack(cols), col_names, gt_keys + return ( + np.empty((len(data), 0)), + [], + [], + InteractionDiagnostics(references, unidentified, skipped_unobserved, disconnected), + ) + return ( + np.column_stack(cols), + col_names, + gt_keys, + InteractionDiagnostics(references, unidentified, skipped_unobserved, disconnected), + ) def _prepare_covariates( @@ -399,17 +854,21 @@ class WooldridgeDiD: to be separately identified from cohort + time FE; ``fit()`` raises ``ValueError`` otherwise. On all-eventually-treated panels the last cohort's trend column is dropped per paper - Section 5.4. ``cohort_trends=True`` + ``survey_design`` raises + Section 5.4 -- but note such panels currently RAISE before that + matters, because the corresponding cell-level normalization is + not implemented and the design is rank-deficient at + fully-treated periods (see REGISTRY, tracked in TODO.md). ``cohort_trends=True`` + ``survey_design`` raises ``NotImplementedError`` at ``fit()`` (deferred follow-up). ``cohort_trends=True`` + ``control_group="never_treated"`` - also raises ``NotImplementedError`` at ``fit()`` because the - OLS + never_treated branch emits ALL ``(g, t)`` placebo cell - dummies (paper Section 4.4 placebo coverage); the appended - ``dg_i · t`` trend columns are linearly spanned by the - per-cohort sum of those cell dummies, so the Section 8 trend - specification is unidentified on this branch. Use - ``control_group="not_yet_treated"`` (the default) for the - cohort_trends surface. + also raises ``NotImplementedError`` at ``fit()``. The + OLS + never_treated branch emits the ``(g, t)`` placebo cell + dummies (paper Section 4.4 placebo coverage) minus each cohort's + reference cell, and the appended ``dg_i · t`` trend columns are + still spanned — jointly by those cells and the unit fixed effects, + which absorb ``1{cohort=g}`` and so recover the omitted reference. + The Section 8 trend specification is therefore unidentified on this + branch. Use ``control_group="not_yet_treated"`` (the default) for + the cohort_trends surface. """ def __init__( @@ -670,12 +1129,14 @@ def fit( # 0d.ii Reject cohort_trends=True + control_group="never_treated". # The OLS + never_treated branch (paper W2025 Section 4.4 / library - # ``_build_interaction_matrix`` ``include_pre=True``) emits ALL + # ``_build_interaction_matrix`` ``include_pre=True``) emits the # ``(g, t)`` cells for each treated cohort as treatment-cell - # indicator dummies — including pre-treatment placebo cells. For - # any treated cohort, ``dg_i · t = Σ_t t · 1{cohort=g, time=t}`` - # is fully spanned by the existing per-cohort sum of cell dummies, - # so the appended trend columns are unidentified. The per-cohort + # indicator dummies — including pre-treatment placebo cells — + # minus each cohort's reference cell. For any treated cohort, + # ``dg_i · t = Σ_t t · 1{cohort=g, time=t}`` is still spanned: + # the emitted cells cover every t except ``ref(g)``, and the unit + # FE absorbs ``1{cohort=g}``, which recovers the omitted one. So + # the appended trend columns remain unidentified. The per-cohort # pre-period guard above counts observed periods but doesn't # catch this collinearity. Fail-close pending a redesigned # design-matrix path that drops the placebo dummies (or restricts @@ -685,11 +1146,13 @@ def fit( raise NotImplementedError( "WooldridgeDiD(cohort_trends=True, " "control_group='never_treated') is not yet supported: " - "the OLS never_treated path emits ALL (g, t) cells (paper " - "W2025 Section 4.4 placebo coverage), and the appended " - "dg_i · t trend columns are linearly spanned by the " - "per-cohort sum of those cell dummies, so the Section 8 " - "trend specification is unidentified on this branch. Use " + "the OLS never_treated path emits the (g, t) placebo cells " + "(paper W2025 Section 4.4 placebo coverage) minus each " + "cohort's reference cell, and the appended dg_i · t trend " + "columns are still spanned - jointly by those cells and the " + "unit fixed effects, which absorb 1{cohort=g} and recover the " + "omitted reference - so the Section 8 trend specification is " + "unidentified on this branch. Use " "control_group='not_yet_treated' (the default) for " "cohort_trends=True, or wait for the deferred follow-up " "tracked in DEFERRED.md." @@ -775,12 +1238,16 @@ def fit( # recommended comparison, not a free upgrade (see REGISTRY.md # §WooldridgeDiD "Nonlinear extensions"). Detection reads the FULL # outcome column: outcome *type* is a property of the variable, so the - # named column is the right signal. This is also identical to the - # estimation sample here — _filter_sample expresses the control-group - # choice through the design matrix, NOT by dropping rows, so the column - # and the fitted sample share the same outcome support (invariant pinned - # by test_filter_sample_preserves_outcome_support); classifying on the - # variable stays correct should that ever change. + # named column is the right signal, and it is DELIBERATELY read before + # any row filtering. `_filter_sample` still expresses the control-group + # choice through the design matrix rather than by dropping rows + # (invariant pinned by test_filter_sample_preserves_outcome_support), + # but the estimation sample is no longer always identical to this + # column: a cohort with no ETWFE reference period has its rows excluded + # downstream (see `_exclude_unidentified_cohorts`). Classifying on the + # variable is the right call regardless — "binary"/"count" is a property + # of how the outcome was measured, not of which rows survive + # identification, so the hint must not flip on a sample reduction. if self.method == "ols": suggested = _suggest_nonlinear_method(df[outcome]) if suggested is not None: @@ -864,19 +1331,204 @@ def fit( f"cohort has at least 2 observed pre-periods." ) - # 2. Build interaction matrix - X_int, int_col_names, gt_keys = _build_interaction_matrix( - sample, - cohort=cohort, - time=time, - anticipation=self.anticipation, - control_group=self.control_group, - method=self.method, - ) - if X_int.shape[1] == 0: + # 2. Build interaction matrix. + # + # Run once to learn which cohorts have no ETWFE reference period. Those + # are unidentified, and their ROWS must go too: the builder can only + # omit columns, and `_filter_sample` keeps every treated row, so a + # cohort left in the frame without cells falls into the omitted baseline + # beside the controls and loads its treatment effect onto the time FE. + # Everything downstream (covariates, within-transform, cluster ids, + # results metadata) reads `sample`, so the exclusion happens here, + # BEFORE any of it, and the matrix is rebuilt on the reduced frame. + # Cell support must be weight-aware (see `_build_interaction_matrix`). + # Survey weights resolve later in fit(), but normalization is + # multiplicative so a raw zero stays zero; reading the column here is + # equivalent for support purposes and avoids reordering the pipeline. + # + # That equivalence covers ZEROS only. An invalid weight (NaN, negative, + # Inf) would also change support -- `_cell_supported` uses `nansum`, so + # an all-NaN or negative cell reads as unsupported -- and support drives + # COHORT EXCLUSION. A poisoned weight confined to the excluded rows + # would then never reach the survey resolution below, and the fit would + # drop a cohort, return a confident finite estimate, and warn that the + # cohort "has no pre-treatment period" when its periods were merely + # poisoned. So validate the raw column against survey.py's own rules + # BEFORE it can influence anything (codex R3 P1). + _cell_w = None + if survey_design is not None: + # Validate the COMPLETE design against the PRE-exclusion sample. + # + # Support decisions below can DELETE an unidentified cohort's rows, + # and every survey validation that runs after that point is blind to + # whatever those rows contained. Patching individual checks into this + # spot one at a time (weights, then weight_type, then zero-weight + # groups) kept missing cases -- missing strata/PSU values, non-nested + # PSU reuse, invalid FPC -- because the defect is the ORDER, not any + # single check. So resolve the whole design here, purely for its + # validation side effects, and let `_resolve_survey_for_wooldridge` + # do the real resolution later on the final sample. Warnings are + # suppressed because that later call is the one whose normalization + # notice the user should see; this pass must not double-warn. + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + survey_design.resolve(sample) + + if getattr(survey_design, "weights", None): + _cell_w = sample[survey_design.weights].to_numpy(dtype=float) + # The weighted within-transform's 0/0 constraint, checked before + # exclusion can delete the offending rows. OLS only: logit and + # Poisson use explicit cohort + time dummies and fit zero-weight + # cells cleanly, so this does not apply to them. + if self.method == "ols": + _reject_zero_weight_groups(_cell_w, sample, unit, time) + + def _build(frame: pd.DataFrame, w: Optional[np.ndarray]): + return _build_interaction_matrix( + frame, + cohort=cohort, + time=time, + anticipation=self.anticipation, + control_group=self.control_group, + method=self.method, + weights=w, + unit=unit, + ) + + X_int, int_col_names, gt_keys, interaction_diagnostics = _build(sample, _cell_w) + + if interaction_diagnostics.unidentified and survey_design is not None: + # Excluding a cohort under a complex survey design is DOMAIN + # estimation, and this path does not implement it. Deleting the rows + # removes their PSUs and strata from the Taylor-linearized meat and + # from `df_survey = n_PSU - n_strata`, so the surviving ATTs would be + # reported with variance computed on a design the user never + # specified (measured: `df_survey` 22 -> 14 on a two-stratum panel, + # with a finite SE and no indication anything was wrong). + # + # `docs/methodology/REGISTRY.md` "Subpopulation Analysis (Phase 6)" + # and Lumley (2004) Section 3.4 require the opposite: zero the + # excluded rows' weights and RETAIN the strata/PSU layout. + # `SurveyDesign.subpopulation()` implements that contract, but + # composing it here also needs the weighted within-transform to + # tolerate zero-weight units -- shared machinery behind several + # estimators. Until that lands, refuse the combination rather than + # return a confident number from the wrong design (TODO.md). + _labels = ", ".join(str(g) for g in sorted(interaction_diagnostics.unidentified)) + raise NotImplementedError( + f"Cohort(s) {_labels} have no pre-treatment period before treatment " + f"begins (accounting for anticipation), so their ATT(g, t) are " + f"unidentified and their observations would have to be excluded. " + f"Combining that exclusion with `survey_design=` is not supported: " + f"dropping the rows would also drop their PSUs/strata from the " + f"survey variance and degrees of freedom, silently changing the " + f"design the estimate is based on. Either restrict the data to " + f"identified cohorts yourself (making the design explicit), reduce " + f"`anticipation` so the cohort(s) keep a reference period, or fit " + f"without `survey_design=`." + ) + + if interaction_diagnostics.unidentified: + # The builder can only omit COLUMNS. A cohort with no reference is + # unidentified and its ROWS must go too, or it joins the omitted + # baseline and loads its effect onto the time FE. Rebuild on the + # reduced frame, and recompute `groups` from it -- every downstream + # consumer (covariate blocks, fit dispatch, results metadata, + # aggregation counts) reads `groups`, and a stale one emits + # all-zero `D{g}_x_{cov}` columns for a cohort that is no longer + # present, which fails outright under rank_deficient_action="error". + keep_mask = ~sample[cohort].isin(interaction_diagnostics.unidentified) + sample = _exclude_unidentified_cohorts( + sample, cohort, time, interaction_diagnostics.unidentified, self.control_group + ) + if _cell_w is not None: + _cell_w = _cell_w[keep_mask.to_numpy()] + groups = sorted(g for g in sample[cohort].unique() if g > 0) + X_int, int_col_names, gt_keys, interaction_diagnostics = _build(sample, _cell_w) + if interaction_diagnostics.skipped_unobserved and self.rank_deficient_action != "silent": + _pairs = ", ".join( + f"({g}, {t})" for g, t in interaction_diagnostics.skipped_unobserved[:8] + ) + _more = ( + f" (+{len(interaction_diagnostics.skipped_unobserved) - 8} more)" + if len(interaction_diagnostics.skipped_unobserved) > 8 + else "" + ) + _msg = ( + f"Skipped {len(interaction_diagnostics.skipped_unobserved)} cohort-time " + f"cell(s) with no observations: {_pairs}{_more}. These would have entered " + "the design as all-zero columns. Their ATTs are not estimable." + ) + # Honor ALL THREE `rank_deficient_action` modes, not just "silent". + # Pre-3.9 these cells reached the design as identically-zero columns + # and QR dropped them, so `"error"` FAILED CLOSED. Skipping them + # earlier is the better behavior, but it must not quietly downgrade + # a caller's explicit `"error"` to a warning (codex R6): the whole + # point of the setting is that an unestimable cell is fatal. + if self.rank_deficient_action == "error": + raise ValueError( + f"{_msg} rank_deficient_action='error' requires every " + "requested cohort-time cell to be estimable. Provide " + "observations for the listed cell(s), restrict the panel to " + "cohorts and periods that are observed, or set " + "rank_deficient_action='warn'." + ) + warnings.warn(_msg, UserWarning, stacklevel=2) + if interaction_diagnostics.disconnected: + # These cells are OBSERVED but not identified: their connected + # component of the cohort's (unit x period) support graph contains no + # unemitted period, so the block sums to that component's unit + # indicators and QR drops one of them -- possibly a real + # post-treatment effect, leaving `overall_att` an average over an + # incomplete cell set. That is issue #724's failure mode reached + # through unit support rather than through the reference, so it fails + # closed here rather than being left to a rank warning. + # + # NOT gated on `rank_deficient_action`: that setting governs how + # rank warnings are surfaced, not whether an unidentified estimand + # may be returned as a number (same convention as the + # unidentified-cohort and survey refusals). Resolving these cells + # properly -- component-aware estimation, or documented row removal + # with the estimand restated -- is tracked in TODO.md. + _cells = ", ".join(f"({g}, {t})" for g, t in interaction_diagnostics.disconnected[:8]) + _extra = ( + f" (+{len(interaction_diagnostics.disconnected) - 8} more)" + if len(interaction_diagnostics.disconnected) > 8 + else "" + ) + raise ValueError( + f"Cohort-time cell(s) {_cells}{_extra} are observed but NOT " + "identified: within their cohort, the units that observe them " + "form a group whose periods are all treatment cells, so those " + "columns are collinear with the unit fixed effects. Per-period " + "observation counts do not detect this -- identification runs " + "through the units. Estimating anyway would drop one cell " + "arbitrarily and average the rest into a partial overall ATT. " + "Restrict the panel to units with overlapping observation " + "periods within each cohort, or drop the disconnected units." + ) + + # A successful fit must retain at least one estimable POST-treatment + # cell. `X_int.shape[1] == 0` is too weak a condition on the + # `never_treated` + OLS path: `include_pre` also emits placebo (t < g) + # columns, so a cohort with no supported post-treatment cell still + # yields a NON-empty design, and the fit returns placebo coefficients + # with `overall_att = NaN` instead of failing. Condition on the + # treatment cells this message already names, which subsumes the + # empty-design case (no columns => no post cells). + post_cells = [(g, t) for g, t in gt_keys if t >= g - self.anticipation] + if not post_cells: + _skipped = interaction_diagnostics.skipped_unobserved + _detail = "" + if _skipped: + _pairs = ", ".join(f"({g}, {t})" for g, t in _skipped[:8]) + _detail = f" Cell(s) skipped for lack of observations: {_pairs}." raise ValueError( - "No valid treatment cells found. Check that treated units " - "have post-treatment observations in the data." + "No estimable post-treatment cells found: no treated cohort has " + "a supported observation at or after its treatment start " + f"(t >= g - anticipation, anticipation={self.anticipation})." + f"{_detail} Check that treated units have post-treatment " + "observations in the data." ) # 3. Covariates @@ -1155,10 +1807,13 @@ def _fit_ols( # "all variables in regression (5.3) involving dT_i get # dropped" when the last cohort serves as control). Drop # the LAST cohort's trend column deterministically — that - # cohort then acts as the trend baseline, mirroring the - # paper's all-treated normalization rule and matching the - # ``_build_interaction_matrix`` last-cohort handling for - # cohort × time interactions. + # cohort then acts as the trend baseline. NOTE: the + # matching last-cohort handling for cohort × time CELLS + # is NOT implemented in ``_build_interaction_matrix``, so + # this branch is currently unreachable end-to-end -- such + # panels are rank-deficient at fully-treated periods and + # fail the completeness gate. Kept because implementing + # the cell half is what restores them (TODO.md). has_never_treated = (sample[cohort] == 0).any() trend_groups = groups if has_never_treated else groups[:-1] trend_cols: List[np.ndarray] = [] @@ -1186,19 +1841,9 @@ def _fit_ols( # interactions + covariates) are demeaned together. wt_weights = survey_weights if survey_weights is not None else np.ones(len(tmp)) - # Guard: zero-weight unit/time groups cause 0/0 in within_transform - if survey_weights is not None and np.any(survey_weights == 0): - sw_series = pd.Series(survey_weights, index=sample.index) - for grp_col, grp_label in [(unit, "unit"), (time, "time period")]: - grp_sums = sw_series.groupby(sample[grp_col]).sum() - zero_grps = grp_sums[grp_sums == 0].index.tolist() - if zero_grps: - raise ValueError( - f"Survey weights sum to zero for {grp_label}(s) " - f"{zero_grps[:3]}. Cannot compute weighted " - f"within-transformation. Remove zero-weight " - f"{grp_label}s or use non-zero weights." - ) + # Guard: zero-weight unit/time groups cause 0/0 in within_transform. + # Also run pre-exclusion in fit(); see _reject_zero_weight_groups. + _reject_zero_weight_groups(survey_weights, sample, unit, time) _w_regressors = [f"_x{i}" for i in range(X_design.shape[1])] _pre_norms = pre_demean_norms(tmp, _w_regressors, weights=wt_weights) @@ -1580,6 +2225,9 @@ def _fit_ols( # ``aggregate(weights="cohort_share")``. n_g_per_cohort = {g: int(sample[sample[cohort] == g][unit].nunique()) for g in groups} + _require_complete_cell_set(gt_keys, gt_effects) + _require_estimable_overall_att(overall, gt_effects, self.anticipation, self.control_group) + results = WooldridgeDiDResults( group_time_effects=gt_effects, overall_att=overall["att"], @@ -1905,6 +2553,9 @@ def _avg_ax0(a, cell_mask): gt_effects, gt_weights, gt_keys_ordered, None, self.alpha, df=df_inf ) + _require_complete_cell_set(gt_keys, gt_effects) + _require_estimable_overall_att(overall, gt_effects, self.anticipation, self.control_group) + return WooldridgeDiDResults( group_time_effects=gt_effects, overall_att=overall["att"], @@ -2158,6 +2809,9 @@ def _avg_ax0(a, cell_mask): gt_effects, gt_weights, gt_keys_ordered, None, self.alpha, df=df_inf ) + _require_complete_cell_set(gt_keys, gt_effects) + _require_estimable_overall_att(overall, gt_effects, self.anticipation, self.control_group) + return WooldridgeDiDResults( group_time_effects=gt_effects, overall_att=overall["att"], diff --git a/diff_diff/wooldridge_results.py b/diff_diff/wooldridge_results.py index 55ed701e0..cdbe3d4bf 100644 --- a/diff_diff/wooldridge_results.py +++ b/diff_diff/wooldridge_results.py @@ -102,8 +102,11 @@ class WooldridgeDiDResults(BaseResults): # never-treated cohort exists, per the all-eventually-treated drop # rule). On all-treated panels the last cohort is intentionally # absent from the dict; its slope is the baseline (zero in deviation - # form). See REGISTRY ``## WooldridgeDiD (ETWFE)`` → "Heterogeneous - # cohort trends" Notes for the exact normalization contract. + # form). The all-treated branch is currently UNREACHABLE: the paper + # rule is applied to the trend columns but not to the cohort × time + # cells, so those panels raise at fit() (TODO.md). See REGISTRY + # ``## WooldridgeDiD (ETWFE)`` → "Heterogeneous cohort trends" Notes + # for the exact normalization contract. cohort_trend_coefs: Dict[Any, float] = field(default_factory=dict, repr=False) # Flag set by ``_fit_ols`` when ``n_bootstrap > 0`` AND the multiplier diff --git a/docs/api/wooldridge_etwfe.rst b/docs/api/wooldridge_etwfe.rst index cd066fa36..c315ea6b1 100644 --- a/docs/api/wooldridge_etwfe.rst +++ b/docs/api/wooldridge_etwfe.rst @@ -12,7 +12,7 @@ This module implements ETWFE via a single saturated regression that: 3. **Uses ASF-based ATT** for nonlinear models: E[f(η₁)] − E[f(η₀)] 4. **Computes delta-method SEs** for all aggregations (event, group, calendar, simple) 5. **Supports paper W2025 cohort-share aggregation** via ``aggregate(weights="cohort_share")`` (Eqs. 7.4 + 7.6; default is cell-count matching Stata ``jwdid_estat``) -6. **Supports paper W2025 Section 8 heterogeneous cohort trends** via ``cohort_trends=True`` (OLS path only; auto-routes to full-dummy mode; requires ``control_group="not_yet_treated"`` — the default — and ``survey_design=None``; the ``never_treated`` and survey paths are fail-closed with ``NotImplementedError`` because the all-(g, t)-cells placebo basis collinearity / unvalidated survey-TSL composition would make the trend specification unidentified or unverified — see Methodology Registry for the full contract) +6. **Supports paper W2025 Section 8 heterogeneous cohort trends** via ``cohort_trends=True`` (OLS path only; auto-routes to full-dummy mode; requires ``control_group="not_yet_treated"`` — the default — and ``survey_design=None``; the ``never_treated`` and survey paths are fail-closed with ``NotImplementedError`` because the placebo-cell basis remaining collinear with the trend columns through the unit fixed effects / unvalidated survey-TSL composition would make the trend specification unidentified or unverified — see Methodology Registry for the full contract) 7. **Follows the Stata jwdid specification** for OLS defaults and nonlinear paths (see Methodology Registry for documented SE/aggregation deviations) **When to use WooldridgeDiD:** @@ -69,8 +69,18 @@ never-treated cohort's trend (when a never-treated cohort exists) OR the last cohort's trend (when no never-treated cohort exists, per paper W2025 Section 5.4's all-eventually-treated drop rule). On all-treated panels the last cohort is intentionally absent from the -dict; its slope is the baseline (zero in deviation form). See -``docs/methodology/REGISTRY.md`` → ``## WooldridgeDiD (ETWFE)`` → +dict; its slope is the baseline (zero in deviation form). + +.. warning:: + + The all-eventually-treated branch is currently **unreachable**: the + paper's Section 5.4 rule is applied to the trend columns but NOT to + the cohort × time cells, so such panels are rank-deficient at + fully-treated periods and ``fit()`` raises rather than returning + relabeled contrasts. Implementing the cell half restores these fits + (tracked in ``TODO.md``). + +See ``docs/methodology/REGISTRY.md`` → ``## WooldridgeDiD (ETWFE)`` → "Heterogeneous cohort trends" for the full normalization contract. .. autoclass:: diff_diff.wooldridge_results.WooldridgeDiDResults diff --git a/docs/methodology/REGISTRY.md b/docs/methodology/REGISTRY.md index 0482d2706..6683c0ab9 100644 --- a/docs/methodology/REGISTRY.md +++ b/docs/methodology/REGISTRY.md @@ -1959,6 +1959,13 @@ The saturated ETWFE regression includes: 4. Additional covariates X_it interacted with cohort×time indicators (optional) The interaction coefficient `δ_{g,t}` identifies `ATT(g, t)` under parallel trends. +- **Note (reference-period normalization, W2025 Eq. 6.1/6.4 — `never_treated` OLS only):** this applies to the **lead-and-lag** specification, i.e. `control_group="never_treated"` on the OLS path, which is the only configuration that emits pre-treatment cells (`include_pre`). The default `not_yet_treated` (and both nonlinear paths) emit `t >= g − anticipation` cells only — the lag-only specification — so no cell is omitted from those designs and no placebo cells are produced; the reference period is still computed there, but solely to detect cohorts that have none (see the unidentified-cohort Note). For the lead-and-lag specification, for each treated cohort `g` the design OMITS one cell — the reference period — because the cohort's full cell block spans the cohort indicator `1{G_i = g}`, which the unit fixed effects absorb. The paper is explicit: Eq. 6.1 excludes `dg_i · f(g-1)_t` "so that `s = g − 1` is the reference period", and Eq. 6.4 describes the saturated regression as a collection of 2×2 DiDs using `g − 1` as reference. The library omits `ref(g) = max{t : t < g − anticipation AND cohort g is OBSERVED at t}` — `g − 1` on a balanced panel, and the cohort's own latest available pre-period otherwise. The support must be **per-cohort**: a panel-wide rule omits an identically-zero column whenever that cohort is unobserved at the panel's latest pre-period, leaving the collinearity intact. Anchored to Stata `jwdid ... never`, which omits the same cell for every cohort (`tests/test_etwfe_cs_stata_parity.py::TestNeverTreatedVsStataJwdid`). **Before v3.9 the reference was emitted and generic QR rank detection dropped an arbitrary column instead** — on `mpdta` that silently removed two genuine post-treatment effects, `(2004, 2004)` and `(2006, 2007)` (issue #724). +- **Note (reference-period sensitivity, W2025 Section 6.1):** the choice of reference is a NORMALIZATION, not an identifying assumption. The paper notes any set of pre-treatment periods may serve, and the pre-trend `t`-test is numerically identical whichever is used. `g − 1` is the paper's canonical choice and the one this library implements. **The point estimates are NOT invariant to it** — only the two-sided pre-trend test is (Section 6.1). Different references give different finite-sample 2×2 contrasts, all consistent for ATT(g, t) under parallel trends: on `mpdta`, ATT(2007, 2007) computed by hand against references 2006/2005/2004/2003 gives −0.0261 / −0.0571 / −0.0599 / −0.0294. This is exactly why issue #724 mattered — QR dropped `g2007_t2005`, silently making 2005 the reference, and the estimator returned −0.0571: a correct 2×2 DiD against the wrong baseline. It also means the unbalanced-panel fallback (the cohort's latest AVAILABLE pre-period) yields a different contrast from `g − 1`, which is legitimate but should be read as such. +- **Note (unidentified cohorts — library identification limit):** a cohort with NO observed period before `g − anticipation` has no reference cell, so none of its ATTs are identified. The library warns naming the cohort and **excludes its observations** from the estimation sample. Excluding only its columns would be worse than the original bug: `_filter_sample` retains every treated row, so the cohort would join the omitted baseline beside the controls and load its treatment effect onto the time fixed effects — measured on `mpdta` at `anticipation=1`, that moved `ATT(2006, 2006)` by 0.0077 and `ATT(2006, 2007)` by 0.0055, past the 5e-3 this library treats as material. Exclusion can cascade (the removed rows may have been another cohort's not-yet-treated comparison), so the surviving sample is re-checked and `fit()` raises when no comparison observations remain rather than returning an all-NaN fit. This is a **library limit, not the paper's last-cohort result** — W2025's "the last cohort `T` plays the role of the never-treated group" is conditioned on there being no never-treated group at all, a different case. +- **Note:** unobserved `(g, t)` pairs are skipped rather than emitted as identically-zero columns, and the skipped pairs are named in a `UserWarning` (gated on `rank_deficient_action`, so `"silent"` remains silent). +- **Note (within-cohort support connectivity — OLS/unit-FE path only):** per-period support is **necessary but not sufficient** for identification. Identification runs through the unit fixed effects, so each emitted `(g, t)` cell must be connected to an **unemitted** period in the bipartite graph whose nodes are that cohort's units and periods (edge = a supported observation). Within any connected component in which *every* period is a treatment cell, those columns sum to the component's unit indicators — absorbed by the unit FE — so the block is rank-deficient and QR drops one, potentially a genuine post-treatment effect, leaving the overall ATT an average over an incomplete cell set. This is issue #724's failure mode reached through **unit** support rather than through the reference period, and per-period observation counts cannot detect it. Measured on a panel where cohort 4's units split into `{2, 4}` and `{1, 5}` groups: `g4_t5` was silently dropped and `overall_att` computed from the single surviving post cell. `fit()` now **fails closed** naming the unidentified cells (not gated on `rank_deficient_action` — that setting governs how rank warnings surface, not whether an unidentified estimand may be returned as a number). On `not_yet_treated` and the nonlinear paths pre-treatment periods are not emitted, so a component holding any pre-period is safe — which is why the condition is "contains an unemitted period", not "contains the reference". The condition applies **only to the OLS path**, whose within-transformation absorbs the unit fixed effects: `logit`/`poisson` use explicit cohort + time dummies, nothing absorbs a component's cell block, and such designs are full rank — so the check is gated on `method="ols"` and those paths estimate the same panel normally. Component-aware estimation (rather than refusal) is tracked in `TODO.md`. +- **Note (unidentified-cohort exclusion is REFUSED under `survey_design=`):** the row exclusion above is naive subsetting, which under a complex survey design is not domain estimation — it removes the excluded rows' PSUs and strata from the Taylor-linearized meat and from `df_survey = n_PSU − n_strata`, so surviving ATTs would be reported with variance computed on a design the user never specified (measured on a two-stratum panel: `df_survey` 22 → 14, finite SE, no diagnostic). *Subpopulation Analysis (Phase 6)* below and Lumley (2004) §3.4 require the opposite — zero the excluded rows' weights and RETAIN the strata/PSU layout — and `SurveyDesign.subpopulation()` implements that contract, but composing it here additionally requires the weighted within-transformation to tolerate zero-weight units (shared machinery). Until that lands, `fit()` raises `NotImplementedError` when `survey_design=` is combined with an unidentified cohort, on all three methods, rather than returning a confident number from the wrong design (tracked in `TODO.md`). Relatedly, the **full** `SurveyDesign` is resolved against the **pre-exclusion** sample for validation only (weights, `weight_type`, strata, PSU, FPC, nesting): every check that ran after exclusion was blind to whatever the deleted rows contained, so invalid metadata confined to an excluded cohort used to vanish silently. +- **Note (finite-overall-ATT invariant, ledger `M-124`):** a successful `fit()` reports a **finite** `overall_att` or raises explaining why it cannot, so a result object never presents a NaN point estimate as a completed fit. The invariant covers the **point estimate only**. `overall_se` and the inference fields may legitimately be NaN on a successful fit where the *variance* is unidentifiable while the ATT is not — Bell-McCaffrey DOF unavailable on the `hc2_bm` path, and `aggregate(weights="cohort_share")`, both of which fail closed to NaN through `safe_inference()` by design (see those entries). `_require_estimable_overall_att` accordingly checks `overall_att` and nothing else. Enforced at two levels because neither subsumes the other: at BUILD time (is any post-treatment cell estimable, `t >= g − anticipation`) and again POST-SOLVE in `_require_estimable_overall_att`, before every results construction. The build-time notion of a treatment cell is deliberately **wider** than the aggregation's — cells in the anticipation window (`g − anticipation <= t < g`) are estimated and reported per-cell, but excluded from the overall average per W2025 — so a cohort never observed once actually treated produces cells yet no overall estimand. Three causes are distinguished by message: no cell survived the solve (collinear design); cells survived but none has `t >= g`; post-treatment cells survived with NaN coefficients (rank-dropped). **Cohort count is not a proxy for comparison support** — a design with two surviving treated cohorts can still be fully degenerate when neither post cell has a same-period comparison unit, so the check is on surviving finite coefficients, not on how many cohorts remain. - **Note:** OLS path uses iterative alternating-projection within-transformation (uniform weights) for exact FE absorption on both balanced and unbalanced panels. One-pass demeaning (`y - ȳ_i - ȳ_t + ȳ`) is only exact for balanced panels. - **Note:** The weighted within-transformation (`utils.within_transform` with `weights`) is invoked on every WooldridgeDiD fit (survey weights when provided, `np.ones` otherwise) and emits a `UserWarning` on non-convergence per the shared convention documented under *Absorbed Fixed Effects with Survey Weights*. - **Note:** NaN values in the `cohort` column are filled with 0 (treated as never-treated), both in `_filter_sample` and in `fit()`. This recategorization now emits a `UserWarning` reporting the affected row count so it is no longer silent (axis-E silent coercion under the Phase 2 audit). Pass `0` directly for never-treated units to avoid the warning. @@ -1973,7 +1980,7 @@ Average Structural Function (ASF) approach. For each treated cell (g, t): where `g(·)` is the link inverse (logistic or exp), `η_i` is the individual linear predictor (fixed effects + controls), and `δ_{g,t}` is the interaction coefficient from the nonlinear model. -- **Note (outcome-fit hint):** When `method="ols"` is used on a **binary** (`{0,1}`) or a **non-negative integer count** outcome, `WooldridgeDiD.fit()` emits a `UserWarning` noting that a matching nonlinear model (`method="logit"` / `method="poisson"`) is often the **more appropriate specification** for such outcomes. The framing is the paper's, not an efficiency heuristic: the nonlinear paths impose parallel trends on the **link/index scale** (logit index / Poisson log-mean) rather than in **levels**, and Wooldridge (2023) states the **linear-PT assumption is only valid for continuous/unbounded outcomes** (Eq. 2.5 vs the Index-PT Eq. 2.6–2.7). In the paper's **Section 5 simulations** the linear model is both **biased** (POLS −0.15 to −0.29 for binary; OLS >30%, >50% in places, for counts) **and less precise** (nonlinear SEs ~30–70% smaller) where the nonlinear mean holds, and the paper notes pre-trends tests often **fail to detect** the linear model's misspecification. This is a **different identifying assumption** than linear OLS — which one fits depends on which parallel-trends restriction holds — so the warning frames the nonlinear model as a **recommended comparison, not an automatic switch or a free upgrade**. Per **Table 1**, Gaussian/OLS remains a valid QMLE for *any* response, so the hint never asserts OLS is mechanically wrong. **Framing provenance (two-sided):** the earlier "canonical link requirement tied to Prop 3.1" reading was **incorrect** (corrected in **PR #453 R1**); the warning is equally careful *not* to over-correct into framing the nonlinear path as a pure efficiency add-on, since the paper's case is primarily about **bias / appropriateness** (codex R1, reconciled against `docs/methodology/papers/wooldridge-2023-review.md` §5). Detection is deliberately high-signal — binary requires support exactly `{0,1}`; the count branch suggests Poisson for non-negative integers with `>2` distinct values (Poisson is the natural *unbounded* nonnegative/count model per Table 1; bounded-support / **binomial** outcomes with a known upper bound are not separately detected — that routing would need user-supplied metadata). Fractional / bounded-continuous outcomes are **not flagged by this heuristic** (fractional responses are themselves a logistic/Bernoulli case in Table 1; the heuristic simply does not attempt to detect them — a coverage limit, not a claim that OLS is preferred for such outcomes). **Detection sample:** the hint reads the *full* outcome column passed to `fit()` — outcome *type* is a property of the variable, so the named column is the right signal. In this estimator that column is also identical to the estimation sample: `WooldridgeDiD._filter_sample` expresses the control-group choice through the design matrix (`_build_interaction_matrix`), **not** by dropping rows, so the full column and the fitted sample always share the same outcome support (invariant pinned by `tests/test_wooldridge.py::TestOutcomeFitHint::test_filter_sample_preserves_outcome_support`). Classifying on the variable keeps the suggestion methodologically correct (a count is a count regardless of which rows enter estimation) should that selection ever change. Always-on (suppress via `warnings.filterwarnings`), implemented as the pure helper `_suggest_nonlinear_method` + the OLS-only fit-time gate `0g`; it never alters the fit or raises. +- **Note (outcome-fit hint):** When `method="ols"` is used on a **binary** (`{0,1}`) or a **non-negative integer count** outcome, `WooldridgeDiD.fit()` emits a `UserWarning` noting that a matching nonlinear model (`method="logit"` / `method="poisson"`) is often the **more appropriate specification** for such outcomes. The framing is the paper's, not an efficiency heuristic: the nonlinear paths impose parallel trends on the **link/index scale** (logit index / Poisson log-mean) rather than in **levels**, and Wooldridge (2023) states the **linear-PT assumption is only valid for continuous/unbounded outcomes** (Eq. 2.5 vs the Index-PT Eq. 2.6–2.7). In the paper's **Section 5 simulations** the linear model is both **biased** (POLS −0.15 to −0.29 for binary; OLS >30%, >50% in places, for counts) **and less precise** (nonlinear SEs ~30–70% smaller) where the nonlinear mean holds, and the paper notes pre-trends tests often **fail to detect** the linear model's misspecification. This is a **different identifying assumption** than linear OLS — which one fits depends on which parallel-trends restriction holds — so the warning frames the nonlinear model as a **recommended comparison, not an automatic switch or a free upgrade**. Per **Table 1**, Gaussian/OLS remains a valid QMLE for *any* response, so the hint never asserts OLS is mechanically wrong. **Framing provenance (two-sided):** the earlier "canonical link requirement tied to Prop 3.1" reading was **incorrect** (corrected in **PR #453 R1**); the warning is equally careful *not* to over-correct into framing the nonlinear path as a pure efficiency add-on, since the paper's case is primarily about **bias / appropriateness** (codex R1, reconciled against `docs/methodology/papers/wooldridge-2023-review.md` §5). Detection is deliberately high-signal — binary requires support exactly `{0,1}`; the count branch suggests Poisson for non-negative integers with `>2` distinct values (Poisson is the natural *unbounded* nonnegative/count model per Table 1; bounded-support / **binomial** outcomes with a known upper bound are not separately detected — that routing would need user-supplied metadata). Fractional / bounded-continuous outcomes are **not flagged by this heuristic** (fractional responses are themselves a logistic/Bernoulli case in Table 1; the heuristic simply does not attempt to detect them — a coverage limit, not a claim that OLS is preferred for such outcomes). **Detection sample:** the hint reads the *full* outcome column passed to `fit()` — outcome *type* is a property of the variable, so the named column is the right signal — and it is read *before* any row filtering. `WooldridgeDiD._filter_sample` still expresses the control-group choice through the design matrix (`_build_interaction_matrix`), **not** by dropping rows (invariant pinned by `tests/test_wooldridge.py::TestOutcomeFitHint::test_filter_sample_preserves_outcome_support`), but as of 3.9 the full column and the fitted sample **no longer always share the same outcome support**: a cohort with no ETWFE reference period has its observations excluded downstream (see the unidentified-cohorts Note above and ledger row `M-123`). Classifying on the variable remains the correct choice precisely because of that — a count is a count regardless of which rows survive identification, so the hint must not flip on a sample reduction. Always-on (suppress via `warnings.filterwarnings`), implemented as the pure helper `_suggest_nonlinear_method` + the OLS-only fit-time gate `0g`; it never alters the fit or raises. *Standard errors:* - OLS: Cluster-robust sandwich estimator at the unit level (default) @@ -1984,7 +1991,7 @@ where `g(·)` is the link inverse (logistic or exp), `η_i` is the individual li - **Note:** QMLE sandwich uses `weight_type="aweight"` which applies `(G/(G-1)) * ((n-1)/(n-k))` small-sample adjustment. Stata `jwdid` uses `G/(G-1)` only. The `(n-1)/(n-k)` term is conservative (inflates SEs slightly). For typical ETWFE panels where n >> k, the difference is negligible. *Variance families (`vcov_type`, OLS path only):* -- `hc1` (default) — CR1 Liang-Zeger cluster-robust on the within-transformed design. Bit-equal to prior behavior (FWL preserves the score). The natural R anchor is `fixest::feols(y ~ | unit + time, cluster=~unit)` or Stata `jwdid` (both within-transform). **Deviation from R `lm + clubSandwich::vcovCR(type="CR1S")`:** the full-dummy `lm` SE differs by a factor of `sqrt((n - k_within) / (n - k_total))` because clubSandwich's `(n-1)/(n-p)` finite-sample correction counts ALL columns (intercept + treatment + unit dummies + time dummies = `k_total`) while WooldridgeDiD's `solve_ols` on the within-transformed design counts only the treatment-cell columns (`k_within`). On the 240-obs / 51-column R-parity fixture this is ~11%; on typical larger panels (n >> k_total) the gap shrinks to <2%. No public WooldridgeDiD code path exposes the `lm + CR1S` (CR1 cluster-robust on the full-dummy design) finite-sample correction — `vcov_type="hc2_bm"` routes to the CR2 Bell-McCaffrey sandwich on the full-dummy design (different variance estimator entirely), not CR1S. Users who need exact `lm + clubSandwich::vcovCR(type="CR1S")` parity must call `solve_ols` directly on a full-dummy design or fit via R. Same deviation pattern as SunAbraham PR #472 (`fixest::sunab` vs `lm + clubSandwich`). +- `hc1` (default) — CR1 Liang-Zeger cluster-robust on the within-transformed design. Bit-equal to prior behavior (FWL preserves the score). The natural R anchor is `fixest::feols(y ~ | unit + time, cluster=~unit)` or Stata `jwdid` (both within-transform). **Deviation from Stata `jwdid` (measured 2026-07-26, `tests/test_etwfe_cs_stata_parity.py`):** the ATT(g,t) POINT estimates match `jwdid` exactly (~3e-8 on the `mpdta` panel, i.e. Stata's log-output rounding), but every `hc1` SE is SMALLER than `jwdid`'s by a factor that is **uniform across cells** (spread < 1e-6 within a fit) and shrinks as the cluster count grows: 1.0280 at G=20, 1.0132 at G=40, 1.0086 at G=60, 1.0046 at G=110, 1.0010 at G=500. **Only the G=500 full-panel ratio is PINNED by CI** (`tests/test_etwfe_cs_stata_parity.py`); the smaller-G figures were measured ad hoc on subsampled panels during the #724 investigation and no committed artifact reproduces them, so treat them as the shape of the trend rather than as regression-gated constants. Committing that subsample ladder as a golden block is part of the derivation work tracked in `TODO.md` — the ladder is the instrument the derivation needs, not a separate chore. The library is therefore systematically **anti-conservative** relative to the reference - negligibly with many clusters (~0.1% at G=500) and materially in few-cluster designs (~2.8% at G=20). **The mechanism is NOT yet identified.** The gap tracks `sqrt(G/(G-1))` closely but lies consistently ABOVE it (by ~0.2% at G=20 down to ~0.001% at G=500), and `solve_ols` already applies the full CR1 `(G/(G-1)) * ((n-1)/(n-k))` - so a missing `G/(G-1)` is ruled out as the explanation. This is recorded as a measured deviation, not a diagnosed one; deriving the true factor from the within-transform `k` accounting against `hdfe`/`reghdfe`'s is tracked in `TODO.md`, and no correction should be applied until it reproduces `jwdid` exactly rather than approximately. This is distinct from the `lm + clubSandwich` deviation below, whose factor is `k`-based. `CallawaySantAnna` shows no such gap - its SEs match Stata `csdid` outright - which localizes this to the ETWFE path rather than a library-wide convention. **Deviation from R `lm + clubSandwich::vcovCR(type="CR1S")`:** the full-dummy `lm` SE differs by a factor of `sqrt((n - k_within) / (n - k_total))` because clubSandwich's `(n-1)/(n-p)` finite-sample correction counts ALL columns (intercept + treatment + unit dummies + time dummies = `k_total`) while WooldridgeDiD's `solve_ols` on the within-transformed design counts only the treatment-cell columns (`k_within`). On the 240-obs / 51-column R-parity fixture this is ~11%; on typical larger panels (n >> k_total) the gap shrinks to <2%. No public WooldridgeDiD code path exposes the `lm + CR1S` (CR1 cluster-robust on the full-dummy design) finite-sample correction — `vcov_type="hc2_bm"` routes to the CR2 Bell-McCaffrey sandwich on the full-dummy design (different variance estimator entirely), not CR1S. Users who need exact `lm + clubSandwich::vcovCR(type="CR1S")` parity must call `solve_ols` directly on a full-dummy design or fit via R. Same deviation pattern as SunAbraham PR #472 (`fixest::sunab` vs `lm + clubSandwich`). - `hc2_bm` — CR2 Bell-McCaffrey via auto-route to full-dummy design (`[intercept, X_design, unit_dummies, time_dummies]`), then `solve_ols(..., vcov_type="hc2_bm")` through the clubSandwich port (PR #475). FWL does NOT preserve the hat matrix; HC2 leverage + BM DOF require the full-projection design. Per-coefficient SE matches `clubSandwich::vcovCR(lm(...), cluster=~unit, type="CR2")` at atol=1e-10. Per-cell `(g, t)` inference fields use `coef_test()$df_Satt` Bell-McCaffrey DOF (pinned at atol=1e-6 from CI half-width inversion). Aggregated inference (overall ATT + `.aggregate("group" | "calendar" | "event")`) uses contrast-specific BM DOFs from `_compute_cr2_bm_contrast_dof` (matches R `Wald_test(constraints=matrix(w, 1), vcov=vcov_CR2, test="HTZ")$df_denom`); the overall ATT contrast DOF is computed at fit time, the other three aggregations lazily on each `.aggregate(...)` call from BM artifacts (the REDUCED kept-column `X` / `cluster_ids` / bread matrix + the reduced-space coef-index map) stored on the Results object — using the reduced design after rank-deficient drops keeps the bread non-singular and matches the subspace `solve_ols` actually estimated in. Fail-closed across all surfaces: when BM DOF is unavailable (helper raises or returns non-finite), the affected inference fields are NaN — not normal-theory fallback (per `feedback_bm_contrast_dof_fail_closed`). - `classical`, `hc2` — supported via auto-route to full-dummy AND auto-drop of the unit auto-cluster (one-way families don't compose with `cluster_ids` per the linalg validator). Set `self.cluster=None` (default) for these; explicit `cluster="state"` + one-way family raises at the linalg validator. SE matches `summary(lm(...))$coefficients` (classical) and `sandwich::vcovHC(type="HC2")` respectively. Per-cell + aggregate p-values/CIs use the residual DOF `n - rank(X)` (matches R `lm()` / `coef_test()` t-distribution under both classical OLS SE and `sandwich::vcovHC` defaults) — not normal-theory, so inference is correct under small samples. - `conley` (spatial-HAC, Conley 1999) — supported on the **OLS path** via the within-transform design (or the full-dummy design when `cohort_trends=True`, like the other full-dummy families — see the cohort-trends row below), threading the `conley_*` params through `solve_ols` / `conley.py` (`conley_lag_cutoff=0` = within-period spatial only; `>0` adds within-unit Bartlett serial — the panel-aware path, since `conley_time`/`conley_unit` are always supplied, not pooled cross-sectional). Reuses the already-`conleyreg`-validated machinery (no new variance code). The unit auto-cluster is dropped on the conley path (an explicit `cluster=` enables the spatial+cluster product kernel); `survey_design=` / `weights` / `n_bootstrap>0` are rejected, and `method ∈ {logit, poisson}` + conley remains rejected (the `method != "ols"` guard — a QMLE-on-pseudo-residuals Conley sandwich is a separate derivation). FWL-composability (the within-transform conley SE equals the full-dummy conley SE) is pinned in `tests/test_conley_vcov.py::TestConleyWooldridge::test_fwl_composability_vs_full_dummy`. @@ -2028,7 +2035,7 @@ where `g(·)` is the link inverse (logistic or exp), `η_i` is the individual li - Never-treated control only: Pre-treatment periods still estimable as placebo ATTs - **Note:** Poisson QMLE with cohort+time dummies (not unit dummies) is consistent even in short panels (Wooldridge 1999, JBES). The exponential mean function is unique in that incidental parameters from group dummies do not cause inconsistency. - **Note:** Logit path uses cohort×time additive dummies (not unit dummies) to avoid incidental parameters bias — a standard limitation of logit FE in short panels. This matches Stata `jwdid method(logit)` which uses `i.gvar i.tvar`. -- **Note:** Nonlinear methods (logit, Poisson) with `control_group="never_treated"` restrict the interaction matrix to post-treatment cells only. Pre-treatment placebo cells are OLS-only (where within-transformation absorbs FE). Including all (g,t) cells in the nonlinear design creates exact collinearity between cohort dummies and cell indicator sums, leading to a data-dependent normalization via QR dropping. +- **Note:** Nonlinear methods (logit, Poisson) with `control_group="never_treated"` restrict the interaction matrix to post-treatment cells only. Pre-treatment placebo cells are OLS-only (where within-transformation absorbs FE). Including all (g,t) cells in the nonlinear design creates exact collinearity between cohort dummies and cell indicator sums. The OLS path resolves this by omitting each cohort's `ref(g)` reference cell explicitly (W2025 Eq. 6.1/6.4); the nonlinear paths avoid it by restricting to post-treatment cells instead, since their explicit cohort + time dummies (no within-transformation) make the reference omission insufficient on its own. *Algorithm:* 1. Identify cohorts G and time periods T from data @@ -2069,8 +2076,8 @@ where `g(·)` is the link inverse (logistic or exp), `η_i` is the individual li - **Result attribute:** `WooldridgeDiDResults.cohort_trend_coefs: Dict[g → δ_g]` populated under `cohort_trends=True`; empty dict otherwise. - **Note:** Polynomial-trend extensions (`"quadratic"`, `"cubic"` per paper p. 2572 footnote) are NOT yet exposed — `cohort_trends` is a binary `True/False` flag for linear `dg_i · t` only. - **Note:** `cohort_trends=True` + `survey_design` is **NOT yet supported** (raises `NotImplementedError` at `fit()`). The full-dummy auto-route composed with the survey TSL variance has not been validated against R-parity goldens. Tracked in DEFERRED.md follow-up. -- **Note:** `cohort_trends=True` + `control_group="never_treated"` is **NOT yet supported** (raises `NotImplementedError` at `fit()`). The OLS + never_treated branch emits ALL `(g, t)` cells as treatment-cell indicator dummies (paper W2025 Section 4.4 placebo coverage); the appended `dg_i · t` trend columns are then linearly spanned by the per-cohort sum of those cell dummies, so the Section 8 trend specification is unidentified on this branch. Use `control_group="not_yet_treated"` (the default) for `cohort_trends=True`. Tracked in DEFERRED.md follow-up. -- **Note:** Identification + baseline normalization for `cohort_trend_coefs` on all-eventually-treated panels: when a never-treated cohort (`g = 0`) is present, **all** `G` treated cohorts get a `dg_i · t` interaction column and `cohort_trend_coefs[g]` reports each cohort's linear slope relative to the never-treated baseline (absorbed by time FE). When **no** never-treated cohort exists, the **last cohort's** trend column is dropped deterministically per paper W2025 Section 5.4 ("all variables in regression (5.3) involving `dT_i` get dropped"); that cohort serves as the trend baseline, and `cohort_trend_coefs` surfaces `G - 1` entries (the last cohort is absent — its slope is the baseline in deviation form). Mirrors the all-treated normalization the library applies to cohort × time interactions. +- **Note:** `cohort_trends=True` + `control_group="never_treated"` is **NOT yet supported** (raises `NotImplementedError` at `fit()`). The OLS + never_treated branch emits the `(g, t)` placebo cells (paper W2025 Section 4.4 placebo coverage) MINUS each cohort's `ref(g)` reference cell (W2025 Eq. 6.1/6.4; see the reference-period Note below). The appended `dg_i · t` trend columns are still spanned — jointly by the emitted cells and the unit fixed effects, which absorb `1{cohort=g}` and thereby recover the omitted reference — so the Section 8 trend specification remains unidentified on this branch. Use `control_group="not_yet_treated"` (the default) for `cohort_trends=True`. Tracked in DEFERRED.md follow-up. +- **Note:** Identification + baseline normalization for `cohort_trend_coefs` on all-eventually-treated panels: when a never-treated cohort (`g = 0`) is present, **all** `G` treated cohorts get a `dg_i · t` interaction column and `cohort_trend_coefs[g]` reports each cohort's linear slope relative to the never-treated baseline (absorbed by time FE). When **no** never-treated cohort exists, the **last cohort's** trend column is dropped deterministically per paper W2025 Section 5.4 ("all variables in regression (5.3) involving `dT_i` get dropped"); that cohort serves as the trend baseline, and `cohort_trend_coefs` surfaces `G - 1` entries (the last cohort is absent — its slope is the baseline in deviation form). **Note (deviation from W2025 Section 5.4):** the paper's rule is applied to the TREND columns only. The matching normalization for the cohort x time CELLS is NOT implemented, so on an all-eventually-treated panel the cells at fully-treated periods are jointly collinear with the time FE and `fit()` RAISES (the completeness gate, ledger `M-124`) rather than returning the relabeled contrasts it used to. The trend-drop path described here is therefore currently unreachable end-to-end; it is retained because implementing the cell half is what restores these fits (tracked in `TODO.md`, High). ### Deviations from the paper / from R / library extensions diff --git a/docs/methodology/papers/wooldridge-2025-review.md b/docs/methodology/papers/wooldridge-2025-review.md index 1744bad21..d9044c132 100644 --- a/docs/methodology/papers/wooldridge-2025-review.md +++ b/docs/methodology/papers/wooldridge-2025-review.md @@ -742,7 +742,7 @@ For nonlinear extensions (logit, Poisson, fractional outcomes, ASF counterfactua 10. **Heterogeneous-trends interpretation when pre-trends bleed into post-treatment.** The paper warns (Eq. 6.7 commentary): "If the violation of PT carries into the treated periods, including the pre-treatment indicators can actually exacerbate the bias compared with not including them." This is a methodological caveat about both leads-and-lags and the heterogeneous-trends Section 8 specification, not currently surfaced in REGISTRY. -11. **All-cohort-treated mechanics.** Section 5.4 explains that when all units are eventually treated and the last cohort `T` serves as control, "all variables in regression (5.3) (or its TWFE version) involving `dT_i` get dropped." The shipped library's `control_group="not_yet_treated"` handles this case automatically, but the mechanical dropping is not explicitly documented in user-facing docs. +11. **All-cohort-treated mechanics.** Section 5.4 explains that when all units are eventually treated and the last cohort `T` serves as control, "all variables in regression (5.3) (or its TWFE version) involving `dT_i` get dropped." The shipped library applies this rule to the cohort-TREND columns only; the matching drop for the cohort x time CELLS is not implemented, so `control_group="not_yet_treated"` on an all-eventually-treated panel now RAISES rather than estimating (previously it returned relabeled cohort contrasts under ATT(g,t) labels). Implementing the cell half is the open work. 12. **Cohort centering for `dg_i · x_i` interactions.** The paper notes (p. 2560) that centering `dg_i · x_i` around `x̄_g` makes the `dg_i` coefficient an "average selection effect" estimate. This is currently not exposed as a user-facing parameter in the shipped library; the existing implementation does NOT center the `dg_i · x_i` interactions by default (the cohort-centering applies only to the `w_it · dg_i · fs_t · ẍ_ig` treatment-interaction term per Eq. 5.2). This is an interpretive nuance, not a numerical issue for ATT estimates. diff --git a/docs/tutorials/16_wooldridge_etwfe.ipynb b/docs/tutorials/16_wooldridge_etwfe.ipynb index 06326aea0..fb279a2da 100644 --- a/docs/tutorials/16_wooldridge_etwfe.ipynb +++ b/docs/tutorials/16_wooldridge_etwfe.ipynb @@ -63,7 +63,7 @@ "cell_type": "markdown", "id": "e5f6a7b8", "metadata": {}, - "source": "## Basic OLS Estimation\n\nThe default `method='ols'` fits a single regression with:\n- Treatment interaction dummies (one per treatment cohort x post-treatment period cell)\n- Unit fixed effects (absorbed via within-transformation)\n- Time fixed effects (absorbed via within-transformation)\n\nWith `control_group='not_yet_treated'` (default), pre-treatment observations from treated units sit in the regression baseline alongside not-yet-treated controls. With `control_group='never_treated'`, pre-treatment interaction indicators are added so only never-treated units define the counterfactual baseline, and pre-treatment coefficients serve as placebo checks." + "source": "## Basic OLS Estimation\n\nThe default `method='ols'` fits a single regression with:\n- Treatment interaction dummies (one per treatment cohort x post-treatment period cell)\n- Unit fixed effects (absorbed via within-transformation)\n- Time fixed effects (absorbed via within-transformation)\n\n**The two control groups produce different designs, and the distinction matters below.**\n\nWith `control_group='not_yet_treated'` (the default, used in this section), only cells at `t >= g - anticipation` are emitted. Pre-treatment observations from treated units sit in the regression baseline alongside not-yet-treated controls, so there are **no placebo cells and no omitted reference cell** — the lag-only specification.\n\nWith `control_group='never_treated'`, pre-treatment interaction indicators are added so only never-treated units define the counterfactual baseline, and pre-treatment coefficients serve as placebo checks. That is the lead-and-lag specification of Wooldridge (2025) Eq. 6.1/6.4, and it is the one that **deliberately omits one cell per cohort — the reference period.** Eq. 6.1 excludes the `g − 1` indicator so that `s = g − 1` is the reference, which is what makes each coefficient a 2×2 DiD against that baseline. The cohort's full block of cell dummies would otherwise sum to its cohort indicator, which the unit fixed effects already absorb, so the design would be rank-deficient and an arbitrary cell — possibly a real post-treatment one — would be dropped instead. Stata `jwdid ... never` omits the same cell. On an unbalanced panel the omitted cell is that cohort's latest *observed* pre-period rather than `g − 1`." }, { "cell_type": "code", @@ -85,13 +85,7 @@ "cell_type": "markdown", "id": "a7b8c9d0", "metadata": {}, - "source": [ - "## Cohort×Time Cell Estimates ATT(g,t)\n", - "\n", - "The raw building blocks are ATT(g,t) — the treatment effect for cohort `g` at calendar time `t`. These are stored in `r.group_time_effects` and correspond to Stata's regression output table (`first_treat#year#c.__tr__`).\n", - "\n", - "Post-treatment cells have `t >= g`; pre-treatment cells (`t < g`) serve as placebo checks." - ] + "source": "## Cohort×Time Cell Estimates ATT(g,t)\n\nThe raw building blocks are ATT(g,t) — the treatment effect for cohort `g` at calendar time `t`. These are stored in `r.group_time_effects` and correspond to Stata's regression output table (`first_treat#year#c.__tr__`).\n\n`r` above is the **default** `not_yet_treated` fit, so it contains post-treatment cells (`t >= g`) only. Pre-treatment placebo cells belong to the `never_treated` specification, which the next cell fits explicitly so the two can be compared." }, { "cell_type": "code", @@ -120,21 +114,7 @@ "id": "c9d0e1f2", "metadata": {}, "outputs": [], - "source": [ - "# Also show pre-treatment placebo cells\n", - "print(\"Pre-treatment placebo ATT(g,t) cells (should be ~0 under parallel trends)\")\n", - "print(\"{:>8} {:>8} | {:>10} {:>10} {:>7} {:>7}\".format(\n", - " \"cohort\", \"year\", \"Coef.\", \"Std.Err.\", \"t\", \"P>|t|\"))\n", - "print(\"-\" * 60)\n", - "\n", - "for (g, t), v in sorted(r.group_time_effects.items()):\n", - " if t >= g:\n", - " continue\n", - " row = \"{:>8} {:>8} | {:>10.4f} {:>10.4f} {:>7.2f} {:>7.3f}\".format(\n", - " int(g), int(t), v['att'], v['se'], v['t_stat'], v['p_value']\n", - " )\n", - " print(row)" - ] + "source": "# Placebo cells exist only under control_group='never_treated' (the lead-and-lag\n# specification). The default not_yet_treated fit `r` above has none.\nprint(f\"default (not_yet_treated) pre-treatment cells: \"\n f\"{sorted(k for k in r.group_time_effects if k[1] < k[0])}\")\n\nr_never = WooldridgeDiD(control_group='never_treated').fit(\n data, outcome='outcome', unit='unit', time='period', cohort='first_treat')\n\nprint(\"\\nPre-treatment placebo ATT(g,t) cells (should be ~0 under parallel trends)\")\nprint(\"{:>8} {:>8} | {:>10} {:>10} {:>7} {:>7}\".format(\n \"cohort\", \"year\", \"Coef.\", \"Std.Err.\", \"t\", \"P>|t|\"))\nprint(\"-\" * 60)\n\nfor (g, t), v in sorted(r_never.group_time_effects.items()):\n if t >= g:\n continue\n print(\"{:>8} {:>8} | {:>10.4f} {:>10.4f} {:>7.2f} {:>7.3f}\".format(\n int(g), int(t), v['att'], v['se'], v['t_stat'], v['p_value']))\n\n# One cell per cohort is absent: the omitted reference period (g-1 here, since\n# this panel is balanced). Every coefficient above is measured against it.\nfor g in sorted(c for c in data['first_treat'].unique() if c > 0):\n present = {t for (gg, t) in r_never.group_time_effects if gg == g}\n missing = sorted(set(data['period'].unique()) - present)\n print(f\"cohort {g}: omitted reference cell -> {[(g, t) for t in missing]}\")" }, { "cell_type": "markdown", diff --git a/docs/v4-deprecations.yaml b/docs/v4-deprecations.yaml index 6600b8660..f4353621c 100644 --- a/docs/v4-deprecations.yaml +++ b/docs/v4-deprecations.yaml @@ -1327,6 +1327,32 @@ rows: test_ref: tests/test_aggregate_contract.py code_refs: [diff_diff/staggered.py, diff_diff/staggered_results.py, diff_diff/aggregation.py] notes: "balance_e moves from fit() onto aggregate() with [M-020]. Previously tracked only as the prose 'balance_e moves to aggregate() in the same PR' inside M-020's notes, which no test could assert - the same un-rowed-obligation class the gating-completeness amendment closed. Applies to event-study aggregation ONLY (the shipped code threads it nowhere else), so aggregate(type='simple'|'group', balance_e=...) raises rather than silently ignoring it. The other three balance_e sites (ImputationDiD, TwoStageDiD, EfficientDiD) get their own rows in the PRs that migrate them." + - id: M-124 + kind: behavior + group: etwfe-reference-period + old: "diff_diff.wooldridge:WooldridgeDiD.fit" + new: null + introduced_in: "3.9" + deprecated_in: null + removed_in: null + status: done + phase: 2 + test_ref: tests/test_wooldridge.py + code_refs: [diff_diff/wooldridge.py, docs/methodology/REGISTRY.md] + notes: "WooldridgeDiD.fit() now FAILS CLOSED whenever it cannot produce a finite overall ATT POINT ESTIMATE, instead of returning a result object whose overall_att is NaN - which reads as a completed estimate and silently defeats downstream aggregation and plotting. Scope is the POINT ESTIMATE only: overall_se and the inference fields may still be NaN on a successful fit when the VARIANCE alone is unidentifiable (hc2_bm with no Bell-McCaffrey DOF; aggregate(weights='cohort_share')), both of which fail closed through safe_inference() by design - so the guard checks overall_att and nothing else (codex R6 DT-1 corrected an earlier overstatement of this row). Enforced as ONE invariant at two levels rather than as per-symptom patches. (1) BUILD time: the pre-3.9 guard tested `X_int.shape[1] == 0`, strictly weaker than the error message it carried, because on the never_treated + OLS path `include_pre` also emits placebo (t < g) columns - so a cohort observed only before its own treatment start still produced a NON-empty design; it now conditions on post-treatment cells (t >= g - anticipation). (2) POST-SOLVE, in `_require_estimable_overall_att`, called before every WooldridgeDiDResults construction (OLS + both nonlinear fitters), which catches what the build-time check cannot see: coefficients removed by rank reduction, and cells that exist but all fall inside the anticipation window. The two levels are NOT redundant - the build-time notion of a treatment cell (t >= g - anticipation) is deliberately WIDER than the aggregation's (t >= g, anticipation leads excluded per W2025), so each catches cases the other misses. Three causes are distinguished with their own messages: no cell survived the solve; cells survived but none has t >= g; post cells survived with NaN coefficients. Surfaced by codex review R2 (finding M1). BOTH post-solve cases were verified PRE-EXISTING - identical all-NaN return, same warnings, at the pre-PR base commit 7e4026a0 - so this row records a deliberate fix of long-standing behavior, not a regression introduced by the reference-period work. Notably a design with TWO surviving treated cohorts can still be fully degenerate (neither post cell has a same-period comparison), so cohort count is not a valid proxy for comparison support; pinned by tests/test_wooldridge.py::TestOverallAttFailsClosed. Separate row from [M-123] because the trigger differs: M-123 changes WHICH ROWS are estimated (unidentified-cohort exclusion), this changes whether a degenerate design RETURNS or RAISES, and it fires on configurations with no unidentified cohort at all. introduced_in gates the 3.9 cut; deprecated_in stays null so the early-flip guard does not fire against the PR that ships it. No opt-out: the alternative is an all-NaN result presented as a successful fit. NOT ALL BRANCHES ARE TERMINAL: the 'no cell has t >= g' branch (every treatment cell inside the anticipation window) discards per-cell ATTs that ARE identified, and the 'post cells all NaN' branch is a post-hoc substitute for the same-period comparison-support diagnostic the estimator lacks. Both are STOPGAPS with follow-up rows in TODO.md; the invalid-weight and no-post-treatment-cell branches are terminal (genuinely unidentified). COMPLETENESS GATE (codex R8, P0): a finite overall ATT does NOT imply the reported cell set is complete. When rank reduction removes SOME treatment columns the survivors are silently re-identified as whatever contrast the reduced design supports and KEEP their ATT(g,t) labels - measured on an all-eventually-treated panel with a true ATT of 1.5 everywhere: ATT(3,8)=-0.0096, ATT(5,9)=-0.2393, overall_att=1.0023, and with rank_deficient_action='silent' no warning at all. _require_complete_cell_set now compares the cells the builder emitted against the coefficients that survived and fails closed on any loss, not gated on rank_deficient_action (that setting governs how rank warnings surface, not whether a mislabeled contrast may be returned as an ATT). CAPABILITY REGRESSION, deliberate: all-eventually-treated panels (no never-treated group) now RAISE where they previously returned those wrong numbers, including under survey designs and cohort_trends=True. W2025 Section 5.4 defines the correct treatment (last cohort serves as control, its columns dropped deliberately rather than by QR); the library already applies it to the cohort-TREND column but not to the cells, so implementing the cell half restores the capability - tracked in TODO.md at High priority." + - id: M-123 + kind: behavior + group: etwfe-reference-period + old: "diff_diff.wooldridge:WooldridgeDiD.fit" + new: null + introduced_in: "3.9" + deprecated_in: null + removed_in: null + status: done + phase: 2 + test_ref: tests/test_wooldridge.py + code_refs: [diff_diff/wooldridge.py, docs/methodology/REGISTRY.md, tests/test_etwfe_cs_stata_parity.py] + notes: "ETWFE reference-period normalization (issue #724). The saturated design now OMITS each cohort's reference cell explicitly - ref(g) = latest supported period before g - anticipation, per-cohort and positive-weight - per Wooldridge 2025 Eq. 6.1/6.4, anchored to Stata jwdid ... never. Previously the reference was emitted and generic QR rank detection dropped an arbitrary column, which on real mpdta silently removed ATT(2004,2004) and ATT(2006,2007) AND left every surviving coefficient measured against the wrong reference (errors of 0.010-0.041, up to 8x the 5e-3 material threshold). ESTIMATION-SAMPLE CHANGE, which is why this carries a row rather than landing as a bug-fix side effect: a cohort with NO valid reference is unidentified and its OBSERVATIONS are now excluded (with a warning naming the cohort), on the DEFAULT not_yet_treated path as well as never_treated. Retaining those rows - the pre-3.9 behavior - put a treated cohort into the omitted baseline and loaded its effect onto the time FE, moving other cohorts' ATTs by up to 0.0077. This falsifies the previously documented invariant that _filter_sample expresses the control-group choice through the design matrix and never by dropping rows, so the fitted sample and the full outcome column no longer share support by construction; the _suggest_nonlinear_method hint that cited that invariant remains correct because outcome TYPE is a property of the variable, not the sample. introduced_in gates the 3.9 cut ([M-091]/[M-092]/[M-122] pattern); deprecated_in stays null so the early-flip guard does not fire against the PR that ships it. No opt-out: the alternative is a silently contaminated estimate. BOUNDARY (codex R5): the exclusion is REFUSED, not performed, when survey_design= is supplied - NotImplementedError across ols/logit/poisson. Deleting rows under a complex design is naive subsetting rather than domain estimation: it removes their PSUs/strata from the Taylor-linearized meat and from df_survey = n_PSU - n_strata, so surviving ATTs would carry variance from a design the user never specified (measured on a two-stratum panel: df_survey 22 -> 14, finite SE, no signal). REGISTRY 'Subpopulation Analysis (Phase 6)' and Lumley (2004) 3.4 require zero-padding excluded rows while RETAINING strata/PSU; SurveyDesign.subpopulation() implements that contract but composing it here also needs the weighted within-transform to tolerate zero-weight units (shared machinery behind several estimators), so it is tracked in TODO.md rather than folded in. Relatedly the FULL SurveyDesign is now resolved against the PRE-exclusion sample for validation only - weights, weight_type, strata, PSU, FPC and nesting - because every check that ran after exclusion was blind to whatever the deleted rows contained." - id: M-122 kind: behavior group: aggregate-postfit diff --git a/tests/test_etwfe_cs_stata_parity.py b/tests/test_etwfe_cs_stata_parity.py new file mode 100644 index 000000000..854053c66 --- /dev/null +++ b/tests/test_etwfe_cs_stata_parity.py @@ -0,0 +1,358 @@ +"""Stata parity for WooldridgeDiD (ETWFE) and CallawaySantAnna ATT(g,t). + +Anchors BOTH staggered estimators against their canonical Stata implementations +-- ``jwdid`` (Rios-Avila, Wooldridge ETWFE) and ``csdid`` (Rios-Avila / +Sant'Anna, Callaway-Sant'Anna) -- on the genuine ``mpdta`` panel. + +WHY THIS ARM EXISTS +------------------- +``tests/test_wooldridge.py`` previously asserted that ETWFE ATT(g,t) EQUALS +CallawaySantAnna ATT(g,t) within ``5e-3``. That claim is FALSE on real data: at +``(g=2007, t=2007)`` the estimators differ by 0.0171 (-0.0431 vs -0.0261). The +assertion only ever passed because ``load_mpdta()`` was silently substituting a +synthetic, effect-homogeneous DGP when its source URL 404'd (issue #722), and on +that DGP the two estimators do coincide. + +Stata's ``jwdid`` and ``csdid`` reproduce the SAME disagreement, which is what +establishes it as a property of the estimators rather than a bug in either +implementation. So instead of one self-referential cross-check that validated +nothing, each estimator is pinned to its own external reference, and the +ETWFE-vs-CS gap is RECORDED rather than asserted away. + +The golden is committed (``benchmarks/data/etwfe_cs_stata_golden.json``) and the +panel is committed alongside it, so CI never needs Stata or network access -- +network dependence being the exact failure mode that produced the false +assertion in the first place. Regenerate with +``benchmarks/stata/generate_etwfe_cs_golden.do``. +""" + +import json +import re +import warnings +from pathlib import Path + +import numpy as np +import pandas as pd +import pytest + +from diff_diff import CallawaySantAnna, WooldridgeDiD + +_GOLDEN = Path(__file__).parent.parent / "benchmarks" / "data" / "etwfe_cs_stata_golden.json" +_PANEL = Path(__file__).parent.parent / "benchmarks" / "data" / "mpdta_stata_panel.csv" + +#: Upstream ``mpdta.csv`` digest. Pinned identically in the generator and in +#: ``diff_diff/datasets.py``; asserted here so a swapped panel cannot silently +#: retarget the parity. +_PANEL_SHA256 = "2283bea1221a152420f98dfa20f633c5d054ea51d881115c8cd702a97bcd3167" + +pytestmark = pytest.mark.skipif( + not (_GOLDEN.exists() and _PANEL.exists()), + reason="Stata golden/panel not present (regenerate with benchmarks/stata/generate_etwfe_cs_golden.do)", +) + + +def _panel() -> pd.DataFrame: + df = pd.read_csv(_PANEL) + return df.rename(columns={"first.treat": "first_treat"}) + + +def _golden() -> dict: + with open(_GOLDEN) as fh: + return json.load(fh) + + +def _stata_csdid_cells(golden: dict) -> dict: + """``g2004:t_2003_2007`` -> ``(2004, 2007)``, post-treatment cells only.""" + out = {} + for key, rec in golden["csdid"].items(): + m = re.match(r"g(\d+):t_(\d+)_(\d+)$", key) + if not m: + continue + g, _base, t = (int(x) for x in m.groups()) + if t >= g: # pre-treatment placebo cells are not compared here + out[(g, t)] = rec + return out + + +def _stata_jwdid_cells(golden: dict, block: str = "jwdid") -> dict: + """``2004bn.first_treat#2005.year#c.__tr__`` -> ``(2004, 2005)``. + + ``block`` selects the arm: ``"jwdid"`` (not-yet-treated, the default) or + ``"jwdid_never"`` (never-treated controls, the issue #724 anchor). Both use + the same coefficient-name shape. + """ + out = {} + for key, rec in golden[block].items(): + m = re.match(r"(\d+)b?n?\.first_treat#(\d+)b?n?\.year", key) + if not m: + continue # _cons and any non-cell terms + g, t = (int(x) for x in m.groups()) + out[(g, t)] = rec + return out + + +@pytest.fixture(scope="module") +def fits(): + df = _panel() + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + etwfe = WooldridgeDiD(method="ols", control_group="not_yet_treated").fit( + df, outcome="lemp", unit="countyreal", time="year", cohort="first_treat" + ) + cs = CallawaySantAnna(control_group="not_yet_treated").fit( + df, outcome="lemp", unit="countyreal", time="year", first_treat="first_treat" + ) + return etwfe, cs + + +def test_panel_matches_the_pinned_upstream_digest(): + """The parity is only meaningful against the authentic panel.""" + import hashlib + + got = hashlib.sha256(_PANEL.read_bytes()).hexdigest() + assert got == _PANEL_SHA256, f"committed panel digest {got} != pinned {_PANEL_SHA256}" + + +def test_golden_records_every_reference_implementation(): + golden = _golden() + assert _stata_csdid_cells(golden), "golden has no csdid cells" + assert _stata_jwdid_cells(golden, "jwdid"), "golden has no jwdid cells" + assert _stata_jwdid_cells(golden, "jwdid_never"), "golden has no jwdid_never cells" + assert golden["meta"]["source_sha256"] == _PANEL_SHA256 + + +def test_golden_records_a_usable_drift_identifier_for_every_ssc_dependency(): + """The SSC packages are UNPINNED, so this metadata is the only thing between + a silent upstream change and an unexplained golden diff. + + ``"MISSING"`` means the package was absent at generation time; ``"unknown"`` + means its ``.ado`` carried no parseable ``*!`` version header, which + silently disables drift detection for that dependency -- ``drdid`` was in + exactly that state. The generator now falls back to a checksum of the + ``.ado``, which is opaque but still changes when upstream does. Codex R6. + """ + versions = _golden()["meta"]["ssc_versions"] + assert set(versions) == {"drdid", "csdid", "jwdid", "hdfe"} + for pkg, ident in versions.items(): + assert ident not in ("unknown", "MISSING", "", None), ( + f"SSC dependency {pkg!r} has no usable drift identifier ({ident!r}); " + "regenerate the golden with benchmarks/stata/generate_etwfe_cs_golden.do" + ) + + +@pytest.fixture(scope="module") +def never_fit(): + """``never_treated`` OLS on the authentic panel - the issue #724 surface.""" + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + return WooldridgeDiD(method="ols", control_group="never_treated").fit( + _panel(), outcome="lemp", unit="countyreal", time="year", cohort="first_treat" + ) + + +class TestNeverTreatedVsStataJwdid: + """Issue #724: the library returned 5 of 7 post-treatment cells because the + ETWFE reference period was left to QR rank detection instead of being + omitted explicitly. ``jwdid ... never`` is the external anchor -- it omits + ``g-1`` per cohort, matching Wooldridge (2025) Eq. 6.1/6.4. + """ + + def test_cell_set_matches_jwdid_exactly(self, never_fit): + """The actual #724 regression: WHICH cells come back, not their values. + + Asserted as a SET, never against a column name -- which cell QR + sacrificed was panel-dependent, so naming one would pin an artifact of + this fixture rather than the contract. + """ + stata = set(_stata_jwdid_cells(_golden(), "jwdid_never")) + ours = set(never_fit.group_time_effects) + assert ours == stata, ( + f"cell set differs from jwdid: only-ours={sorted(ours - stata)}, " + f"only-stata={sorted(stata - ours)}" + ) + + def test_all_post_treatment_cells_present(self, never_fit): + """Pre-fix this returned 5 of 7, with (2004, 2004) absent entirely.""" + post = {k for k in never_fit.group_time_effects if k[1] >= k[0]} + assert post == { + (2004, 2004), + (2004, 2005), + (2004, 2006), + (2004, 2007), + (2006, 2006), + (2006, 2007), + (2007, 2007), + } + + def test_no_rank_deficiency(self): + """The reference is now omitted by construction, so nothing is left for + QR to drop -- the warning that accompanied #724 must be gone.""" + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + WooldridgeDiD(method="ols", control_group="never_treated").fit( + _panel(), outcome="lemp", unit="countyreal", time="year", cohort="first_treat" + ) + rank = [w for w in caught if "Rank-deficient" in str(w.message)] + assert rank == [], f"unexpected rank deficiency: {[str(w.message)[:120] for w in rank]}" + + def test_omitted_cell_is_the_g_minus_1_reference(self, never_fit): + """W2025 Eq. 6.1 omits ``g-1``; jwdid does the same. Derived from the + golden rather than hardcoded, so it tracks the reference implementation. + """ + stata = _stata_jwdid_cells(_golden(), "jwdid_never") + by_cohort: dict = {} + for g, t in stata: + by_cohort.setdefault(g, set()).add(t) + all_times = {2003, 2004, 2005, 2006, 2007} + for g, times in by_cohort.items(): + missing = all_times - times + assert missing == {g - 1}, f"cohort {g} omits {missing}, expected {{{g - 1}}}" + assert (g, g - 1) not in never_fit.group_time_effects + + def test_att_matches_jwdid(self, never_fit): + stata = _stata_jwdid_cells(_golden(), "jwdid_never") + for key, rec in stata.items(): + np.testing.assert_allclose( + never_fit.group_time_effects[key]["att"], + rec["att"], + rtol=0, + atol=1e-9, + err_msg=f"never_treated ATT{key} != Stata jwdid never", + ) + + def test_se_gap_matches_the_not_yet_treated_arm(self, never_fit): + """The SE gap is a control-group-INDEPENDENT finite-sample factor. + + The golden stores ``jwdid_never`` SEs, so the path this PR actually + changed gets its own SE gate rather than inheriting the not-yet-treated + one. Measured here across all 12 never-treated cells (7 post + 5 + placebo): the ratio is uniform to ~1e-14 -- two orders tighter than the + 1e-5 the sibling test allows -- and its mean, 1.00100634, is the SAME + factor the not-yet-treated arm shows. + + That agreement is the point. Whatever produces the gap does NOT depend + on the control group, on the cell being post-treatment vs placebo, or + on the cell count (12 here vs 7 there), which constrains any mechanism + later proposed for it (tracked in TODO.md). As with the sibling test, + no closed form is asserted -- only the observed factor. + """ + stata = _stata_jwdid_cells(_golden(), "jwdid_never") + ratios = [rec["se"] / never_fit.group_time_effects[key]["se"] for key, rec in stata.items()] + assert len(ratios) == 12, f"expected 12 never_treated cells, got {len(ratios)}" + assert max(ratios) - min(ratios) < 1e-5, f"SE ratio is not uniform: {ratios}" + mean_ratio = float(np.mean(ratios)) + assert mean_ratio > 1.0, "library SE should be below jwdid's, not above" + np.testing.assert_allclose(mean_ratio, 1.001006, rtol=1e-4) + + +class TestETWFEvsStataJwdid: + def test_att_matches_jwdid(self, fits): + """Point estimates are the tight gate.""" + etwfe, _ = fits + stata = _stata_jwdid_cells(_golden()) + compared = 0 + for key, rec in stata.items(): + assert key in etwfe.group_time_effects, f"library is missing cell {key}" + np.testing.assert_allclose( + etwfe.group_time_effects[key]["att"], + rec["att"], + rtol=0, + atol=1e-6, + err_msg=f"ETWFE ATT{key} != Stata jwdid", + ) + compared += 1 + assert compared == 7, f"expected 7 ETWFE cells, compared {compared}" + + def test_se_is_uniformly_below_jwdid(self, fits): + """MEASURED deviation, recorded without asserting a mechanism. + + Every ``hc1`` SE is smaller than ``jwdid``'s by a factor that is + UNIFORM across cells (spread < 1e-6 within a fit), so it is a + finite-sample convention difference rather than noise or a per-cell + bug. The magnitude shrinks as the cluster count grows -- 1.0280 at + G=20, 1.0132 at G=40, 1.0010 at G=500 -- so the library is + systematically anti-conservative relative to the reference, negligibly + with many clusters and materially with few. + + Deliberately NOT asserted: a closed form. The gap tracks + ``sqrt(G/(G-1))`` closely but is consistently ABOVE it (by ~0.2% at + G=20, ~0.001% at G=500), and ``solve_ols`` already applies + ``(G/(G-1)) * ((n-1)/(n-k))`` -- so "the library omits G/(G-1)" is + NOT the explanation, and no verified formula has been derived. Pinning + a wrong mechanism here would be worse than pinning none: this test + locks the OBSERVED gap at this panel's G, and the derivation is tracked + in TODO.md. See the REGISTRY WooldridgeDiD note. + + SCOPE: this pins the FULL-PANEL ratio (G=500) only. The smaller-G + figures quoted in the REGISTRY note were measured ad hoc on subsampled + panels and are not gated here, so a regression in the few-cluster + behavior -- where the gap is materially largest -- would not fail CI. + Committing that ladder is listed as a required artifact on the + derivation row in TODO.md. + """ + etwfe, _ = fits + stata = _stata_jwdid_cells(_golden()) + + ratios = [rec["se"] / etwfe.group_time_effects[key]["se"] for key, rec in stata.items()] + assert len(ratios) == 7 + # Uniform across every cell: a finite-sample factor, not noise. + assert max(ratios) - min(ratios) < 1e-5, f"SE ratio is not uniform: {ratios}" + # Stata is LARGER (we are anti-conservative), by the amount observed on + # this G=500 panel. Regenerating against a different panel moves this. + mean_ratio = float(np.mean(ratios)) + assert mean_ratio > 1.0, "library SE should be below jwdid's, not above" + np.testing.assert_allclose(mean_ratio, 1.001006, rtol=1e-4) + + +class TestCallawaySantAnnaVsStataCsdid: + def test_att_matches_csdid(self, fits): + _, cs = fits + stata = _stata_csdid_cells(_golden()) + compared = 0 + for key, rec in stata.items(): + assert key in cs.group_time_effects, f"library is missing cell {key}" + np.testing.assert_allclose( + cs.group_time_effects[key]["effect"], + rec["att"], + rtol=0, + atol=1e-6, + err_msg=f"CS ATT{key} != Stata csdid", + ) + compared += 1 + assert compared == 7, f"expected 7 CS cells, compared {compared}" + + def test_se_matches_csdid(self, fits): + """Unlike ETWFE, the CS SEs agree outright - which is what makes the + ETWFE SE gap a real finding rather than a convention difference the + whole library shares.""" + _, cs = fits + stata = _stata_csdid_cells(_golden()) + for key, rec in stata.items(): + np.testing.assert_allclose( + cs.group_time_effects[key]["se"], + rec["se"], + rtol=1e-5, + atol=0, + err_msg=f"CS SE{key} != Stata csdid", + ) + + +def test_etwfe_and_cs_genuinely_disagree_on_real_data(fits): + """The claim the retired cross-check got wrong. + + Both estimators match their own reference implementation exactly, and Stata's + two implementations disagree with each other by the same amount - so this + divergence is a property of the estimands, not a defect. Pinned so nobody + re-derives an equivalence assertion from a homogeneous synthetic DGP. + """ + etwfe, cs = fits + shared = set(etwfe.group_time_effects) & set(cs.group_time_effects) + gaps = { + k: abs(etwfe.group_time_effects[k]["att"] - cs.group_time_effects[k]["effect"]) + for k in shared + } + worst_key = max(gaps, key=lambda k: gaps[k]) + assert worst_key == (2007, 2007), f"expected the terminal cell to diverge most, got {worst_key}" + # Far beyond the 5e-3 the retired test asserted. + assert gaps[worst_key] > 5e-3 + np.testing.assert_allclose(gaps[worst_key], 0.017052, rtol=1e-3) diff --git a/tests/test_methodology_wooldridge.py b/tests/test_methodology_wooldridge.py index 303c1d7ba..cb89cdc2f 100644 --- a/tests/test_methodology_wooldridge.py +++ b/tests/test_methodology_wooldridge.py @@ -570,11 +570,12 @@ def test_pols_runs_with_never_treated_control_group(self) -> None: Paper Section 4.4 / Procedure 4.1 establishes that POLS / cohort imputation is consistent for ATT identification under either never-treated controls OR last-cohort-as-control. The library's - OLS + never_treated branch includes ALL (g, t) cells as placebos - (wooldridge.py:200) which produces a saturated design that can - hit rank-deficient column drops on small panels — the specific - ATT values from the drop-affected fit aren't a paper-equivalence - property. This test locks only the SURFACE invariant: the + OLS + never_treated branch includes the (g, t) placebo cells minus + each cohort's reference cell (``_build_interaction_matrix``). It + formerly emitted the reference too and hit rank-deficient column + drops (issue #724); those are now gone, but the specific ATT values + still aren't a paper-equivalence property. This test locks only the + SURFACE invariant: the never_treated path completes + returns a well-formed Results object (no exception, finite overall_att, ≥1 finite treated cell). Numerical recovery of ATTs is exercised by @@ -1485,12 +1486,13 @@ def test_cohort_trends_true_plus_weights_cohort_share_simple_excludes_trend_colu def test_cohort_trends_true_rejects_never_treated_control_group(self) -> None: """CI R9 P1 fix: ``cohort_trends=True`` + ``control_group="never_treated"`` raises. - The OLS + never_treated branch emits ALL (g, t) cells as - treatment-cell dummies (paper W2025 Section 4.4 placebo - coverage). For each treated cohort g, the trend column - ``dg_i · t`` is fully spanned by the per-cohort sum of those - cell dummies, so ``dg_i · t`` is unidentified. The library - fail-closes the combination with ``NotImplementedError``. + The OLS + never_treated branch emits the (g, t) placebo cells + (paper W2025 Section 4.4 placebo coverage) minus each cohort's + reference cell. The trend column ``dg_i · t`` is STILL spanned — + jointly by the emitted cells and the unit fixed effects, which + absorb ``1{cohort=g}`` and recover the omitted reference — so + ``dg_i · t`` remains unidentified and the library fail-closes the + combination with ``NotImplementedError``. """ rng = np.random.default_rng(_BASE_SEED_SECTION8 + 23) panel = _make_heterogeneous_trends_panel(rng, n_per_cohort=80, sigma=0.05) @@ -1747,8 +1749,8 @@ def test_plot_event_study_propagates_weights_kwarg(self) -> None: "leads." ) - def test_cohort_trends_true_all_treated_drops_last_cohort_trend(self) -> None: - """CI R4 P1 fix: ``cohort_trends=True`` on all-eventually-treated panel drops last-cohort trend. + def test_cohort_trends_true_all_treated_panel_is_refused(self) -> None: + """``cohort_trends=True`` on an all-eventually-treated panel is REFUSED. Paper W2025 Section 5.4: when all units are eventually treated and the last cohort serves as control, "all variables in @@ -1774,33 +1776,24 @@ def test_cohort_trends_true_all_treated_drops_last_cohort_trend(self) -> None: 3, 4, ], f"DGP precondition: treated cohorts should be [3, 4], got {treated_cohorts}" - with warnings.catch_warnings(): - warnings.filterwarnings("ignore", category=UserWarning) - res = WooldridgeDiD(method="ols", cohort_trends=True).fit( + # The trend-drop path is currently UNREACHABLE through fit(): on an + # all-eventually-treated panel the treatment CELLS at fully-treated + # periods are jointly collinear with the time FE, and since codex R8 a + # lost cell fails closed rather than returning relabeled contrasts. The + # last-cohort trend drop below is the same W2025 Section 5.4 + # normalization applied to the TREND columns only -- applying it to the + # cells is what would make these panels estimable again (TODO.md). Until + # then this test pins the refusal, so the trend logic is not silently + # deleted while it is untestable end-to-end. + with pytest.raises(ValueError, match="not identified and were removed"): + WooldridgeDiD(method="ols", cohort_trends=True).fit( full_panel, outcome="y", unit="unit", time="time", cohort="cohort", ) - # Last cohort (g=4) dropped per paper Section 5.4 — only g=3 in dict - assert set(res.cohort_trend_coefs.keys()) == {3}, ( - f"cohort_trend_coefs should drop the last cohort (g=4) on " - f"all-treated panels; got keys={set(res.cohort_trend_coefs.keys())}" - ) - assert np.isfinite( - res.cohort_trend_coefs[3] - ), f"cohort_trend_coefs[3]={res.cohort_trend_coefs[3]} should be finite" - # Stable ATT output: at least one treated cell estimated cleanly - treated_cells_finite = [ - res.group_time_effects[k]["att"] - for k in res.group_time_effects - if k[0] > 0 and k[1] >= k[0] and np.isfinite(res.group_time_effects[k]["att"]) - ] - assert len(treated_cells_finite) >= 1, ( - "all-treated panel + cohort_trends=True should produce at " - "least one finite treated-cell ATT" - ) + return def test_cohort_trends_true_with_never_treated_keeps_all_cohort_trends(self) -> None: """CI R4 P1 fix companion: with never-treated baseline, all G cohorts surface. diff --git a/tests/test_v4_matrix.py b/tests/test_v4_matrix.py index 5d504a6f4..d089d42eb 100644 --- a/tests/test_v4_matrix.py +++ b/tests/test_v4_matrix.py @@ -117,7 +117,7 @@ # Ids are never reused and terminal rows are never # deleted, so the ledger only grows - raise the floor when rows are added; a # lower parse count means scanner/format drift or an illegal row deletion. -ROW_COUNT_FLOOR = 101 +ROW_COUNT_FLOOR = 103 # Committed snapshot of the shipped id set ("ids are never deleted or reused" # contract - a delete-one-add-one edit keeps the count above the floor but trips @@ -129,7 +129,10 @@ # (97,115) = its completeness sweep over module-level public functions # (twowayfeweights, three HAD pretest entry points, trim_weights) plus the # dCDH results mirror and the fourth `robust` site; (117,117)/(122,122) = -# Phase 2b PR 1 (balance_e onto aggregate(), AggregationResult). M-116 and +# Phase 2b PR 1 (balance_e onto aggregate(), AggregationResult); (123,124) = +# the ETWFE reference-period fix (unidentified-cohort exclusion, and the +# fail-closed guard on a design with no estimable post-treatment cell). +# M-116 and # M-118..M-121 are reserved for the later 2b PRs, not deleted - ids are # never reused, so a gap here is intentional. _INITIAL_ID_RANGES = [ @@ -145,7 +148,7 @@ (94, 96), (97, 115), (117, 117), - (122, 122), + (122, 124), ] EXPECTED_INITIAL_IDS = frozenset( f"M-{n:03d}" for lo, hi in _INITIAL_ID_RANGES for n in range(lo, hi + 1) @@ -547,7 +550,7 @@ def test_initial_ids_never_deleted(): public-function completeness sweep).""" missing = sorted(EXPECTED_INITIAL_IDS - set(_ROW_IDS)) assert not missing, f"ledger rows deleted (ids are permanent): {missing}" - assert len(EXPECTED_INITIAL_IDS) == 101 + assert len(EXPECTED_INITIAL_IDS) == 103 def test_version_tuple_pads_to_three_components(): diff --git a/tests/test_wooldridge.py b/tests/test_wooldridge.py index a10b06f9f..c5c53ef56 100644 --- a/tests/test_wooldridge.py +++ b/tests/test_wooldridge.py @@ -224,7 +224,7 @@ def test_filter_sample_never_treated(self): def test_build_interaction_matrix_columns(self): df = _make_panel() filtered = _filter_sample(df, "unit", "time", "cohort", "not_yet_treated", anticipation=0) - X_int, col_names, gt_keys = _build_interaction_matrix( + X_int, col_names, gt_keys, _ = _build_interaction_matrix( filtered, cohort="cohort", time="time", anticipation=0 ) # Each column should be a valid (g, t) pair with t >= g @@ -234,7 +234,7 @@ def test_build_interaction_matrix_columns(self): def test_build_interaction_matrix_binary(self): df = _make_panel() filtered = _filter_sample(df, "unit", "time", "cohort", "not_yet_treated", anticipation=0) - X_int, col_names, gt_keys = _build_interaction_matrix( + X_int, col_names, gt_keys, _ = _build_interaction_matrix( filtered, cohort="cohort", time="time", anticipation=0 ) # All values should be 0 or 1 @@ -482,11 +482,17 @@ def test_ols_att_sign_direction(self): from diff_diff.datasets import load_mpdta df = load_mpdta() - # never_treated with within-transformation can produce collinear - # pre-treatment interactions; use silent to avoid warnings - est = WooldridgeDiD(control_group="never_treated", rank_deficient_action="silent") - r = est.fit(df, outcome="lemp", unit="countyreal", time="year", cohort="first_treat") + # No `rank_deficient_action="silent"` here any more. That workaround was + # added because never_treated emitted the reference cell and left QR to + # drop something arbitrary (#724); the reference is now omitted by + # construction, so a clean fit must produce NO rank warning. + est = WooldridgeDiD(control_group="never_treated") + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + r = est.fit(df, outcome="lemp", unit="countyreal", time="year", cohort="first_treat") assert np.isfinite(r.overall_att) + rank = [w for w in caught if "Rank-deficient" in str(w.message)] + assert rank == [], f"unexpected rank deficiency: {[str(w.message)[:100] for w in rank]}" def test_never_treated_produces_event_effects_with_placebo_leads(self): """With never_treated, event aggregation should include negative event @@ -571,34 +577,6 @@ def test_poisson_delta_method_se_finite(self): assert cell["se"] >= 0, f"Negative SE at {key}" assert np.isfinite(cell["se"]), f"Non-finite SE at {key}" - def test_ols_etwfe_att_matches_callaway_santanna(self): - """OLS ETWFE ATT(g,t) equals CallawaySantAnna ATT(g,t) (Proposition 3.1).""" - from diff_diff import CallawaySantAnna - from diff_diff.datasets import load_mpdta - - mpdta = load_mpdta() - - etwfe = WooldridgeDiD(method="ols", control_group="not_yet_treated") - cs = CallawaySantAnna(control_group="not_yet_treated") - - er = etwfe.fit(mpdta, outcome="lemp", unit="countyreal", time="year", cohort="first_treat") - cr = cs.fit( - mpdta, outcome="lemp", unit="countyreal", time="year", first_treat="first_treat" - ) - - matched = 0 - for key, effect in er.group_time_effects.items(): - if key in cr.group_time_effects: - cs_att = cr.group_time_effects[key]["effect"] - np.testing.assert_allclose( - effect["att"], - cs_att, - atol=5e-3, - err_msg=f"ATT mismatch at {key}: ETWFE={effect['att']:.4f}, CS={cs_att:.4f}", - ) - matched += 1 - assert matched > 0, "No matching (g,t) keys found between ETWFE and CS" - class TestExports: def test_top_level_import(self): @@ -701,12 +679,28 @@ def test_xgvar_fit_runs(self): class TestAllEventuallyTreated: def test_no_never_treated_not_yet_treated_control(self): - """All units eventually treated, using not_yet_treated control group. - - With not_yet_treated control, the latest cohort's post-treatment - cells are rank-deficient (no controls remain). The estimator drops - those columns, so we check that at least the earlier cohort cells - produce finite ATT effects and the overall ATT is computed from them. + """All units eventually treated: the fit is REFUSED, not approximated. + + This test previously asserted that the earlier cohorts' cells came back + finite once the latest cohort's rank-deficient columns were dropped. + They were finite and WRONG. Measured on this DGP (true ATT = 1.5 for + every post cell) before the completeness gate:: + + ATT(3, 8) = -0.0096 ATT(3, 9) = -0.1038 + ATT(5, 8) = -0.0214 ATT(5, 9) = -0.2393 + overall_att = 1.0023 (true 1.5) + + Once QR drops one of the jointly-collinear cells at a period where every + unit is treated, the survivors identify contrasts BETWEEN treated + cohorts while keeping their ATT(g, t) labels. Asserting finiteness is + exactly what let that through (codex R8 P0). + + W2025 Section 5.4 does define the right answer for these panels -- the + last cohort serves as the control, so its columns are dropped + DELIBERATELY rather than by QR, which leaves the rest identified. The + library already does this for the cohort-TREND column but not for the + cells, so until that lands the honest behavior is to refuse. Tracked in + TODO.md; landing it turns this test back into a fit. """ rng = np.random.default_rng(7) rows = [] @@ -725,15 +719,8 @@ def test_no_never_treated_not_yet_treated_control(self): rows.append({"unit": u, "time": t, "cohort": cohort, "y": y}) df = pd.DataFrame(rows) est = WooldridgeDiD(control_group="not_yet_treated") - r = est.fit(df, outcome="y", unit="unit", time="time", cohort="cohort") - # At least some cells should have finite ATT - finite_cells = [k for k, v in r.group_time_effects.items() if np.isfinite(v["att"])] - assert len(finite_cells) > 0 - # Early cohort (g=3) should have identifiable effects - early_finite = [k for k in finite_cells if k[0] == 3] - assert len(early_finite) > 0 - for k in early_finite: - assert np.isfinite(r.group_time_effects[k]["att"]) + with pytest.raises(ValueError, match="not identified and were removed"): + est.fit(df, outcome="y", unit="unit", time="time", cohort="cohort") class TestEmptyCells: @@ -1092,7 +1079,7 @@ def test_parity_with_dummy_ols_not_yet_treated(self, unbalanced_data): # Build explicit dummy regression on same sample sample = _filter_sample(df, "unit", "time", "cohort", "not_yet_treated", 0) - X_int, _, gt_keys = _build_interaction_matrix( + X_int, _, gt_keys, _ = _build_interaction_matrix( sample, cohort="cohort", time="time", @@ -1199,9 +1186,13 @@ def test_interaction_matrix_fewer_cols_for_nonlinear(self): rows.append({"unit": u, "time": t, "cohort": g, "y": 0.0}) df = pd.DataFrame(rows) - X_ols, _, _ = _build_interaction_matrix(df, "cohort", "time", 0, "never_treated", "ols") - X_logit, _, _ = _build_interaction_matrix(df, "cohort", "time", 0, "never_treated", "logit") - X_nyt, _, _ = _build_interaction_matrix(df, "cohort", "time", 0, "not_yet_treated", "ols") + X_ols, _, _, _ = _build_interaction_matrix(df, "cohort", "time", 0, "never_treated", "ols") + X_logit, _, _, _ = _build_interaction_matrix( + df, "cohort", "time", 0, "never_treated", "logit" + ) + X_nyt, _, _, _ = _build_interaction_matrix( + df, "cohort", "time", 0, "not_yet_treated", "ols" + ) # OLS never_treated > nonlinear never_treated == not_yet_treated assert X_ols.shape[1] > X_logit.shape[1] assert X_logit.shape[1] == X_nyt.shape[1] @@ -1247,7 +1238,7 @@ def test_ols_covariate_parity_with_full_basis_dummy_ols(self, cov_data): # Build explicit-dummy regression with full basis sample = _filter_sample(df, "unit", "time", "cohort", "not_yet_treated", 0) - X_int, _, gt_keys = _build_interaction_matrix( + X_int, _, gt_keys, _ = _build_interaction_matrix( sample, "cohort", "time", 0, "not_yet_treated", "ols" ) n_int = X_int.shape[1] @@ -1598,8 +1589,13 @@ def test_poisson_zero_weight_cell(self, survey_panel): from diff_diff.survey import SurveyDesign df = survey_panel.copy() - # Zero out weights for one treated cohort so some cells have zero weight - df.loc[df["cohort"] == 3, "weight"] = 0.0 + # Zero out ONE treated cell's weights. Deliberately NOT the whole cohort: + # that would leave cohort 3 with no supported reference period, and + # exclusion under a survey design is now refused outright (see + # test_survey_plus_unidentified_cohort_is_refused). Cohort 3 keeps its + # t=1,2 pre-periods, so this still exercises what the test is for -- + # a zero-weight TREATED CELL inside an otherwise identified design. + df.loc[(df["cohort"] == 3) & (df["time"] == 5), "weight"] = 0.0 sd = SurveyDesign(weights="weight", strata="stratum", psu="unit") r = WooldridgeDiD(method="poisson").fit( df, @@ -1612,23 +1608,32 @@ def test_poisson_zero_weight_cell(self, survey_panel): assert np.isfinite(r.overall_att) assert np.isfinite(r.overall_se) - def test_ols_survey_rank_deficient(self, survey_panel): - """Survey OLS handles rank-deficient all-eventually-treated designs.""" + def test_ols_survey_rank_deficient_now_fails_closed(self, survey_panel): + """An all-eventually-treated survey design is REFUSED, not approximated. + + This test previously asserted only that `overall_att` / `overall_se` + were finite on a deliberately rank-deficient design. They were finite + and WRONG: once rank reduction drops one of the jointly-collinear + final-period cells, the survivors identify a contrast between two + treated cohorts and keep their ATT(g, t) labels. Measured on an + equivalent panel with true effects 1.0 and 3.0, the surviving cell came + back as -1.94. Asserting finiteness is what let that through, so the + assertion is now the refusal itself (codex R8 P0). + """ from diff_diff.survey import SurveyDesign # Remove never-treated (cohort=0) to create rank-deficient design df = survey_panel[survey_panel["cohort"] > 0].copy() sd = SurveyDesign(weights="weight", strata="stratum", psu="unit") - r = WooldridgeDiD(control_group="not_yet_treated").fit( - df, - outcome="y", - unit="unit", - time="time", - cohort="cohort", - survey_design=sd, - ) - assert np.isfinite(r.overall_att) - assert np.isfinite(r.overall_se) + with pytest.raises(ValueError, match="not identified and were removed"): + WooldridgeDiD(control_group="not_yet_treated").fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=sd, + ) def test_ols_survey_zero_weight_unit_rejected(self, survey_panel): """Zero-weight unit raises ValueError before within_transform.""" @@ -1653,7 +1658,13 @@ def test_logit_survey_zero_weight_cell(self, survey_panel): from diff_diff.survey import SurveyDesign df = survey_panel.copy() - df.loc[df["cohort"] == 3, "weight"] = 0.0 + # Zero out ONE treated cell's weights. Deliberately NOT the whole cohort: + # that would leave cohort 3 with no supported reference period, and + # exclusion under a survey design is now refused outright (see + # test_survey_plus_unidentified_cohort_is_refused). Cohort 3 keeps its + # t=1,2 pre-periods, so this still exercises what the test is for -- + # a zero-weight TREATED CELL inside an otherwise identified design. + df.loc[(df["cohort"] == 3) & (df["time"] == 5), "weight"] = 0.0 sd = SurveyDesign(weights="weight", strata="stratum", psu="unit") r = WooldridgeDiD(method="logit").fit( df, @@ -2011,15 +2022,39 @@ def test_hc2_bm_plus_bootstrap_rank_deficient(self): n_units, n_periods = 20, 8 units = np.repeat(np.arange(n_units), n_periods) periods = np.tile(np.arange(1, n_periods + 1), n_units) - cohorts = rng.choice([3, 5, 7], size=n_units) + # Rank deficiency is induced by a DUPLICATED COVARIATE, not by dropping + # treatment cells. The panel keeps a never-treated group so every + # ATT(g, t) stays identified: a lost treatment cell now fails closed + # (codex R8 P0), and it always meant the survivors were relabeled + # contrasts -- which is not what this test is about. The subject here is + # the coef_offset / dropped-column indexing in the bootstrap closure, + # and a collinear covariate exercises exactly that on a design whose + # estimands are sound. + cohorts = rng.choice([0, 3, 5, 7], size=n_units) cohort_per_obs = cohorts[units] - tau = np.where(periods >= cohort_per_obs, 0.5 + 0.2 * (periods - cohort_per_obs), 0.0) + treated = (cohort_per_obs > 0) & (periods >= cohort_per_obs) + tau = np.where(treated, 0.5 + 0.2 * (periods - cohort_per_obs), 0.0) y = 1.0 + 0.1 * periods + tau + 0.1 * rng.normal(size=len(units)) - df = pd.DataFrame({"unit": units, "time": periods, "cohort": cohort_per_obs, "y": y}) + xc = rng.normal(size=len(units)) + df = pd.DataFrame( + { + "unit": units, + "time": periods, + "cohort": cohort_per_obs, + "y": y, + "xc": xc, + "xc_dup": xc, # exactly collinear -> one covariate column dropped + } + ) with warnings.catch_warnings(): warnings.simplefilter("ignore", UserWarning) res = WooldridgeDiD(method="ols", vcov_type="hc2_bm", n_bootstrap=50, seed=0).fit( - df, outcome="y", unit="unit", time="time", cohort="cohort" + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + exovar=["xc", "xc_dup"], ) assert np.isfinite(res.overall_att) assert np.isfinite(res.overall_se) @@ -2232,28 +2267,47 @@ def test_aggregate_calendar_under_hc2_bm_uses_bm_contrast_dof(self): assert np.isfinite(eff["conf_int"][0]) assert np.isfinite(eff["conf_int"][1]) - def test_hc2_bm_handles_rank_deficient_all_eventually_treated(self): - """All-eventually-treated panel with not_yet_treated control: late - cohorts have no valid post-treatment comparison and get dropped by - solve_ols's rank-deficiency handling. hc2_bm must compute BM DOF - on the REDUCED design (kept-column subspace) — operating on the - unreduced full-dummy bread would LinAlgError and fail-close every - inference field to NaN (codex R3 P1). Per-cell + aggregate - inference on identified cells must remain finite.""" + def test_hc2_bm_handles_rank_deficient_reduced_design(self): + """hc2_bm must compute BM DOF on the REDUCED design (kept-column + subspace) — operating on the unreduced full-dummy bread would + LinAlgError and fail-close every inference field to NaN (codex R3 P1). + Per-cell + aggregate inference on identified cells must remain finite. + + Rank deficiency comes from a DUPLICATED COVARIATE rather than from + dropping late-cohort treatment cells. The original all-eventually-treated + panel produced finite-but-WRONG ATTs (the survivors were relabeled + cohort contrasts) and now fails closed (codex R8 P0); a collinear + covariate reaches the same reduced-design code path on a panel whose + estimands are sound.""" rng = np.random.default_rng(42) n_units, n_periods = 20, 8 units = np.repeat(np.arange(n_units), n_periods) periods = np.tile(np.arange(1, n_periods + 1), n_units) - cohorts = rng.choice([3, 5, 7], size=n_units) + cohorts = rng.choice([0, 3, 5, 7], size=n_units) cohort_per_obs = cohorts[units] - tau = np.where(periods >= cohort_per_obs, 0.5 + 0.2 * (periods - cohort_per_obs), 0.0) + treated = (cohort_per_obs > 0) & (periods >= cohort_per_obs) + tau = np.where(treated, 0.5 + 0.2 * (periods - cohort_per_obs), 0.0) y = 1.0 + 0.1 * periods + tau + 0.1 * rng.normal(size=len(units)) - df = pd.DataFrame({"unit": units, "time": periods, "cohort": cohort_per_obs, "y": y}) - # Expect a rank-deficient warning from solve_ols (late-cohort drop). + xc = rng.normal(size=len(units)) + df = pd.DataFrame( + { + "unit": units, + "time": periods, + "cohort": cohort_per_obs, + "y": y, + "xc": xc, + "xc_dup": xc, + } + ) with warnings.catch_warnings(): warnings.simplefilter("ignore", UserWarning) res = WooldridgeDiD(method="ols", vcov_type="hc2_bm").fit( - df, outcome="y", unit="unit", time="time", cohort="cohort" + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + exovar=["xc", "xc_dup"], ) # Per-cell inference: all identified cells finite (att + se + p + # CI). solve_ols already excluded the dropped cells from @@ -2525,3 +2579,1138 @@ def test_unit_constant_exovar_snaps_with_cause_warning(self): ) assert np.isfinite(res.overall_att) assert np.isfinite(res.overall_se) + + +class TestReferencePeriodNormalization: + """Issue #724 and the edge cases the reference-period fix introduced. + + The ETWFE design omits the ``g-1`` cell as the reference (Wooldridge 2025, + Eq. 6.1/6.4). The library previously emitted it and left generic QR rank + detection to drop something arbitrary, which silently removed genuine + post-treatment effects. Cross-implementation parity against Stata + ``jwdid ... never`` lives in ``tests/test_etwfe_cs_stata_parity.py``; this + class covers the edges that have no Stata counterpart. + """ + + @staticmethod + def _panel(cohorts, times, n_per=20, seed=0, drop=None): + """Balanced panel unless ``drop`` names ``(cohort, time)`` pairs to omit.""" + rng = np.random.default_rng(seed) + drop = set(drop or ()) + rows = [] + uid = 0 + for g in cohorts: + for _ in range(n_per): + for t in times: + if (g, t) in drop: + continue + treated = g != 0 and t >= g + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + "x1": rng.normal(), + "y": 0.3 * t + (1.0 if treated else 0.0) + rng.normal(0, 0.2), + } + ) + uid += 1 + return pd.DataFrame(rows) + + def test_gapped_cohort_keeps_every_post_cell(self): + """A cohort unobserved at the panel-wide latest pre-period. + + A panel-wide reference rule would omit an all-zero column here, leaving + the collinearity intact and reproducing #724. The per-cohort rule uses + that cohort's own support instead. + """ + df = self._panel([0, 4], [1, 2, 3, 4, 5], drop={(4, 3)}) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + post = {k for k in res.group_time_effects if k[1] >= k[0]} + assert post == {(4, 4), (4, 5)} + assert [w for w in caught if "Rank-deficient" in str(w.message)] == [] + + def test_unobserved_pairs_are_skipped_and_reported(self): + """Identically-zero columns are not emitted, and the skip is not silent.""" + df = self._panel([0, 4], [1, 2, 3, 4, 5], drop={(4, 3)}) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + skipped = [w for w in caught if "no observations" in str(w.message)] + assert len(skipped) == 1 + assert "(4, 3)" in str(skipped[0].message) + + def test_skip_warning_respects_rank_deficient_action_silent(self): + """``silent`` is a documented, supported value; the new notice must not + become an unsuppressible replacement for the rank warning it removes.""" + df = self._panel([0, 4], [1, 2, 3, 4, 5], drop={(4, 3)}) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + WooldridgeDiD( + method="ols", control_group="never_treated", rank_deficient_action="silent" + ).fit(df, outcome="y", unit="unit", time="time", cohort="cohort") + assert [w for w in caught if "no observations" in str(w.message)] == [] + + def test_cohort_without_a_reference_is_excluded_not_silently_rebaselined(self): + """The P0 guard. + + A cohort with no period before ``g - anticipation`` is unidentified. + Dropping only its COLUMNS leaves its rows in the omitted baseline, so its + treatment effect loads onto the time FE and biases the cohorts that ARE + identified. The estimates must therefore match a fit where that cohort + was removed by hand -- asserting only the warning would not catch this. + """ + df = self._panel([0, 3, 5], [1, 2, 3, 4, 5, 6]) + kw = dict(outcome="y", unit="unit", time="time", cohort="cohort") + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + auto = WooldridgeDiD(method="ols", control_group="never_treated", anticipation=2).fit( + df, **kw + ) + excluded = [w for w in caught if "unidentified under the ETWFE" in str(w.message)] + assert len(excluded) == 1, "the excluded cohort must be named" + assert "3" in str(excluded[0].message) + + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + manual = WooldridgeDiD(method="ols", control_group="never_treated", anticipation=2).fit( + df[df["cohort"] != 3].copy(), **kw + ) + + shared = set(auto.group_time_effects) & set(manual.group_time_effects) + assert shared, "no cells survived to compare" + for key in shared: + np.testing.assert_allclose( + auto.group_time_effects[key]["att"], + manual.group_time_effects[key]["att"], + rtol=0, + atol=1e-12, + err_msg=f"ATT{key} differs from an explicit-exclusion fit -- " + "the excluded cohort's rows are still in the baseline", + ) + + def test_excluding_every_comparison_fails_closed(self): + """Exclusion can cascade. On an all-treated ``{1, 3}`` panel, removing + cohort 1 leaves cohort 3 alone with its cells collinear with the time FE + -- which previously produced an all-NaN fit and no error at all. + """ + df = self._panel([1, 3], [1, 2, 3, 4]) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match="no comparison observations remain"): + WooldridgeDiD(method="ols", control_group="not_yet_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + + def test_no_post_treatment_cell_fails_closed(self): + """A cohort observed only BEFORE its own treatment start has no + estimable ATT, and the fit must say so rather than return NaN. + + This is the case the old ``X_int.shape[1] == 0`` guard could not see. + On ``never_treated`` + OLS, ``include_pre`` also emits placebo + (``t < g``) columns, so the design is NON-empty even when not one + post-treatment cell survives -- the fit returned placebo coefficients + with ``overall_att = NaN``. The first half of this test pins that + mechanism (columns exist, no post cell among them) so the guard cannot + be weakened back to a column count without failing here. + """ + df = self._panel([0, 4], [1, 2, 3, 4], drop=[(4, 3), (4, 4)]) + + X_int, _, gt_keys, _ = _build_interaction_matrix( + df, cohort="cohort", time="time", anticipation=0, control_group="never_treated" + ) + assert X_int.shape[1] > 0, "design is empty -- test no longer exercises the guard" + assert [(g, t) for g, t in gt_keys if t >= g] == [] + + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match="No estimable post-treatment cells"): + WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + + def test_not_yet_treated_is_also_fixed(self): + """#724 was NOT confined to never_treated: any cohort whose cell block + spans its own cohort dummy hits it, which anticipation makes reachable + on the default path.""" + df = self._panel([0, 3, 5], [1, 2, 3, 4, 5, 6]) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = WooldridgeDiD(method="ols", control_group="not_yet_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + assert [w for w in caught if "Rank-deficient" in str(w.message)] == [] + assert {k for k in res.group_time_effects if k[1] >= k[0]} + + @pytest.mark.parametrize("control_group", ["never_treated", "not_yet_treated"]) + def test_covariate_path_keeps_every_cell(self, control_group): + """The cell x covariate block is built from the interaction columns, so + the reference omission propagates into it; nothing may be rank-dropped.""" + df = self._panel([0, 3, 5], [1, 2, 3, 4, 5, 6]) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = WooldridgeDiD(method="ols", control_group=control_group).fit( + df, outcome="y", unit="unit", time="time", cohort="cohort", exovar=["x1"] + ) + assert [w for w in caught if "Rank-deficient" in str(w.message)] == [] + assert {k for k in res.group_time_effects if k[1] >= k[0]} + + @pytest.mark.parametrize("method", ["logit", "poisson"]) + def test_nonlinear_methods_unaffected(self, method): + """The builder runs before method dispatch, so logit/Poisson see the + same changes. They restrict to post-treatment cells, so rule 1 is + unreachable there -- this pins that they still fit cleanly.""" + df = self._panel([0, 3, 5], [1, 2, 3, 4, 5, 6]) + df["y"] = (df["y"] > df["y"].median()).astype(int) if method == "logit" else df["y"].abs() + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = WooldridgeDiD(method=method, control_group="never_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + assert {k for k in res.group_time_effects if k[1] >= k[0]} + + def test_zero_weight_reference_cell_does_not_lose_a_post_cell(self): + """Survey weights scale the design by ``sqrt(w)``, so a reference cell + whose rows all carry zero ``pweight`` is absent from the EFFECTIVE + regression even though its rows are in the frame. Choosing the reference + from raw presence therefore omitted nothing, the cohort's block again + spanned the absorbed cohort dummy, and QR dropped a real post cell -- + returning a finite `overall_att` from an incomplete cell set, silently + under the supported ``rank_deficient_action="silent"``. + """ + from diff_diff import SurveyDesign + + rng = np.random.default_rng(1) + rows = [] + uid = 0 + for g in (0, 4): + for _ in range(25): + for t in range(1, 6): + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + # cohort 4's natural reference (4, 3) carries no weight + "w": 0.0 if (g == 4 and t == 3) else 1.0, + "y": 0.3 * t + (1.0 if (g and t >= g) else 0.0) + rng.normal(0, 0.2), + } + ) + uid += 1 + df = pd.DataFrame(rows) + + for action in ("warn", "silent"): + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = WooldridgeDiD( + method="ols", control_group="never_treated", rank_deficient_action=action + ).fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=SurveyDesign(weights="w"), + ) + post = {k for k in res.group_time_effects if k[1] >= k[0]} + assert post == {(4, 4), (4, 5)}, f"{action}: lost a post cell, got {sorted(post)}" + assert [w for w in caught if "Rank-deficient" in str(w.message)] == [] + + def test_excluded_cohort_leaves_no_stale_derived_state(self): + """``groups`` is read by covariate construction, fit dispatch, results + metadata and aggregation counts. Left stale after an exclusion it + reports a phantom zero-member cohort and emits an all-zero + ``D{g}_x_{cov}`` column, which fails outright under + ``rank_deficient_action="error"``. + """ + rng = np.random.default_rng(0) + rows = [] + uid = 0 + for g in (0, 3, 5): + for _ in range(20): + for t in range(1, 7): + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + "x1": rng.normal(), + "y": 0.3 * t + (1.0 if (g and t >= g) else 0.0) + rng.normal(0, 0.2), + } + ) + uid += 1 + df = pd.DataFrame(rows) + kw = dict(outcome="y", unit="unit", time="time", cohort="cohort") + + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = WooldridgeDiD(method="ols", control_group="never_treated", anticipation=2).fit( + df, **kw + ) + assert 3 not in res.groups, "excluded cohort still advertised in results.groups" + assert set(res.groups) == {k[0] for k in res.group_time_effects} + counts = getattr(res, "_n_g_per_cohort", {}) + assert 3 not in counts, f"phantom zero-member cohort in {counts}" + + # The strict path must fit rather than trip on an artificial column. + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + strict = WooldridgeDiD( + method="ols", + control_group="never_treated", + anticipation=2, + rank_deficient_action="error", + ).fit(df, exovar=["x1"], **kw) + assert {k for k in strict.group_time_effects if k[1] >= k[0]} + + +class TestOverallAttFailsClosed: + """A successful fit reports a finite overall ATT, or raises saying why not. + + Three distinct paths used to return a result object with + ``overall_att = overall_se = NaN`` and no error, which reads as a completed + estimate. Two of them (rank reduction, the anticipation window) are only + visible AFTER the solve, so the build-time cell guard cannot see them -- + hence the separate last-line-of-defence check. Surfaced by codex R2 (M1); + both post-solve cases were verified PRE-EXISTING against the pre-PR base + commit before being fixed here (ledger row M-124). + """ + + @staticmethod + def _unbalanced(support, n_per=15, seed=0): + """``{cohort: [observed times]}`` -> panel. ``0`` is never-treated.""" + rng = np.random.default_rng(seed) + rows = [] + uid = 0 + for g, times in support.items(): + for _ in range(n_per): + for t in times: + rows.append( + {"unit": uid, "time": t, "cohort": g, "y": 0.3 * t + rng.normal(0, 0.2)} + ) + uid += 1 + return pd.DataFrame(rows) + + def test_all_post_columns_rank_dropped_fails_closed(self): + """One treated cohort, no never-treated group: every post cell equals a + time indicator, so QR drops them all and nothing is left to average. + + The upstream ``has_untreated`` guard passes here because it counts this + cohort's OWN pre-treatment rows as comparison observations -- which is + precisely why the check has to also run after the solve. + """ + df = self._unbalanced({3: [1, 2, 3, 4]}) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match="not identified and were removed"): + WooldridgeDiD(method="ols", control_group="not_yet_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + + def test_two_cohorts_without_same_period_controls_fail_closed(self): + """Cohort COUNT is not a proxy for comparison support. + + Two treated cohorts survive identification here -- so any guard keyed on + "fewer than 2 cohorts" passes -- yet neither post cell has a usable + same-period comparison and both are dropped as collinear. Regression + gate for exactly that substitution. + """ + df = self._unbalanced({3: [1, 2, 3], 5: [3, 4, 5]}) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match="not identified and were removed"): + WooldridgeDiD(method="ols", control_group="not_yet_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + + def test_only_anticipation_window_cells_fails_closed(self): + """Cells inside the anticipation window are ESTIMATED (t >= g - k) but + excluded from the overall ATT (t >= g), so a cohort never observed once + actually treated yields cells and a NaN headline. + + This is the gap the build-time guard cannot close: its notion of a + treatment cell is deliberately the wider one, so the two checks are not + redundant. The first half pins that the fit really does produce cells, + so the test cannot pass for the trivial reason of an empty design. + """ + df = self._unbalanced({0: [1, 2, 3, 4, 5, 6], 5: [1, 2, 3, 4]}) + + _, _, gt_keys, _ = _build_interaction_matrix( + df, cohort="cohort", time="time", anticipation=2, control_group="never_treated" + ) + assert [(g, t) for g, t in gt_keys if t >= g - 2] != [], "no treatment cell was emitted" + assert [(g, t) for g, t in gt_keys if t >= g] == [] + + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match="No cohort-time cell contributes"): + WooldridgeDiD(method="ols", control_group="never_treated", anticipation=2).fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + + def test_valid_fit_is_unaffected(self): + """The guard must not fire on an ordinary identified design.""" + df = self._unbalanced({0: [1, 2, 3, 4], 3: [1, 2, 3, 4]}) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + assert np.isfinite(res.overall_att) + assert np.isfinite(res.overall_se) + + +class TestInvalidWeightsCannotHideBehindExclusion: + """Invalid survey weights must be REJECTED, never silently reshape the sample. + + Cell support is decided from RAW weights (a zero-weight cell is absent from + the effective regression), and an unsupported reference period excludes the + cohort's rows. So an invalid weight confined to those rows used to be + consumed by the support logic and then never reach ``SurveyDesign.resolve``: + the fit dropped the cohort, returned a confident finite ``overall_att``, and + warned that the cohort "has no pre-treatment period" -- when in truth its + pre-periods were merely poisoned. Measured before the fix: all three invalid + values below produced overall_att = 0.021416714008590323. + + Surfaced by codex R3 (M1); introduced by this PR's weight-aware support, so + it is gated here rather than tracked. `validate_raw_weights` is shared with + survey.py so both paths raise identically. + """ + + @staticmethod + def _poisoned(bad, cohorts=(0, 3, 4), poisoned_cohort=4, n_per=12, seed=0): + """Panel whose bad weights sit ONLY in ``poisoned_cohort``'s pre-periods.""" + rng = np.random.default_rng(seed) + rows = [] + uid = 0 + for g in cohorts: + for _ in range(n_per): + for t in (1, 2, 3, 4): + w = bad if (g == poisoned_cohort and t < poisoned_cohort) else 1.0 + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + "w": w, + "y": float(rng.integers(0, 2)), + } + ) + uid += 1 + return pd.DataFrame(rows) + + @pytest.mark.parametrize( + "bad,message", + [ + (np.nan, "Weights contain NaN values"), + (-1.0, "Weights must be non-negative"), + (-np.inf, "Weights contain Inf values"), + (np.inf, "Weights contain Inf values"), + ], + ) + @pytest.mark.parametrize("control_group", ["never_treated", "not_yet_treated"]) + @pytest.mark.parametrize("method", ["ols", "logit", "poisson"]) + def test_invalid_weights_raise_instead_of_excluding_the_cohort( + self, bad, message, control_group, method + ): + from diff_diff.survey import SurveyDesign + + df = self._poisoned(bad) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match=message): + WooldridgeDiD(method=method, control_group=control_group).fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=SurveyDesign(weights="w"), + ) + + def test_rejection_precedes_any_exclusion_warning(self): + """The error must arrive INSTEAD of the misleading exclusion warning, + not after it -- a user who sees only the warning would conclude their + panel lacks pre-periods and 'fix' the wrong thing.""" + from diff_diff.survey import SurveyDesign + + df = self._poisoned(np.nan) + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + with pytest.raises(ValueError, match="Weights contain NaN values"): + WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=SurveyDesign(weights="w"), + ) + assert [w for w in caught if "no pre-treatment period" in str(w.message)] == [] + + def test_valid_weights_still_fit(self): + """The guard must not reject a legitimate weighted design.""" + from diff_diff.survey import SurveyDesign + + df = self._poisoned(1.0) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=SurveyDesign(weights="w"), + ) + assert np.isfinite(res.overall_att) + assert set(res.groups) == {3, 4}, "no cohort should be excluded on valid weights" + + def test_zero_weights_still_shape_support_without_excluding(self): + """Zero is VALID and keeps its support semantics -- the guard must not + over-reject. + + Zeroing the LATEST pre-period moves the reference to the next supported + one rather than eliminating it, so the cohort stays identified and the + fit proceeds. This is the weight-aware support this PR added, exercised + without tripping the survey-exclusion refusal below. + """ + from diff_diff.survey import SurveyDesign + + rng = np.random.default_rng(0) + rows = [] + uid = 0 + for g in (0, 4): + for _ in range(12): + for t in (1, 2, 3, 4): + # cohort 4's latest pre-period (t=3) carries zero weight -> + # reference falls back to t=2, cohort remains identified. + w = 0.0 if (g == 4 and t == 3) else 1.0 + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + "w": w, + "y": 0.3 * t + rng.normal(0, 0.2), + } + ) + uid += 1 + df = pd.DataFrame(rows) + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=SurveyDesign(weights="w"), + ) + assert np.isfinite(res.overall_att) + assert (4, 3) not in res.group_time_effects, "zero-weight cell should be unsupported" + assert [w for w in caught if "no pre-treatment period" in str(w.message)] == [] + + @pytest.mark.parametrize("method", ["ols", "logit", "poisson"]) + def test_survey_plus_unidentified_cohort_is_refused(self, method): + """Excluding a cohort under a survey design is DOMAIN estimation, and + this path does not implement it -- so it is refused, not approximated. + + Deleting the rows also deletes their PSUs/strata from the TSL meat and + from ``df_survey = n_PSU - n_strata``. Measured before this refusal on a + two-stratum panel: the fit returned a finite SE with ``df_survey = 14`` + against a full-design 22, i.e. variance from a design the user never + specified. Applies to ALL THREE methods -- the earlier zero-weight guard + caught only zero-weight units, and every unit here carries POSITIVE + weight. Codex R5. + """ + from diff_diff.survey import SurveyDesign + + rng = np.random.default_rng(3) + rows = [] + uid = 0 + # cohort 4 observed only at t=3,4; with anticipation=1 it needs t < 3. + for g, times in ((0, [1, 2, 3, 4]), (3, [1, 2, 3, 4]), (4, [3, 4])): + for k in range(8): + stratum = f"s{k % 2}" + for t in times: + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + "stratum": stratum, + "psu": f"{stratum}_g{g}_p{k}", + "w": 1.0 + 0.1 * (uid % 3), + "y": float(rng.integers(0, 2)), + } + ) + uid += 1 + df = pd.DataFrame(rows) + assert (df.groupby("unit")["w"].sum() > 0).all(), "fixture must have no zero-weight unit" + + sd = SurveyDesign(weights="w", strata="stratum", psu="psu") + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(NotImplementedError, match="not supported"): + WooldridgeDiD(method=method, control_group="never_treated", anticipation=1).fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=sd, + ) + + @pytest.mark.parametrize( + "field,mutate,message", + [ + ( + "psu", + lambda d: d.assign(psu=np.where(d["cohort"] == 4, None, d["psu"])), + "PSU column", + ), + ( + "strata", + lambda d: d.assign(stratum=np.where(d["cohort"] == 4, None, d["stratum"])), + "[Ss]trat", + ), + ], + ) + def test_invalid_survey_structure_cannot_hide_in_an_excluded_cohort( + self, field, mutate, message + ): + """Structural survey metadata is validated on the PRE-exclusion sample. + + Weight validation alone was not enough: missing strata/PSU values, PSU + reuse and invalid FPC were all still checked inside + ``SurveyDesign.resolve()``, which ran after the rows were deleted. So + invalid metadata confined to the excluded cohort vanished and the fit + returned finite inference. Resolving the whole design up front fixes the + class, not the instance -- and preserves the SPECIFIC error rather than + a generic refusal. Codex R5. + """ + from diff_diff.survey import SurveyDesign + + rng = np.random.default_rng(3) + rows = [] + uid = 0 + for g, times in ((0, [1, 2, 3, 4]), (3, [1, 2, 3, 4]), (4, [3, 4])): + for k in range(8): + stratum = f"s{k % 2}" + for t in times: + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + "stratum": stratum, + "psu": f"{stratum}_g{g}_p{k}", + "w": 1.0, + "y": 0.3 * t + rng.normal(0, 0.2), + } + ) + uid += 1 + df = mutate(pd.DataFrame(rows)) + + sd = SurveyDesign(weights="w", strata="stratum", psu="psu") + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match=message): + WooldridgeDiD(method="ols", control_group="never_treated", anticipation=1).fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=sd, + ) + + @pytest.mark.parametrize("control_group", ["never_treated", "not_yet_treated"]) + def test_fractional_fweight_is_rejected_before_exclusion(self, control_group): + """Every documented weight-TYPE rule must also run before exclusion. + + `fweight` must be a non-negative integer. That check originally lived + only in ``SurveyDesign.resolve()``, i.e. AFTER support-based exclusion -- + so a cohort whose reference periods carry zero weight (excluded) and + whose remaining rows carry a fractional fweight slipped through and the + fit returned finite estimates on the reduced sample. Codex R4. + """ + from diff_diff.survey import SurveyDesign + + rng = np.random.default_rng(0) + rows = [] + uid = 0 + for g in (0, 3, 4): + for _ in range(12): + for t in (1, 2, 3, 4): + # cohort 4: zero-weight pre-periods (kills its reference, so + # exclusion would fire) + a FRACTIONAL fweight after. + w = (0.0 if t < 4 else 0.5) if g == 4 else 2.0 + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + "w": w, + "y": 0.3 * t + rng.normal(0, 0.2), + } + ) + uid += 1 + df = pd.DataFrame(rows) + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + with pytest.raises(ValueError, match="must be non-negative integers"): + WooldridgeDiD(method="ols", control_group=control_group).fit( + df, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=SurveyDesign(weights="w", weight_type="fweight"), + ) + assert [w for w in caught if "no pre-treatment period" in str(w.message)] == [] + + def test_zero_weight_units_cannot_be_deleted_by_cohort_exclusion(self): + """A zero-weight UNIT must be rejected even when exclusion would remove it. + + The library rejects zero-weight units rather than estimating around them + (`test_ols_survey_zero_weight_unit_rejected`); genuine domain estimation + is `SurveyDesign.subpopulation`, which this within-transform path does + not implement. That guard ran only just before the within-transform, so + unidentified-cohort exclusion could delete the offending rows first: a + subpopulation-style fit whose zero-weight units made up a whole + unidentified cohort silently degraded to naive subsetting and reported a + survey df computed from a design missing entire PSUs (measured: 22 -> + 14, with a misleading "no pre-treatment period" warning) instead of + raising. Codex R4. + """ + from diff_diff.survey import SurveyDesign + + rng = np.random.default_rng(7) + rows = [] + uid = 0 + for g in (0, 3, 4): + for k in range(8): + stratum = f"s{k % 2}" + for t in (1, 2, 3, 4): + rows.append( + { + "unit": uid, + "time": t, + "cohort": g, + "stratum": stratum, + "psu": f"{stratum}_g{g}_p{k}", + "w": 1.0, + "y": 0.3 * t + rng.normal(0, 0.2), + } + ) + uid += 1 + df = pd.DataFrame(rows) + sd = SurveyDesign(weights="w", strata="stratum", psu="psu") + sub_design, sub_data = sd.subpopulation(df, (df["cohort"] != 4).to_numpy()) + + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + with pytest.raises(ValueError, match="Survey weights sum to zero for unit"): + WooldridgeDiD(method="ols", control_group="never_treated").fit( + sub_data, + outcome="y", + unit="unit", + time="time", + cohort="cohort", + survey_design=sub_design, + ) + assert [w for w in caught if "no pre-treatment period" in str(w.message)] == [] + + +class TestRankDeficientActionHonoredOnSkippedCells: + """`rank_deficient_action` must mean the same thing after the skip rewrite. + + Pre-3.9, an unobserved (g, t) reached the design as an identically-zero + column and QR dropped it, so ``"error"`` FAILED CLOSED. This branch skips + those cells earlier -- better behavior, but the first version gated only on + ``!= "silent"``, collapsing ``"warn"`` and ``"error"`` into a warning and + silently downgrading a caller's explicit fail-closed request. Measured on a + cohort-4-gapped panel: base RAISED under ``"error"``; the skip returned + att=0.009034 with only a warning. Codex R6. + """ + + @staticmethod + def _gapped(): + rng = np.random.default_rng(0) + rows = [] + uid = 0 + for g, times in ((0, [1, 2, 3, 4]), (4, [1, 2, 4])): # cohort 4 unobserved at t=3 + for _ in range(12): + for t in times: + rows.append( + {"unit": uid, "time": t, "cohort": g, "y": 0.3 * t + rng.normal(0, 0.2)} + ) + uid += 1 + return pd.DataFrame(rows) + + def test_error_raises_naming_the_skipped_pair(self): + with pytest.raises(ValueError, match=r"Skipped 1 cohort-time cell\(s\).*\(4, 3\)"): + WooldridgeDiD( + method="ols", control_group="never_treated", rank_deficient_action="error" + ).fit(self._gapped(), outcome="y", unit="unit", time="time", cohort="cohort") + + def test_warn_still_fits_and_warns(self): + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = WooldridgeDiD( + method="ols", control_group="never_treated", rank_deficient_action="warn" + ).fit(self._gapped(), outcome="y", unit="unit", time="time", cohort="cohort") + assert np.isfinite(res.overall_att) + assert [w for w in caught if "Skipped 1 cohort-time" in str(w.message)] != [] + + def test_silent_fits_without_warning(self): + with warnings.catch_warnings(record=True) as caught: + warnings.simplefilter("always") + res = WooldridgeDiD( + method="ols", control_group="never_treated", rank_deficient_action="silent" + ).fit(self._gapped(), outcome="y", unit="unit", time="time", cohort="cohort") + assert np.isfinite(res.overall_att) + assert [w for w in caught if "Skipped" in str(w.message)] == [] + + @staticmethod + def _gapped_post(): + """Cohort 4 unobserved at t=5, a POST-treatment cell. + + ``_gapped()``'s hole is at t=3, which is PRE-treatment for cohort 4 and + so is only ever emitted on the ``never_treated`` + OLS path + (``include_pre``). The nonlinear paths emit post cells only, so the gate + has to be exercised there with a gapped POST cell or the test passes + vacuously. + """ + rng = np.random.default_rng(0) + rows = [] + uid = 0 + for g, times in ((0, [1, 2, 3, 4, 5]), (4, [1, 2, 4])): + for _ in range(12): + for t in times: + rows.append( + {"unit": uid, "time": t, "cohort": g, "y": 0.3 * t + rng.normal(0, 0.2)} + ) + uid += 1 + return pd.DataFrame(rows) + + @pytest.mark.parametrize("method", ["logit", "poisson"]) + def test_error_is_honored_on_the_nonlinear_paths_too(self, method): + """The builder runs before method dispatch, so all three share the gate.""" + df = self._gapped_post() + df["y"] = (df["y"] > df["y"].median()).astype(float) + with pytest.raises(ValueError, match=r"Skipped 1 cohort-time cell\(s\).*\(4, 5\)"): + WooldridgeDiD( + method=method, control_group="never_treated", rank_deficient_action="error" + ).fit(df, outcome="y", unit="unit", time="time", cohort="cohort") + + +class TestWithinCohortSupportConnectivity: + """Per-period support does not imply identification -- connectivity does. + + A cohort can have positive-weight observations in every period while its + UNITS split into groups with disjoint observation windows. Identification + runs through the unit fixed effects, so a group whose periods are all + treatment cells contributes columns that sum to its own unit indicators: + rank-deficient, and QR drops one -- possibly a genuine post-treatment + effect, leaving `overall_att` an average over an incomplete cell set. That + is issue #724's failure mode reached through unit support instead of the + reference period. Codex R7. + """ + + @staticmethod + def _split_support(): + """Cohort 4: half the units observe {2,4}, the other half {1,5}. + + Aggregate support covers 1,2,4,5 so ref=2 and cells 1,4,5 are emitted, + but {1,5} is a closed component: both its periods are cells. + """ + rng = np.random.default_rng(11) + rows = [] + uid = 0 + for _ in range(15): + for t in (1, 2, 3, 4, 5): + rows.append( + {"unit": uid, "time": t, "cohort": 0, "y": 0.3 * t + rng.normal(0, 0.2)} + ) + uid += 1 + for times in ((2, 4), (1, 5)): + for _ in range(10): + for t in times: + rows.append( + {"unit": uid, "time": t, "cohort": 4, "y": 0.3 * t + 1 + rng.normal(0, 0.2)} + ) + uid += 1 + return pd.DataFrame(rows) + + def test_disconnected_component_fails_closed(self): + """Measured before the check: `g4_t5` was dropped by QR and the fit + returned overall_att=0.0589 from the single surviving post cell.""" + with pytest.raises(ValueError, match="observed but NOT identified"): + WooldridgeDiD(method="ols", control_group="never_treated").fit( + self._split_support(), outcome="y", unit="unit", time="time", cohort="cohort" + ) + + def test_the_builder_names_the_unidentified_cells(self): + """The diagnostic must name the closed component's cells (1 and 5), not + the connected ones -- otherwise the error cannot guide a fix.""" + df = self._split_support() + _, _, _, diag = _build_interaction_matrix( + df, + cohort="cohort", + time="time", + anticipation=0, + control_group="never_treated", + method="ols", + unit="unit", + ) + assert sorted(t for _, t in diag.disconnected) == [1, 5] + + def test_connected_unbalanced_panel_is_not_flagged(self): + """Guards against a false positive that would reject legitimate data: + an unbalanced cohort whose units still share a period stays connected, + so it must fit normally.""" + rng = np.random.default_rng(5) + rows = [] + uid = 0 + for _ in range(15): + for t in (1, 2, 3, 4, 5): + rows.append( + {"unit": uid, "time": t, "cohort": 0, "y": 0.3 * t + rng.normal(0, 0.2)} + ) + uid += 1 + # Both subsets observe t=2, so the cohort graph is one component. + for times in ((2, 3, 4), (1, 2, 5)): + for _ in range(10): + for t in times: + rows.append( + {"unit": uid, "time": t, "cohort": 4, "y": 0.3 * t + 1 + rng.normal(0, 0.2)} + ) + uid += 1 + df = pd.DataFrame(rows) + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + assert np.isfinite(res.overall_att) + assert (4, 5) in res.group_time_effects + + @pytest.mark.parametrize("method", ["logit", "poisson"]) + def test_nonlinear_paths_are_not_subject_to_the_unit_fe_condition(self, method): + """The connectivity condition is collinearity with ABSORBED UNIT FEs. + + Only the OLS path within-transforms unit fixed effects away. + logit/Poisson use explicit cohort + time dummies, so nothing absorbs a + component's cell block and the design is full rank. Applying the check + there rejected valid fits (codex R9). Same split-support panel the OLS + test refuses -- here it must estimate. + """ + rng = np.random.default_rng(3) + rows = [] + uid = 0 + for _ in range(20): + for t in (1, 2, 4, 5): + rows.append({"unit": uid, "time": t, "cohort": 0, "y": float(rng.integers(0, 2))}) + uid += 1 + # cohort 4's pre-period units are DISJOINT from its post-period units, + # so its post-only component has every period emitted. + for times in ((1, 2), (4, 5)): + for _ in range(10): + for t in times: + rows.append( + {"unit": uid, "time": t, "cohort": 4, "y": float(rng.integers(0, 2))} + ) + uid += 1 + df = pd.DataFrame(rows) + + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = WooldridgeDiD(method=method, control_group="never_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + assert {k for k in res.group_time_effects if k[1] >= k[0]} == {(4, 4), (4, 5)} + assert np.isfinite(res.overall_att) + + def test_ols_still_refuses_the_same_panel(self): + """The OLS half of the contrast: the same data IS unidentified there, + so gating on method must not weaken the OLS guard.""" + rng = np.random.default_rng(3) + rows = [] + uid = 0 + for _ in range(20): + for t in (1, 2, 4, 5): + rows.append( + {"unit": uid, "time": t, "cohort": 0, "y": 0.3 * t + rng.normal(0, 0.2)} + ) + uid += 1 + for times in ((1, 2), (4, 5)): + for _ in range(10): + for t in times: + rows.append( + {"unit": uid, "time": t, "cohort": 4, "y": 0.3 * t + rng.normal(0, 0.2)} + ) + uid += 1 + with pytest.raises(ValueError, match="observed but NOT identified"): + WooldridgeDiD(method="ols", control_group="never_treated").fit( + pd.DataFrame(rows), outcome="y", unit="unit", time="time", cohort="cohort" + ) + + +class TestPartialRankLossFailsClosed: + """A finite overall ATT does not mean the reported cell set is complete. + + When rank reduction removes SOME treatment columns, the survivors are + re-identified as whatever contrast the reduced design supports and keep + their `ATT(g, t)` labels. Codex R8 P0. + """ + + @staticmethod + def _no_comparison_final_period(): + """Never-treated observed through t=4; cohorts 3 and 4 through t=5. + + At t=5 every observed unit is treated, so `g3_t5 + g4_t5` equals the + t=5 indicator and the pair is collinear with the time fixed effect. + Effects are deliberately far apart (1.0 vs 3.0) so a relabeled + contrast is unmistakable. + """ + rng = np.random.default_rng(4) + eff = {3: 1.0, 4: 3.0} + rows = [] + uid = 0 + for _ in range(20): + for t in (1, 2, 3, 4): + rows.append( + {"unit": uid, "time": t, "cohort": 0, "y": 0.2 * t + rng.normal(0, 0.05)} + ) + uid += 1 + for g in (3, 4): + for _ in range(20): + for t in (1, 2, 3, 4, 5): + y = 0.2 * t + (eff[g] if t >= g else 0.0) + rng.normal(0, 0.05) + rows.append({"unit": uid, "time": t, "cohort": g, "y": y}) + uid += 1 + return pd.DataFrame(rows) + + @pytest.mark.parametrize( + "action,expected", + [ + ("warn", "not identified and were removed"), + ("silent", "not identified and were removed"), + # "error" fails EARLIER, inside solve_ols, with its own message + # naming the dependent column -- also fail-closed, just a different + # path. Pinned explicitly so a future refactor cannot quietly move + # this mode onto the softer branch. + ("error", "rank-deficient"), + ], + ) + def test_partial_cell_loss_raises_for_every_rank_deficient_action(self, action, expected): + """Including `"silent"`: that setting controls how rank WARNINGS + surface, not whether a mislabeled contrast may be returned as an ATT. + Measured before the gate under `"silent"`: ATT(3,5) came back as + -1.9393 (true +1.0, wrong SIGN) inside overall_att=+0.7703 (true ~1.8), + with no warning of any kind. + """ + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match=expected): + WooldridgeDiD( + method="ols", control_group="never_treated", rank_deficient_action=action + ).fit( + self._no_comparison_final_period(), + outcome="y", + unit="unit", + time="time", + cohort="cohort", + ) + + def test_the_error_names_the_lost_cell(self): + """A user cannot act on 'something was dropped' -- the message has to + say which cell, since the fix is to restrict the panel or add controls + for that period.""" + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + with pytest.raises(ValueError, match=r"\(4, 5\)|\(3, 5\)"): + WooldridgeDiD(method="ols", control_group="never_treated").fit( + self._no_comparison_final_period(), + outcome="y", + unit="unit", + time="time", + cohort="cohort", + ) + + def test_the_same_panel_fits_once_the_bad_period_is_removed(self): + """The gate must not be a blanket refusal: dropping the period that has + no comparison group leaves a design that estimates correctly.""" + df = self._no_comparison_final_period() + df = df[df["time"] <= 4] + with warnings.catch_warnings(): + warnings.simplefilter("ignore") + res = WooldridgeDiD(method="ols", control_group="never_treated").fit( + df, outcome="y", unit="unit", time="time", cohort="cohort" + ) + assert np.isfinite(res.overall_att) + # True ATT(3,3)=ATT(3,4)=1.0, ATT(4,4)=3.0 -- recovered, not contrasts. + assert res.group_time_effects[(3, 3)]["att"] == pytest.approx(1.0, abs=0.1) + assert res.group_time_effects[(4, 4)]["att"] == pytest.approx(3.0, abs=0.1) + + +class TestControlGroupCellSets: + """The two control groups are DIFFERENT specifications, not a tuning knob. + + `never_treated` + OLS is Wooldridge (2025) Eq. 6.1/6.4's lead-and-lag + design: it emits pre-treatment cells and omits one per cohort as the + reference. The default `not_yet_treated` is the lag-only design: post cells + only, so it emits no placebo cells and omits no reference. The REGISTRY note + and tutorial previously applied the `g-1` reference claim to the default, + which does not exhibit it (codex R9). + """ + + @staticmethod + def _panel(): + rng = np.random.default_rng(2) + rows = [] + uid = 0 + for g in (0, 3, 5): + for _ in range(20): + for t in range(1, 7): + rows.append( + {"unit": uid, "time": t, "cohort": g, "y": 0.3 * t + rng.normal(0, 0.2)} + ) + uid += 1 + return pd.DataFrame(rows) + + def test_default_emits_no_pre_cells_and_omits_no_reference(self): + df = self._panel() + _, _, gt, diag = _build_interaction_matrix( + df, cohort="cohort", time="time", anticipation=0, control_group="not_yet_treated" + ) + assert [k for k in gt if k[1] < k[0]] == [], "lag-only design must emit no placebo cells" + # A reference is still COMPUTED (it gates unidentified-cohort detection) + # but it is never in the emitted set, so nothing is omitted from this design. + assert set(diag.references) == {3, 5} + for g, ref in diag.references.items(): + assert (g, ref) not in gt + + def test_never_treated_emits_pre_cells_except_the_reference(self): + df = self._panel() + _, _, gt, diag = _build_interaction_matrix( + df, cohort="cohort", time="time", anticipation=0, control_group="never_treated" + ) + assert [k for k in gt if k[1] < k[0]] != [], "lead-and-lag design must emit placebo cells" + for g, ref in diag.references.items(): + assert ref == g - 1, "balanced panel: the reference is g-1" + assert (g, ref) not in gt, "the reference cell must be omitted" + emitted = {t for (gg, t) in gt if gg == g} + expected = set(range(1, 7)) - {ref} + assert emitted == expected, f"cohort {g}: expected all periods but {ref}"