Fix US program statistics variable mappings#327
Conversation
ReviewCore logic is correct — variable name swaps, validation, and Blocking
Worth addressing
Verified correct
Minor suggestions
|
There was a problem hiding this comment.
Follow-up review on the new commits (fe78636 … 0c39e5a).
Still worth addressing
-
create_erroris a thin wrapper that adds nothing. Body isreturn error_type(message), and the test (test_create_error_returns_requested_error_type) just re-asserts Python's exception constructor behavior. Bothraise create_error(ValueError, msg)andraise ValueError(msg)produce identical results. The indirection makes the call site longer without helping callers. Suggest removingcreate_error(and its test) and inliningraise ValueError(...).format_conditional_error_detailshould stay — it has real logic (sort + dedupe + None-on-empty). -
Asymmetric baseline/reform configs still merge errors.
_validate_program_statistics_configputs missing variables and missing outputs from both simulations into the same sets, so a config where baseline lacksmedicare_costand reform lackssocial_securityproduces one message that doesn't say which simulation each is missing from. Was minor before, still minor — just noting it's unchanged. -
Inconsistent return annotations on the new aggregate helpers.
require_output_columnhas-> None;get_aggregate_variableandget_output_entity_datadon't. Easy fix — returnVariableand the entity data type respectively, or at leastAny.
New observation
The match=\"Aggregate.variable\" / match=\"ChangeAggregate.filter_variable\" test assertions now lock the internal context= strings into the test contract. If those labels ever get rewritten to be more user-facing (which was a prior minor suggestion), the tests need to update too. Not a blocker — just flagging the new coupling.
Looks mergeable once create_error is removed (or kept with a note explaining what it gives over ValueError(msg)).
vahid-ahmadi
left a comment
There was a problem hiding this comment.
Follow-up after ed1ab89.
Addressed
create_errorremoved everywhere (utils/errors.py, utils/init.py re-export, analysis.py import, the tautological test). Inlineraise ValueError(...)used in the validator.- Return annotations added on the new aggregate helpers —
get_aggregate_variable(...) -> Variable,get_output_entity_data(...) -> Any, plusdata: Anyonrequire_output_column. - Test contract loosened nicely: instead of
match=\"Aggregate.variable\"(which would have locked the internalcontext=label into the test), the tests now assert on the user-facing fragments\"references missing variable\"plus the variable name. That defuses the coupling I flagged last round and leaves room to rewordcontext=later without churn.
The asymmetric baseline/reform-config message is still merged, but that was minor and explicitly noted as such — fine to leave.
LGTM. Nothing else from me; ready to merge.
Fixes #325
This draft PR addresses only the immediate causes of the US
economic_impact_analysisprogram-statistics breakage observed while following up on Vahid's JOSS PR (#264). It is not intended to solve the broader design problem of country package program variables changing over time; that follow-up is tracked in #326.Changes:
ProgramStatistics.program_nameas the direct model/output variable to aggregate; no separate alias field is introduced.policyengine-usvariables:payroll_tax->employee_payroll_taxmedicare->medicare_coststate_income_taxremainsstate_income_taxpayroll_taxwas configured as a person-level program, whileemployee_payroll_taxis a tax-unit-level variable. This means the payroll-tax program-statistics row now reports tax-unit recipient/winner/loser counts, using tax-unit weights, rather than person counts.medicare_costandstate_income_taxto the US output variables materialized by.py.medicare_costandstate_income_taxoutput keys.next(...)lookup failures with descriptiveValueErrors.Verification:
make formatuv run --python 3.13 --extra dev python -m pytest tests/test_household_calculator_snapshot.py::test_us_household_snapshot tests/test_errors.py tests/test_aggregate.py tests/test_change_aggregate.py tests/test_us_program_statistics.pyuv run --python 3.13 --extra dev pytest tests/test_change_aggregate.py tests/test_us_program_statistics.pyuv run --python 3.13 --extra dev ruff check tests/test_us_program_statistics.py tests/test_change_aggregate.pygit diff --check