Convert the SPSS delivery, drop the respondents that failed quality control from both waves, append, re-weight the combined sample and fill in the derived variables — in about twenty seconds, with a report you can hand to whoever signs the data off.
Same six files, same columns, same canonical sort order, same CRLF and quoting BAV ships — just with the bad respondents gone and the refield respondents added. Built for the China 2026 Concepts refield; nothing in it is study-specific beyond a handful of defaults you can override on the command line.
run_pipeline.py chains them for the common case. Run them individually when you need to inspect or adjust an intermediate — every stage writes its own directory and its own timestamped log.
convert_sav.pyReads every .sav in a directory and writes the three respondent-level BAV CSVs. Files are classified by their variable signature rather than their filename, so a renamed or re-cut delivery converts without configuration.
filter_respondents.pyReads the quality-control workbook and filters each export down to its good respondents. _wts.csv is the authoritative roster: the respondents that survive there become the set the other files are filtered against.
merge_datasets.pyAppends one or more exports onto a base and re-sorts into canonical BAV order, so the result is indistinguishable in structure from a single-wave export. Respondent keys must be disjoint — a collision means two different people would share an identity, so the merge aborts.
--study-key restamps all six files, because a merged export is a new study and the source key is already taken.weight_respondents.pyRim-weights the combined sample to population targets — gender × age interlocked, city tier, income — within each version cell, normalised to mean 1.0. The refield delivery carries no weights at all, so this is what makes the merged export usable for weighted analysis.
derive_variables.pyComputes the variables a refield delivery leaves out even when the source variable came through fine — Income Group is the standard case. Percentiles are weighted by default, because that is the distribution the study claims to represent.
The China 2026 targets were reverse-engineered from the original wave's own weights. Raking each version cell to them reproduces all 19,591 published weights to three decimal places — correlation 0.99999999, maximum relative error 0.016%.
"50% women" and "18% aged 18–24" are both satisfied by a sample where every young respondent is a woman. Crossing gender and age into one dimension closes that hole; the cost is thinner cells, which is a judgement call the anomaly report gives you the evidence for.
Response rates thin out at the top of the age range, so the study is defined over 18–74. Respondents outside it take no part in the raking — so a handful of 80-year-olds cannot pull everyone else's weights around — but keep their rows and a neutral weight.
When weights run away the tool names the thin band and suggests merging it into a neighbour. Capping hides the symptom and quietly stops you hitting the targets you asked for. It is still available if you want it, which is why achieved marginals are always reported.
Weighting is usually where fieldwork faults first surface. A band that is empty in some version cells but not others rarely says anything about the population — it usually means those cells were never fielded as designed.
| Check | What it means | Default |
|---|---|---|
empty | A target band has no respondents in some weighting group. Its share is silently absorbed by the other bands, corrupting every weight in the group. | Aborts the run |
sparse | Fewer than 30 respondents in a cell. Thin cells are what produce runaway weights. | Warning |
extreme | A respondent weighs more than 5× the average. | Warning |
inefficient | A weighting group falls below 50% efficiency. | Warning |
| Key collision | The same respondent key appears in both waves, so two different people would share an identity. | Aborts the merge |
| Orphan respondent | Someone answers questions but has no row in the roster file. | Aborts the filter |
| Derivation drift | The derived variable disagrees with the published values it was tested against. | Fails below 99% |
Published under study key 1258. Weighted at 82.7% efficiency, effective sample size 15,506 against the 18,756 respondents in the universe, with weights ranging 0.39–4.84 and every version cell averaging exactly 1.0.
| File | Original kept / total | Refield kept / total | Merged |
|---|---|---|---|
_wts.csv · respondents | 11,506 / 19,591 | 7,268 / 8,238 | 18,774 |
_df.csv · demographic answers | 844,362 / 1,410,535 | 457,953 / 515,644 | 1,302,315 |
_bf.csv · brand facts | 4,612,740 / 6,437,205 | 2,957,297 / 3,214,044 | 7,570,037 |
The scripts carry PEP 723 inline dependencies, so uv resolves them on first run. Python 3.11+ for TOML configs; pandas, pyreadstat, openpyxl and numpy otherwise. Every script has --help, --dry-run, and prompts for anything missing when the terminal is interactive.
# the whole thing, end to end uv run ./run_pipeline.py \ --existing-dir "existing data" \ --refield-dir "refield data" \ --mapping "respondent mappings/refield and original good resps.xlsx" \ --output-dir "output" \ --prefix "China 2026 Concepts" \ --new-study-key 1258 \ --weight-config config/china-2026-targets.toml \ --derive-income-group # or audit the weights already in a delivery, without recomputing anything uv run ./weight_respondents.py --input-dir "output/final" --audit-only
0 success, 1 unexpected error, 2 aborted on a failed guardrail, 130 interrupted — so it drops into CI as it is.