Survey Platform Data Extraction & JSON-to-CSV Conversion
Overview
A shell-based data-extraction and conversion toolkit for a survey/questionnaire platform. It pulls dozens of related tables (beneficiaries, questionnaires, questions, answer options, evaluations, reports, email logs) out of a remote CRUD API and converts the resulting JSON into CSV for analysis.
Why It Exists
Extracting a full relational dataset out of a hosted survey application, table by table, and getting it into an analyzable form is tedious and error-prone by hand. These scripts automate the bulk pull and the JSON-to-CSV conversion so the entire dataset can be snapshotted and analyzed offline.
What We Built
A small set of shell scripts driving the extraction: do.sh issues parameterized POSTs to a remote JsTruster/Crud endpoint (accepting a table name argument and a SELECT payload, with an auth token), process_tables.sh orchestrates pulling many tables, and json_to_csv.sh flattens the resulting JSON into CSV. The output is ~47 JSON files covering the full schema, questionnaires (chestionare*), beneficiaries (beneficiari*), respondents (completatori*), evaluations (evaluari*), reports (rapoarte*), notifications, email metering, and access logs, plus consolidated all.json / all.csv. It’s a focused extraction-and-transform toolkit.
Technologies & Approach
Pure Bash with curl for REST extraction (parameterized per table), simple shell-based JSON-to-CSV conversion, and a flat-file output model. The design favors transparency and repeatability, re-run the scripts to refresh a full dataset snapshot.
Outcome / Impact
The toolkit produced a complete, analysis-ready snapshot of a survey platform’s relational data with a handful of scripts, turning a manual, multi-table export chore into a one-command refresh. It demonstrates pragmatic data-extraction and ETL automation against a third-party API.
Capabilities Demonstrated
- Automated bulk extraction from remote REST/CRUD APIs
- Parameterized, table-by-table export orchestration
- JSON-to-CSV transformation across complex relational schemas
- Reproducible, scriptable data snapshotting