Treating my running data like a production data platform | Marián Kamendy

My training data lives in Runalyze. I wanted weekly numbers from it without exporting spreadsheets, so I built a small pipeline on Databricks, and I built it the way I would build a data product at work. This is how it is put together, and where it is deliberately simple.

Why treat a hobby like production

At personal scale a script and a CSV file would do. I treated it as a production system anyway, because the habits were the point: raw data that can be replayed, steps that can be re-run safely, secrets out of the code, and deployments that come from the repository, not from clicking around in a workspace.

Three layers, one job

The pipeline uses the usual medallion layout. Bronze keeps what the source sent, Silver holds clean and typed records, and Gold holds the answers I actually look at. Each layer is one notebook, run as dependent tasks of a single Databricks job on serverless compute, every night at 02:00 UTC. If a step fails, the later steps do not run.

Bronze: keep the raw payload, insert only what is new

The first notebook reads the API token from a Databricks secret scope, pings the API to fail fast on bad authentication, and fetches activities. Each activity is stored as its raw JSON string, with an ingestion timestamp and the activity ID pulled out of the payload. Duplicates inside the API response are dropped, and the rows go into the Bronze Delta table with a MERGE:

bronze.alias("t").merge(
    df_raw.alias("s"),
    "t.activity_id = s.activity_id"
).whenNotMatchedInsertAll().execute()

An activity that is already in Bronze stays exactly as it was; a new one is inserted. Running the notebook twice in a row changes nothing the second time. That idempotence makes a nightly schedule boring in the good sense: retries and manual re-runs are safe. And because the raw JSON is kept, a parsing bug means fixing the Silver notebook and rebuilding from Bronze, not fetching the history again.

Silver: an explicit schema, then types and de-duplication

The Silver notebook parses the raw JSON with from_json against a schema I wrote down instead of letting Spark infer one: ID, sport ID, distance, duration, date and time, average and maximum heart rate, and title. That keeps the table shape stable: if the payload changes, a field I depend on shows up empty in a known column instead of quietly reshaping the table.

From there the notebook converts the date to a proper timestamp, keeps only running activities (selected by Runalyze's sport ID), drops duplicates on the activity ID, and upserts into Silver with another MERGE that updates matching rows and inserts new ones.

Gold: answers, rebuilt every night

The Gold notebook groups runs by calendar week and writes three small tables:

Weekly mileage: total distance and number of runs per week.
Weekly pace: total time divided by total distance, shown both as minutes per kilometre and as a readable m:ss min/km string.
Heart-rate zones: how many runs fall into each of five zones.

The pace is distance-weighted on purpose. Averaging each run's pace would give a short recovery jog the same weight as a long run; dividing the week's total time by its total distance does not.

(col("total_duration_sec") / 60) / col("total_distance_km")

Deployment as code

The job, its three tasks, their order, the compute and the schedule are all defined in databricks.yml as a Databricks Asset Bundle. A GitHub Actions workflow deploys it on every push to main: it installs the Databricks CLI, runs databricks bundle destroy and then databricks bundle deploy, with the workspace host and token taken from GitHub secrets. The workspace always matches the repository, nothing is edited by hand, and every change goes live the same way.

Honest limitations

Heart-rate zones come from average heart rate. The activity endpoint the pipeline uses returns only an average and a maximum per run, so a whole run lands in one zone. An interval session with a moderate average is filed as one steady aerobic run. The table counts runs per zone, not time spent in each zone, and the zone limits are fixed numbers in the notebook rather than my personal thresholds.
Gold is rebuilt in full on every run. Each table is overwritten from all of Silver, and Silver is merged from all of Bronze. At personal-data scale that is simpler and obviously correct; at work volumes I would aggregate incrementally.
Bronze is insert-only. An activity edited in Runalyze after ingestion never reaches Bronze again.
Ingestion is a single request. The notebook does not page through results or ask for a date range; it stores whatever the activity endpoint returns.
Running is one hard-coded sport ID, which may differ between Runalyze accounts.
Deploys are blunt. Destroying the bundle before deploying keeps the workspace clean, but the job is torn down and recreated on every push.
No tests or data-quality checks yet. That is the first thing I would add before calling it production for anyone other than myself.

What I take from it

The project is small, but it asks the same questions as a data platform at work. Can I replay the raw data? Can I re-run any step safely? Where do the secrets live? How does a change reach production? Answering them for my own running data is a cheap way to keep those habits sharp. The code is on GitHub: montanis/runalyze-databricks.