Datasets & Models

Dataset¶

The available datasets are listed in the data folder, with metadata provided in data/metadata.xlsx.

Simple initialization example: Run this in Python. On macOS, start Python with python3 if python is not available.

import pynnlf

pynnlf.init("example_project")

In this example, only the sample dataset (ds0) is included by default. Additional datasets are downloaded only if you enable downloads.

To download all datasets, use:

import pynnlf

pynnlf.init(
    "example_project",
    download_data=True,
    all_data=True,
)

To download a specific subset of datasets:

import pynnlf

pynnlf.init(
    "example_project",
    download_data=True,
    datasets=["ds14", "ds15"],
)

The download is typically fast because the dataset files are relatively small.

Below are some of the datasets currently included in the library.

Ausgrid Solar Home Datasets¶

This dataset has been widely used in net load forecasting research since 2016. It includes data from 300 solar-equipped households within the Ausgrid network in Sydney. The original source is here.

Dataset Name	Description
`ds1_ashd.csv`	Ausgrid Solar Home Dataset, aggregate of 300 household data in Ausgrid network
`ds4_ashd_with_weather.csv`	`ds1` enhanced with temperature, relative humidity, and wind speed data
`ds13_ashd_with_cloud_solcast.csv`	`ds4` further enriched with cloud data from Solcast
`ds19_ashd_148hh.csv`	ASHD aggregate of the 148 households nearest to Sydney Airport, selected by postcode-centroid distance
`ds20_ashd_148hh_with_weather.csv`	`ds19` enhanced with Sydney Airport temperature, relative humidity, and wind speed data
`ds21_ashd_148hh_with_weather_cloud_solcast.csv`	`ds20` further enriched with Sydney Airport Solcast cloud opacity data

Australian Centre for Advanced Photovoltaics (ACAP), Energy Data Platform (ACAP-EDP) Datasets¶

These datasets were compiled by UNSW Sydney from the Australian Centre for Advanced Photovoltaics (ACAP), Energy Data Platform (ACAP-EDP), which gathers household monitoring data from Solar Analytics and Wattwatchers. Sensitive information such as customer addresses, names, and NMIs has been removed. Only aggregated load profiles are distributed; the single-household samples (ds25-ds27) are not shared publicly at the data owner's request. We acknowledge the Australian Centre for Advanced Photovoltaics (ACAP), Energy Data Platform (ACAP-EDP) for the household net load data compiled from Solar Analytics and Wattwatchers.

Dataset Name	Description
`ds10_acap-edp_cluster2_30min.csv`	ACAP-EDP dataset for Cluster 2 (148 households) with 30-minute resolution
`ds11_acap-edp_cluster2_30min_with_weather.csv`	`ds10` enhanced with weather data including temperature, humidity, etc.
`ds28`-`ds36` ACAP-EDP aggregation datasets	Publication datasets for ACAP-EDP aggregation-level experiments at 10, 100, and synthetic-bootstrap 1000 households, with 3 samples per level and Sydney Airport weather variables. These are generated by `publication/journal_article_1/notebooks/03_acap_edp_aggregation_level` from 139 valid per-household ACAP-EDP checkpoints; household series are forward-filled before aggregation. The same notebooks also produce the single-household samples `ds25`-`ds27`, which are not distributed.

Solar Analytics CICCADA BESS Datasets¶

These datasets were generated for the publication workflow from Solar Analytics CICCADA BESS data. The 100-household variants are retained for exploration because 56 of the 100 selected households contain meaningful negative reconstructed underlying-load intervals. The 44-household variants keep only households with no intervals below the -0.05 kW meaningful-negative threshold.

Dataset Name	Description
`ds16_sa_bess_underlying_load_30min.csv`	Exploratory 100-household aggregate underlying load
`ds17_sa_bess_net_load_with_pv_30min.csv`	Exploratory 100-household aggregate net load with PV
`ds18_sa_bess_net_load_with_pv_battery_30min.csv`	Exploratory 100-household aggregate net load with PV and battery
`ds22_sa_bess_44hh_pos_underlying_load_30min.csv`	Clean 44-household aggregate underlying load
`ds23_sa_bess_44hh_pos_net_load_with_pv_30min.csv`	Clean 44-household aggregate net load with PV
`ds24_sa_bess_44hh_pos_net_load_with_pv_battery_30min.csv`	Clean 44-household aggregate net load with PV and battery

Ausgrid Zone Substation (ZS) Datasets - Mascot¶

Unlike the previous household-focused datasets, this dataset covers a zone substation, which includes residential, commercial and industrial (C&I), and major customers. The original source is here.

Dataset Name	Description
`ds14_ausgrid_zs_mascot.csv`	Zone Substation data for Mascot
`ds15_ausgrid_zs_mascot_30min_with_weather.csv`	`ds14` enhanced with weather data at 30-minute resolution

International Datasets¶

Two aggregate datasets from outside Australia, built by the notebooks in publication/journal_article_1/notebooks/00_data_exploration_and_processing/02_process_international_datasets/ with the same recipe as the ASHD datasets: sum the sites, convert energy to average power, attach weather under the same column names, and validate a regular index with no missing values.

Dataset Name	Description
`ds37_sundance_hourly.csv`	UMass Smart* SunDance release: hourly net load for calendar 2015, summed over the 87 of 100 North American solar sites that are hourly, complete and residential in scale. Sites in several time zones are summed on their local standard clock, so the aggregate is synthetic rather than a physical feeder.
`ds38_sundance_hourly_with_weather.csv`	`ds37` enhanced with the site-mean temperature, relative humidity and wind speed from the weather files shipped with the release
`ds39_fluvius_pv_30min.csv`	Fluvius digital-meter release: half-hourly net load (offtake minus injection) for calendar 2024, summed over the 300 residential meters with solar panels and neither a heat pump nor an electric vehicle, on Central European Time
`ds40_fluvius_pv_30min_with_weather.csv`	`ds39` enhanced with Brussels-Uccle temperature, relative humidity and wind speed from the Open-Meteo historical weather API

SunDance is released under CC BY 4.0 by the UMass Trace Repository; cite Chen, D. and Irwin, D. (2017), SunDance: Black-box Behind-the-Meter Solar Disaggregation, ACM e-Energy '17. The Fluvius data (dataset 1_50, last updated by Fluvius on 2025-09-29) are reused under the Fluvius open data licence, which permits redistribution and commercial use provided Fluvius and the date of the last update are named. Open-Meteo data are CC BY 4.0.

Model¶

Forecasting Models Overview¶

Model ID	Model Name	Short Description
`m1_naive`	Naive	Forecast equals the last observed value
`m2_snaive`	Seasonal Naive	Forecast equals the value from the same season in the previous cycle

Statistical Models¶

Model ID	Model Name	Short Description
`m3_ets`	ETS	Exponential smoothing model with error, trend, and seasonality components
`m4_arima`	ARIMA	Autoregressive Integrated Moving Average model for time series forecasting
`m5_sarima`	SARIMA	Seasonal ARIMA model with seasonal components
`m6_lr`	Linear Regression	Predicts future values using a linear combination of input features

Machine Learning Models¶

Model ID	Model Name	Short Description
`m7_ann`	ANN	Basic Artificial Neural Network with one hidden layer
`m8_dnn`	Deep Neural Network	ANN with more than one hidden layer
`m9_rt`	Regression Tree	Decision tree model for regression tasks
`m10_rf`	Random Forest	Ensemble of regression trees for improved accuracy and robustness
`m11_svr`	Support Vector Regression	Uses support vectors to perform regression with margin of tolerance
`m12_rnn`	Recurrent Neural Network	Neural network with feedback loops for sequential data
`m13_lstm`	Long Short-Term Memory	RNN variant designed to capture long-term dependencies
`m14_gru`	Gated Recurrent Unit	Simplified LSTM with fewer parameters
`m15_transformer`	Transformer	Attention-based model for sequence modeling without recurrence
`m16_prophet`	Prophet	Time series model developed by Facebook for business forecasting
`m17_xgb`	XGBoost	Gradient boosting framework optimized for speed and performance
`m18_nbeats`	N-BEATS	Deep learning model for univariate time series forecasting
`m19_nhits`	N-HiTS	Multi-rate MLP stack with backcast residuals (Challu et al., 2023)
`m20_patchtst`	PatchTST	Transformer encoder over patches of the lookback window (Nie et al., 2023)

Hyperparameter¶

The list of available models and their hyperparameters can be seen in models/hyperparameters.yaml. The values currently available are the hyperparameter values mostly used in academic literature, but not necessarily the optimum value.

Data Provenance¶

The datasets in data/ are derived from the following sources, all cleared for public release:

Source	Used for
Ausgrid Solar Home Dataset (ASHD)	`ds1`, `ds4`, `ds12`, `ds13`, `ds19`–`ds21`
Australian Centre for Advanced Photovoltaics (ACAP), Energy Data Platform (ACAP-EDP)	`ds2`, `ds3`, `ds5`–`ds11`, `ds28`–`ds36`
Ausgrid zone substation, Mascot	`ds14`, `ds15`
Solar Analytics CICCADA battery energy storage system (BESS) cohort	South Australian BESS datasets
Solcast and the Australian Bureau of Meteorology (BOM)	weather and cloud variables in the Australian `_with_weather` variants
UMass Smart* SunDance release (CC BY 4.0)	`ds37`, `ds38`
Fluvius Open Data (Fluvius open data licence)	`ds39`, `ds40`
Open-Meteo historical weather API (CC BY 4.0)	weather variables in `ds40`

Downloading datasets¶

A new workspace ships with the ds0 sample only. To add more:

import pynnlf

pynnlf.init("my_project", download_data=True, all_data=True)   # everything
pynnlf.init("my_project", download_data=True, datasets=["ds15"])  # a specific set

Source data for the publication notebooks¶

The unaggregated source files these datasets are built from are not distributed with the repository: they are large, and the raw feeds are not ours to redistribute. The data preparation notebooks under publication/journal_article_1/notebooks/ therefore read from a local directory instead.

Set PYNNLF_RAW_DATA_DIR to the directory holding your source data folders before running them:

$env:PYNNLF_RAW_DATA_DIR = "D:\pynnlf_source_data"

Those notebooks expect the 1. raw, 2. processed and 3. cleaned folders beneath that root. The international datasets read 1. raw/SunDance/SunDance_data_release/ (the unpacked SunDance archive), 1. raw/Fluvius/ (the Fluvius Deel_1 ZIP and legend workbook) and 1. raw/Open-Meteo/ (fetched by the notebook when absent). A notebook that needs the variable raises an error naming it if it is unset.