it4lia/predictive_irrigation_models

Predictive Irrigation Models

This repository contains end-to-end pipelines for predictive irrigation, combining:

• field sensor measurements (tensiometers and related sensors),
• irrigation logs,
• weather observations and generated forecasts,
• optional crop/soil metadata,
• optional satellite-derived vegetation indices,
• and two modeling tracks: XGBoost-based forecasting and AquaCrop-based simulation.

The implementation is orchestrated with Prefect flows in the pipelines/ folder and configured through YAML files in config/.

What Is In This Repository

• main.py: runs the full workflow in sequence.
• pipelines/: data preparation and model flows.
• tools/: weather, geospatial, authentication, and Copernicus helpers.
• aquacrop/: local AquaCrop engine implementation used by the AquaCrop pipeline.
• config/: runtime configuration (plus some private config files expected at runtime, see below).
• notebooks/notebook_demo.ipynb: interactive demo notebook.

Pipeline Overview

The default execution path in main.py is:

1. model_preparation_pipeline()
2. aquacrop_preparation_pipeline()
3. xgcast_preparation_pipeline()
4. aquacrop_pipeline()
5. xgcast_model_pipeline()

Main data flow (paths are created/used by the pipelines):

• data/03_primary/: primary prepared inputs (sensor, weather, irrigation, locations, crop, soil, satellite).
• data/04_model_input/: merged model-ready tables (full_table_<consortium>.parquet).
• data/05_xgcast_input/: normalized training/validation/test artifacts for XGBoost.
• data/05_aquacrop_input/<consortium>/<sensor>/: weather/irrigation/settings for AquaCrop runs.
• data/06_xgcast_output/: trained XGBoost model files.
• data/06_aquacrop_output/<consortium>/<sensor>/: AquaCrop simulation outputs.

Requirements

• Python 3.11+
• uv (recommended for environment/dependency management)

Project dependencies are declared in pyproject.toml.

Setup

From the repository root:

uv sync

If you prefer running with the virtual environment directly:

uv run python -m main

Configuration

Base configuration files committed in this repo:

• config/params.yml
• config/aquacrop_params.yml
• config/xgcast_params.yml
• config/fieldsensor_irrigator_mapping_anonym.yaml
• config/request_scripts/*.js

Important: private/non-versioned configs

Some optional pipeline branches (especially satellite download/auth flows) expect private files that are intentionally gitignored, for example:

• config/copernicus_oauth_config.json
• config/keycloak_config.json
• config/pre_anonym_params.yml
• potentially consortium-specific field mapping files (for field-level satellite aggregation)

If those files are missing, full satellite acquisition workflows will not run.

Running The Pipelines

Full run (same sequence as main.py)

uv run python -m main

Run individual flows

uv run python -m pipelines.model_preparation_pipeline
uv run python -m pipelines.aquacrop_preparation_pipeline
uv run python -m pipelines.aquacrop_pipeline
uv run python -m pipelines.xgcast_pipeline

Expected Input Artifacts

At minimum, model preparation expects consortium-scoped files such as:

• data/03_primary/field_sensor_data_<consortium>.parquet
• data/03_primary/irrigation_data_<consortium>.parquet
• data/03_primary/locations_ids_<consortium>.parquet
• data/03_primary/historical_weather_data_<consortium>.parquet
• data/03_primary/forecasted_weather_data_<consortium>.parquet

Optional inputs are controlled through data_availability in config/params.yml:

• weather sensor data,
• crop type data,
• soil type data,
• remote sensing data.

Models

XGCast (XGBoost)

• training configuration in config/xgcast_params.yml.
• outputs model file data/06_xgcast_output/xgcast_<consortium>.json.

AquaCrop

• uses local aquacrop/ package implementation.
• simulation outputs include water flux, water storage, crop growth, and summary tables under data/06_aquacrop_output/.

Notebook Demo

Open notebooks/notebook_demo.ipynb for an interactive walkthrough using consortium/sensor examples and model outputs.

Known Caveats

• data/ is gitignored; expected datasets are not included in this repository.
• several remote-sensing paths depend on private configuration and credentials.

Contributing

Contributions are welcome. Open an issue or submit a pull request with:

• the problem statement,
• reproducible steps,
• and any data/config assumptions required to test the change.