Kobo Backuper

A Python-based backup solution for KoboToolbox data. NGOs and organizations using KoboToolbox often lack good backup options for their data. This tool uses the KoboToolbox API to download all project data, including databases and attachments, and stores them locally or on WebDAV-compatible services like Nextcloud.

Problem

KoboToolbox is widely used by NGOs to collect field data, but there's no built-in way to create comprehensive backups of your data. This tool solves that by:

• Downloading complete database exports as CSV
• Backing up all attachments (images, files)
• Storing form structure definitions
• Supporting local storage and WebDAV (Nextcloud, ownCloud, etc.)
• Running automatically on schedules or on-demand

Architecture

The project follows a modular architecture:

• Pipeline: Orchestrates the backup process (pipelines/kobo_backup_pipeline.py)
• Modules: Core functionality
  • kobo_client.py: KoboToolbox API interaction
  • storage_local.py: Local file system storage
  • storage_webdav.py: WebDAV/Nextcloud storage
• Config: JSON configuration files define pipelines and their settings
• Launcher: Executes pipelines immediately or on schedules

Installation

pip install -r requirements.txt

Configuration

Create a JSON configuration file for your pipeline(s). Each pipeline backs up one KoboToolbox project.

Basic Local Storage Example

{
  "name": "My Project Backup",
  "trigger": "immediate",
  "project_name": "My Project",
  "kobo_client": {
    "environment_var_name": "KOBO_API_TOKEN",
    "base_url": "https://eu.kobotoolbox.org"
  },
  "storage_local": {
    "base_path": "./backups"
  }
}

WebDAV/Nextcloud Storage Example

{
  "name": "My Project Backup",
  "trigger": "immediate",
  "project_name": "My Project",
  "kobo_client": {
    "environment_var_name": "KOBO_API_TOKEN",
    "base_url": "https://eu.kobotoolbox.org"
  },
  "storage_webdav": {
    "base_url": "https://nextcloud.example.com/remote.php/dav/files/username",
    "username_env": "NEXTCLOUD_USERNAME",
    "password_env": "NEXTCLOUD_PASSWORD"
  }
}

Scheduled Backup Example

{
  "pipelines": [
    {
      "name": "Weekly Backup",
      "trigger": "scheduled",
      "schedule": {
        "interval_weeks": 1,
        "weekday": "sunday",
        "time": "02:00"
      },
      "timeout_minutes": 60,
      "project_name": "My Project",
      "kobo_client": {
        "environment_var_name": "KOBO_API_TOKEN",
        "base_url": "https://eu.kobotoolbox.org"
      },
      "storage_local": {
        "base_path": "./backups"
      }
    }
  ]
}

Configuration Parameters

Pipeline Configuration

• name (required): Pipeline name for logging
• trigger (optional, default: "immediate"): "immediate" or "scheduled"
• project_name (required): Name of the KoboToolbox project to backup
• timeout_minutes (optional, default: 60): Maximum runtime for pipeline
• max_attachments_per_run (optional, default: 100): Limit attachments downloaded per run
• attachment_batch_size (optional, default: 10): Attachments per batch before delay
• attachment_batch_delay (optional, default: 1.0): Seconds to wait between batches

Schedule Configuration (for trigger: "scheduled")

• interval_weeks: How many weeks between runs
• weekday: Day of week (e.g., "monday", "sunday")
• time: Time in 24h format (e.g., "02:00")

Kobo Client Configuration

• environment_var_name (required): Environment variable name for API token
• base_url (required): KoboToolbox server URL (e.g., "https://eu.kobotoolbox.org")
• export_timeout (optional, default: 300): Seconds to wait for export completion

Local Storage Configuration

• base_path (required): Base directory for backups
• target_path (optional): Custom folder name (defaults to project_name)

WebDAV Storage Configuration

• base_url (required): WebDAV base URL
• username_env (required): Environment variable name for username
• password_env (required): Environment variable name for password
• target_path (optional): Custom folder name (defaults to project_name)

Usage

Environment Setup

Create a .env file:

KOBO_API_TOKEN=your_api_token_here
NEXTCLOUD_USERNAME=your_username  # If using WebDAV
NEXTCLOUD_PASSWORD=your_password   # If using WebDAV

Get your API token from: KoboToolbox → Account Settings → Security → API Key

Run Immediate Backup

./launcher.sh configs/my_backup.json

Run Scheduled Backups (Daemon Mode)

The launcher automatically detects scheduled pipelines and runs as a daemon:

./launcher.sh configs/scheduled_backups.json

Development Tools

# Export CSV for a project
./devtools/kobo_backup.sh Sandbox

# List all attachments
./devtools/list_attachments.sh Sandbox

Storage Structure

Backups are stored as:

{target_path}/
  database/
    {project}_{timestamp}.zip  # Contains CSV + form.json
  attachments/
    {attachment_id}-{filename}

License

MIT License - see LICENSE file for details.