Skip to content

VictoriaMetrics

High-Performance Time Series Database

VictoriaMetrics is a fast, cost-effective, and scalable time series database written in Go. It is designed for high-throughput monitoring workloads and is Prometheus-compatible, making it a drop-in replacement for Prometheus storage. This instance runs the single-node version with vmagent for scrape-based metrics collection.

Why Choose VictoriaMetrics?

  • Prometheus-compatible - works with PromQL, Grafana, and existing exporters
  • High performance - optimized for high-cardinality time series data
  • Low resource usage - single binary with minimal memory footprint
  • Built-in downsampling - reduces storage costs over time
  • vmagent included - lightweight scraper with remote write support

Install

Infrastructure as Code

This VictoriaMetrics instance is deployed using OpenTofu with custom Incus images. The infrastructure configuration manages container provisioning and persistent storage volumes.

The canonical configuration is maintained at benoitjpnet/OpenTofu.

Custom VictoriaMetrics Image

The VictoriaMetrics Incus image is built and maintained at forgejo.benoit.jp.net/benoitjpnet/Laminar.

Infrastructure Configuration

The OpenTofu configuration provisions:

  • Incus instance running custom VictoriaMetrics image
  • Two persistent storage volumes:
    • /var/lib/victoriametrics - Time series database storage (retention: 12 months)
    • /var/backups/victoriametrics - Backup storage
OpenTofu Configuration
resource "incus_storage_volume" "victoriametrics_var_lib_victoriametrics" {
  name = "victoriametrics_var_lib_victoriametrics"
  pool = incus_storage_pool.default.name
  config = {
    "initial.gid" = "1001"
    "initial.uid" = "1001"
    "initial.mode" = "755"
  }
}

resource "incus_storage_volume" "victoriametrics_var_backups_victoriametrics" {
  name = "victoriametrics_var_backups_victoriametrics"
  pool = incus_storage_pool.default.name
}

resource "incus_instance" "victoriametrics" {
  name  = "victoriametrics"
  image = "laminar.incus:victoriametrics-1.147.0-1benoitjpnet"

  device {
    name = "var_lib_victoriametrics"
    type = "disk"
    properties = {
      path   = "/var/lib/victoriametrics"
      source = incus_storage_volume.victoriametrics_var_lib_victoriametrics.name
      pool   = incus_storage_pool.default.name
    }
  }

  device {
    name = "var_backups_victoriametrics"
    type = "disk"
    properties = {
      path   = "/var/backups/victoriametrics"
      source = incus_storage_volume.victoriametrics_var_backups_victoriametrics.name
      pool   = incus_storage_pool.default.name
    }
  }
}

Deploy Infrastructure

Apply OpenTofu configuration
tofu apply

Services

The container runs two systemd services:

  • victoriametrics (port 8428) - Single-node VictoriaMetrics time series database with 12-month retention
  • vmagent (port 8429) - Scrape agent that collects metrics and writes to the local VictoriaMetrics instance

The vmagent scrape configuration is at /etc/victoriametrics/promscrape.yml on the persistent config volume.

Access

VictoriaMetrics is restricted to Tailnet/internal access via HAProxy ACLs (allowed_ips only). The web UI and API are available at:

  • https://victoriametrics.benoit.jp.net - VictoriaMetrics web UI and API (port 8428)

Internal Access Only

VictoriaMetrics is not publicly accessible. It is restricted to Tailscale and monitoring IPs via HAProxy allowed_ips ACL. No public DNS rewrite is needed for external access.

Upgrade

Incus Image Upgrade

Upgrading VictoriaMetrics is straightforward since all data volumes persist independently of the container.

Backup current data before upgrading:

Enter Incus container
incus shell victoriametrics

Backup configuration (if customized)

cp /etc/victoriametrics/promscrape.yml /var/backups/victoriametrics/

Snapshot data (optional)

# VictoriaMetrics data is on a persistent volume, so it survives container recreation.
# For extra safety, you can use vmbackup:
/opt/victoriametrics/vmbackup-prod -storageDataPath=/var/lib/victoriametrics \
  -dst=/var/backups/victoriametrics/snapshot

Deploy to new Incus image and restore data:

Update OpenTofu configuration

Edit your OpenTofu configuration file to update the image version:

resource "incus_instance" "victoriametrics" {
  name  = "victoriametrics"
  image = "laminar.incus:victoriametrics-NEW-VERSION"  # Update this line
  # ...
}

Apply infrastructure changes

Provision new Incus container
tofu apply
Enter Incus container
incus shell victoriametrics

Restore configuration (if backed up)

cp /var/backups/victoriametrics/promscrape.yml /etc/victoriametrics/

Restart services

systemctl restart victoriametrics
systemctl restart vmagent

Upgrade Complete

Your VictoriaMetrics instance is now running on the new Incus image with all metrics data intact!

Using VictoriaMetrics

Querying Metrics

VictoriaMetrics supports PromQL queries via the web UI at https://victoriametrics.benoit.jp.net and via HTTP API:

Query metrics via API
curl -s 'https://victoriametrics.benoit.jp.net/api/v1/query?query=up' | jq .

Grafana Integration

To use VictoriaMetrics as a Grafana data source:

  1. Add a Prometheus data source in Grafana
  2. Set the URL to http://victoriametrics.incus:8428 (from within the Incus network)
  3. Save and test the connection

Adding Scrape Targets

Edit /etc/victoriametrics/promscrape.yml inside the container to add new scrape targets:

Example promscrape.yml
scrape_configs:
  - job_name: victoriametrics
    static_configs:
      - targets:
          - localhost:8428
  - job_name: node_exporter
    static_configs:
      - targets:
          - some-host:9100

After editing, restart vmagent:

incus exec victoriametrics -- systemctl restart vmagent

Related Documentation:

External Resources: