Planning Infra Nodes for Monitoring

Overview

This document explains how to plan and configure infra nodes for the Monitoring plugins. Use plugin configuration to place Monitoring workloads on infra nodes instead of patching generated workloads after installation.

Supported Configuration Methods

  • For ACP Monitoring with Prometheus, configure placement either in the console through Advanced Configuration or in YAML through spec.config.components.nodeSelector and spec.config.components.tolerations in Installation.
  • For ACP Monitoring with VictoriaMetrics, configure placement either in the console through Advanced Configuration or in YAML through spec.config.components.nodeSelector and spec.config.components.tolerations in Installation.

Do not use patching generated Deployments, StatefulSets, or other plugin-managed workloads as the standard way to place Monitoring workloads on infra nodes.

Before You Configure Placement

Before configuring placement, ensure the following conditions are met:

  • Plan the infra nodes according to Cluster Node Planning.
  • Confirm whether your storage uses LocalVolume or other persistent volumes with spec.nodeAffinity.
  • Make sure the selected infra nodes satisfy both the scheduling rules and the storage placement constraints.

Configure Placement in the Console

Prometheus

When installing or upgrading ACP Monitoring with Prometheus from the console, expand Advanced Configuration and configure the following fields:

Console fieldDescription
Node SelectorsSets plugin-level node selector rules for the Prometheus workloads.
Node TolerationsSets plugin-level toleration rules for the Prometheus workloads.

VictoriaMetrics

When installing or upgrading ACP Monitoring with VictoriaMetrics from the console, expand Advanced Configuration and configure the following fields:

Console fieldDescription
Node SelectorsSets plugin-level node selector rules for the VictoriaMetrics workloads.
Node TolerationsSets plugin-level toleration rules for the VictoriaMetrics workloads.

Configure Placement in YAML

Prometheus

If you want the Prometheus plugin workloads to run on dedicated infra nodes, configure plugin-level scheduling rules during installation or upgrade.

Example:

spec:
  config:
    components:
      nodeSelector:
        - key: kubernetes.io/os
          value: linux
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          operator: Exists

VictoriaMetrics

If you want the VictoriaMetrics plugin workloads to run on dedicated infra nodes, configure plugin-level scheduling rules during installation or upgrade.

Example:

spec:
  config:
    components:
      nodeSelector:
        - key: kubernetes.io/os
          value: linux
      tolerations:
        - effect: NoSchedule
          key: node-role.kubernetes.io/infra
          operator: Exists

When storage.type is LocalVolume, you can select one or more nodes. Make sure every selected storage node also matches the configured node selector rules.

Troubleshooting

Monitoring workloads are still scheduled on general nodes

Check the following items:

  • The target nodes have the expected labels.
  • The configured tolerations match the taints on the infra nodes.
  • The plugin has been upgraded or re-applied with the latest scheduling configuration.

Monitoring workloads cannot be scheduled on the selected infra nodes

This issue usually indicates that the selected nodes do not satisfy one or more scheduling or storage constraints.

Common causes:

  • The infra nodes do not have the labels referenced by nodeSelector.
  • The infra nodes have taints that are not covered by the configured tolerations.
  • The selected LocalVolume nodes or PV nodeAffinity rules point to nodes outside the infra node group.

Learn More

Subsequent Actions

  • Verify that the Monitoring workloads are running on the expected infra nodes.
  • Review whether the selected infra nodes still meet your capacity planning targets.