Planning Infra Nodes for Monitoring
TOC
OverviewSupported Configuration MethodsBefore You Configure PlacementConfigure Placement in the ConsolePrometheusVictoriaMetricsConfigure Placement in YAMLPrometheusVictoriaMetricsTroubleshootingMonitoring workloads are still scheduled on general nodesMonitoring workloads cannot be scheduled on the selected infra nodesLearn MoreSubsequent ActionsOverview
This document explains how to plan and configure infra nodes for the Monitoring plugins. Use plugin configuration to place Monitoring workloads on infra nodes instead of patching generated workloads after installation.
Supported Configuration Methods
- For ACP Monitoring with Prometheus, configure placement either in the console through Advanced Configuration or in YAML through
spec.config.components.nodeSelectorandspec.config.components.tolerationsin Installation. - For ACP Monitoring with VictoriaMetrics, configure placement either in the console through Advanced Configuration or in YAML through
spec.config.components.nodeSelectorandspec.config.components.tolerationsin Installation.
Do not use patching generated Deployments, StatefulSets, or other plugin-managed workloads as the standard way to place Monitoring workloads on infra nodes.
Before You Configure Placement
Before configuring placement, ensure the following conditions are met:
- Plan the infra nodes according to Cluster Node Planning.
- Confirm whether your storage uses
LocalVolumeor other persistent volumes withspec.nodeAffinity. - Make sure the selected infra nodes satisfy both the scheduling rules and the storage placement constraints.
Configure Placement in the Console
Prometheus
When installing or upgrading ACP Monitoring with Prometheus from the console, expand Advanced Configuration and configure the following fields:
VictoriaMetrics
When installing or upgrading ACP Monitoring with VictoriaMetrics from the console, expand Advanced Configuration and configure the following fields:
Configure Placement in YAML
Prometheus
If you want the Prometheus plugin workloads to run on dedicated infra nodes, configure plugin-level scheduling rules during installation or upgrade.
Example:
VictoriaMetrics
If you want the VictoriaMetrics plugin workloads to run on dedicated infra nodes, configure plugin-level scheduling rules during installation or upgrade.
Example:
When storage.type is LocalVolume, you can select one or more nodes. Make sure every selected storage node also matches the configured node selector rules.
Troubleshooting
Monitoring workloads are still scheduled on general nodes
Check the following items:
- The target nodes have the expected labels.
- The configured tolerations match the taints on the infra nodes.
- The plugin has been upgraded or re-applied with the latest scheduling configuration.
Monitoring workloads cannot be scheduled on the selected infra nodes
This issue usually indicates that the selected nodes do not satisfy one or more scheduling or storage constraints.
Common causes:
- The infra nodes do not have the labels referenced by
nodeSelector. - The infra nodes have taints that are not covered by the configured tolerations.
- The selected
LocalVolumenodes or PVnodeAffinityrules point to nodes outside the infra node group.
Learn More
Subsequent Actions
- Verify that the Monitoring workloads are running on the expected infra nodes.
- Review whether the selected infra nodes still meet your capacity planning targets.