Filtering Metrics Data in Splunk and how It differs from Log Event Filtering
.png)
Background
Our work with Splunk Enterprise 9.3.3, required filtering specific data before it entered a metrics index. For example, we wanted to keep only the metrics we needed, like dropping everything that matched “metric_name:k8s.container.cpu*” before it was stored and counted against our license.
But the catch is, metrics filtering in Splunk, is different from normal event data filtering. Using props/transforms for log events, will not work on metric indexes.
Why Metrics Filtering Is Different
With Events, Splunk processes raw events. That means you can use regex, props, and transforms to mess with the data, even drop events entirely before they are indexed.
With metrics, Splunk stores data as structured numbers, not as raw events. Standard indexing is bypassed, which means props.conf and transforms.Conf fails to work effectively and fails to erase an entire record.
The solution is to use Splunk’s log-to-metrics conversion settings, where you tell Splunk which fields should be considered as metrics, and which you want to ignore.
Working Config Example: How We Dropped Unwanted Metrics
To drop metrics like k8s.container.cpu* or anything with "otelcol" in the name, here is what worked for us:
props.conf:
[<your sourcetype>]
METRIC-SCHEMA-TRANSFORMS = metric-schema:filter_metrics
transforms.conf:
[metric-schema:filter_metrics]
METRIC-SCHEMA-BLACKLIST-DIMS = *otelcol*
METRIC-SCHEMA-MEASURES = os.type
Explanation:
• The blacklist line tells Splunk to drop any metric with a name matching "otelcol*".
• The measures settings makes sure Splunk knows which field is a measurement.
• It than treats others as dimensions and applies the blacklist, respectively.
Note: If you use METRIC-SCHEMA-MEASURES = *, the blacklist will not work right. You must specify the measure, so the blacklist gets used for dimensions (which is where name-based filters work).
What This Means for Your License:
Metrics you filter out this way do not use up your Splunk license volume. If you check license usage before and after, you should see a reduction if your filter is working.
Validation In our tests:
• With no filter, all metrics came in (Outlined in yellow).
• Filtering out "otelcol*" reduced license use (Outlined in red).
• Filtering out "k8s" metrics made the biggest difference (Outlined in blue).
This was clear both in the numbers and in license usage charts.
Summary:
• Splunk metrics filtering happens at field level, not at full event level.
• Use the specific metrics schema settings, not the classic event filters.
In advance of releasing anything into production, always validate it in a development environment, because sometimes license reductions are not obvious if your unwanted metrics are a small portion of all incoming data.
If you get stuck, Splunk’s own support docs are quite good and worth a look.
For more information, explore our Splunk Professional Services.
References:
Set up ingest-time log-to-metrics conversion with configuration files | Splunk Docs
