Last reviewed: 2026-10-15

Setting up Prometheus Blackbox Exporter Probes

Table of Contents

Introduction

Prometheus Blackbox Exporter allows you to monitor external endpoints by probing them from your Kubernetes cluster. This guide will help you set up blackbox probes for HTTP endpoints, TCP services, and more.

Prerequisites

  • Access to an ArgoCD deployment
  • Permissions to create applications in your ArgoCD project
  • Basic understanding of Helm charts and Kubernetes

Setting up Blackbox Probes

Creating the Required Files

To set up blackbox probes, you need to create two key files in your ArgoCD deployment directory:

  1. application.yaml - Defines the Helm chart to use
  2. values.yaml - Configures the probes

application.yaml

Create this file in your deployment directory:

apiVersion: v2
name: prometheus-blackbox-probes
description: Prometheus blackbox probes
version: 0.1.0

helm:
  chart: helm/idp-blackbox-probe
  chartVersion: "1.0.3" # Check https://github.com/jppol-idp/helm-idp/tree/main/charts/idp-blackbox-probe for the latest version

values.yaml

Create this file in the same directory:

probes:
  # -- The name of the probe. Multiple probes can be defined.
  # @default -- N/A
  - name: guide-example-probes
    # -- jobName is optional, if not set it will default to the name of the probe.
    # The jobname is used to group the probes in the prometheus metrics
    jobName: "guide-example-probes"
    # -- type is optional, if not set it will default to http_2xx
    #  - http_2xx will check if the response code is in the 200-299 range
    #  - http_403_as_ok will check if the response code is 403 and treat it as a success
    #  - tcp_connect will check if it can connect to the specified port using tcp
    type: http_2xx
    # -- interval is optional and defines how long between probes, as a duration such as 30s. If not set it will default to 60s
    interval: 60s
    # -- urls is a list of URLs to probe
    urls:
      - 'https://example.com/app/'
      - 'https://api.example.com/health'
      - 'https://status.example.com/'

Configuring Probes

You can define multiple probe configurations in the same values.yaml file. Each probe configuration should include:

  • name: A unique identifier for the probe group
  • jobName (optional): Groups metrics in Prometheus; defaults to the probe name if not specified
  • type: The type of probe to perform (see Probe Types)
  • interval (optional): How frequently to run the probe, as a duration such as 30s; defaults to 60s if not specified
  • urls: A list of endpoints to probe

Example of multiple probe configurations:

probes:
  - name: production-websites
    jobName: "production-website-probes"
    type: http_2xx
    interval: 30s
    urls:
      - 'https://example.com'
      - 'https://blog.example.com'
      
  - name: api-endpoints
    jobName: "api-health-probes"
    type: http_2xx
    interval: 15s
    urls:
      - 'https://api.example.com/health'
      - 'https://api.example.com/status'
      
  - name: database-connectivity
    jobName: "db-connection-probes"
    type: tcp_connect
    urls:
      - 'db.example.com:5432'

Probe Types

The Blackbox Exporter supports various probe types for different monitoring needs:

Type Description
http_2xx Verifies that HTTP endpoints return a status code in the 200-299 range
http_403_as_ok Treats only 403 (Forbidden) as a successful response. Useful for endpoints behind an IP allowlist or authentication, where a 403 shows the endpoint is up
tcp_connect Checks if a TCP connection can be established to the specified host and port

For HTTP probes, provide the full URL including http:// or https://. For TCP probes, use the format host:port without a scheme, for example db.example.com:5432 or 10.0.0.10:5432.

Each probe attempt times out after 5 seconds.

Probing in-cluster services

When probing a service that runs inside the same Kubernetes cluster, the URL must use the service’s fully qualified domain name (FQDN). The short service name form is not accepted by the probe’s URL validation.

Use the FQDN form:

http://<service>.<namespace>.svc.cluster.local:<port>/<path>

For example, instead of:

http://my-service:5000/healthz/live

use:

http://my-service.my-namespace.svc.cluster.local:5000/healthz/live

Note: Need additional probe types? If you require custom probe configurations (such as treating HTTP 404 as a success indicator or other specific validation logic), please contact the IDP team directly. The IDP team can implement and support custom probe modules tailored to your specific monitoring requirements.

Viewing Probe Results

After setting up your probes, you can view the results in Grafana:

  1. Access your cluster’s Grafana instance (e.g., https://grafana.<cluster-name>.idp.jppol.dk)
  2. Navigate to the “Blackbox Exporter (HTTP prober)” dashboard
  3. Filter by job name to see your specific probe results

The dashboard provides information about:

  • Probe success/failure rates
  • Response times
  • Status code distributions
  • SSL/TLS certificate expiration (for HTTPS endpoints)

Troubleshooting

Common Issues

Issue Possible Solution
Probes failing with timeout errors Check network connectivity between the cluster and the target. The timeout is fixed at 5 seconds; contact the IDP team if your endpoint needs longer
SSL certificate validation failures Ensure the certificate is valid and issued by a trusted certificate authority. Skipping verification is not supported
Inconsistent results Check for load balancers or CDNs that might route requests differently
No data appearing in Grafana Check that the application is synced in ArgoCD and see Checking Probe Configuration

Checking Probe Configuration

To verify your probe configuration has been applied:

  1. Open your application in ArgoCD and check that it is synced. Invalid values, such as an unknown probe type or a URL in the wrong format, are rejected by the chart and shown as a sync error.
  2. Access your cluster’s Grafana instance (e.g., https://grafana.<cluster-name>.idp.jppol.dk)
  3. Navigate to the “Explore” section (compass icon in the left sidebar)
  4. Select “Prometheus” as the data source
  5. Enter the following query, replacing my-probes with the jobName of your probe: 
    probe_success{job="my-probes"}
  6. Click “Run query”. Each URL appears as a separate series, with 1 for success and 0 for failure

Additional Resources