Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ jobs:
echo "DOCKER_CLI_EXPERIMENTAL=enabled" >> $GITHUB_ENV

- name: Set up Golang
uses: actions/setup-go@v4
uses: actions/setup-go@v5
id: go
with:
go-version: ${{ env.GOVERSION }}
Expand Down
2 changes: 1 addition & 1 deletion .golangci.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: "2"

run:
timeout: 5m
go: '1.23'
go: '1.25'

linters:
enable:
Expand Down
2 changes: 1 addition & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# Build stage
FROM golang:1.23.10 AS builder
FROM golang:1.25.9 AS builder

# Set non-root user for build
RUN useradd -u 10001 -m builder
Expand Down
56 changes: 37 additions & 19 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,31 +1,49 @@
# Github Prometheus Exporter (promgithub)

The `promgithub` service is a lightweight service designed to receive and process GitHub webhook events (commits, pull requests, workflow jobs and workflow runs). The webhook events are converted to prometheus metrics, allowing monitoring and insights into GitHub activities.
`promgithub` is a service that receives GitHub webhook events and exposes Prometheus metrics for repository activity, workflow runs, workflow jobs, commits, and pull requests.

## Metrics Exported by the Service
It is designed to be simple to deploy and can run either:
- as a single instance
- as multiple instances with Redis for shared deduplication and state

The `promgithub` service exports the following Prometheus metrics:
## Metrics exported

| Name | Type | Labels | Description |
|------------------------------------|-----------|---------------------------------------------------------------|-------------------------------------------|
| `promgithub_workflow_status` | Counter | `repository`, `branch`, `workflow_name`, `workflow_status`, `conclusion`| Total number of workflow runs with status |
| `promgithub_workflow_duration` | Histogram | `repository`, `branch`, `workflow_name`, `workflow_status`, `conclusion`| Duration of workflow runs |
| `promgithub_workflow_queued` | Gauge | `repository`, `branch`, `workflow_name` | Number of workflow runs queued |
| `promgithub_workflow_in_progress` | Gauge | `repository`, `branch`, `workflow_name` | Number of workflow runs in progress |
| `promgithub_workflow_completed` | Gauge | `repository`, `branch`, `workflow_conclusion`,`workflow_name` | Number of workflow runs completed |
`promgithub` exports the following metrics:

| Name | Type | Labels | Description |
|------------------------------------|-----------|-------------------------------------------------------------------------|-------------------------------------------|
| `promgithub_workflow_status` | Counter | `repository`, `branch`, `workflow_name`, `workflow_status`, `conclusion` | Total number of workflow runs with status |
| `promgithub_workflow_duration` | Histogram | `repository`, `branch`, `workflow_name`, `workflow_status`, `conclusion` | Duration of workflow runs |
| `promgithub_workflow_queued` | Gauge | `repository`, `branch`, `workflow_name` | Number of workflow runs queued |
| `promgithub_workflow_in_progress` | Gauge | `repository`, `branch`, `workflow_name` | Number of workflow runs in progress |
| `promgithub_workflow_completed` | Gauge | `repository`, `branch`, `workflow_conclusion`, `workflow_name` | Number of workflow runs completed |
| `promgithub_job_status` | Counter | `repository`, `branch`, `workflow_name`, `job_status`, `job_conclusion` | Total number of jobs with status |
| `promgithub_job_duration` | Histogram | `repository`, `branch`, `workflow_name`, `job_status`, `job_conclusion` | Duration of jobs runs in seconds |
| `promgithub_job_queued` | Gauge | `repository`, `branch`, `workflow_name` | Number of jobs queued |
| `promgithub_job_in_progress` | Gauge | `repository`, `branch`, `workflow_name` | Number of jobs in progress |
| `promgithub_job_completed` | Gauge | `repository`, `branch`, `job_conclusion`, `workflow_name` | Number of jobs completed |
| `promgithub_commit_pushed` | Counter | `repository` | Total number of commits pushed |
| `promgithub_pull_request` | Counter | `repository`, `base_branch`, `pull_request_status` | Total number of pull requests |
| `promgithub_job_queued` | Gauge | `repository`, `branch`, `workflow_name` | Number of jobs queued |
| `promgithub_job_in_progress` | Gauge | `repository`, `branch`, `workflow_name` | Number of jobs in progress |
| `promgithub_job_completed` | Gauge | `repository`, `branch`, `job_conclusion`, `workflow_name` | Number of jobs completed |
| `promgithub_commit_pushed` | Counter | `repository` | Total number of commits pushed |
| `promgithub_pull_request` | Counter | `repository`, `base_branch`, `pull_request_status` | Total number of pull requests |

## Metric model

The exporter focuses on repository and workflow health signals while avoiding noisy per-entity labels such as runner names, job names, commit author identities, and pull request authors.

This keeps the default metric set compact and practical for Prometheus while still preserving the `branch` label for branch-specific workflow and job visibility.

## Redis-backed multi-instance mode

When Redis is configured, `promgithub` uses it for:
- webhook delivery deduplication using `X-GitHub-Delivery`
- shared workflow run state storage
- shared workflow job state storage

This allows multiple `promgithub` instances to share delivery and run state through a common backend.

## Using `promgithub` service
## Using promgithub

For usage information see [Usage documentation](./docs/usage.md)
See [Usage documentation](./docs/usage.md) for deployment and configuration examples.

## Contributing to `promgithub` service
## Contributing

For contributing guidelines see [Contributing documentation](./docs/contributing.md)
See [Contributing documentation](./docs/contributing.md).
207 changes: 125 additions & 82 deletions docs/usage.md
Original file line number Diff line number Diff line change
@@ -1,134 +1,179 @@
# Using `promgithub` service

## Deploying the service
## Overview

The service can be deployed in your choice of infrastructure. To allow webhooks to be pushed to `promgithub` make sure your service deployment is accessible from your Github instance.
`promgithub` receives GitHub webhook events and exposes Prometheus metrics over HTTP.

### Service Parameters
It can be deployed:
- as a single instance with only the webhook secret configured
- as a multi-instance deployment with Redis configured for shared deduplication and state

The service expects the following parameters to be set:
## Configuration

- **Environment Variables**:
- `PROMGITHUB_WEBHOOK_SECRET`: The secret used to validate incoming GitHub webhook requests.
- `PROMGITHUB_SERVICE_PORT` (optional): Service API port (default is `8080`).
### Environment variables

## Deploying in kubernetes
The service supports the following environment variables:

To deploy the service in a kubernetes cluster you can use the provided helm chart from the [promgithub chart repository](https://github.com/darthfork/promgithub/pkgs/container/promgithub-charts%2Fpromgithub)
- `PROMGITHUB_WEBHOOK_SECRET`: Secret used to validate incoming GitHub webhook requests.
- `PROMGITHUB_SERVICE_PORT` (optional): HTTP port for the service, default `8080`.
- `PROMGITHUB_REDIS_ADDR` (optional): Redis address in `host:port` form.
- `PROMGITHUB_REDIS_PASSWORD` (optional): Redis password.
- `PROMGITHUB_REDIS_DB` (optional): Redis database number, default `0`.
- `PROMGITHUB_REDIS_KEY_PREFIX` (optional): Prefix used for Redis keys, default `promgithub`.
- `PROMGITHUB_REDIS_DELIVERY_TTL` (optional): TTL for webhook delivery dedupe keys, default `24h`.

When deploying with kubernetes add the following resources and configurations to your `promgithub` deployment
If Redis is configured, the service stores delivery and run state in Redis.

### Chart.yaml
## Running the service

Add the helm repository as a dependency to your chart deployment:
### Run the binary

```yaml
apiVersion: v2
name: promgithub
description: Deployment of promgithub
type: application
version: <chart version>

dependencies:
- name: promgithub
version: "<promgithub-charts version>"
repository: "oci://ghcr.io/darthfork/promgithub-charts"
```bash
PROMGITHUB_WEBHOOK_SECRET="<your webhook secret>" \
PROMGITHUB_SERVICE_PORT="8080" \
/path/to/binary/promgithub
```

### Ingress
### Run the binary with Redis

Add an Ingress configuration allowing your github instance to access `promgithub` deployment. More details can be found [here](https://kubernetes.io/docs/concepts/services-networking/ingress/)
```bash
PROMGITHUB_WEBHOOK_SECRET="<your webhook secret>" \
PROMGITHUB_REDIS_ADDR="<redis-host:6379>" \
PROMGITHUB_REDIS_PASSWORD="<redis password>" \
PROMGITHUB_REDIS_DB="0" \
PROMGITHUB_REDIS_KEY_PREFIX="promgithub" \
PROMGITHUB_REDIS_DELIVERY_TTL="24h" \
PROMGITHUB_SERVICE_PORT="8080" \
/path/to/binary/promgithub
```

### Docker

### Values
```bash
docker run \
-e PROMGITHUB_WEBHOOK_SECRET=<your webhook secret> \
-e PROMGITHUB_SERVICE_PORT=8080 \
-p 8080:8080 \
ghcr.io/darthfork/promgithub:<version>
```

Create a values file with the webhook secret and (optional) service port values for your chart
### Docker with Redis

```yaml
promgithub:
secrets:
github_webhook_secret: <your webhook secret> # Mounted as PROMGITHUB_WEBHOOK_SECRET in the deployment
service:
port: <service port> # optional (default is 8080)
```bash
docker run \
-e PROMGITHUB_WEBHOOK_SECRET=<your webhook secret> \
-e PROMGITHUB_REDIS_ADDR=<redis-host:6379> \
-e PROMGITHUB_REDIS_PASSWORD=<redis password> \
-e PROMGITHUB_REDIS_DB=0 \
-e PROMGITHUB_REDIS_KEY_PREFIX=promgithub \
-e PROMGITHUB_REDIS_DELIVERY_TTL=24h \
-e PROMGITHUB_SERVICE_PORT=8080 \
-p 8080:8080 \
ghcr.io/darthfork/promgithub:<version>
```

**Default values**: The default values file for promgithub can be found [here](https://github.com/darthfork/promgithub/blob/main/helm/promgithub/values.yaml)
### Docker Compose with Redis

### Metrics Scraping
```yaml
services:
redis:
image: redis:7
command: ["redis-server", "--appendonly", "yes"]
ports:
- "6379:6379"

Create prometheus configuration resource with the chart for scraping metrics from the `/metrics` endpoint from promgithub service. For more details see the [Prometheus scraping configuration](#prometheus-scraping-configuration) below.
promgithub:
image: ghcr.io/darthfork/promgithub:<version>
environment:
PROMGITHUB_WEBHOOK_SECRET: <your webhook secret>
PROMGITHUB_REDIS_ADDR: redis:6379
PROMGITHUB_REDIS_PASSWORD: <redis password>
PROMGITHUB_REDIS_DB: 0
PROMGITHUB_REDIS_KEY_PREFIX: promgithub
PROMGITHUB_REDIS_DELIVERY_TTL: 24h
PROMGITHUB_SERVICE_PORT: 8080
ports:
- "8080:8080"
depends_on:
- redis
```

## Deploying service in a container
## Deploying with Kubernetes

To deploy the service in a container using a container management environment like fargate/docker-compose, you can use the `promgithub` container from the [GHCR container repository](https://github.com/darthfork/promgithub/pkgs/container/promgithub)
`promgithub` includes a Helm chart.

### Docker CLI
### Add the chart dependency

Run the service using docker cli as follows:
```yaml
apiVersion: v2
name: promgithub
description: Deployment of promgithub
type: application
version: <chart version>

```bash
docker run\
-e PROMGITHUB_WEBHOOK_SECRET=<your webhook secret>\
-e PROMGITHUB_SERVICE_PORT=<service port>\
-p <HOST_PORT>:<CONTAINER_PORT>\
ghcr.io/darthfork/promgithub:<version>
dependencies:
- name: promgithub
version: "<promgithub-charts version>"
repository: "oci://ghcr.io/darthfork/promgithub-charts"
```

### Docker Compose

To run the service in docker compose, create a compose file as below:
### Values for an external Redis instance

```yaml
# promgithub-compose.yaml
services:
promgithub:
image: ghcr.io/darthfork/promgithub:<version>
hostname: promgithub
stdin_open: false
tty: false
environment:
- PROMGITHUB_WEBHOOK_SECRET=<your webhook secret>
- PROMGITHUB_SERVICE_PORT=<service port (optional)>
ports:
- <HOST_PORT>:<CONTAINER_PORT>
promgithub:
secrets:
github_webhook_secret: <your webhook secret>
redis_password: <redis password>
redisConfig:
addr: redis.example.internal:6379
db: 0
keyPrefix: promgithub
deliveryTTL: 24h
```

To start the `promgithub` container with compose run:
### Values for a bundled Redis deployment

```bash
docker-compose -f promgithub-compose.yaml run --rm promgithub
```yaml
promgithub:
secrets:
github_webhook_secret: <your webhook secret>
redis:
enabled: true
auth:
enabled: true
password: <redis password>
redisConfig:
db: 0
keyPrefix: promgithub
deliveryTTL: 24h
```

## Deploying service binary
When `redis.enabled=true`, the chart deploys Redis as a dependency and configures `promgithub` to connect to it automatically.

The service binaries are also available under [github releases](https://github.com/darthfork/promgithub/releases) which can be deployed as the user wishes.
### Ingress

```bash
PROMGITHUB_WEBHOOK_SECRET="<your webhook secret>" PROMGITHUB_SERVICE_PORT="<service port>" /path/to/binary/promgithub
```
Expose the `/webhook` endpoint to GitHub using your preferred Kubernetes ingress setup.

## Setting up the Webhook in GitHub (Repository/Organization)
## Setting up the GitHub webhook

1. Navigate to your GitHub repository or organization settings.
2. Under **Settings**, find **Webhooks** and click **Add webhook**.
3. Enter the payload URL pointing to your `promgithub` service, e.g., `http://<your-service-url>/webhook`.
4. Set the **Content type** to `application/json`.
5. Add the **Secret**: Use the value of `PROMGITHUB_WEBHOOK_SECRET`.
6. Select the following events to trigger the webhook:
2. Under **Settings**, open **Webhooks** and click **Add webhook**.
3. Set the payload URL to your `promgithub` webhook endpoint, for example `https://<your-service-url>/webhook`.
4. Set **Content type** to `application/json`.
5. Set the **Secret** to the value used for `PROMGITHUB_WEBHOOK_SECRET`.
6. Subscribe to these events:
- **push**
- **pull request**
- **workflow job**
- **workflow runs**
7. Click **Add webhook** to save.
7. Save the webhook.

## Prometheus scraping configuration
## Scraping metrics

Configure prometheus to scrape `promgithub`'s `/metrics` endpoint to extract metrics.
`promgithub` exposes Prometheus metrics on `/metrics`.

### Prometheus configuration

To allow prometheus to scrape `promgithub`'s `/metrics` endpoint, add the following configuration to your prometheus setup:

```yaml
scrape_configs:
- job_name: 'promgithub'
Expand All @@ -142,8 +187,6 @@ scrape_configs:

### VictoriaMetrics configuration

If you use victoria-metrics as your metrics provider, add a `vmservicescrape` configuration to your `promgithub` chart deployment

```yaml
apiVersion: operator.victoriametrics.com/v1beta1
kind: VMServiceScrape
Expand Down
4 changes: 3 additions & 1 deletion go.mod
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
module promgithub

go 1.23.10
go 1.25.9

require (
github.com/gorilla/mux v1.8.1
github.com/prometheus/client_golang v1.20.5
github.com/redis/go-redis/v9 v9.7.3
github.com/stretchr/testify v1.9.0
go.uber.org/zap v1.27.0
)
Expand All @@ -13,6 +14,7 @@ require (
github.com/beorn7/perks v1.0.1 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/davecgh/go-spew v1.1.1 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/klauspost/compress v1.17.9 // indirect
github.com/kylelemons/godebug v1.1.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
Expand Down
Loading
Loading