285 lines
18 KiB
Markdown
285 lines
18 KiB
Markdown
## Prometheus 2.x
|
|
|
|
Prometheus version 2.x has made changes to alertmanager, storage and recording rules. Check out the migration guide [here](https://prometheus.io/docs/prometheus/2.0/migration/)
|
|
|
|
Users of this chart will need to update their alerting rules to the new format before they can upgrade.
|
|
|
|
## Upgrading from previous chart versions.
|
|
|
|
As of version 5.0, this chart uses Prometheus 2.1. This version of prometheus introduces a new data format and is not compatible with prometheus 1.x. It is recommended to install this as a new release, as updating existing releases will not work. See the [prometheus docs](https://prometheus.io/docs/prometheus/latest/migration/#storage) for instructions on retaining your old data.
|
|
|
|
### Example migration
|
|
|
|
Assuming you have an existing release of the prometheus chart, named `prometheus-old`. In order to update to prometheus 2.1 while keeping your old data do the following:
|
|
|
|
1. Update the `prometheus-old` release. Disable scraping on every component besides the prometheus server, similar to the configuration below:
|
|
|
|
```
|
|
alertmanager:
|
|
enabled: false
|
|
alertmanagerFiles:
|
|
alertmanager.yml: ""
|
|
kubeStateMetrics:
|
|
enabled: false
|
|
nodeExporter:
|
|
enabled: false
|
|
pushgateway:
|
|
enabled: false
|
|
server:
|
|
extraArgs:
|
|
storage.local.retention: 720h
|
|
serverFiles:
|
|
alerts: ""
|
|
prometheus.yml: ""
|
|
rules: ""
|
|
```
|
|
|
|
1. Deploy a new release of the chart with version 5.0+ using prometheus 2.x. In the values.yaml set the scrape config as usual, and also add the `prometheus-old` instance as a remote-read target.
|
|
|
|
```
|
|
prometheus.yml:
|
|
...
|
|
remote_read:
|
|
- url: http://prometheus-old/api/v1/read
|
|
...
|
|
```
|
|
|
|
Old data will be available when you query the new prometheus instance.
|
|
|
|
## Configuration
|
|
|
|
The following table lists the configurable parameters of the Prometheus chart and their default values.
|
|
|
|
Parameter | Description | Default
|
|
--------- | ----------- | -------
|
|
`basic.namespace.server` | GIN Server Namespace | `gin-server`
|
|
`basic.namespace.stateless` | GIN Stateless Namespace | `gin-stateless`
|
|
`basic.runVersion` | Set test version | `stable`
|
|
`basic.runController` | Set test job | `all`
|
|
`basic.gatewayPort` | Set port | `8080`
|
|
`basic.runDebugMode` | Enable debug | `false`
|
|
`basic.dockerTag` | Set test revision | `4.8.3-0`
|
|
`basic.disablePostgres` | If true, tests will be run locally | `true`
|
|
`basic.repository` | Set repository | `nexus.iqser.com:5001/`
|
|
`nextcloud.username` | Set username | `***`
|
|
`nextcloud.password` | Set password | `***`
|
|
`nextcloud.url` | Set url | `https://cloud.iqser.com/remote.php/webdav`
|
|
`test.config.content_processing_timeout` | Set content processing timeout | `10`
|
|
`test.config.warmup_wait_time` | Set warmup wait time | `0`
|
|
`test.config.discovery.retries` | Set number of retries for discovery | `20`
|
|
`test.config.discovery.timeout` | Set timeout for discovery | `20`
|
|
`test.config.loadtest.users` | alertmanager Ingress TLS configuration (YAML) | `10`
|
|
`test.config.loadtest.duration` | node labels for alertmanager pod assignment | `10`
|
|
|
|
`alertmanager.persistentVolume.storageClass` | alertmanager data Persistent Volume Storage Class | `unset`
|
|
`alertmanager.persistentVolume.subPath` | Subdirectory of alertmanager data Persistent Volume to mount | `""`
|
|
`alertmanager.podAnnotations` | annotations to be added to alertmanager pods | `{}`
|
|
`alertmanager.replicaCount` | desired number of alertmanager pods | `1`
|
|
`alertmanager.resources` | alertmanager pod resource requests & limits | `{}`
|
|
`alertmanager.service.annotations` | annotations for alertmanager service | `{}`
|
|
`alertmanager.service.clusterIP` | internal alertmanager cluster service IP | `""`
|
|
`alertmanager.service.externalIPs` | alertmanager service external IP addresses | `[]`
|
|
`alertmanager.service.loadBalancerIP` | IP address to assign to load balancer (if supported) | `""`
|
|
`alertmanager.service.loadBalancerSourceRanges` | list of IP CIDRs allowed access to load balancer (if supported) | `[]`
|
|
`alertmanager.service.servicePort` | alertmanager service port | `80`
|
|
`alertmanager.service.type` | type of alertmanager service to create | `ClusterIP`
|
|
`alertmanagerFiles.alertmanager.yml` | Prometheus alertmanager configuration | example configuration
|
|
`configmapReload.name` | configmap-reload container name | `configmap-reload`
|
|
`configmapReload.image.repository` | configmap-reload container image repository | `jimmidyson/configmap-reload`
|
|
`configmapReload.image.tag` | configmap-reload container image tag | `v0.1`
|
|
`configmapReload.image.pullPolicy` | configmap-reload container image pull policy | `IfNotPresent`
|
|
`configmapReload.extraArgs` | Additional configmap-reload container arguments | `{}`
|
|
`configmapReload.extraConfigmapMounts` | Additional configmap-reload configMap mounts | `[]`
|
|
`configmapReload.resources` | configmap-reload pod resource requests & limits | `{}`
|
|
`initChownData.enabled` | If false, don't reset data ownership at startup | true
|
|
`initChownData.name` | init-chown-data container name | `init-chown-data`
|
|
`initChownData.image.repository` | init-chown-data container image repository | `busybox`
|
|
`initChownData.image.tag` | init-chown-data container image tag | `latest`
|
|
`initChownData.image.pullPolicy` | init-chown-data container image pull policy | `IfNotPresent`
|
|
`initChownData.resources` | init-chown-data pod resource requests & limits | `{}`
|
|
`kubeStateMetrics.enabled` | If true, create kube-state-metrics | `true`
|
|
`kubeStateMetrics.name` | kube-state-metrics container name | `kube-state-metrics`
|
|
`kubeStateMetrics.image.repository` | kube-state-metrics container image repository| `k8s.gcr.io/kube-state-metrics`
|
|
`kubeStateMetrics.image.tag` | kube-state-metrics container image tag | `v1.1.0`
|
|
`kubeStateMetrics.image.pullPolicy` | kube-state-metrics container image pull policy | `IfNotPresent`
|
|
`kubeStateMetrics.args` | kube-state-metrics container arguments | `{}`
|
|
`kubeStateMetrics.nodeSelector` | node labels for kube-state-metrics pod assignment | `{}`
|
|
`kubeStateMetrics.podAnnotations` | annotations to be added to kube-state-metrics pods | `{}`
|
|
`kubeStateMetrics.tolerations` | node taints to tolerate (requires Kubernetes >=1.6) | `[]`
|
|
`kubeStateMetrics.replicaCount` | desired number of kube-state-metrics pods | `1`
|
|
`kubeStateMetrics.resources` | kube-state-metrics resource requests and limits (YAML) | `{}`
|
|
`kubeStateMetrics.service.annotations` | annotations for kube-state-metrics service | `{prometheus.io/scrape: "true"}`
|
|
`kubeStateMetrics.service.clusterIP` | internal kube-state-metrics cluster service IP | `None`
|
|
`kubeStateMetrics.service.externalIPs` | kube-state-metrics service external IP addresses | `[]`
|
|
`kubeStateMetrics.service.loadBalancerIP` | IP address to assign to load balancer (if supported) | `""`
|
|
`kubeStateMetrics.service.loadBalancerSourceRanges` | list of IP CIDRs allowed access to load balancer (if supported) | `[]`
|
|
`kubeStateMetrics.service.servicePort` | kube-state-metrics service port | `80`
|
|
`kubeStateMetrics.service.type` | type of kube-state-metrics service to create | `ClusterIP`
|
|
`nodeExporter.enabled` | If true, create node-exporter | `true`
|
|
`nodeExporter.name` | node-exporter container name | `node-exporter`
|
|
`nodeExporter.image.repository` | node-exporter container image repository| `prom/node-exporter`
|
|
`nodeExporter.image.tag` | node-exporter container image tag | `v0.15.2`
|
|
`nodeExporter.image.pullPolicy` | node-exporter container image pull policy | `IfNotPresent`
|
|
`nodeExporter.extraArgs` | Additional node-exporter container arguments | `{}`
|
|
`nodeExporter.extraHostPathMounts` | Additional node-exporter hostPath mounts | `[]`
|
|
`nodeExporter.extraConfigmapMounts` | Additional node-exporter configMap mounts | `[]`
|
|
`nodeExporter.nodeSelector` | node labels for node-exporter pod assignment | `{}`
|
|
`nodeExporter.podAnnotations` | annotations to be added to node-exporter pods | `{}`
|
|
`nodeExporter.tolerations` | node taints to tolerate (requires Kubernetes >=1.6) | `[]`
|
|
`nodeExporter.resources` | node-exporter resource requests and limits (YAML) | `{}`
|
|
`nodeExporter.securityContext` | securityContext for containers in pod | `{}`
|
|
`nodeExporter.service.annotations` | annotations for node-exporter service | `{prometheus.io/scrape: "true"}`
|
|
`nodeExporter.service.clusterIP` | internal node-exporter cluster service IP | `None`
|
|
`nodeExporter.service.externalIPs` | node-exporter service external IP addresses | `[]`
|
|
`nodeExporter.service.loadBalancerIP` | IP address to assign to load balancer (if supported) | `""`
|
|
`nodeExporter.service.loadBalancerSourceRanges` | list of IP CIDRs allowed access to load balancer (if supported) | `[]`
|
|
`nodeExporter.service.servicePort` | node-exporter service port | `9100`
|
|
`nodeExporter.service.type` | type of node-exporter service to create | `ClusterIP`
|
|
`pushgateway.enabled` | If true, create pushgateway | `true`
|
|
`pushgateway.name` | pushgateway container name | `pushgateway`
|
|
`pushgateway.image.repository` | pushgateway container image repository | `prom/pushgateway`
|
|
`pushgateway.image.tag` | pushgateway container image tag | `v0.4.0`
|
|
`pushgateway.image.pullPolicy` | pushgateway container image pull policy | `IfNotPresent`
|
|
`pushgateway.extraArgs` | Additional pushgateway container arguments | `{}`
|
|
`pushgateway.ingress.enabled` | If true, pushgateway Ingress will be created | `false`
|
|
`pushgateway.ingress.annotations` | pushgateway Ingress annotations | `{}`
|
|
`pushgateway.ingress.hosts` | pushgateway Ingress hostnames | `[]`
|
|
`pushgateway.ingress.tls` | pushgateway Ingress TLS configuration (YAML) | `[]`
|
|
`pushgateway.nodeSelector` | node labels for pushgateway pod assignment | `{}`
|
|
`pushgateway.podAnnotations` | annotations to be added to pushgateway pods | `{}`
|
|
`pushgateway.tolerations` | node taints to tolerate (requires Kubernetes >=1.6) | `[]`
|
|
`pushgateway.replicaCount` | desired number of pushgateway pods | `1`
|
|
`pushgateway.resources` | pushgateway pod resource requests & limits | `{}`
|
|
`pushgateway.service.annotations` | annotations for pushgateway service | `{}`
|
|
`pushgateway.service.clusterIP` | internal pushgateway cluster service IP | `""`
|
|
`pushgateway.service.externalIPs` | pushgateway service external IP addresses | `[]`
|
|
`pushgateway.service.loadBalancerIP` | IP address to assign to load balancer (if supported) | `""`
|
|
`pushgateway.service.loadBalancerSourceRanges` | list of IP CIDRs allowed access to load balancer (if supported) | `[]`
|
|
`pushgateway.service.servicePort` | pushgateway service port | `9091`
|
|
`pushgateway.service.type` | type of pushgateway service to create | `ClusterIP`
|
|
`rbac.create` | If true, create & use RBAC resources | `true`
|
|
`server.name` | Prometheus server container name | `server`
|
|
`server.image.repository` | Prometheus server container image repository | `prom/prometheus`
|
|
`server.image.tag` | Prometheus server container image tag | `v2.1.0`
|
|
`server.image.pullPolicy` | Prometheus server container image pull policy | `IfNotPresent`
|
|
`server.extraArgs` | Additional Prometheus server container arguments | `{}`
|
|
`server.prefixURL` | The prefix slug at which the server can be accessed | ``
|
|
`server.baseURL` | The external url at which the server can be accessed | ``
|
|
`server.extraHostPathMounts` | Additional Prometheus server hostPath mounts | `[]`
|
|
`server.extraConfigmapMounts` | Additional Prometheus server configMap mounts | `[]`
|
|
`server.extraSecretMounts` | Additional Prometheus server Secret mounts | `[]`
|
|
`server.configMapOverrideName` | Prometheus server ConfigMap override where full-name is `{{.Release.Name}}-{{.Values.server.configMapOverrideName}}` and setting this value will prevent the default server ConfigMap from being generated | `""`
|
|
`server.ingress.enabled` | If true, Prometheus server Ingress will be created | `false`
|
|
`server.ingress.annotations` | Prometheus server Ingress annotations | `[]`
|
|
`server.ingress.hosts` | Prometheus server Ingress hostnames | `[]`
|
|
`server.ingress.tls` | Prometheus server Ingress TLS configuration (YAML) | `[]`
|
|
`server.nodeSelector` | node labels for Prometheus server pod assignment | `{}`
|
|
`server.tolerations` | node taints to tolerate (requires Kubernetes >=1.6) | `[]`
|
|
`server.persistentVolume.enabled` | If true, Prometheus server will create a Persistent Volume Claim | `true`
|
|
`server.persistentVolume.accessModes` | Prometheus server data Persistent Volume access modes | `[ReadWriteOnce]`
|
|
`server.persistentVolume.annotations` | Prometheus server data Persistent Volume annotations | `{}`
|
|
`server.persistentVolume.existingClaim` | Prometheus server data Persistent Volume existing claim name | `""`
|
|
`server.persistentVolume.mountPath` | Prometheus server data Persistent Volume mount root path | `/data`
|
|
`server.persistentVolume.size` | Prometheus server data Persistent Volume size | `8Gi`
|
|
`server.persistentVolume.storageClass` | Prometheus server data Persistent Volume Storage Class | `unset`
|
|
`server.persistentVolume.subPath` | Subdirectory of Prometheus server data Persistent Volume to mount | `""`
|
|
`server.podAnnotations` | annotations to be added to Prometheus server pods | `{}`
|
|
`server.replicaCount` | desired number of Prometheus server pods | `1`
|
|
`server.resources` | Prometheus server resource requests and limits | `{}`
|
|
`server.service.annotations` | annotations for Prometheus server service | `{}`
|
|
`server.service.clusterIP` | internal Prometheus server cluster service IP | `""`
|
|
`server.service.externalIPs` | Prometheus server service external IP addresses | `[]`
|
|
`server.service.loadBalancerIP` | IP address to assign to load balancer (if supported) | `""`
|
|
`server.service.loadBalancerSourceRanges` | list of IP CIDRs allowed access to load balancer (if supported) | `[]`
|
|
`server.service.nodePort` | Port to be used as the service NodePort (ignored if `server.service.type` is not `NodePort`) | `0`
|
|
`server.service.servicePort` | Prometheus server service port | `80`
|
|
`server.service.type` | type of Prometheus server service to create | `ClusterIP`
|
|
`serviceAccounts.alertmanager.create` | If true, create the alertmanager service account | `true`
|
|
`serviceAccounts.alertmanager.name` | name of the alertmanager service account to use or create | `{{ prometheus.alertmanager.fullname }}`
|
|
`serviceAccounts.kubeStateMetrics.create` | If true, create the kubeStateMetrics service account | `true`
|
|
`serviceAccounts.kubeStateMetrics.name` | name of the kubeStateMetrics service account to use or create | `{{ prometheus.kubeStateMetrics.fullname }}`
|
|
`serviceAccounts.nodeExporter.create` | If true, create the nodeExporter service account | `true`
|
|
`serviceAccounts.nodeExporter.name` | name of the nodeExporter service account to use or create | `{{ prometheus.nodeExporter.fullname }}`
|
|
`serviceAccounts.pushgateway.create` | If true, create the pushgateway service account | `true`
|
|
`serviceAccounts.pushgateway.name` | name of the pushgateway service account to use or create | `{{ prometheus.pushgateway.fullname }}`
|
|
`serviceAccounts.server.create` | If true, create the server service account | `true`
|
|
`serviceAccounts.server.name` | name of the server service account to use or create | `{{ prometheus.server.fullname }}`
|
|
`server.terminationGracePeriodSeconds` | Prometheus server Pod termination grace period | `300`
|
|
`server.retention` | (optional) Prometheus data retention | `""`
|
|
`serverFiles.alerts` | Prometheus server alerts configuration | `{}`
|
|
`serverFiles.rules` | Prometheus server rules configuration | `{}`
|
|
`serverFiles.prometheus.yml` | Prometheus server scrape configuration | example configuration
|
|
`networkPolicy.enabled` | Enable NetworkPolicy | `false` |
|
|
|
|
Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example,
|
|
|
|
```console
|
|
$ helm install stable/prometheus --name my-release \
|
|
--set server.terminationGracePeriodSeconds=360
|
|
```
|
|
|
|
Alternatively, a YAML file that specifies the values for the above parameters can be provided while installing the chart. For example,
|
|
|
|
```console
|
|
$ helm install stable/prometheus --name my-release -f values.yaml
|
|
```
|
|
|
|
> **Tip**: You can use the default [values.yaml](values.yaml)
|
|
|
|
### RBAC Configuration
|
|
Roles and RoleBindings resources will be created automatically for `server` and `kubeStateMetrics` services.
|
|
|
|
To manually setup RBAC you need to set the parameter `rbac.create=false` and specify the service account to be used for each service by setting the parameters: `serviceAccounts.{{ component }}.create` to `false` and `serviceAccounts.{{ component }}.name` to the name of a pre-existing service account.
|
|
|
|
> **Tip**: You can refer to the default `*-clusterrole.yaml` and `*-clusterrolebinding.yaml` files in [templates](templates/) to customize your own.
|
|
|
|
### ConfigMap Files
|
|
AlertManager is configured through [alertmanager.yml](https://prometheus.io/docs/alerting/configuration/). This file (and any others listed in `alertmanagerFiles`) will be mounted into the `alertmanager` pod.
|
|
|
|
Prometheus is configured through [prometheus.yml](https://prometheus.io/docs/operating/configuration/). This file (and any others listed in `serverFiles`) will be mounted into the `server` pod.
|
|
|
|
### Ingress TLS
|
|
If your cluster allows automatic creation/retrieval of TLS certificates (e.g. [kube-lego](https://github.com/jetstack/kube-lego)), please refer to the documentation for that mechanism.
|
|
|
|
To manually configure TLS, first create/retrieve a key & certificate pair for the address(es) you wish to protect. Then create a TLS secret in the namespace:
|
|
|
|
```console
|
|
kubectl create secret tls prometheus-server-tls --cert=path/to/tls.cert --key=path/to/tls.key
|
|
```
|
|
|
|
Include the secret's name, along with the desired hostnames, in the alertmanager/server Ingress TLS section of your custom `values.yaml` file:
|
|
|
|
```yaml
|
|
server:
|
|
ingress:
|
|
## If true, Prometheus server Ingress will be created
|
|
##
|
|
enabled: true
|
|
|
|
## Prometheus server Ingress hostnames
|
|
## Must be provided if Ingress is enabled
|
|
##
|
|
hosts:
|
|
- prometheus.domain.com
|
|
|
|
## Prometheus server Ingress TLS configuration
|
|
## Secrets must be manually created in the namespace
|
|
##
|
|
tls:
|
|
- secretName: prometheus-server-tls
|
|
hosts:
|
|
- prometheus.domain.com
|
|
```
|
|
|
|
### NetworkPolicy
|
|
|
|
Enabling Network Policy for Prometheus will secure connections to Alert Manager
|
|
and Kube State Metrics by only accepting connections from Prometheus Server.
|
|
All inbound connections to Prometheus Server are still allowed.
|
|
|
|
To enable network policy for Prometheus, install a networking plugin that
|
|
implements the Kubernetes NetworkPolicy spec, and set `networkPolicy.enabled` to true.
|
|
|
|
If NetworkPolicy is enabled for Prometheus' scrape targets, you may also need
|
|
to manually create a networkpolicy which allows it.
|