根据Nginx Ingress指标对指定后端进行HPA

本文分享自华为云社区《根据Nginx Ingress指标对指定后端进行HPA》，作者： 可以交个朋友。

背景

Nginx Ingress作为K8s流量入口，可以通过域名和path路径转发流量至不同的后端pod。nginx-ingress暴露nginx_ingress_controller_requests指标，以此指标可以采集到不同域名访问的QPS值，可以根据该指标作为对应后端工作负载HPA策略，实现根据域名的访问QPS对关联后端实例进行HPA。

简介

本方案基于开源nginx-ingress默认提供的nginx_ingress_controller_requests指标，该指标包含的标签可以区分每个域名的访问量，再通过Prometheus-adapter转化成QPS指标，HPA利用QPS指标对域名关联的后端工作负载做弹性策略
注意：由于HPA规则中scaleTargetRef和describedObject两个字段关联的资源都无法指定命名空间，所以指标来源、HPA和弹性目标需在同一命名空间，而nginx-ingress和业务工作负载一般处在不同命名空间，本次方案将采用external类型的HPA，可以忽略指标来源的命名空间

操作步骤

1. 创建演示需要的弹性目标工作负载，service以及ingress

apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: app1
  labels: 
    app: app1 
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: app1
  template: 
    metadata: 
      labels: 
        app: app1
    spec: 
     s: 
      - image: nginx:v1
        name: nginx
        ports: 
        - name: http 
         Port: 80 
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: app1-service 
  namespace: default 
  labels: 
    app: app1-service 
spec: 
  ports: 
    - port: 8080 
      name: http 
      protocol: TCP 
      targetPort: 80
  selector: 
    app: app1
  type: ClusterIP 

--- 
apiVersion: apps/v1 
kind: Deployment 
metadata: 
  name: app2
  labels: 
    app: app2
spec: 
  replicas: 1 
  selector: 
    matchLabels: 
      app: app2
  template: 
    metadata: 
      labels: 
        app: app2 
    spec: 
     s: 
      - image: nginx:v2
        name: nginx
        ports: 
        - name: http 
         Port: 80
--- 
apiVersion: v1 
kind: Service 
metadata: 
  name: app2-service
  namespace: default 
  labels: 
    app: app2-service
spec: 
  ports: 
    - port: 80 
      name: http 
      protocol: TCP 
      targetPort: 80
  selector: 
    app: app2
  type: ClusterIP 
--- 
apiVersion: networking.k8s.io/v1 
kind: Ingress 
metadata: 
  name: test-ingress 
  namespace: default 
spec: 
  ingressClassName: nginx
  rules:
  - host: www.test-hpa.com
    http:
      paths:
        - path: /
          pathType: ImplementationSpecific
          backend:
            service:
              name: app1-service
              port:
                number: 8080
          property:
            ingress.beta.kubernetes.io/url-match-mode: STARTS_WITH
        - path: /home/
          pathType: ImplementationSpecific
          backend:
            service:
              name: app2-service
              port:
                number: 80

2、分别查询www.test-hpa.com/和www.test-hpa.com/home/的nginx_ingress_controller_requests指标，指标正常

3、创建external类型的apiservices资源；创建后apiservices的状态为false是正常现象，添加externalRules后状态变为true

apiVersion: apiregistration.k8s.io/v1 
kind: APIService 
metadata: 
  name: v1beta1.external.metrics.k8s.io 
spec: 
  group: external.metrics.k8s.io 
  groupPriorityMinimum: 100 
  insecureSkipTLSVerify: true 
  service:         #指定prometheus-adapter对应的service，华为CCE插件中adapter名称为custom-metrics-apiserver 
    name: custom-metrics-apiserver 
    namespace: monitoring 
    port: 443 
  version: v1beta1 
  versionPriority: 100

4、将externalRules规则添加到adapter的configmap中，修改后需要重启prometheus-adapter服务

kubectl -n monitoring edit configmap user-adapter-config

externalRules: 
- metricsQuery: sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>) 
  name: 
    as: _per_second 
    matches: ^(.*) 
  resources: 
    namespaced: false  #忽略指标来源的命名空间，该配置不适用rules规则 
  seriesQuery: nginx_ingress_controller_requests

seriesQuery：原始指标；可以直接写指标名称，也可以使用{labelKey=labelValue}的方式筛选出原始指标
metricsQuery：用PromQL对指标进行筛选汇聚；.Series表示原始指标，.LabelMatchers表示对指标进行标签筛选，hpa中可以配置具体的筛选规则
name：将指标重命名
resources：hpa查询指标时通过api的方式调用，调用路径为：，而resources的作用就是将指标中命名空间标签的值替换路径中的${namespace}，而我们本次方案需要忽略指标来源命名空间。

5、custom-metrics-apiserver服务重启后需要等待1分钟左右，执行命令查看指标是否正常

kubectl get --raw /apis/external.metrics.k8s.io/v1beta1/namespaces/*/nginx_ingress_controller_requests_per_second

6、创建HPA规则

apiVersion: autoscaling/v2 
kind: HorizontalPodAutoscaler 
metadata: 
  name: app1-hpa
spec: 
  scaleTargetRef: 
    apiVersion: apps/v1 
    kind: Deployment 
    name: app1 
  minReplicas: 1 
  maxReplicas: 10 
  metrics: 
    - type: External 
      external: 
        metric: 
          name: nginx_ingress_controller_requests_per_second 
          selector: 
            matchLabels:    #可以通过该字段对指标进行过滤，这里标签筛选条件会加入到externalRules的<<.LabelMatchers>>中。 
              exported_service: app1-service  #筛选后端服务为sample-app的请求 
              host: www.test-hpa.com        #筛选访问域名为test.example.com的请求 
        target: 
          type: AverageValue   #External指标类型下只支持Value和AverageValue类型的目标值。 
          averageValue: 30 
--- 
apiVersion: autoscaling/v2 
kind: HorizontalPodAutoscaler 
metadata: 
  name: app2-hpa 
spec: 
  scaleTargetRef: 
    apiVersion: apps/v1 
    kind: Deployment 
    name: app2
  minReplicas: 1 
  maxReplicas: 10 
  metrics: 
    - type: External 
      external: 
        metric: 
          name: nginx_ingress_controller_requests_per_second 
          selector: 
            matchLabels: 
              exported_service: app2-service 
              host: www.test-hpa.com
        target: 
          type: AverageValue 
          averageValue: 30

弹性演示

1. 使用命令压测app1对应的访问域名和路径，正常触发弹性；请自行配置域名与ELB地址的映射

ab -c 50 -n 5000 http://www.test-hpa.com/

2、用同样的方式压测app2，正常触发弹性

ab -c 50 -n 5000 http://www.test-hpa.com/home/

 

点击关注，第一时间了解华为云新鲜技术~