2020-09-05

最新推荐文章于 2025-05-05 15:42:49 发布

原创最新推荐文章于 2025-05-05 15:42:49 发布 · 305 阅读

0 ·

本内容遵循CC 4.0 BY-SA版权协议

k8s 专栏收录该内容

2 篇文章

订阅专栏

k8s HPA扩缩容

环境k8s 1.83

kube-controller-manage 启动参数
可以先登录kube-controller-manage的pod查看帮助 kube-controller-manage -h

[root@k8s-master ~]# vim /etc/kubernetes/manifests/kube-controller-manager.yaml

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-controller-manager
      # 自动扩缩容配置
      # pod启动后等待多久开始cpu取样, 默认5m
    - --horizontal-pod-autoscaler-cpu-initialization-period=5m0s
      # 执行缩容操作的等待时长，默认5m
    - --horizontal-pod-autoscaler-downscale-stabilization=3m0s
      # 首次探测Pod达到Ready状态的时延，默认30s
    - --horizontal-pod-autoscaler-initial-readiness-delay=30s
      # HPA控制器同步Pod副本数量的时间间隔，默认15s
    - --horizontal-pod-autoscaler-sync-period=10s
      # 设置扩缩容忍度，默认值0.1(10%)，表示基于算法得到的结果在0.9-1.1(-10%-10%)，控制器都不会进行扩缩容
    - --horizontal-pod-autoscaler-tolerance=0.1
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=10.244.0.0/16
    - --cluster-name=kubernetes
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt

HPA 资源文件查看

[root@k8s-master ~]# kubectl api-versions | grep autoscaling
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2

一共三个版本,v1版本支持对cpu使用率做扩缩容
看一下能指定的配置

[root@k8s-master ~]# kubectl explain hpa.spec --api-version=autoscaling/v1 --recursive
KIND:     HorizontalPodAutoscaler
VERSION:  autoscaling/v1

RESOURCE: spec <Object>

DESCRIPTION:
     behaviour of autoscaler. More info:
     https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#spec-and-status.

     specification of a horizontal pod autoscaler.

FIELDS:
   maxReplicas	<integer>   #pod副本数的上限
   minReplicas	<integer>   # 下限
   scaleTargetRef	<Object> # 针对目标
      apiVersion	<string>
      kind	<string>
      name	<string>
   targetCPUUtilizationPercentage	<integer>

例子
1.拿一个pod做试验，设置request.cpu=100m
本文用的istio1.7的bookinfo demo中的productpage，对应的svc是productpage:9080
2.创建一个hpa
3.监控 hpa 与pod资源消耗以及pod创建

[root@k8s-master kube]# kubectl get po productpage-v1-68f4ffc876-b2hn9 -o jsonpath={.spec.containers[0].resources} 
map[limits:map[cpu:400m] requests:map[cpu:100m]]
#创建一个hpa，目标cpu 50% 最小1 最大10 
[root@k8s-master hpa]# kubectl autoscale deployment productpage-v1 --cpu-percent=50 --min=1 --max=10
horizontalpodautoscaler.autoscaling/productpage-v1 autoscaled
#1.开一个shell监控hpa
[root@k8s-master hpa]# kubectl get hpa -w
NAME             REFERENCE                   TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
productpage-v1   Deployment/productpage-v1   10%/50%   1         10        1          59s
#2.开一个shell监控po
[root@k8s-master ~]# kubectl get po -l app=productpage -w
NAME                              READY   STATUS    RESTARTS   AGE
productpage-v1-68f4ffc876-b2hn9   2/2     Running   0          40m
3.开一个shell 监控pod资源
 [root@k8s-master kube]# watch -n 1 kubectl top po

4.加压以及结果
创建一个pod一直访问这个微服务
kubectl run app --image=busybox -it – sh -c ‘while true; do wget -q -O- http://productpage:9080; done’

[root@k8s-node1 ~]# kubectl run app --image=busybox -it -- sh -c 'while true; do wget -q -O- http://productpage:9080; done'

[root@k8s-master hpa]# kubectl get hpa -w
NAME             REFERENCE                   TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
productpage-v1   Deployment/productpage-v1   10%/50%   1         10        1          11m
productpage-v1   Deployment/productpage-v1   458%/50%   1         10        1          12m
productpage-v1   Deployment/productpage-v1   458%/50%   1         10        4          13m
productpage-v1   Deployment/productpage-v1   458%/50%   1         10        8          13m
productpage-v1   Deployment/productpage-v1   458%/50%   1         10        10         13m
productpage-v1   Deployment/productpage-v1   328%/50%   1         10        10         13m
productpage-v1   Deployment/productpage-v1   130%/50%   1         10        10         14m
productpage-v1   Deployment/productpage-v1   54%/50%    1         10        10         15m
productpage-v1   Deployment/productpage-v1   54%/50%    1         10        10         16m
productpage-v1   Deployment/productpage-v1   49%/50%    1         10        10         19m
productpage-v1   Deployment/productpage-v1   54%/50%    1         10        10         20m

[root@k8s-master ~]# kubectl get po -l app=productpage -w
NAME                              READY   STATUS    RESTARTS   AGE
productpage-v1-68f4ffc876-b2hn9   2/2     Running   0          40m
productpage-v1-68f4ffc876-rgp9c   0/2     Pending   0          0s
productpage-v1-68f4ffc876-rgp9c   0/2     Pending   0          0s
productpage-v1-68f4ffc876-zmj8t   0/2     Pending   0          0s
productpage-v1-68f4ffc876-s74bj   0/2     Pending   0          0s
productpage-v1-68f4ffc876-zmj8t   0/2     Pending   0          0s
productpage-v1-68f4ffc876-rgp9c   0/2     Init:0/1   0          0s
productpage-v1-68f4ffc876-s74bj   0/2     Pending    0          0s
productpage-v1-68f4ffc876-zmj8t   0/2     Init:0/1   0          0s
productpage-v1-68f4ffc876-s74bj   0/2     Init:0/1   0          0s
productpage-v1-68f4ffc876-zd6ml   0/2     Pending    0          0s
productpage-v1-68f4ffc876-zd6ml   0/2     Pending    0          0s
productpage-v1-68f4ffc876-2v2rf   0/2     Pending    0          0s
productpage-v1-68f4ffc876-jx6st   0/2     Pending    0          0s
productpage-v1-68f4ffc876-2v2rf   0/2     Pending    0          0s
productpage-v1-68f4ffc876-jx6st   0/2     Pending    0          0s
productpage-v1-68f4ffc876-zd6ml   0/2     Init:0/1   0          0s
productpage-v1-68f4ffc876-vx9m5   0/2     Pending    0          0s
productpage-v1-68f4ffc876-vx9m5   0/2     Pending    0          0s
productpage-v1-68f4ffc876-2v2rf   0/2     Init:0/1   0          0s
productpage-v1-68f4ffc876-jx6st   0/2     Init:0/1   0          0s
productpage-v1-68f4ffc876-vx9m5   0/2     Init:0/1   0          0s
productpage-v1-68f4ffc876-rgp9c   0/2     PodInitializing   0          17s
productpage-v1-68f4ffc876-zmj8t   0/2     PodInitializing   0          17s
productpage-v1-68f4ffc876-z6l4s   0/2     Pending           0          0s
productpage-v1-68f4ffc876-z6l4s   0/2     Pending           0          0s
productpage-v1-68f4ffc876-pcm2c   0/2     Pending           0          0s
productpage-v1-68f4ffc876-z6l4s   0/2     Init:0/1          0          0s
productpage-v1-68f4ffc876-pcm2c   0/2     Pending           0          0s
productpage-v1-68f4ffc876-pcm2c   0/2     Init:0/1          0          0s
productpage-v1-68f4ffc876-s74bj   0/2     PodInitializing   0          32s
productpage-v1-68f4ffc876-vx9m5   0/2     PodInitializing   0          23s
productpage-v1-68f4ffc876-zd6ml   0/2     PodInitializing   0          38s
productpage-v1-68f4ffc876-2v2rf   0/2     PodInitializing   0          38s
productpage-v1-68f4ffc876-jx6st   0/2     PodInitializing   0          53s
productpage-v1-68f4ffc876-zmj8t   1/2     Running           0          63s
productpage-v1-68f4ffc876-zmj8t   2/2     Running           0          66s
productpage-v1-68f4ffc876-rgp9c   1/2     Running           0          78s
productpage-v1-68f4ffc876-pcm2c   0/2     PodInitializing   0          59s
productpage-v1-68f4ffc876-rgp9c   2/2     Running           0          80s
productpage-v1-68f4ffc876-z6l4s   0/2     PodInitializing   0          74s
productpage-v1-68f4ffc876-vx9m5   1/2     Running           0          84s
productpage-v1-68f4ffc876-vx9m5   2/2     Running           0          86s
productpage-v1-68f4ffc876-s74bj   1/2     Running           0          109s
productpage-v1-68f4ffc876-2v2rf   1/2     Running           0          99s
productpage-v1-68f4ffc876-2v2rf   2/2     Running           0          101s
productpage-v1-68f4ffc876-s74bj   2/2     Running           0          112s
productpage-v1-68f4ffc876-zd6ml   1/2     Running           0          115s
productpage-v1-68f4ffc876-pcm2c   1/2     Running           0          105s
productpage-v1-68f4ffc876-pcm2c   2/2     Running           0          106s
productpage-v1-68f4ffc876-zd6ml   2/2     Running           0          117s
productpage-v1-68f4ffc876-jx6st   1/2     Running           0          2m10s
productpage-v1-68f4ffc876-jx6st   2/2     Running           0          2m11s
productpage-v1-68f4ffc876-z6l4s   1/2     Running           0          2m16s
productpage-v1-68f4ffc876-z6l4s   2/2     Running           0          2m20s
#改成5秒复制。。。
[root@k8s-master kube]# watch -n 5 kubectl top po
 Every 5.0s: kubectl top po -l app=productpage                                                                                   k8s-master: Sat Sep  5 13:59:07 2020

NAME                              CPU(cores)   MEMORY(bytes)
productpage-v1-68f4ffc876-2v2rf   53m          104Mi
productpage-v1-68f4ffc876-b2hn9   53m          116Mi
productpage-v1-68f4ffc876-jx6st   66m          107Mi
productpage-v1-68f4ffc876-pcm2c   53m          103Mi
productpage-v1-68f4ffc876-rgp9c   64m          108Mi
productpage-v1-68f4ffc876-s74bj   64m          106Mi
productpage-v1-68f4ffc876-vx9m5   53m          105Mi
productpage-v1-68f4ffc876-z6l4s   63m          106Mi
productpage-v1-68f4ffc876-zd6ml   64m          106Mi
productpage-v1-68f4ffc876-zmj8t   53m          106Mi

5.删掉加压的pod，可以观察自动缩容

kubectl delete po app

[root@k8s-master hpa]# kubectl get hpa -w
NAME             REFERENCE                   TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
productpage-v1   Deployment/productpage-v1   9%/50%    1         10        10         48m
productpage-v1   Deployment/productpage-v1   11%/50%   1         10        10         48m
productpage-v1   Deployment/productpage-v1   11%/50%   1         10        10         49m
productpage-v1   Deployment/productpage-v1   11%/50%   1         10        7          50m
productpage-v1   Deployment/productpage-v1   11%/50%   1         10        7          50m
productpage-v1   Deployment/productpage-v1   9%/50%    1         10        3          50m
productpage-v1   Deployment/productpage-v1   11%/50%   1         10        3          51m
productpage-v1   Deployment/productpage-v1   10%/50%   1         10        3          51m
productpage-v1   Deployment/productpage-v1   10%/50%   1         10        3          52m
productpage-v1   Deployment/productpage-v1   10%/50%   1         10        2          53m

可以看kube-controller-manager的日志，截取扩缩容的部分

kubectl logs kube-controller-manager-k8s-master -n=kube-system | grep scale
I0905 05:42:56.552760       1 horizontal.go:664] Successful rescale of productpage-v1, old size: 1, new size: 4, reason: cpu resource utilization (percentage of request) above target
I0905 05:42:56.553751       1 event.go:278] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"productpage-v1", UID:"3b252b6a-23a8-4a22-9d9c-1895728c98a7", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1368628", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 4; reason: cpu resource utilization (percentage of request) above target
I0905 05:43:07.240683       1 horizontal.go:664] Successful rescale of productpage-v1, old size: 4, new size: 8, reason: cpu resource utilization (percentage of request) above target
I0905 05:43:07.244838       1 event.go:278] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"productpage-v1", UID:"3b252b6a-23a8-4a22-9d9c-1895728c98a7", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1368830", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 8; reason: cpu resource utilization (percentage of request) above target
I0905 05:43:17.255349       1 horizontal.go:664] Successful rescale of productpage-v1, old size: 8, new size: 10, reason: cpu resource utilization (percentage of request) above target
I0905 05:43:17.255635       1 event.go:278] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"productpage-v1", UID:"3b252b6a-23a8-4a22-9d9c-1895728c98a7", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1368892", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 10; reason: cpu resource utilization (percentage of request) above target
I0905 06:19:56.649629       1 horizontal.go:664] Successful rescale of productpage-v1, old size: 10, new size: 7, reason: All metrics below target
I0905 06:19:56.650038       1 event.go:278] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"productpage-v1", UID:"3b252b6a-23a8-4a22-9d9c-1895728c98a7", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1376179", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 7; reason: All metrics below target
I0905 06:20:48.008878       1 horizontal.go:664] Successful rescale of productpage-v1, old size: 7, new size: 3, reason: All metrics below target
I0905 06:20:48.009488       1 event.go:278] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"productpage-v1", UID:"3b252b6a-23a8-4a22-9d9c-1895728c98a7", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1376447", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 3; reason: All metrics below target
I0905 06:23:00.740184       1 horizontal.go:664] Successful rescale of productpage-v1, old size: 3, new size: 2, reason: All metrics below target
I0905 06:23:00.740535       1 event.go:278] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"productpage-v1", UID:"3b252b6a-23a8-4a22-9d9c-1895728c98a7", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1376862", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 2; reason: All metrics below target
I0905 06:23:52.094693       1 horizontal.go:664] Successful rescale of productpage-v1, old size: 2, new size: 1, reason: All metrics below target
I0905 06:23:52.094974       1 event.go:278] Event(v1.ObjectReference{Kind:"HorizontalPodAutoscaler", Namespace:"default", Name:"productpage-v1", UID:"3b252b6a-23a8-4a22-9d9c-1895728c98a7", APIVersion:"autoscaling/v2beta2", ResourceVersion:"1377118", FieldPath:""}): type: 'Normal' reason: 'SuccessfulRescale' New size: 1; reason: All metrics below target

先这样吧