前言
学完前面三章的apiserver源码，我对准入控制机制有了深入理解。但总有种"纸上得来终觉浅"的感觉——知道Webhook是怎么工作的，但自己动手写一个又是另一回事。
于是决定从零开发一个MutatingAdmissionWebhook，实现类似Istio的自动Sidecar注入功能。目标很简单：当在特定namespace创建Pod时，自动注入一个Nginx Sidecar容器。
花了两个周末时间，踩了不少坑（证书问题、TLS配置、JSON Patch格式等），最终成功跑通。今天就把完整过程分享出来。
什么是自动Sidecar注入？
典型应用场景
Service Mesh（如Istio）的核心能力之一就是自动注入Envoy Sidecar：
用户创建Pod:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
containers:
- name: app
image: my-app:1.0
MutatingWebhook拦截后，自动变成:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
containers:
- name: app
image: my-app:1.0
- name: nginx-sidecar ← 自动注入！
image: nginx:alpine
优势：
- 应用无感知，不需要修改业务代码
- 统一的管理和控制
- 可以添加代理、监控、日志收集等功能
我们的目标
开发一个Webhook，实现：
- 监听指定namespace（如
sidecar-inject=true标签的namespace） - 在该namespace创建Pod时，自动注入Nginx Sidecar
- Nginx作为反向代理，转发流量到主容器
整体架构
┌─────────────────────────────────────────────────────────────────┐
│ Kubernetes Cluster │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌──────────────┐ ┌──────────────────────────────┐ │
│ │ Pod │ │ MutatingWebhookConfiguration │ │
│ │ my-app │◄───────│ name: nginx-sidecar-injector │ │
│ └──────┬───────┘ │ namespaceSelector: │ │
│ │ │ matchLabels: │ │
│ │ │ sidecar-inject: enabled │ │
│ │ └──────────────┬─────────────────┘ │
│ │ │ │
│ │ 1. 创建Pod │ 2. 匹配到Webhook │
│ │ │ │
│ ┌──────▼───────────────────────────────▼─────────────────┐ │
│ │ kube-apiserver │ │
│ │ │ │
│ │ MutatingAdmissionWebhook │ │
│ │ │ │ │
│ │ │ 3. HTTP POST AdmissionReview │ │
│ │ ▼ │ │
│ │ ┌─────────────────────────────────────────────────────┐│ │
│ │ │ Nginx Sidecar Injector Webhook ││ │
│ │ │ (运行在集群内或集群外) ││ │
│ │ │ ││ │
│ │ │ - 接收AdmissionReview请求 ││ │
│ │ │ - 构造JSON Patch添加sidecar容器 ││ │
│ │ │ - 返回AdmissionResponse ││ │
│ │ └─────────────────────────────────────────────────────┘│ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
实施步骤
整个实现分为6个步骤：
- 检查集群Webhook支持
- 开发Webhook服务（Go实现HTTP server）
- 生成TLS证书（Kubernetes需要HTTPS）
- 部署Webhook服务
- 创建MutatingWebhookConfiguration
- 验证注入效果
第一步：检查集群Webhook支持
首先确认集群是否启用了MutatingAdmissionWebhook控制器：
# 检查apiserver启动参数
kubectl get pods -n kube-system -l component=kube-apiserver -o yaml | grep enable-admission-plugins
# 输出应该包含MutatingAdmissionWebhook和ValidatingAdmissionWebhook
# --enable-admission-plugins=...,MutatingAdmissionWebhook,ValidatingAdmissionWebhook
如果没有启用，需要修改apiserver启动参数（Kubernetes 1.9+默认启用）。
第二步：开发Webhook服务
项目结构
sidecar-injector/
├── main.go # 主程序
├── go.mod # Go模块
├── Dockerfile # 容器镜像
├── deploy/
│ ├── deployment.yaml # Webhook部署
│ ├── service.yaml # Service暴露
│ └── webhook.yaml # MutatingWebhookConfiguration
└── certs/
└── generate.sh # 证书生成脚本
核心代码：main.go
package main
import (
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"k8s.io/api/admission/v1"
corev1 "k8s.io/api/core/v1"
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
const (
port = 8443
certFile = "/etc/webhook/certs/tls.crt"
keyFile = "/etc/webhook/certs/tls.key"
)
func main() {
http.HandleFunc("/mutate", mutateHandler)
http.HandleFunc("/health", healthHandler)
fmt.Printf("Starting webhook server on port %d...\n", port)
if err := http.ListenAndServeTLS(fmt.Sprintf(":%d", port), certFile, keyFile, nil); err != nil {
fmt.Fprintf(os.Stderr, "Failed to start server: %v\n", err)
os.Exit(1)
}
}
// healthHandler 健康检查
func healthHandler(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("ok"))
}
// mutateHandler 处理准入控制请求
func mutateHandler(w http.ResponseWriter, r *http.Request) {
// 读取请求体
body, err := io.ReadAll(r.Body)
if err != nil {
http.Error(w, fmt.Sprintf("could not read request body: %v", err), http.StatusBadRequest)
return
}
// 解析AdmissionReview
var admissionReview v1.AdmissionReview
if err := json.Unmarshal(body, &admissionReview); err != nil {
http.Error(w, fmt.Sprintf("could not parse admission review: %v", err), http.StatusBadRequest)
return
}
// 获取请求信息
admissionRequest := admissionReview.Request
if admissionRequest == nil {
http.Error(w, "admission request is nil", http.StatusBadRequest)
return
}
fmt.Printf("Received admission request: %s/%s %s\n",
admissionRequest.Namespace,
admissionRequest.Name,
admissionRequest.Operation)
// 解析Pod对象
var pod corev1.Pod
if err := json.Unmarshal(admissionRequest.Object.Raw, &pod); err != nil {
http.Error(w, fmt.Sprintf("could not parse pod: %v", err), http.StatusBadRequest)
return
}
// 构造响应
admissionResponse := &v1.AdmissionResponse{
UID: admissionRequest.UID,
Allowed: true,
}
// 检查是否需要注入sidecar
if shouldInject(&pod) {
// 构造JSON Patch
patch, err := createPatch(&pod)
if err != nil {
admissionResponse.Allowed = false
admissionResponse.Result = &metav1.Status{
Message: fmt.Sprintf("could not create patch: %v", err),
}
} else {
patchType := v1.PatchTypeJSONPatch
admissionResponse.PatchType = &patchType
admissionResponse.Patch = patch
fmt.Printf("Injected sidecar into pod %s/%s\n", pod.Namespace, pod.Name)
}
}
// 构造返回的AdmissionReview
responseReview := v1.AdmissionReview{
TypeMeta: admissionReview.TypeMeta,
Response: admissionResponse,
}
// 序列化响应
responseBytes, err := json.Marshal(responseReview)
if err != nil {
http.Error(w, fmt.Sprintf("could not marshal response: %v", err), http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
w.Write(responseBytes)
}
// shouldInject 判断是否需要注入sidecar
func shouldInject(pod *corev1.Pod) bool {
// 检查是否已经有sidecar
for _, container := range pod.Spec.Containers {
if container.Name == "nginx-sidecar" {
return false
}
}
// 检查annotation
if pod.Annotations == nil {
return true
}
// 可以通过annotation控制是否注入
if inject, ok := pod.Annotations["sidecar-injector/inject"]; ok {
return inject == "true"
}
return true
}
// createPatch 创建JSON Patch
func createPatch(pod *corev1.Pod) ([]byte, error) {
// 获取主容器的端口
mainPort := int32(8080)
if len(pod.Spec.Containers) > 0 && len(pod.Spec.Containers[0].Ports) > 0 {
mainPort = pod.Spec.Containers[0].Ports[0].ContainerPort
}
// 构造nginx配置
nginxConf := fmt.Sprintf(`
server {
listen 80;
location / {
proxy_pass http://localhost:%d;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}`, mainPort)
// 构造sidecar容器
sidecar := corev1.Container{
Name: "nginx-sidecar",
Image: "nginx:alpine",
Ports: []corev1.ContainerPort{
{
Name: "http",
ContainerPort: 80,
Protocol: corev1.ProtocolTCP,
},
},
VolumeMounts: []corev1.VolumeMount{
{
Name: "nginx-config",
MountPath: "/etc/nginx/conf.d",
},
},
}
// 构造ConfigMap Volume
configVolume := corev1.Volume{
Name: "nginx-config",
VolumeSource: corev1.VolumeSource{
ConfigMap: &corev1.ConfigMapVolumeSource{
LocalObjectReference: corev1.LocalObjectReference{
Name: "nginx-sidecar-config",
},
},
},
}
// 构造JSON Patch
patch := []map[string]interface{}{}
// 1. 添加sidecar容器
patch = append(patch, map[string]interface{}{
"op": "add",
"path": "/spec/containers/-",
"value": sidecar,
})
// 2. 添加volume（如果不存在）
if len(pod.Spec.Volumes) == 0 {
patch = append(patch, map[string]interface{}{
"op": "add",
"path": "/spec/volumes",
"value": []corev1.Volume{configVolume},
})
} else {
patch = append(patch, map[string]interface{}{
"op": "add",
"path": "/spec/volumes/-",
"value": configVolume,
})
}
// 3. 添加annotation标记已注入
if pod.Annotations == nil {
patch = append(patch, map[string]interface{}{
"op": "add",
"path": "/metadata/annotations",
"value": map[string]string{
"sidecar-injector/injected": "true",
},
})
} else {
patch = append(patch, map[string]interface{}{
"op": "add",
"path": "/metadata/annotations/sidecar-injector~1injected",
"value": "true",
})
}
return json.Marshal(patch)
}
go.mod
module sidecar-injector
go 1.21
require (
k8s.io/api v0.28.0
k8s.io/apimachinery v0.28.0
)
第三步：生成TLS证书
Webhook必须使用HTTPS，Kubernetes会验证证书。我们有两种方案：
方案1：使用集群CA签名（推荐）
#!/bin/bash
# certs/generate.sh
set -e
SERVICE_NAME="nginx-sidecar-injector"
NAMESPACE="kube-system"
# 生成CA私钥
openssl genrsa -out ca.key 2048
# 生成CA证书
openssl req -new -x509 -key ca.key -out ca.crt -days 365 \
-subj "/CN=${SERVICE_NAME}-ca"
# 生成Webhook服务器私钥
openssl genrsa -out tls.key 2048
# 创建证书签名请求配置
cat > csr.conf <<EOF
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
[req_distinguished_name]
CN = ${SERVICE_NAME}.${NAMESPACE}.svc
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = ${SERVICE_NAME}
DNS.2 = ${SERVICE_NAME}.${NAMESPACE}
DNS.3 = ${SERVICE_NAME}.${NAMESPACE}.svc
DNS.4 = ${SERVICE_NAME}.${NAMESPACE}.svc.cluster.local
EOF
# 生成证书签名请求
openssl req -new -key tls.key -out tls.csr -config csr.conf
# 使用CA签名
openssl x509 -req -in tls.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
-out tls.crt -days 365 -extensions v3_req -extfile csr.conf
# 创建Kubernetes Secret
echo "Creating Kubernetes secret..."
kubectl create secret tls ${SERVICE_NAME}-tls \
--cert=tls.crt \
--key=tls.key \
-n ${NAMESPACE} \
--dry-run=client -o yaml > secret.yaml
# 输出CA Bundle（base64编码，用于MutatingWebhookConfiguration）
echo ""
echo "CA Bundle (for MutatingWebhookConfiguration):"
cat ca.crt | base64 | tr -d '\n'
echo ""
# 清理临时文件
rm -f tls.csr csr.conf ca.srl
echo ""
echo "Done! Files generated:"
echo " - ca.crt: CA certificate"
echo " - ca.key: CA private key"
echo " - tls.crt: Webhook server certificate"
echo " - tls.key: Webhook server private key"
echo " - secret.yaml: Kubernetes TLS secret"
运行生成证书：
chmod +x certs/generate.sh
cd certs && ./generate.sh
方案2：使用cert-manager自动管理
如果集群安装了cert-manager，可以自动管理证书：
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx-sidecar-injector-tls
namespace: kube-system
spec:
secretName: nginx-sidecar-injector-tls
dnsNames:
- nginx-sidecar-injector
- nginx-sidecar-injector.kube-system
- nginx-sidecar-injector.kube-system.svc
- nginx-sidecar-injector.kube-system.svc.cluster.local
issuerRef:
name: kubernetes-ca
kind: ClusterIssuer
第四步：部署Webhook服务
Dockerfile
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY main.go .
RUN CGO_ENABLED=0 GOOS=linux go build -o webhook main.go
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /
COPY --from=builder /app/webhook /webhook
# 创建证书目录
RUN mkdir -p /etc/webhook/certs
EXPOSE 8443
ENTRYPOINT ["/webhook"]
构建镜像：
docker build -t your-registry/nginx-sidecar-injector:v1.0 .
docker push your-registry/nginx-sidecar-injector:v1.0
部署配置
# deploy/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: kube-system
labels:
sidecar-inject: enabled
---
# deploy/configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: nginx-sidecar-config
namespace: kube-system
data:
default.conf: |
server {
listen 80;
location / {
proxy_pass http://localhost:8080;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
}
---
# deploy/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-sidecar-injector
namespace: kube-system
labels:
app: nginx-sidecar-injector
spec:
replicas: 1
selector:
matchLabels:
app: nginx-sidecar-injector
template:
metadata:
labels:
app: nginx-sidecar-injector
spec:
containers:
- name: webhook
image: your-registry/nginx-sidecar-injector:v1.0
imagePullPolicy: Always
ports:
- containerPort: 8443
name: https
volumeMounts:
- name: tls-certs
mountPath: /etc/webhook/certs
readOnly: true
resources:
limits:
cpu: 500m
memory: 256Mi
requests:
cpu: 100m
memory: 128Mi
volumes:
- name: tls-certs
secret:
secretName: nginx-sidecar-injector-tls
---
# deploy/service.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx-sidecar-injector
namespace: kube-system
labels:
app: nginx-sidecar-injector
spec:
ports:
- port: 443
targetPort: 8443
name: https
selector:
app: nginx-sidecar-injector
部署：
kubectl apply -f deploy/configmap.yaml
kubectl apply -f deploy/secret.yaml # 使用生成的secret.yaml
kubectl apply -f deploy/deployment.yaml
kubectl apply -f deploy/service.yaml
第五步：创建MutatingWebhookConfiguration
这是最关键的一步，告诉Kubernetes什么时候调用我们的Webhook。
# deploy/webhook.yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: nginx-sidecar-injector
webhooks:
- name: sidecar-injector.webhook.example.com
# 匹配规则
rules:
- operations: ["CREATE", "UPDATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
scope: "Namespaced"
# 只拦截特定namespace
namespaceSelector:
matchLabels:
sidecar-inject: enabled
# 或根据对象标签选择
# objectSelector:
# matchLabels:
# inject-sidecar: "true"
clientConfig:
service:
name: nginx-sidecar-injector
namespace: kube-system
path: "/mutate"
port: 443
# CA Bundle，替换为ca.crt的base64编码
caBundle: ${CA_BUNDLE}
admissionReviewVersions: ["v1", "v1beta1"]
sideEffects: None
timeoutSeconds: 5
failurePolicy: Ignore # 生产环境建议用Fail
reinvocationPolicy: Never
注入CA Bundle：
# 获取CA Bundle
CA_BUNDLE=$(cat certs/ca.crt | base64 | tr -d '\n')
# 替换并应用
sed "s/\${CA_BUNDLE}/$CA_BUNDLE/g" deploy/webhook.yaml | kubectl apply -f -
第六步：验证注入效果
1. 标记namespace
kubectl label namespace default sidecar-inject=enabled
2. 创建测试Pod
# test-pod.yaml
apiVersion: v1
kind: Pod
metadata:
name: test-app
namespace: default
spec:
containers:
- name: main-app
image: hashicorp/http-echo
args:
- "-text=Hello from main app"
ports:
- containerPort: 5678
kubectl apply -f test-pod.yaml
3. 查看注入结果
# 查看Pod详情
kubectl get pod test-app -o yaml
# 应该看到两个容器
kubectl get pod test-app -o jsonpath='{.spec.containers[*].name}'
# 输出: main-app nginx-sidecar
# 查看sidecar日志
kubectl logs test-app -c nginx-sidecar
# 测试访问（通过sidecar访问主应用）
kubectl port-forward pod/test-app 8080:80
curl http://localhost:8080
# 输出: Hello from main app
4. 查看Webhook日志
kubectl logs -n kube-system -l app=nginx-sidecar-injector
应该看到：
Received admission request: default/test-app CREATE
Injected sidecar into pod default/test-app
踩坑实录
坑1：证书CN和DNS名称不匹配
现象：Kubernetes调用Webhook时报错x509: certificate is valid for ... not for nginx-sidecar-injector.kube-system.svc
解决：确保证书的Subject Alternative Name包含Service的所有可能域名。
坑2：JSON Patch格式错误
现象：Webhook返回成功，但Pod没有变化
解决：JSON Patch路径要使用RFC 6901格式，特别是特殊字符要转义：
// annotation中的/要转义为~1
"/metadata/annotations/sidecar-injector~1injected"
坑3：failurePolicy设置不当
现象：Webhook服务挂了，导致所有Pod创建失败
解决：开发阶段用Ignore，生产环境用Fail但要注意高可用。
坑4：忽略系统namespace
现象：kube-system中的Pod也被注入sidecar，导致集群异常
解决：使用namespaceSelector排除系统namespace：
namespaceSelector:
matchExpressions:
- key: kubernetes.io/metadata.name
operator: NotIn
values: ["kube-system", "kube-public", "kube-node-lease"]
坑5：Webhook超时
现象：Pod创建慢，或超时失败
解决：优化Webhook处理逻辑，设置合理的timeout：
timeoutSeconds: 5
进阶优化
1. 添加Metrics监控
import "github.com/prometheus/client_golang/prometheus"
var (
injectionCounter = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "sidecar_injection_total",
Help: "Total number of sidecar injections",
},
[]string{"namespace", "status"},
)
)
func init() {
prometheus.MustRegister(injectionCounter)
}
2. 支持配置化
apiVersion: v1
kind: ConfigMap
metadata:
name: sidecar-injector-config
data:
config.yaml: |
sidecarContainer:
image: nginx:alpine
resources:
limits:
cpu: 100m
memory: 128Mi
port: 80
3. 使用Controller运行时框架
可以用controller-runtime简化Webhook开发：
import (
"sigs.k8s.io/controller-runtime/pkg/webhook"
"sigs.k8s.io/controller-runtime/pkg/webhook/admission"
)
type PodMutator struct {
Client client.Client
Decoder *admission.Decoder
}
func (m *PodMutator) Default(ctx context.Context, obj runtime.Object) error {
pod := obj.(*corev1.Pod)
// 注入sidecar逻辑
return nil
}
总结
通过这个项目，我们完整实践了K8s准入控制器的开发：
- 理解原理：知道Webhook在API请求链中的位置和作用
- 开发实现：用Go实现了MutatingWebhook服务器
- 证书管理：生成了自签名证书并配置到Kubernetes
- 部署验证：完整部署流程并验证注入效果
实际开发中还有很多细节要考虑（高可用、配置管理、错误处理等），但这个基础版本已经可以工作。
扫一扫
1021
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
