绯诲垪鏂囩珷
- Cilium 绯诲垪鏂囩珷
鍓嶈█
灏?Kubernetes 鐨?CNI 浠庡叾浠栫粍浠跺垏鎹负 Cilium, 宸茬粡鍙互鏈夋晥鍦版彁鍗囩綉缁滅殑鎬ц兘銆備絾鏄€氳繃瀵?Cilium 涓嶅悓妯″紡鐨勫垏鎹?鍔熻兘鐨勫惎鐢紝鍙互杩涗竴姝ユ彁鍗?Cilium 鐨勭綉缁滄€ц兘銆傚叿浣撹皟浼橀」鍖呮嫭涓嶉檺浜庯細
- 鍚敤鏈湴璺敱 (Native Routing)
- 瀹屽叏鏇挎崲 KubeProxy
- IP 鍦板潃浼 (Masquerading) 鍒囨崲涓哄熀浜?eBPF 鐨勬ā寮?/li>
- Kubernetes NodePort 瀹炵幇鍦?DSR(Direct Server Return) 妯″紡涓嬭繍琛?/li>
- 缁曡繃 iptables 杩炴帴璺熻釜 (Bypass iptables Connection Tracking)
- 涓绘満璺敱 (Host Routing) 鍒囨崲涓哄熀浜?BPF 鐨勬ā寮?锛堥渶瑕?Linux Kernel >= 5.10锛?/li>
- 鍚敤 IPv6 BIG TCP 锛堥渶瑕?Linux Kernel >= 5.19锛?/li>
绂佺敤 Hubble锛堜絾鏄笉寤鸿锛屽彲瑙傚療鎬ф瘮涓€鐐圭偣鐨勬€ц兘鎻愬崌鏇撮噸瑕侊級- 淇敼 MTU 涓哄法鍨嬪抚 (jumbo frames) 锛堥渶瑕佺綉缁滄潯浠跺厑璁革級
- 鍚敤甯﹀绠＄悊鍣?(Bandwidth Manager) 锛堥渶瑕?Kernel >= 5.1锛?/li>
- 鍚敤 Pod 鐨?BBR 鎷ュ鎺у埗 锛堥渶瑕?Kernel >= 5.18锛?/li>
- 鍚敤 XDP 鍔犻€?锛堥渶瑕?鏀寔鏈湴 XDP 椹卞姩绋嬪簭锛?/li>
- 锛堥珮绾х敤鎴峰彲閫夛級璋冩暣 eBPF Map Size
- Linux Kernel 浼樺寲鍜屽崌绾?
CONFIG_PREEMPT_NONE=y
- 鍏朵粬锛?
- tuned network-* profiles, 濡傦細
tuned-adm profile network-latency鎴?network-throughput - CPU 璋冧负鎬ц兘妯″紡
- 鍋滄
irqbalance锛屽皢缃戝崱涓柇寮曡剼鎸囧悜鐗瑰畾 CPU
- tuned network-* profiles, 濡傦細
鍦ㄧ綉缁?缃戝崱璁惧/OS 绛夋潯浠舵弧瓒崇殑鎯呭喌涓嬶紝鎴戜滑灏藉彲鑳藉鍦板惎鐢ㄨ繖浜涜皟浼橀€夐」锛岀浉鍏充紭鍖栭」浼氬湪鍚庣画鏂囩珷閫愪竴鏇存柊銆傛暚璇锋湡寰呫€?/p>
浠婂ぉ鎴戜滑鏉ュ皢 Cilium 鐨?NodePort 瀹炵幇浠?SNAT 鏀逛负 DSR 浠ユ彁鍗囩綉缁滄€ц兘銆?/p>
娴嬭瘯鐜
- Cilium 1.13.4
- K3s v1.26.6+k3s1
- OS
- 3 鍙?Ubuntu 23.04 VM, Kernel 6.2, x86
Direct Server Return (DSR)
榛樿鎯呭喌涓嬶紝Cilium 鐨?eBPF NodePort 瀹炵幇浠?SNAT 妯″紡杩愯銆備篃灏辨槸璇达紝褰撹妭鐐瑰閮ㄦ祦閲忓埌杈炬椂锛屽鏋滆妭鐐圭‘瀹氳礋杞藉钩琛″櫒銆丯odePort 鎴栧叿鏈夊閮?IP 鐨勬湇鍔＄殑鍚庣浣嶄簬杩滅▼鑺傜偣锛岄偅涔堣妭鐐瑰氨浼氶€氳繃鎵ц SNAT 灏嗚姹傞噸瀹氬悜鍒颁唬琛ㄨ嚜宸辩殑杩滅▼鍚庣銆傝繖涓嶉渶瑕侀澶栨洿鏀?MTU銆備唬浠锋槸锛屾潵鑷悗绔殑鍥炲闇€瑕?strong>棰濆璺冲洖鑺傜偣锛屽湪閭ｉ噷鎵ц鍙嶅悜 SNAT 杞崲锛岀劧鍚庡啀灏嗘暟鎹寘鐩存帴杩斿洖缁欏閮ㄥ鎴风銆?/p>
绀轰緥濡備笅锛孋ilium 鐨?eBPF NodePort 瀹炵幇浠?SNAT 妯″紡杩愯锛?/p>
$ kubectl -n kube-system exec ds/cilium -- cilium status --verbose
...
KubeProxyReplacement Details:
Status: Strict
Socket LB: Enabled
Socket LB Tracing: Enabled
Socket LB Coverage: Full
Devices: eth0 192.168.2.3 (Direct Routing)
Mode: SNAT
SNAT 妯″紡涓嬶紝NodePort 鍚庣 pod 鍦ㄥ叾浠栬妭鐐瑰叆鍚戞祦閲忥細
鍑哄悜娴侀噺锛?/p>
璇ヨ缃彲閫氳繃 loadBalancer.mode Helm 閫夐」鏇存敼涓?dsr锛屼互渚胯 Cilium 鐨?eBPF NodePort 瀹炵幇鍦?DSR 妯″紡涓嬭繍琛屻€傚湪杩欑妯″紡涓嬶紝鍚庣鐩存帴鍥炲澶栭儴瀹㈡埛绔紝鑰屼笉闇€瑕侀澶栫殑璺宠浆锛屼篃灏辨槸璇达紝鍚庣閫氳繃浣跨敤鏈嶅姟 IP/port 浣滀负婧愭潵鍥炲銆侱SR 鐩墠瑕佹眰 Cilium 浠?strong>鏈湴璺敱妯″紡閮ㄧ讲锛屼篃灏辨槸璇达紝瀹冧笉鑳藉湪浠讳綍涓€绉嶉毀閬撴ā寮忎笅宸ヤ綔銆?/p>
DSR 妯″紡娴侀噺濡備笅锛?/p>
DSR 妯″紡鐨勫彟涓€涓紭鐐规槸淇濈暀浜嗗鎴风鐨勬簮 IP锛屽洜姝ゅ彲浠ュ湪鍚庣鑺傜偣瀵瑰叾杩涜绛栫暐鍖归厤銆傝€屽湪 SNAT 妯″紡涓嬪垯鏃犳硶鍋氬埌杩欎竴鐐广€傞壌浜庝竴涓壒瀹氱殑鍚庣鍙澶氫釜鏈嶅姟浣跨敤锛屽悗绔渶瑕佺煡閬撳畠浠渶瑕佸洖澶嶇殑鏈嶅姟 IP/绔彛銆?/p>
璇锋敞鎰忥紝鐢变簬 Cilium 鐗瑰畾鐨?IP 閫夐」鍙兘浼氳搴曞眰缃戠粶缁撴瀯涓㈠純锛屽洜姝?DSR 妯″紡鍦ㄦ煇浜涘叕鍏变簯鎻愪緵鍟嗙幆澧冧腑鍙兘鏃犳硶浣跨敤銆傚鏋滃悗绔綅浜庝笌澶勭悊缁欏畾 NodePort 璇锋眰鐨勮妭鐐圭浉璺濊緝杩滅殑鑺傜偣涓婏紝鍦ㄥ嚭鐜版湇鍔¤繛鎺ラ棶棰樻椂锛岄鍏堣妫€鏌?NodePort 璇锋眰鏄惁瀹為檯鍒拌揪浜嗗寘鍚悗绔殑鑺傜偣銆傚鏋滀笉鏄紝鍒欏缓璁垏鎹㈠洖榛樿 SNAT 妯″紡浣滀负涓€绉嶈В鍐虫柟娉曘€?/p>
姝ゅ锛屽湪鏌愪簺瀹炴柦婧?鐩爣 IP 鍦板潃妫€鏌ョ殑鍏叡浜戞彁渚涘晢鐜涓紙濡?AWS锛夛紝蹇呴』绂佺敤妫€鏌ユ墠鑳戒娇鐢?DSR 妯″紡銆?/p>
鍚敤 DSR 瀹炴柦姝ラ
鍦ㄥ惎鐢ㄤ粎 DSR 妯″紡鐨勬棤 kube proxy 鐜涓紝涓婅堪 Helm 绀轰緥閰嶇疆濡備笅锛?/p>
helm upgrade cilium cilium/cilium --version 1.13.4 \
--namespace kube-system \
--reuse-values \
--set loadBalancer.mode=dsr
馃惥Warning
鍓嶆彁鏄細
- 鍚敤鏈湴璺敱
- Cilium 瀹屽叏鏇挎崲 KubeProxy
楠岃瘉
$ kubectl -n kube-system exec ds/cilium -- cilium status --verbose|grep DSR
Mode: DSR
鎬ц兘鎻愬崌
鎬ц兘鎻愬崌鍙互鏌ョ湅瀹樻柟鐨?benchmark:
馃憤锔忦煈嶏笍馃憤锔?/p>
鎬荤粨
鏈枃鎴戜滑灏?Cilium 鐨?NodePort 瀹炵幇浠?SNAT 妯″紡鍒囨崲涓?DSR 鐨勬ā寮忋€傜浉姣?SNAT 妯″紡锛孌SR 鍦?NodePort 鍚庣 pod 鍦ㄥ叾浠栬妭鐐?鏃朵紭鍔挎槑鏄撅細
- 缃戠粶灏戜簡鑷冲皯涓€璺?/li>
- 鍙互淇濈暀瀹㈡埛绔殑婧?IP
浣嗘槸鍦ㄥ叕鏈変簯鐜涓婇渶瑕佹敞鎰忓惎鐢ㄧ殑鏉′欢銆?/p>
鑷虫锛屾€ц兘璋冧紭宸插畬鎴愶細
- 鉁旓笍 鍚敤鏈湴璺敱 (Native Routing)
- 鉁旓笍 瀹屽叏鏇挎崲 KubeProxy
- 鉁旓笍 IP 鍦板潃浼 (Masquerading) 鍒囨崲涓哄熀浜?eBPF 鐨勬ā寮?/li>
- 鉁旓笍 Kubernetes NodePort 瀹炵幇鍦?DSR(Direct Server Return) 妯″紡涓嬭繍琛?/li>
- 缁曡繃 iptables 杩炴帴璺熻釜 (Bypass iptables Connection Tracking)
- 涓绘満璺敱 (Host Routing) 鍒囨崲涓哄熀浜?BPF 鐨勬ā寮?(闇€瑕?Linux Kernel >= 5.10)
- 鍚敤 IPv6 BIG TCP (闇€瑕?Linux Kernel >= 5.19)
- 淇敼 MTU 涓哄法鍨嬪抚 (jumbo frames) 锛堥渶瑕佺綉缁滄潯浠跺厑璁革級
- 鍚敤甯﹀绠＄悊鍣?(Bandwidth Manager) (闇€瑕?Kernel >= 5.1)
- 鍚敤 Pod 鐨?BBR 鎷ュ鎺у埗 (闇€瑕?Kernel >= 5.18)
- 鍚敤 XDP 鍔犻€?锛堥渶瑕?鏀寔鏈湴 XDP 椹卞姩绋嬪簭锛?/li>
馃摎锔忓弬鑰冩枃妗?/h2>
- DSR Mode - Kubernetes Without kube-proxy 鈥?Cilium 1.13.4 documentation
- Cilium 1.10: WireGuard, BGP Support, Egress IP Gateway, New Cilium CLI, XDP Load Balancer, Alibaba Cloud Integration and more
涓変汉琛? 蹇呮湁鎴戝笀; 鐭ヨ瘑鍏变韩, 澶╀笅涓哄叕. 鏈枃鐢变笢椋庡井楦ｆ妧鏈崥瀹?EWhisper.cn 缂栧啓.
涓変汉琛? 蹇呮湁鎴戝笀; 鐭ヨ瘑鍏变韩, 澶╀笅涓哄叕. 鏈枃鐢变笢椋庡井楦ｆ妧鏈崥瀹?EWhisper.cn 缂栧啓.