關(guān)于開發(fā)中負載均衡(代理)的一些筆記
發(fā)布時間：2021-11-23 點擊數(shù)：757
v
- 負載這一塊的知識有些亂, 聽人家常說軟負載、硬負載，四層負載，七層負載、客戶端負載,服務(wù)端負載之類的,所以梳理一下。
- 負載均衡在系統(tǒng)架構(gòu)中是一個非常重要,通過負載均衡可以提高系統(tǒng)的高可用，緩解網(wǎng)絡(luò)、硬件資源的限制。
- 博文主要涉及一些項目中常用的負載方式，很淺，不涉及負載算法啥的，做負載離不來代理，所以文中提到代理服務(wù)器即負載服務(wù)器。
-
這里梳理的方式從 Dev和Ops的概念出發(fā)。這里簡單分類一下：
-
軟負載
-
客戶端負載
- Spring Cloud Ribbon
-
服務(wù)端負載
- Ngixn (4-7)層負載
- Hyproxy (4-7)層負載
- LVS (4)層負載
- kube-proxy (4-7)層負載
-
-
硬負載
- F5
-
關(guān)于 LVS和kube-proxy、F5我們這里之后在和小伙伴分享，F(xiàn)5沒有接觸過，LVS的demo容器的方式一直沒有成功，kube-proxy這一塊我還沒學到，只是簡單的了解.
如果能深刻理解苦難，苦難就會給人帶來崇高感 。 ——路遙
一、軟負載
處理傳輸層到應(yīng)用層的數(shù)據(jù)，為了能通一個URL將前端的訪問分發(fā)到后臺的多個服務(wù)器上
1、客戶端負載
Dev 即開發(fā)角度的負載均衡。開發(fā)中的負載均衡一般是在微服務(wù)中涉及。服務(wù)提供方一般以多實例的形式提供服務(wù)，負載均衡功能能夠讓服務(wù)調(diào)用方連接到合適的服務(wù)節(jié)點。 并且，服務(wù)節(jié)點選擇的過程對服務(wù)調(diào)用方來說是透明的。
所以這里理解為是客戶端的負載均衡，是相對服務(wù)端負載均衡而言。
客戶端負載均衡來講,就是調(diào)用的客戶端本身是知道所有服務(wù)信息,當需要調(diào)用服務(wù)上的接口的時候,客戶端從自身所維護的服務(wù)列表中,根據(jù)提前配置好的負載均衡策略,自己挑選一個服務(wù)來調(diào)用,此時,客戶端知道它所調(diào)用的是哪一個服務(wù).
在 Spring Cloud 中使用在RestTemplate進行服務(wù)調(diào)用,要想使用負載均衡功能,需要使用Spring Cloud Ribbon。
Spring Cloud Ribbon是一個基于HTTP和TCP的客戶端負載均衡工具,它基于Nettlix Ribbon實現(xiàn)。通過Spring Cloud的封裝,可以讓我們輕松地將面向服務(wù)的REST模板請求自動轉(zhuǎn)換成客戶端負載均衡的服務(wù)調(diào)用。
使用時需要給RestTemplate實例上添加一個@LoadBalanced注解即可,此時, RestTemplate就會自動具備負載均衡功能,這個負載均衡就是客戶端負載均衡。
package com.liruilong.consulcon; import org.springframework.boot.SpringApplication; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.cloud.client.discovery.EnableDiscoveryClient; import org.springframework.context.annotation.Bean; import org.springframework.web.client.RestTemplate; @SpringBootApplication @EnableDiscoveryClient public class ConsulConApplication { public static void main(String[] args) {
SpringApplication.run(ConsulConApplication.class, args);
} @Bean @LoadBalanced RestTemplate restTemplate(){ return new RestTemplate();
}
}
二、服務(wù)端負載
Ops 即運維角度的負載均衡,這里的負載我們也稱為服務(wù)端負載
所謂服務(wù)端負載均衡,比如傳統(tǒng)的Nginx的方式，調(diào)用的客戶端并不知道具體是哪個服務(wù)提供的服務(wù),它也不關(guān)心,反正請求發(fā)送給Nginx, 或者hyproxy作為代理的服務(wù)器，然后 Ngixn 在請求負載任意服務(wù),客戶端只需要記著Nginx的地址即可。
1、Nginx負載
七層(應(yīng)用層)負載
Nginx 7層負載是最常見的一種負載,所謂7層負載，即應(yīng)用層負載，即基于應(yīng)用層協(xié)議(TELNET,SSH,HTTP,SMTP,POP...)做的代理，7層負載需要解析數(shù)據(jù)包的具體內(nèi)容，需要消耗額外的cpu，然后根據(jù)具體內(nèi)容(url, 參數(shù), cookie, 請求頭)匹配相應(yīng)的路徑，然后轉(zhuǎn)發(fā)到相應(yīng)的服務(wù)器。轉(zhuǎn)發(fā)的過程是：建立和目標機器的連接，然后轉(zhuǎn)發(fā)請求，收到響應(yīng)數(shù)據(jù)在轉(zhuǎn)發(fā)給請求客戶端。
使用docker構(gòu)建一個內(nèi)部網(wǎng)絡(luò)
┌──[root@liruilongs.github.io]-[~] └─$ docker network create --subnet 10.1.1.1/24 load_balancing 0e0cdf9c70b038f9bcd44fd282ddc3e5bff77403ca28ce5b9006c20793ae2f8d
內(nèi)網(wǎng)里運行兩個httpd服務(wù)
10.1.1.22
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker run -itd --name=web1 --net=load_balancing -p 80 -h web1 --ip 10.1.1.22 httpd
ccaa091f295d40c61e50f103e9d84b86caddf9f98d6e5075de3690d93ab48f70
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ echo 10.1.1.22 >index.html;cat index.html
10.1.1.22
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker cp ./index.html web1:/usr/local/apache2/htdocs/
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
ccaa091f295d httpd "httpd-foreground" 29 seconds ago Up 28 seconds 0.0.0.0:49153->80/tcp, :::49153->80/tcp web1
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ curl 127.0.0.1:49153
10.1.1.22
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$
10.1.1.33
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker run -itd --name=web2 --net=load_balancing -p 80 -h web2 --ip 10.1.1.33 httpd
5b08d54cf4983f6f6ce69cc0cee4b2eab2684cfde8deee89796196760924e434
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ echo 10.1.1.33 >index.html;cat index.html
10.1.1.33
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker cp ./index.html web2:/usr/local/apache2/htdocs/
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5b08d54cf498 httpd "httpd-foreground" 40 seconds ago Up 38 seconds 0.0.0.0:49154->80/tcp, :::49154->80/tcp web2
ccaa091f295d httpd "httpd-foreground" 6 minutes ago Up 6 minutes 0.0.0.0:49153->80/tcp, :::49153->80/tcp web1
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ curl 127.0.0.1:49154
10.1.1.33
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$
Ngixn實現(xiàn)到上面兩個httpd服務(wù)的負載
| ng配置文件 |
|---|
|
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ cat nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid; #daemon off; events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$upstream_addr - $remote_addr - $remote_user [$time_local] "$request" ' '$status $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" ';
access_log /var/log/nginx/nginx_access.log main;
sendfile on;
keepalive_timeout 65;
include /etc/nginx/conf.d/*.conf;
server {
listen 8099;
server_name localhost;
root /var/www/html/;
index index.html index.htm;
access_log /var/log/nginx/default_access.log main;
error_log /var/log/nginx/default_error.log;
location / {
proxy_pass http://backend;
}
location ~ .* {
proxy_pass http://backend;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
upstream backend {
server web2:80;
server web1:80;
}
}
運行Nginx容器
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker run -itd -p 8099:8099 --name=nginx --network=load_balancing -v $PWD/nginx.conf:/etc/nginx/nginx.conf nginx
0af20ed5c390e81398037a498fc7d385cac96cd2f403a8b08f6f4e09d7a20ee0
測試一下
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0af20ed5c390 nginx "/docker-entrypoint.…" 4 seconds ago Up 3 seconds 80/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp nginx
b16bcb89e0a8 httpd "httpd-foreground" 32 minutes ago Up 32 minutes 0.0.0.0:49155->80/tcp, :::49155->80/tcp web1
5b08d54cf498 httpd "httpd-foreground" 39 minutes ago Up 39 minutes 0.0.0.0:49154->80/tcp, :::49154->80/tcp web2
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ curl 127.0.0.1:8099
10.1.1.33
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ curl 127.0.0.1:8099
10.1.1.22
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$
四層(傳輸層)負載
所謂四層負載，即在傳輸層協(xié)議的基礎(chǔ)上來做負載，基于TCP,UDP等協(xié)議，傳輸層的作用是確保數(shù)據(jù)被可靠的傳輸送到目標地址，能夠讓應(yīng)用程序之間實現(xiàn)通信，所以彼此傳遞的是數(shù)據(jù)包，標識的只有IP+端口。不涉及具體的url其他結(jié)構(gòu)解析。路徑匹配等，不會涉及具體的應(yīng)用層協(xié)議，所以理論上四層負載要比七成負載快。
nginx 四層代理是nginx1.9.0開始新增的功能，需要開啟--with-stream模塊，可以實現(xiàn)四層協(xié)議的轉(zhuǎn)發(fā)、代理、負載等功能。
這里的話，我們還是用容器的方式。配置方式和七層主要是配置文件的區(qū)別
| ng配置文件 |
|---|
|
└─$ cat nginx.conf
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log notice;
pid /var/run/nginx.pid;
daemon off;
events {
worker_connections 1024;
} # 四層代理的方式 stream{
server {
listen 8088;
proxy_pass backend;
}
upstream backend {
server web1:80;
server web2:80;
}
}
http { # 這個是協(xié)議級別 include mime.types;
default_type application/octet-stream;
keepalive_timeout 65;
gzip on;
server { #這個是服務(wù)器級別 listen 80;
server_name localhost;
location / { #這個是請求級別 root html;
index index.html index.htm;
}
}
}
啟動4層負載的Nginx
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker run -itd -p 8088:8088 --name=nginx4 --network=load_balancing -v $PWD/nginx.conf:/etc/nginx/nginx.conf nginx nginx
7c342f86752c5fe494b5a142983503d82dd11ea54e2968da7172f2201d1c45ea
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker logs nginx4
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/11/16 16:25:50 [notice] 1#1: using the "epoll" event method 2021/11/16 16:25:50 [notice] 1#1: nginx/1.21.3 2021/11/16 16:25:50 [notice] 1#1: built by gcc 8.3.0 (Debian 8.3.0-6) 2021/11/16 16:25:50 [notice] 1#1: OS: Linux 3.10.0-693.el7.x86_64 2021/11/16 16:25:50 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576 2021/11/16 16:25:50 [notice] 1#1: start worker processes 2021/11/16 16:25:50 [notice] 1#1: start worker process 31 2021/11/16 16:25:50 [notice] 1#1: start worker process 32 ┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
7c342f86752c nginx "/docker-entrypoint.…" 21 seconds ago Up 19 seconds 80/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp nginx4
0af20ed5c390 nginx "/docker-entrypoint.…" 2 hours ago Up 2 hours 80/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp nginx
b16bcb89e0a8 httpd "httpd-foreground" 2 hours ago Up 2 hours 0.0.0.0:49155->80/tcp, :::49155->80/tcp web1
5b08d54cf498 httpd "httpd-foreground" 2 hours ago Up 2 hours 0.0.0.0:49154->80/tcp, :::49154->80/tcp web2
測試一下
┌──[root@liruilongs.github.io]-[~/load_balancing] └─$ curl 127.0.0.1:8088 10.1.1.22 ┌──[root@liruilongs.github.io]-[~/load_balancing] └─$ curl 127.0.0.1:8088 10.1.1.33 ┌──[root@liruilongs.github.io]-[~/load_balancing] └─$ curl 127.0.0.1:8088 10.1.1.22
2、HAProxy負載
HAProxy 是一款提供高可用性、負載均衡以及基于TCP（第四層）和HTTP（第七層）應(yīng)用的代理軟件，支持虛擬主機，它是免費、快速并且可靠的一種解決方案。
HAProxy特別適用于那些負載特大的web站點，這些站點通常又需要會話保持或七層處理。HAProxy完全可以支持數(shù)以萬計的并發(fā)連接。
7 層(應(yīng)用層)負載
這里我們還用之前的連個httpd服務(wù)演示
┌──[root@liruilongs.github.io]-[~/load_balancing] └─$ docker pull haproxy
haproxy.cfg配置文件
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ cat haproxy.cfg
global log 127.0.0.1 local0 log 127.0.0.1 local1 notice
maxconn 4096
defaults log global
mode http
option httplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen admin_stats bind 0.0.0.0:8070
mode http
stats enable stats hide-version
stats scope .
stats realm Haproxy\ Statistics
stats refresh 30s
stats uri /
stats auth user:pass
frontend balancer bind 0.0.0.0:8077
mode http
default_backend web_backends
backend web_backends
mode http
option forwardfor
balance roundrobin
server web1 web1:80 check
server web2 web2:80 check
option httpchk GET /
http-check expect status 200
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker run -itd -p 8070 -p 8077 --name=haproxy --net=load_balancing -v $PWD/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg haproxy
a467636f6fdc75a7cf7538c32caab78c5a9e465235e2baec212bc9274a2cd534
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a467636f6fdc haproxy "docker-entrypoint.s…" 2 seconds ago Up 1 second 0.0.0.0:49163->8070/tcp, :::49163->8070/tcp, 0.0.0.0:49162->8077/tcp, :::49162->8077/tcp haproxy
7c342f86752c nginx "/docker-entrypoint.…" 40 minutes ago Up 40 minutes 80/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp nginx4
0af20ed5c390 nginx "/docker-entrypoint.…" 2 hours ago Up 2 hours 80/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp nginx
b16bcb89e0a8 httpd "httpd-foreground" 3 hours ago Up 3 hours 0.0.0.0:49155->80/tcp, :::49155->80/tcp web1
5b08d54cf498 httpd "httpd-foreground" 3 hours ago Up 3 hours 0.0.0.0:49154->80/tcp, :::49154->80/tcp
測試下
┌──[root@liruilongs.github.io]-[~/load_balancing] └─$ curl 127.0.0.1:49162 10.1.1.22 ┌──[root@liruilongs.github.io]-[~/load_balancing] └─$ curl 127.0.0.1:49162 10.1.1.33 ┌──[root@liruilongs.github.io]-[~/load_balancing] └─$
| 統(tǒng)計頁面 |
|---|
|
4 層(傳輸層)負載
四層負載和七層負載也是配置文件的區(qū)別
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ cat haproxy.cfg
global log 127.0.0.1 local0 log 127.0.0.1 local1 notice
maxconn 4096
defaults log global
mode http
option httplog
option dontlognull
timeout connect 5000ms
timeout client 50000ms
timeout server 50000ms
listen admin_stats bind 0.0.0.0:8070
mode http
stats enable stats hide-version
stats scope .
stats realm Haproxy\ Statistics
stats refresh 30s
stats uri /
stats auth user:pass
listen web bind *:3306
mode tcp
balance roundrobin
server web1 10.1.1.22:80 weight 1 check inter 1s rise 2 fall 2
server web2 10.1.1.33:80 weight 1 check inter 1s rise 2 fall 2
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$
運行容器并測試
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker run -itd -p 8070 -p 3306 --name=haproxy4 --net=load_balancing -v $PWD/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg haproxy
602eebc6ad525c420bad6061d9465fa7cdf2036d7db177efaf5450edf8c9db8b
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
602eebc6ad52 haproxy "docker-entrypoint.s…" 8 seconds ago Up 6 seconds 0.0.0.0:49165->3306/tcp, :::49165->3306/tcp, 0.0.0.0:49164->8070/tcp, :::49164->8070/tcp haproxy4
a467636f6fdc haproxy "docker-entrypoint.s…" 19 hours ago Up 19 hours 0.0.0.0:49163->8070/tcp, :::49163->8070/tcp, 0.0.0.0:49162->8077/tcp, :::49162->8077/tcp haproxy
7c342f86752c nginx "/docker-entrypoint.…" 20 hours ago Up 20 hours 80/tcp, 0.0.0.0:8088->8088/tcp, :::8088->8088/tcp
nginx4
0af20ed5c390 nginx "/docker-entrypoint.…" 21 hours ago Up 21 hours 80/tcp, 0.0.0.0:8099->8099/tcp, :::8099->8099/tcp
nginx
b16bcb89e0a8 httpd "httpd-foreground" 22 hours ago Up 22 hours 0.0.0.0:49155->80/tcp, :::49155->80/tcp
web1
5b08d54cf498 httpd "httpd-foreground" 22 hours ago Up 22 hours 0.0.0.0:49154->80/tcp, :::49154->80/tcp
web2
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ curl 127.0.0.1:49165
10.1.1.22
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$ curl 127.0.0.1:49165
10.1.1.33
┌──[root@liruilongs.github.io]-[~/load_balancing]
└─$