docker跨节点通信flannel

Posted on Views:

目的

etcd和flannel实现docker跨物理机通信

实验目标：
跨物理机的容器之间能直接访问
docker通过Flannel可以实现各容器间的相互通信，即宿主机和容器，容器和容器之间都能相互通信

实验环境：
192.168.3.50  //etcd、flannel、docker
192.168.3.51 //flannel、docker

1.系统统一设置

1.1
hosts\selinux\firewalld\iptalbes

systemctl stop firewalld.service

1.2
iptables -P INPUT ACCEPT && iptables -P FORWARD ACCEPT && iptables -F

2.install etcd

2.1download etcd
wget https://github.com/coreos/etcd/releases/download/v2.3.2/etcd-v2.3.2-linux-amd64.tar.gz
tar xvf etcd-v2.3.2-linux-amd64.tar.gz
cd etcd-v2.3.2-linux-amd64
mv etcd* /usr/local/bin/

2.2configure etcd.service
加入systemd管理，并且为了以后扩展，我还设置了集群，大家可以不必添加集群设置
cat > /usr/lib/systemd/system/etcd.service <<-'EOF'
[Unit]
Description=etcd
[Service]
Environment=ETCD_NAME=kubernetes
Environment=ETCD_DATA_DIR=/var/lib/etcd
Environment=ETCD_LISTEN_CLIENT_URLS=http://192.168.3.50:4001,http://localhost:4001
Environment=ETCD_LISTEN_PEER_URLS=http://192.168.3.50:7001,http://localhost:7001
Environment=ETCD_INITIAL_ADVERTISE_PEER_URLS=http://192.168.3.50:7001,http://localhost:7001
Environment=ETCD_ADVERTISE_CLIENT_URLS=http://192.168.3.50:4001,http://localhost:4001
Environment=ETCD_INITIAL_CLUSTER_STATE=new
Environment=ETCD_INITIAL_CLUSTER_TOKEN=Kubernetes
Environment=ETCD_INITIAL_CLUSTER=kubernetes=http://192.168.3.50:7001,kubernetes=http://localhost:7001
ExecStart=/usr/local/bin/etcd
[Install]
WantedBy=multi-user.target
EOF

2.3reload etcd
systemctl daemon-reload && systemctl start etcd

2.4testing
etcdctl ls /
etcdctl mkdir /test  //测试etcd可用性

2.5add subnework
etcd添加网段设置分配给docker网络的网段
etcdctl mk /coreos.com/network/config '{"Network":"172.20.0.0/16", "SubnetMin": "172.20.1.0", "SubnetMax": "172.20.254.0"}'
etcdctl get /coreos.com/network/config

3.install docker-ce

4.install flannel

注意：所有主机均需要安装flannel。

4.1二进制安装flannel
wget https://github.com/coreos/flannel/releases/download/v0.5.5/flannel-0.5.5-linux-amd64.tar.gz
[root@c_3 ~]# tar zxf flannel-0.5.5-linux-amd64.tar.gz 
[root@c_3 ~]# mv flannel-0.5.5 /opt/flannel
[root@c_3 ~]# 
[root@c_3 ~]# ll /opt/flannel/
total 16212
-rwxr-xr-x. 1 1000 1000 16581152 Nov 13  2015 flanneld
-rwxrwxr-x. 1 1000 1000     2008 Nov 13  2015 mk-docker-opts.sh
-rw-rw-r--. 1 1000 1000    11843 Nov 13  2015 README.md

4.2添加一个flannel服务的System单元，简单的就可以
cat > /usr/lib/systemd/system/flanneld.service <<-'EOF'
[Unit]
Description=flannel
[Service]
ExecStart=/opt/flannel/flanneld \
    -etcd-endpoints=http://192.168.3.50:4001
[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload && systemctl start flanneld && systemctl status flanneld

4.3testing
[root@c_3 ~]# ip a |grep flannel
4: flannel0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1472 qdisc pfifo_fast state UNKNOWN group default qlen 500
    inet 172.20.56.0/16 scope global flannel0

ot@c_3 opt]# etcdctl ls /coreos.com/network/subnets    //etcd查看已经分配的网段
/coreos.com/network/subnets/172.20.6.0-24
/coreos.com/network/subnets/172.20.56.0-24

5.docker-ce修改

5.1flannel help
cd /opt/flannel

[root@c_3 flannel]# ./mk-docker-opts.sh --help
./mk-docker-opts.sh: illegal option -- -
./mk-docker-opts.sh [-f FLANNEL-ENV-FILE] [-d DOCKER-ENV-FILE] [-i] [-c] [-m] [-k COMBINED-KEY]

Generate Docker daemon options based on flannel env file
OPTIONS:
    -f	Path to flannel env file. Defaults to /run/flannel/subnet.env   //flannel0网卡的配置信息
    -d	Path to Docker env file to write to. Defaults to /run/docker_opts.env
    -i	Output each Docker option as individual var. e.g. DOCKER_OPT_MTU=1500
    -c	Output combined Docker options into DOCKER_OPTS var       //网络信息写入 /run/docker_opts.env
    -k	Set the combined options key to this value (default DOCKER_OPTS=)
    -m	Do not output --ip-masq (useful for older Docker version)

5.2configure
[root@c_3 flannel]# ./mk-docker-opts.sh -c
[root@c_3 flannel]# cat /run/docker_opts.env    //保存环境键值对
DOCKER_OPTS=" --bip=172.20.56.1/24 --ip-masq=true --mtu=1472 "

5.3docker使用flannel的网络传递数据修改docker启动参数
[root@c_3 ~]# systemctl cat docker | head -1  //查看docker读取的配置文件
# /usr/lib/systemd/system/docker.service

/usr/lib/systemd/system/docker.service
EnvironmentFile=-/run/docker_opts.env
ExecStart=/usr/bin/dockerd $DOCKER_OPTS

systemctl daemon-reload && systemctl restart docker

5.4check
[root@c_3 flannel]# ip a|grep docker0   //docker0网络信息已经变化
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    inet 172.20.56.1/24 brd 172.20.56.255 scope global docker0

[root@c_3 flannel]# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
3c6b97123f83        bridge              bridge              local
4501ec9545f6        host                host                local
ccd8eef9f50b        none                null                local
[root@c_3 flannel]# docker network inspect 3c6b97123f83 |grep -i subnet
                    "Subnet": "172.20.56.1/24",

6.测试

docker run --rm -it busybox sh
ping ip

注意：
如上面操作后，发现各容器内分配的ip之间相互ping不通，基本就是由于防火墙问题引起的！
通过"systemctl stop firewalld.service"关闭了防火墙，为什么还有防火墙问题？？
这是因为linux还有底层的iptables，所以解决办法是在各节点上执行下面操作：
[root@node-1 ~]# iptables -P INPUT ACCEPT
[root@node-1 ~]# iptables -P FORWARD ACCEPT
[root@node-1 ~]# iptables -F
执行上面操作后，基本各容器间就能相互ping通了。
docker通过Flannel可以实现各容器间的相互通信，即宿主机和容器，容器和容器之间都能相互通信

维护命令

flannel生成的vxlan设备的删除方法
停止flanneld服务并确认
# systemctl stop flanneld
# status flanneld

使用ifconfig将设备停止
# ip addr s flannel.1
# ifconfig flannel.1 down

使用ip link del删除vxlan设备
# ip link del flannel.1
# ip addr s flannel.1

[root@c_3 ~]# etcdctl rm --recurive /coreos.com
Incorrect Usage.

NAME:
   etcdctl rm - remove a key or a directory

USAGE:
   etcdctl rm [command options] <key>

OPTIONS:
   --dir		removes the key if it is an empty directory or a key-value pair
   --recursive		removes the key and all child keys(if it is a directory)
   --with-value 	previous value
   --with-index "0"	previous index
   
[root@c_3 ~]# etcdctl rm --recursive /coreos.com

etcdctl mk /coreos.com/network/config '{"Network":"172.20.0.0/16", "SubnetMin": "172.20.1.0", "SubnetMax": "172.20.254.0"}'