kubeadm是Kubernetes官方提供的用于快速安装Kubernetes集群的工具，伴随Kubernetes每个版本的发布都会同步更新，kubeadm会对集群配置方面的一些实践做调整，通过实验kubeadm可以学习到Kubernetes官方在集群配置上一些新的最佳实践。最近发布的Kubernetes 1.15中，kubeadm对HA集群的配置已经达到beta可用，说明kubeadm距离生产环境中可用的距离越来越近了。1.准备1.1系统配置在安装之前，需要先做如下准备。两台CentOS 7.6主机如下：cat /etc/hosts 172.20.0.11 ip-172-20-0-11.cu-sd-cn.vm.7x34.com 172.20.0.12 ip-172-20-0-12.cu-sd-cn.vm.7x34.com 如果各个主机启用了防火墙，需要开放Kubernetes各个组件所需要的端口，可以查看Installing kubeadm中的”Check required ports”一节。 这里简单起见在各节点禁用防火墙：systemctl stop firewalld systemctl disable firewalld禁用SELINUX：setenforce 0 vi /etc/selinux/config SELINUX=disabled创建/etc/sysctl.d/k8s.conf文件，添加如下内容：net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1执行命令使修改生效。modprobe br_netfilter sysctl -p /etc/sysctl.d/k8s.conf1.2kube-proxy开启ipvs的前置条件由于ipvs已经加入到了内核的主干，所以为kube-proxy开启ipvs的前提需要加载以下的内核模块：ip_vs ip_vs_rr ip_vs_wrr ip_vs_sh nf_conntrack_ipv4在所有的Kubernetes节点node1和node2上执行以下脚本:cat > /etc/sysconfig/modules/ipvs.modules <<EOF #!/bin/bash modprobe -- ip_vs modprobe -- ip_vs_rr modprobe -- ip_vs_wrr modprobe -- ip_vs_sh modprobe -- nf_conntrack_ipv4 EOF chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep -e ip_vs -e nf_conntrack_ipv4上面脚本创建了的/etc/sysconfig/modules/ipvs.modules文件，保证在节点重启后能自动加载所需模块。 使用lsmod | grep -e ip_vs -e nf_conntrack_ipv4命令查看是否已经正确加载所需的内核模块。接下来还需要确保各个节点上已经安装了ipset软件包yum install ipset。 为了便于查看ipvs的代理规则，最好安装一下管理工具ipvsadm yum install ipvsadm。如果以上前提条件如果不满足，则即使kube-proxy的配置开启了ipvs模式，也会退回到iptables模式。1.3安装DockerKubernetes从1.6开始使用CRI(Container Runtime Interface)容器运行时接口。默认的容器运行时仍然是Docker，使用的是kubelet中内置dockershim CRI实现。安装docker的yum源:yum install -y yum-utils device-mapper-persistent-data lvm2 sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo查看最新的Docker版本：yum list docker-ce.x86_64 --showduplicates |sort -r docker-ce.x86_64 3:18.09.7-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.6-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.5-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.4-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.3-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.2-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.1-3.el7 docker-ce-stable docker-ce.x86_64 3:18.09.0-3.el7 docker-ce-stable docker-ce.x86_64 18.06.3.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.2.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable ...Kubernetes 1.15当前支持的docker版本列表是1.13.1, 17.03, 17.06, 17.09, 18.06, 18.09。 这里在各节点安装docker的18.09.7版本。yum makecache fast yum install -y --setopt=obsoletes=0 \ docker-ce-18.09.7-3.el7 systemctl start docker systemctl enable docker确认一下iptables filter表中FOWARD链的默认策略(pllicy)为ACCEPT。iptables -nvL Chain INPUT (policy ACCEPT 263 packets, 19209 bytes) pkts bytes target prot opt in out source destination Chain FORWARD (policy ACCEPT 0 packets, 0 bytes) pkts bytes target prot opt in out source destination 0 0 DOCKER-USER all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 DOCKER-ISOLATION-STAGE-1 all -- * * 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- * docker0 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED 0 0 DOCKER all -- * docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 !docker0 0.0.0.0/0 0.0.0.0/0 0 0 ACCEPT all -- docker0 docker0 0.0.0.0/0 0.0.0.0/01.4 修改docker cgroup driver为systemd根据文档CRI installation中的内容，对于使用systemd作为init system的Linux的发行版，使用systemd作为docker的cgroup driver可以确保服务器节点在资源紧张的情况更加稳定，因此这里修改各个节点上docker的cgroup driver为systemd。创建或修改/etc/docker/daemon.json：{ "exec-opts": ["native.cgroupdriver=systemd"] }重启docker：systemctl restart docker docker info | grep Cgroup Cgroup Driver: systemd2.使用kubeadm部署Kubernetes2.1 安装kubeadm和kubelet下面在各节点安装kubeadm和kubelet：cat <<EOF > /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF安装kubeadm、kubectl、kubeletyum makecache fast yum install -y kubelet kubeadm kubectl ... Installed: kubeadm.x86_64 0:1.15.0-0 kubectl.x86_64 0:1.15.0-0 kubelet.x86_64 0:1.15.0-0 Dependency Installed: conntrack-tools.x86_64 0:1.4.4-4.el7 cri-tools.x86_64 0:1.12.0-0 kubernetes-cni.x86_64 0:0.7.5-0 libnetfilter_cthelper.x86_64 0:1.0.0-9.el7 libnetfilter_cttimeout.x86_64 0:1.0.0-6.el7 libnetfilter_queue.x86_64 0:1.0.2-2.el7_2 socat.x86_64 0:1.7.3.2-2.el7 从安装结果可以看出还安装了cri-tools, kubernetes-cni, socat三个依赖：官方从Kubernetes 1.14开始将cni依赖升级到了0.7.5版本socat是kubelet的依赖cri-tools是CRI(Container Runtime Interface)容器运行时接口的命令行工具运行kubelet --help可以看到原来kubelet的绝大多数命令行flag参数都被DEPRECATED了，如：...... --address 0.0.0.0 The IP address for the Kubelet to serve on (set to 0.0.0.0 for all IPv4 interfaces and `::` for all IPv6 interfaces) (default 0.0.0.0) (DEPRECATED: This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.) ......而官方推荐我们使用--config指定配置文件，并在配置文件中指定原来这些flag所配置的内容。具体内容可以查看这里Set Kubelet parameters via a config file。这也是Kubernetes为了支持动态Kubelet配置（Dynamic Kubelet Configuration）才这么做的，参考Reconfigure a Node’s Kubelet in a Live Cluster。kubelet的配置文件必须是json或yaml格式，具体可查看这里。Kubernetes 1.8开始要求关闭系统的Swap，如果不关闭，默认配置下kubelet将无法启动。 关闭系统的Swap方法如下:swapoff -a修改 /etc/fstab 文件，注释掉 SWAP 的自动挂载，使用free -m确认swap已经关闭。 swappiness参数调整，修改/etc/sysctl.d/k8s.conf添加下面一行：vm.swappiness=0执行sysctl -p /etc/sysctl.d/k8s.conf使修改生效。因为这里本次用于测试两台主机上还运行其他服务，关闭swap可能会对其他服务产生影响，所以这里修改kubelet的配置去掉这个限制。 使用kubelet的启动参数--fail-swap-on=false去掉必须关闭Swap的限制，修改/etc/sysconfig/kubelet，加入：KUBELET_EXTRA_ARGS=--fail-swap-on=false2.2 使用kubeadm init初始化集群在各节点开机启动kubelet服务：systemctl enable kubelet.service使用kubeadm config print init-defaults可以打印集群初始化默认的使用的配置：apiVersion: kubeadm.k8s.io/v1beta2 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 1.2.3.4 bindPort: 6443 nodeRegistration: criSocket: /var/run/dockershim.sock name: node1 taints: - effect: NoSchedule key: node-role.kubernetes.io/master --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta2 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: type: CoreDNS etcd: local: dataDir: /var/lib/etcd imageRepository: k8s.gcr.io kind: ClusterConfiguration kubernetesVersion: v1.14.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 scheduler: {} 从默认的配置中可以看到，可以使用imageRepository定制在集群初始化时拉取k8s所需镜像的地址。基于默认配置定制出本次使用kubeadm初始化集群所需的配置文件kubeadm.yaml：apiVersion: kubeadm.k8s.io/v1beta2 kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.99.11 bindPort: 6443 nodeRegistration: taints: - effect: PreferNoSchedule key: node-role.kubernetes.io/master --- apiVersion: kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: v1.15.0 networking: podSubnet: 10.244.0.0/16使用kubeadm默认配置初始化的集群，会在master节点打上node-role.kubernetes.io/master:NoSchedule的污点，阻止master节点接受调度运行工作负载。这里测试环境只有两个节点，所以将这个taint修改为node-role.kubernetes.io/master:PreferNoSchedule。在开始初始化集群之前可以使用kubeadm config images pull预先在各个节点上拉取所k8s需要的docker镜像。接下来使用kubeadm初始化集群，选择node1作为Master Node，在node1上执行下面的命令：kubeadm init --config kubeadm.yaml --ignore-preflight-errors=Swap [init] Using Kubernetes version: v1.15.0 [preflight] Running pre-flight checks [WARNING Swap]: running with swap on is not supported. Please disable swap [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Activating the kubelet service [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "etcd/ca" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [node1 localhost] and IPs [192.168.99.11 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [node1 localhost] and IPs [192.168.99.11 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.99.11] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 26.004907 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.15" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node node1 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:PreferNoSchedule] [bootstrap-token] Using token: 4qcl2f.gtl3h8e5kjltuo0r [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.99.11:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \ --discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e 上面记录了完成的初始化输出的内容，根据输出的内容基本上可以看出手动初始化安装一个Kubernetes集群所需要的关键步骤。 其中有以下关键内容：[kubelet-start] 生成kubelet的配置文件”/var/lib/kubelet/config.yaml”[certs]生成相关的各种证书[kubeconfig]生成相关的kubeconfig文件[control-plane]使用/etc/kubernetes/manifests目录中的yaml文件创建apiserver、controller-manager、scheduler的静态pod[bootstraptoken]生成token记录下来，后边使用kubeadm join往集群中添加节点时会用到下面的命令是配置常规用户如何使用kubectl访问集群：mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config最后给出了将节点加入集群的命令kubeadm join 192.168.99.11:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \ --discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e查看一下集群状态，确认个组件都处于healthy状态：kubectl get cs NAME STATUS MESSAGE ERROR controller-manager Healthy ok scheduler Healthy ok etcd-0 Healthy {"health":"true"} 集群初始化如果遇到问题，可以使用下面的命令进行清理：kubeadm reset ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1 rm -rf /var/lib/cni/2.3 安装Pod Network接下来安装flannel network add-on：mkdir -p ~/k8s/ cd ~/k8s curl -O https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml kubectl apply -f kube-flannel.yml clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.extensions/kube-flannel-ds-amd64 created daemonset.extensions/kube-flannel-ds-arm64 created daemonset.extensions/kube-flannel-ds-arm created daemonset.extensions/kube-flannel-ds-ppc64le created daemonset.extensions/kube-flannel-ds-s390x created这里注意kube-flannel.yml这个文件里的flannel的镜像是0.11.0，quay.io/coreos/flannel:v0.11.0-amd64如果Node有多个网卡的话，参考flannel issues 39701，目前需要在kube-flannel.yml中使用--iface参数指定集群主机内网网卡的名称，否则可能会出现dns无法解析。需要将kube-flannel.yml下载到本地，flanneld启动参数加上--iface=<iface-name>...... containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.11.0-amd64 command: - /opt/bin/flanneld args: - --ip-masq - --kube-subnet-mgr - --iface=eth1 ......使用kubectl get pod --all-namespaces -o wide确保所有的Pod都处于Running状态。kubectl get pod -n kube-system NAME READY STATUS RESTARTS AGE coredns-5c98db65d4-dr8lf 1/1 Running 0 52m coredns-5c98db65d4-lp8dg 1/1 Running 0 52m etcd-node1 1/1 Running 0 51m kube-apiserver-node1 1/1 Running 0 51m kube-controller-manager-node1 1/1 Running 0 51m kube-flannel-ds-amd64-mm296 1/1 Running 0 44s kube-proxy-kchkf 1/1 Running 0 52m kube-scheduler-node1 1/1 Running 0 51m2.4 测试集群DNS是否可用kubectl run curl --image=radial/busyboxplus:curl -it kubectl run --generator=deployment/apps.v1beta1 is DEPRECATED and will be removed in a future version. Use kubectl create instead. If you don't see a command prompt, try pressing enter. [ root@curl-5cc7b478b6-r997p:/ ]$ 进入后执行nslookup kubernetes.default确认解析正常:nslookup kubernetes.default Server: 10.96.0.10 Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local Name: kubernetes.default Address 1: 10.96.0.1 kubernetes.default.svc.cluster.local2.5 向Kubernetes集群中添加Node节点下面将node2这个主机添加到Kubernetes集群中，在node2上执行:kubeadm join 192.168.99.11:6443 --token 4qcl2f.gtl3h8e5kjltuo0r \ --discovery-token-ca-cert-hash sha256:7ed5404175cc0bf18dbfe53f19d4a35b1e3d40c19b10924275868ebf2a3bbe6e \ --ignore-preflight-errors=Swap [preflight] Running pre-flight checks [WARNING Swap]: running with swap on is not supported. Please disable swap [WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service' [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' [kubelet-start] Downloading configuration for the kubelet from the "kubelet-config-1.15" ConfigMap in the kube-system namespace [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Activating the kubelet service [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.node2加入集群很是顺利，下面在master节点上执行命令查看集群中的节点：kubectl get node NAME STATUS ROLES AGE VERSION node1 Ready master 57m v1.15.0 node2 Ready <none> 11s v1.15.02.5.1 如何从集群中移除Node如果需要从集群中移除node2这个Node执行下面的命令：在master节点上执行：kubectl drain node2 --delete-local-data --force --ignore-daemonsets kubectl delete node node2在node2上执行：kubeadm reset ifconfig cni0 down ip link delete cni0 ifconfig flannel.1 down ip link delete flannel.1 rm -rf /var/lib/cni/在node1上执行：kubectl delete node node22.6 kube-proxy开启ipvs修改ConfigMap的kube-system/kube-proxy中的config.conf，mode: "ipvs"：kubectl edit cm kube-proxy -n kube-system之后重启各个节点上的kube-proxy pod：kubectl get pod -n kube-system | grep kube-proxy | awk '{system("kubectl delete pod "$1" -n kube-system")}'kubectl get pod -n kube-system | grep kube-proxy kube-proxy-7fsrg 1/1 Running 0 3s kube-proxy-k8vhm 1/1 Running 0 9s kubectl logs kube-proxy-7fsrg -n kube-system I0703 04:42:33.308289 1 server_others.go:170] Using ipvs Proxier. W0703 04:42:33.309074 1 proxier.go:401] IPVS scheduler not specified, use rr by default I0703 04:42:33.309831 1 server.go:534] Version: v1.15.0 I0703 04:42:33.320088 1 conntrack.go:52] Setting nf_conntrack_max to 131072 I0703 04:42:33.320365 1 config.go:96] Starting endpoints config controller I0703 04:42:33.320393 1 controller_utils.go:1029] Waiting for caches to sync for endpoints config controller I0703 04:42:33.320455 1 config.go:187] Starting service config controller I0703 04:42:33.320470 1 controller_utils.go:1029] Waiting for caches to sync for service config controller I0703 04:42:33.420899 1 controller_utils.go:1036] Caches are synced for endpoints config controller I0703 04:42:33.420969 1 controller_utils.go:1036] Caches are synced for service config controller日志中打印出了Using ipvs Proxier，说明ipvs模式已经开启。3.Kubernetes常用组件部署越来越多的公司和团队开始使用Helm这个Kubernetes的包管理器，这里也将使用Helm安装Kubernetes的常用组件。3.1 Helm的安装Helm由客户端命helm令行工具和服务端tiller组成，Helm的安装十分简单。 下载helm命令行工具到master节点node1的/usr/local/bin下，这里下载的2.14.1版本：curl -O https://get.helm.sh/helm-v2.14.1-linux-amd64.tar.gz tar -zxvf helm-v2.14.1-linux-amd64.tar.gz cd linux-amd64/ cp helm /usr/local/bin/为了安装服务端tiller，还需要在这台机器上配置好kubectl工具和kubeconfig文件，确保kubectl工具可以在这台机器上访问apiserver且正常使用。 这里的node1节点已经配置好了kubectl。因为Kubernetes APIServer开启了RBAC访问控制，所以需要创建tiller使用的service account: tiller并分配合适的角色给它。 详细内容可以查看helm文档中的Role-based Access Control。 这里简单起见直接分配cluster-admin这个集群内置的ClusterRole给它。创建helm-rbac.yaml文件：apiVersion: v1 kind: ServiceAccount metadata: name: tiller namespace: kube-system --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: tiller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: tiller namespace: kube-systemubectl create -f helm-rbac.yaml serviceaccount/tiller created clusterrolebinding.rbac.authorization.k8s.io/tiller created接下来使用helm部署tiller:helm init --service-account tiller --skip-refresh Creating /root/.helm Creating /root/.helm/repository Creating /root/.helm/repository/cache Creating /root/.helm/repository/local Creating /root/.helm/plugins Creating /root/.helm/starters Creating /root/.helm/cache/archive Creating /root/.helm/repository/repositories.yaml Adding stable repo with URL: https://kubernetes-charts.storage.googleapis.com Adding local repo with URL: http://127.0.0.1:8879/charts $HELM_HOME has been configured at /root/.helm. Tiller (the Helm server-side component) has been installed into your Kubernetes Cluster. Please note: by default, Tiller is deployed with an insecure 'allow unauthenticated users' policy. To prevent this, run `helm init` with the --tiller-tls-verify flag. For more information on securing your installation see: https://docs.helm.sh/using_helm/#securing-your-helm-installation Happy Helming!tiller默认被部署在k8s集群中的kube-system这个namespace下：kubectl get pod -n kube-system -l app=helm NAME READY STATUS RESTARTS AGE tiller-deploy-c4fd4cd68-dwkhv 1/1 Running 0 83shelm version Client: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"} Server: &version.Version{SemVer:"v2.14.1", GitCommit:"5270352a09c7e8b6e8c9593002a73535276507c0", GitTreeState:"clean"}注意由于某些原因需要网络可以访问gcr.io和kubernetes-charts.storage.googleapis.com，如果无法访问可以通过helm init --service-account tiller --tiller-image <your-docker-registry>/tiller:v2.13.1 --skip-refresh使用私有镜像仓库中的tiller镜像最后在node1上修改helm chart仓库的地址为azure提供的镜像地址：helm repo add stable http://mirror.azure.cn/kubernetes/charts "stable" has been added to your repositories helm repo list NAME URL stable http://mirror.azure.cn/kubernetes/charts local http://127.0.0.1:8879/charts 3.2 使用Helm部署Nginx Ingress为了便于将集群中的服务暴露到集群外部，需要使用Ingress。接下来使用Helm将Nginx Ingress部署到Kubernetes上。 Nginx Ingress Controller被部署在Kubernetes的边缘节点上，关于Kubernetes边缘节点的高可用相关的内容可以查看之前整理的Bare metal环境下Kubernetes Ingress边缘节点的高可用,Ingress Controller使用hostNetwork。我们将node1(192.168.99.11)做为边缘节点，打上Label：kubectl label node node1 node-role.kubernetes.io/edge= node/node1 labeled kubectl get node NAME STATUS ROLES AGE VERSION node1 Ready edge,master 138m v1.15.0 node2 Ready <none> 82m v1.15.0stable/nginx-ingress chart的值文件ingress-nginx.yaml如下：controller: replicaCount: 1 hostNetwork: true nodeSelector: node-role.kubernetes.io/edge: '' affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app operator: In values: - nginx-ingress - key: component operator: In values: - controller topologyKey: kubernetes.io/hostname tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule defaultBackend: nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedulenginx ingress controller的副本数replicaCount为1，将被调度到node1这个边缘节点上。这里并没有指定nginx ingress controller service的externalIPs，而是通过hostNetwork: true设置nginx ingress controller使用宿主机网络。helm repo update helm install stable/nginx-ingress \ -n nginx-ingress \ --namespace ingress-nginx \ -f ingress-nginx.yamlkubectl get pod -n ingress-nginx -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES nginx-ingress-controller-cc9b6d55b-pr8vr 1/1 Running 0 10m 192.168.99.11 node1 <none> <none> nginx-ingress-default-backend-cc888fd56-bf4h2 1/1 Running 0 10m 10.244.0.14 node1 <none> <none>如果访问http://192.168.99.11返回default backend，则部署完成。3.3 使用Helm部署dashboardkubernetes-dashboard.yaml：image: repository: k8s.gcr.io/kubernetes-dashboard-amd64 tag: v1.10.1 ingress: enabled: true hosts: - k8s.frognew.com annotations: nginx.ingress.kubernetes.io/ssl-redirect: "true" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS" tls: - secretName: frognew-com-tls-secret hosts: - k8s.frognew.com nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedule rbac: clusterAdminRole: truehelm install stable/kubernetes-dashboard \ -n kubernetes-dashboard \ --namespace kube-system \ -f kubernetes-dashboard.yamlkubectl -n kube-system get secret | grep kubernetes-dashboard-token kubernetes-dashboard-token-pkm2s kubernetes.io/service-account-token 3 3m7s kubectl describe -n kube-system secret/kubernetes-dashboard-token-pkm2s Name: kubernetes-dashboard-token-pkm2s Namespace: kube-system Labels: <none> Annotations: kubernetes.io/service-account.name: kubernetes-dashboard kubernetes.io/service-account.uid: 2f0781dd-156a-11e9-b0f0-080027bb7c43 Type: kubernetes.io/service-account-token Data ==== ca.crt: 1025 bytes namespace: 11 bytes token: eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.[binary].24ad6ZgZMxdydpwlmYAiMxZ9VSIN7dDR7Q6-RLW0qC81ajXoQKHAyrEGpIonfld3gqbE0xO8nisskpmlkQra72-9X6sBPoByqIKyTsO83BQlME2sfOJemWD0HqzwSCjvSQa0x-bUlq9HgH2vEXzpFuSS6Svi7RbfzLXlEuggNoC4MfA4E2hF1OX_ml8iAKx-49y1BQQe5FGWyCyBSi1TD_-ZpVs44H5gIvsGK2kcvi0JT4oHXtWjjQBKLIWL7xxyRCSE4HmUZT2StIHnOwlX7IEIB0oBX4mPg2_xNGnqwcu-8OERU9IoqAAE2cZa0v3b5O2LMcJPrcxrVOukvRIumA在dashboard的登录窗口使用上面的token登录。3.4 使用Helm部署metrics-server从Heapster的github https://github.com/kubernetes/heapster中可以看到已经，heapster已经DEPRECATED。 这里是heapster的deprecation timeline。 可以看出heapster从Kubernetes 1.12开始从Kubernetes各种安装脚本中移除。Kubernetes推荐使用metrics-server。我们这里也使用helm来部署metrics-server。metrics-server.yaml:args: - --logtostderr - --kubelet-insecure-tls - --kubelet-preferred-address-types=InternalIP nodeSelector: node-role.kubernetes.io/edge: '' tolerations: - key: node-role.kubernetes.io/master operator: Exists effect: NoSchedule - key: node-role.kubernetes.io/master operator: Exists effect: PreferNoSchedulehelm install stable/metrics-server \ -n metrics-server \ --namespace kube-system \ -f metrics-server.yaml使用下面的命令可以获取到关于集群节点基本的指标信息：kubectl top node NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% node1 650m 32% 1276Mi 73% node2 73m 3% 527Mi 30% Codekubectl top pod -n kube-system NAME CPU(cores) MEMORY(bytes) coredns-5c98db65d4-dr8lf 8m 7Mi coredns-5c98db65d4-lp8dg 6m 8Mi etcd-node1 44m 46Mi kube-apiserver-node1 74m 295Mi kube-controller-manager-node1 35m 50Mi kube-flannel-ds-amd64-7lwm9 2m 8Mi kube-flannel-ds-amd64-mm296 5m 9Mi kube-proxy-7fsrg 1m 11Mi kube-proxy-k8vhm 3m 11Mi kube-scheduler-node1 8m 15Mi kubernetes-dashboard-848b8dd798-c4sc2 2m 14Mi metrics-server-8456fb6676-fwh2t 10m 19Mi tiller-deploy-7bf78cdbf7-9q94c 1m 16Mi 遗憾的是，当前Kubernetes Dashboard还不支持metrics-server。因此如果使用metrics-server替代了heapster，将无法在dashboard中以图形展示Pod的内存和CPU情况(实际上这也不是很重要，当前我们是在Prometheus和Grafana中定制的Kubernetes集群中各个Pod的监控，因此在dashboard中查看Pod内存和CPU也不是很重要)。 Dashboard的github上有很多这方面的讨论，如https://github.com/kubernetes/dashboard/issues/2986，Dashboard已经准备在将来的某个时间点支持metrics-server。但由于metrics-server和metrics pipeline肯定是Kubernetes在monitor方面未来的方向，所以推荐使用metrics-server。4.总结本次安装涉及到的Docker镜像：# network and dns quay.io/coreos/flannel:v0.11.0-amd64 k8s.gcr.io/coredns:1.3.1 # helm and tiller gcr.io/kubernetes-helm/tiller:v2.14.1 # nginx ingress quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.24.1 k8s.gcr.io/defaultbackend:1.5 # dashboard and metric-sever k8s.gcr.io/kubernetes-dashboard-amd64:v1.10.1 gcr.io/google_containers/metrics-server-amd64:v0.3.2参考Installing kubeadmUsing kubeadm to Create a ClusterGet Docker CE for CentOSkubernetes: k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm/v1beta2
1. 命令行参数-v,–verbose 详细模式，如果命令执行成功，输出详细的结果(-vv –vvv -vvvv) -i PATH,–inventory=PATH 指定host文件的路径，默认是在/etc/ansible/hosts(生产环境经常用到) -f NUM,–forks=NU NUM是指定一个整数，默认是5，指定fork开启同步进程的个数。 -m NAME,–module-name=NAME 指定使用的module名称，默认是command -m DIRECTORY,–module-path=DIRECTORY 指定module的目录来加载module，默认是/usr/share/ansible, -a,MODULE_ARGS 指定module模块的参数 -k,-ask-pass 提示输入ssh的密码，而不是使用基于ssh的密钥认证 -sudo 指定使用sudo获得root权限(生产环境经常用到) -K,-ask-sudo-pass 提示输入sudo密码，与–sudo一起使用 (生产环境经常用到) -u USERNAME,-user=USERNAME 指定移动端的执行用户 -C,-check 测试此命令执行会改变什么内容，不会真正的去执行2. command模块测试环境[root@temp01 ~]# ansible all -a "pwd" 192.168.3.70 | SUCCESS | rc=0 >> /root 192.168.3.60 | SUCCESS | rc=0 >> /root [root@temp01 ~]# ansible all -m command -a "pwd" #-m command 原来是可以省略的，ansible 默认模块就是command 192.168.3.70 | SUCCESS | rc=0 >> /root 192.168.3.60 | SUCCESS | rc=0 >> /root3. 再用ping模块#-f 参数用法 [root@temp01 ~]# ansible all -m ping -f 1 #all 表示hosts里面所有主机 -m 调用模块 -f 指定同步进程数 192.168.3.60 | SUCCESS => { "changed": false, "ping": "pong" } 192.168.3.70 | SUCCESS => { "changed": false, "ping": "pong" } #-C参数用法 [root@temp01 ~]# ansible all -m ping -f 1 -C 192.168.3.60 | SUCCESS => { "changed": false, "ping": "pong" } 192.168.3.70 | SUCCESS => { "changed": false, "ping": "pong" }4. user模块[root@temp01 ~]# ansible db -m user -a 'name=DBA uid=505 home=/home/dba shell=/sbin/nologin' 192.168.3.70 | SUCCESS => { "changed": true, "comment": "", "createhome": true, "group": 505, "home": "/home/dba", "name": "DBA", "shell": "/sbin/nologin", "state": "present", "system": false, "uid": 505 } [root@temp01 ~]# ansible db -m user -a 'name=DBA state=absent' #state属性有：present表示添加,absend表示删除，state属性默认是present 192.168.3.70 | SUCCESS => { "changed": true, "force": false, "name": "DBA", "remove": false, "state": "absent" } 修改用户密码[root@temp01 ansible_playbook]# ansible all -m shell -a 'echo "123456" | passwd foo --stdin' 192.168.3.60 | SUCCESS | rc=0 >> Changing password for user foo. passwd: all authentication tokens updated successfully. 192.168.3.70 | SUCCESS | rc=0 >> Changing password for user foo. passwd: all authentication tokens updated successfully.看看下面这个坑[root@temp01 ~]# ansible all -m cron -a "name=cat minute='*/1' job='ls /root'" 192.168.3.70 | FAILED! => { "changed": false, "failed": true, "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!" } 192.168.3.60 | FAILED! => { "changed": false, "failed": true, "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!" }[root@temp01 ~]# ansible all -m copy -a "src=/etc/ansible/hosts dest=/root mode=600" 192.168.3.60 | FAILED! => { "changed": false, "checksum": "8add7a4e56ae6f7a1c6d0218ce757510d195bb3c", "failed": true, "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!" } 192.168.3.70 | FAILED! => { "changed": false, "checksum": "8add7a4e56ae6f7a1c6d0218ce757510d195bb3c", "failed": true, "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!" }需要在客户端安装libselinux-python库yum模块[root@temp01 ~]# ansible all -m yum -a 'name=libselinux-python state=present' 192.168.3.70 | SUCCESS => { "changed": true, "msg": "warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY\nImporting GPG key 0xC105B9DE:\n Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>\n Package: centos-release-6-5.el6.centos.11.1.x86_64 (@anaconda-CentOS-201311272149.x86_64/6.5)\n From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6\n", "rc": 0, "results": [ "Loaded plugins: fastestmirror, security\nLoading mirror speeds from cached hostfile\n * base: mirrors.zju.edu.cn\n * extras: centos.ustc.edu.cn\n * updates: mirrors.zju.edu.cn\nSetting up Install Process\nResolving Dependencies\n--> Running transaction check\n---> Package libselinux-python.x86_64 0:2.0.94-7.el6 will be installed\n--> Processing Dependency: libselinux = 2.0.94-7.el6 for package: libselinux-python-2.0.94-7.el6.x86_64\n--> Running transaction check\n---> Package libselinux.x86_64 0:2.0.94-5.3.el6_4.1 will be updated\n--> Processing Dependency: libselinux = 2.0.94-5.3.el6_4.1 for package: libselinux-utils-2.0.94-5.3.el6_4.1.x86_64\n---> Package libselinux.x86_64 0:2.0.94-7.el6 will be an update\n--> Running transaction check\n---> Package libselinux-utils.x86_64 0:2.0.94-5.3.el6_4.1 will be updated\n---> Package libselinux-utils.x86_64 0:2.0.94-7.el6 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n libselinux-python x86_64 2.0.94-7.el6 base 203 k\nUpdating for dependencies:\n libselinux x86_64 2.0.94-7.el6 base 109 k\n libselinux-utils x86_64 2.0.94-7.el6 base 82 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package(s)\nUpgrade 2 Package(s)\n\nTotal download size: 394 k\nDownloading Packages:\n--------------------------------------------------------------------------------\nTotal 210 kB/s | 394 kB 00:01 \nRetrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6\nRunning rpm_check_debug\nRunning Transaction Test\nTransaction Test Succeeded\nRunning Transaction\n\r Updating : libselinux-2.0.94-7.el6.x86_64 1/5 \n\r Installing : libselinux-python-2.0.94-7.el6.x86_64 2/5 \n\r Updating : libselinux-utils-2.0.94-7.el6.x86_64 3/5 \n\r Cleanup : libselinux-utils-2.0.94-5.3.el6_4.1.x86_64 4/5 \n\rCleanup : libselinux-2.0.94-5.3.el6_4.1.x86_64 5/5 \n\r Verifying : libselinux-python-2.0.94-7.el6.x86_64 1/5 \n\r Verifying : libselinux-utils-2.0.94-7.el6.x86_64 2/5 \n\r Verifying : libselinux-2.0.94-7.el6.x86_64 3/5 \n\r Verifying : libselinux-2.0.94-5.3.el6_4.1.x86_64 4/5 \n\r Verifying : libselinux-utils-2.0.94-5.3.el6_4.1.x86_64 5/5 \n\nInstalled:\n libselinux-python.x86_64 0:2.0.94-7.el6 \n\nDependency Updated:\n libselinux.x86_64 0:2.0.94-7.el6 libselinux-utils.x86_64 0:2.0.94-7.el6 \n\nComplete!\n" ] } 192.168.3.60 | SUCCESS => { "changed": true, "msg": "warning: rpmts_HdrFromFdno: Header V3 RSA/SHA1 Signature, key ID c105b9de: NOKEY\nImporting GPG key 0xC105B9DE:\n Userid : CentOS-6 Key (CentOS 6 Official Signing Key) <centos-6-key@centos.org>\n Package: centos-release-6-5.el6.centos.11.1.x86_64 (@anaconda-CentOS-201311272149.x86_64/6.5)\n From : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6\n", "rc": 0, "results": [ "Loaded plugins: fastestmirror, security\nLoading mirror speeds from cached hostfile\n * base: mirrors.cn99.com\n * extras: mirrors.cn99.com\n * updates: mirrors.cn99.com\nSetting up Install Process\nResolving Dependencies\n--> Running transaction check\n---> Package libselinux-python.x86_64 0:2.0.94-7.el6 will be installed\n--> Processing Dependency: libselinux = 2.0.94-7.el6 for package: libselinux-python-2.0.94-7.el6.x86_64\n--> Running transaction check\n--->Package libselinux.x86_64 0:2.0.94-5.3.el6_4.1 will be updated\n--> Processing Dependency: libselinux = 2.0.94-5.3.el6_4.1 for package: libselinux-utils-2.0.94-5.3.el6_4.1.x86_64\n---> Package libselinux.x86_64 0:2.0.94-7.el6 will be an update\n--> Running transaction check\n---> Package libselinux-utils.x86_64 0:2.0.94-5.3.el6_4.1 will be updated\n---> Package libselinux-utils.x86_64 0:2.0.94-7.el6 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n libselinux-python x86_64 2.0.94-7.el6 base 203 k\nUpdatingfor dependencies:\n libselinux x86_64 2.0.94-7.el6 base 109 k\n libselinux-utils x86_64 2.0.94-7.el6 base 82 k\n\nTransaction Summary\n================================================================================\nInstall 1 Package(s)\nUpgrade 2 Package(s)\n\nTotal download size: 394 k\nDownloading Packages:\n--------------------------------------------------------------------------------\nTotal 70 kB/s | 394 kB 00:05 \nRetrieving key from file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-6\nRunning rpm_check_debug\nRunning Transaction Test\nTransaction Test Succeeded\nRunning Transaction\n\r Updating : libselinux-2.0.94-7.el6.x86_64 1/5 \n\r Installing : libselinux-python-2.0.94-7.el6.x86_64 2/5 \n\r Updating : libselinux-utils-2.0.94-7.el6.x86_64 3/5 \n\r Cleanup : libselinux-utils-2.0.94-5.3.el6_4.1.x86_64 4/5 \n\r Cleanup : libselinux-2.0.94-5.3.el6_4.1.x86_64 5/5 \n\r Verifying : libselinux-python-2.0.94-7.el6.x86_64 1/5 \n\r Verifying : libselinux-utils-2.0.94-7.el6.x86_64 2/5 \n\r Verifying : libselinux-2.0.94-7.el6.x86_64 3/5 \n\r Verifying : libselinux-2.0.94-5.3.el6_4.1.x86_64 4/5 \n\r Verifying : libselinux-utils-2.0.94-5.3.el6_4.1.x86_64 5/5 \n\nInstalled:\n libselinux-python.x86_64 0:2.0.94-7.el6 \n\nDependency Updated:\n libselinux.x86_64 0:2.0.94-7.el6 libselinux-utils.x86_64 0:2.0.94-7.el6 \n\nComplete!\n" ] }service 模块[root@temp01 ~]# ansible all -m service -a 'name=sshd state=restarted' #state属性有running,started,stopped,restarted,reloaded 192.168.3.60 | SUCCESS => { "changed": true, "name": "sshd", "state": "started" } 192.168.3.70 | SUCCESS => { "changed": true, "name": "sshd", "state": "started" }yum模块还是介绍下[root@temp01 ~]# ansible all -m yum -a 'name=ntp state=present' #state属性有absent,present,installed,removed,latest 192.168.3.70 | SUCCESS => { "changed": false, "msg": "", "rc": 0, "results": [ "ntp-4.2.6p5-1.el6.centos.x86_64 providing ntp is already installed" ] } 192.168.3.60 | SUCCESS => { "changed": false, "msg": "", "rc": 0, "results": [ "ntp-4.2.6p5-1.el6.centos.x86_64 providing ntp is already installed" ] }script模块 远程执行脚本[root@temp01 ~]# cat ansible.sh #脚本内容 #!/bin/bash echo "Hello world" >>/root/ansible.log [root@temp01 ~]# ansible all -m script -a 'ansible.sh' 192.168.3.70 | SUCCESS => { "changed": true, "rc": 0, "stderr": "", "stdout": "", "stdout_lines": [] } 192.168.3.60 | SUCCESS => { "changed": true, "rc": 0, "stderr": "", "stdout": "", "stdout_lines": [] } [root@example_02 ~]# pwd #跳转到客户端验证 /root [root@example_02 ~]# cat ansible.log Hello worldshell模块shell 模块与command模块区别 ，shell支持管道符和变更，下面这个例子说明问题[root@temp01 ~]# ansible all -m mommand -a 'cat /etc/passwd | grep root' ERROR! this task 'mommand' has extra params, which is only allowed in the following modules: command, win_command, shell, win_shell, script, include, include_vars, add_host, group_by, set_fact, raw, meta [root@temp01 ~]# ansible all -m shell -a 'cat /etc/passwd | grep root' 192.168.3.70 | SUCCESS | rc=0 >> root:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/root:/sbin/nologin 192.168.3.60 | SUCCESS | rc=0 >> root:x:0:0:root:/root:/bin/bash operator:x:11:0:operator:/root:/sbin/nologinfile模块 修改文件属性[root@temp01 ~]# ansible all -m file -a 'path=/root/ansible.log owner=root mode=777 group=root' #修改文件属性 192.168.3.60 | SUCCESS => { "changed": true, "gid": 0, "group": "root", "mode": "0777", "owner": "root", "path": "/root/ansible.log", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 12, "state": "file", "uid": 0 } 192.168.3.70 | SUCCESS => { "changed": true, "gid": 0, "group": "root", "mode": "0777", "owner": "root", "path": "/root/ansible.log", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 12, "state": "file", "uid": 0 }新建目录：类 mkdir -p[root@temp01 ~]# ansible all -m file -a 'dest=/root/1/2/3/4/ mode=755 owner=root group=root state=directory' 192.168.3.60 | SUCCESS => { "changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/1/2/3/4/", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 4096, "state": "directory", "uid": 0 } 192.168.3.70 | SUCCESS => { "changed": true, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/1/2/3/4/", "secontext": "unconfined_u:object_r:admin_home_t:s0", "size": 4096, "state": "directory", "uid": 0 } [root@example_02 ~]# ll total 48 -rw-------. 1 root root 1106 Jul 11 05:26 anaconda-ks.cfg -rwxrwxrwx. 1 root root 12 Dec 30 02:53 ansible.log -rw-------. 1 root root 1052 Dec 30 01:09 hosts -rw-r--r--. 1 root root 23089 Jul 11 05:26 install.log -rw-r--r--. 1 root root 6240 Jul 11 05:24 install.log.syslog [root@example_02 ~]# cd 1/2/3/4/ #转到客户端验证 You have new mail in /var/spool/mail/root [root@example_02 4]# pwd /root/1/2/3/4删除目录文件[root@temp01 ~]# ansible all -m file -a 'dest=/root/1/2/3/4/ state=absent' 192.168.3.60 | SUCCESS => { "changed": true, "path": "/root/1/2/3/4/", "state": "absent" } 192.168.3.70 | SUCCESS => { "changed": true, "path": "/root/1/2/3/4/", "state": "absent" } [root@example_02 ~]# cd 1/2/3/4 #跳转验证 -bash: cd: 1/2/3/4: No such file or directory [root@example_02 ~]#
环境准备操作系统：Centos 7 1810安装方式：YUM配置epel的YUM源安装cobbleryum install cobbler cobbler-web pykickstart httpd dhcp tftp-server启动httpd、tftp、cobblerd服务，并设置为开机启动systemctl start cobblerd systemctl start httpd systemctl start tftp #设置相关服务开机启动 systemctl enable cobblerd systemctl enable httpd systemctl enable tftp systemctl enable rsyncd配置cobbler执行cobbler check命令检查cobbler设置这里可能会报如下错误httpd does not appear to be running and proxying cobbler, or SELinux is in the way. Original traceback: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cobbler/cli.py", line 251, in check_setup s.ping() File "/usr/lib64/python2.7/xmlrpclib.py", line 1233, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.7/xmlrpclib.py", line 1591, in __request verbose=self.__verbose File "/usr/lib64/python2.7/xmlrpclib.py", line 1273, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python2.7/xmlrpclib.py", line 1321, in single_request response.msg, ProtocolError: <ProtocolError for 127.0.0.1:80/cobbler_api: 503 Service Unavailable>根据提示，可以确定是selinux引起的问题，可以将selinux关掉或者执行如下操作setsebool -P httpd_can_network_connect true这里建议关掉selinux，如果不想关的话，则参考如下链接的设置方法https://github.com/cobbler/cobbler/wiki/Selinux再次执行cobbler check命令检查，输出信息如下The following are potential configuration items that you may want to fix: 1 : The 'server' field in /etc/cobbler/settings must be set to something other than localhost, or kickstarting features will not work. This should be a resolvable hostname or IP for the boot server as reachable by all machines that will use it. 2 : For PXE to be functional, the 'next_server' field in /etc/cobbler/settings must be set to something other than 127.0.0.1, and should match the IP of the boot server on the PXE network. 3 : SELinux is enabled. Please review the following wiki page for details on ensuring cobbler works correctly in your SELinux environment: https://github.com/cobbler/cobbler/wiki/Selinux 4 : change 'disable' to 'no' in /etc/xinetd.d/tftp 5 : Some network boot-loaders are missing from /var/lib/cobbler/loaders, you may run 'cobbler get-loaders' to download them, or, if you only want to handle x86/x86_64 netbooting, you may ensure that you have installed a *recent* version of the syslinux package installed and can ignore this message entirely. Files in this directory, should you want to support all architectures, should include pxelinux.0, menu.c32, elilo.efi, and yaboot. The 'cobbler get-loaders' command is the easiest way to resolve these requirements. 6 : debmirror package is not installed, it will be required to manage debian deployments and repositories 7 : The default password used by the sample templates for newly installed machines (default_password_crypted in /etc/cobbler/settings) is still set to 'cobbler' and should be changed, try: "openssl passwd -1 -salt 'random-phrase-here' 'your-password-here'" to generate new one 8 : fencing tools were not found, and are required to use the (optional) power management features. install cman or fence-agents to use them Restart cobblerd and then run 'cobbler sync' to apply changes.下面我们便依据以上检查出的问题逐项进行修复及配置修改配置文件/etc/cobbler/settings （问题1、2）#更改监听IP。 sed -i 's/server: 127.0.0.1/server: 172.25.68.8/' /etc/cobbler/settings #next_server，用Cobbler管理DHCP需要修改此项。 sed -i 's/next_server: 127.0.0.1/next_server: 172.25.68.8/' /etc/cobbler/settings #用Cobbler管理DHCP。 sed -i 's/manage_dhcp: 0/manage_dhcp: 1/' /etc/cobbler/settings #防止循环装系统。 sed -i 's/pxe_just_once: 0/pxe_just_once: 1/' /etc/cobbler/settings关闭或设置selinux（问题3）#临时关闭selinux setenforce 0 #永久关闭selinux sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config修改tftp相关设置（问题4）编辑配置文件/etc/xinetd.d/tftp，如果里面是disable=yes的话，需要修改为disable=no同步cobbler设置（问题5）执行如下命令，从github上同步cobbler需要用到的一些引导文件及加载项cobbler get-loaders安装debian系统相关支持（问题6）yum -y install debmirror修改默认密码（问题7）这里设置的密码是使用cobbler默认安装的操作系统的密码，用如下命令生成，其中random-phrase-here为干扰码，可以自行设置，your-password-here为自己设置的密码，执行完成后，用生成的值将/etc/cobbler/settings中default_password_crypted的值替换掉即可。openssl passwd -1 -salt 'random-phrase-here' 'your-password-here'"安装fence相关设备支持（问题8）yum -y install fence-agents再次重启cobblerd服务并执行cobbler check进行配置项检查检查还有如下的报错The following are potential configuration items that you may want to fix: 1 : comment out 'dists' on /etc/debmirror.conf for proper debian support 2 : comment out 'arches' on /etc/debmirror.conf for proper debian support Restart cobblerd and then run 'cobbler sync' to apply changes.解决办法：将/etc/debmirror.conf配置文件中包含@dists和@arches的两行注释掉即可再次检查已经没有报错DHCP的配置由于上文中已经配置由cobbler管理DHCP，因此，DHCP的配置就应该使用cobbler的模板来生成，我们直接修改/etc/cobbler/dhcp.template文件即可，这里需要注意的一点就是DHCP的配置必须要有一段包含本子网的配置，否则DHCP会报错。新增的网段直接复制subnet相关的配置就行Cobbler命令行的使用Cobbler Web的使用上文中已经安装过cobbler-web了，因此，只要启动httpd服务，cobbler web服务便可以使用了。新版的cobbler web界面需要用HTTPS来访问，访问地址是https://ip/cobbler_web但是，我们在访问的时候，发现服务器报500错误，经过检查httpd的日志，发现报错如下：[Thu Jan 24 16:01:25.739471 2019] [:error] [pid 5466] [remote 172.25.120.32:96] mod_wsgi (pid=5466): Exception occurred processing WSGI script '/usr/share/cobbler/web/cobbler.wsgi'. [Thu Jan 24 16:01:25.739567 2019] [:error] [pid 5466] [remote 172.25.120.32:96] Traceback (most recent call last): [Thu Jan 24 16:01:25.739618 2019] [:error] [pid 5466] [remote 172.25.120.32:96] File "/usr/share/cobbler/web/cobbler.wsgi", line 26, in application [Thu Jan 24 16:01:25.739687 2019] [:error] [pid 5466] [remote 172.25.120.32:96] _application = get_wsgi_application() [Thu Jan 24 16:01:25.739719 2019] [:error] [pid 5466] [remote 172.25.120.32:96] File "/usr/lib/python2.7/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application [Thu Jan 24 16:01:25.739753 2019] [:error] [pid 5466] [remote 172.25.120.32:96] django.setup(set_prefix=False) [Thu Jan 24 16:01:25.739768 2019] [:error] [pid 5466] [remote 172.25.120.32:96] File "/usr/lib/python2.7/site-packages/django/__init__.py", line 22, in setup [Thu Jan 24 16:01:25.739791 2019] [:error] [pid 5466] [remote 172.25.120.32:96] configure_logging(settings.LOGGING_CONFIG, settings.LOGGING) [Thu Jan 24 16:01:25.739804 2019] [:error] [pid 5466] [remote 172.25.120.32:96] File "/usr/lib/python2.7/site-packages/django/conf/__init__.py", line 56, in __getattr__ [Thu Jan 24 16:01:25.739826 2019] [:error] [pid 5466] [remote 172.25.120.32:96] self._setup(name) [Thu Jan 24 16:01:25.739843 2019] [:error] [pid 5466] [remote 172.25.120.32:96] File "/usr/lib/python2.7/site-packages/django/conf/__init__.py", line 41, in _setup [Thu Jan 24 16:01:25.739907 2019] [:error] [pid 5466] [remote 172.25.120.32:96] self._wrapped = Settings(settings_module) [Thu Jan 24 16:01:25.739933 2019] [:error] [pid 5466] [remote 172.25.120.32:96] File "/usr/lib/python2.7/site-packages/django/conf/__init__.py", line 110, in __init__ [Thu Jan 24 16:01:25.739955 2019] [:error] [pid 5466] [remote 172.25.120.32:96] mod = importlib.import_module(self.SETTINGS_MODULE) [Thu Jan 24 16:01:25.739968 2019] [:error] [pid 5466] [remote 172.25.120.32:96] File "/usr/lib64/python2.7/importlib/__init__.py", line 37, in import_module [Thu Jan 24 16:01:25.739990 2019] [:error] [pid 5466] [remote 172.25.120.32:96] __import__(name) [Thu Jan 24 16:01:25.740032 2019] [:error] [pid 5466] [remote 172.25.120.32:96] File "/usr/share/cobbler/web/settings.py", line 89, in <module> [Thu Jan 24 16:01:25.740062 2019] [:error] [pid 5466] [remote 172.25.120.32:96] from django.conf.global_settings import TEMPLATE_CONTEXT_PROCESSORS [Thu Jan 24 16:01:25.740298 2019] [:error] [pid 5466] [remote 172.25.120.32:96] ImportError: cannot import name TEMPLATE_CONTEXT_PROCESSORS这个报错是由于centos 7默认安装的python-django版本过高，cobbler还存在一些bug未修复，因此，我们需要将djaogo框架降到低版本。首先，查询当前系统安装的django版本rpm -qa |grep django然后卸载rpm -e python2-django-1.11.18-1.el7.noarch --nodeps安装较低版本的djangoyum -y install python2-django16重启httpd服务systemctl restart httpd自定义的kickstart文件
新配置的nginx，做proxy转发时遇到错误，页面无法打开，access日志中报502错误，从nginx可以curl到后端页面，经查error日志，发现如下报错2015/12/28 16:42:00 [crit] 1850#0: *1 connect() to 192.168.1.1:8080 failed (13: Permission denied) while connecting to upstream, client: ***.***.***.***, server: *****.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://*******", host: "tl.cm-inf.com", referrer: "http://*******"可以看出，错误中提示 Permission denied ，但nginx是可以curl出结果的，初步判断为selinux的问题，解决方法如下：关闭selinux调整selinux配置，在终端执行以下命令：setsebool -P httpd_can_network_connect 1推荐使用第二种方法，selinux尽量不要一刀切关掉，在安全方面还是非常有用的。
zabbix启动的时候报错如下：28395:20170122:112700.799 using configuration file: /etc/zabbix/zabbix_agentd.conf28395:20170122:112700.799 cannot set resource limit: [13] Permission denied28395:20170122:112700.799 cannot disable core dump, exiting...基本能确定是selinux的问题，Centos6处理方法关闭selinux（不推荐）setenforce 0更新selinux策略即可yum update selinux-policy.noarch selinux-policy-targeted.noarch更新完成之后重新启动zabbix即可Centos 7下处理方法今天发现centos 7 1611使用此方法并不管用，现新增方法如下：安装selinux相关工具yum install policycoreutils-python如果是server_agent端，则按照以下操作进行cat /var/log/audit/audit.log | grep zabbix_agentd | grep denied | audit2allow -M zabbix_agent_setrlimit执行上述命令后，会在当前目录生成一个名为zabbix_agent_setrlimit.pp的文件，接下来执行以下命令semodule -i zabbix_agent_setrlimit.pp如果是server端，则按照如下方法执行即可cat /var/log/audit/audit.log | grep zabbix_server | grep denied | audit2allow -M zabbix_server_setrlimit semodule -i zabbix_server_setrlimit.pp最后重启zabbix即可