部署ceph系统为k8s提供存储平台

发表于 | 分类于 | 评论数： | 阅读次数：

背景

PersistentVolume（PV）是集群中已由管理员配置的一段网络存储。 集群中的资源就像一个节点是一个集群资源。 PV是诸如卷之类的卷插件，但是具有独立于使用PV的任何单个pod的生命周期。 该API对象包含存储的实现细节，即NFS，iSCSI或云提供商特定的存储系统。

PersistentVolumeClaim（PVC）是用户存储的请求。 它类似于pod。Pod消耗节点资源，PVC消耗存储资源。 pod可以请求特定级别的资源（CPU和内存）。 权限要求可以请求特定的大小和访问模式。

我们使用3个Centos 7系统 ,关闭防火墙，搭建一套ceph系统为k8s提供存储平台。
master: 192.168.80.198
node01: 192.168.80.199
node02: 192.168.80.200

Ceph安装

新建磁盘

给虚拟机新增一块硬盘sdb，需要硬盘目录不存在数据

主机ip映射

  • 设置主机名

    1
    2
    3
    4
    5
    6
    # 在master上执行
    hostnamectl set-hostname master
    # 下面在 node01 执行
    hostnamectl set-hostname node01
    # 下面在 node02 执行
    hostnamectl set-hostname node02

  • 增加映射

    1
    2
    3
    4
    5
    6
    7
    8
    9
    # master 中执行命令
    cat >> /etc/hosts << EOF
    192.168.80.198    master
    192.168.80.199    node01
    192.168.80.200    node02
    EOF
    # 将hosts文件拷贝到node01 和 node02中,注意在这两个命令中按照提示输入密码
    scp /etc/hosts root@node01:/etc
    scp /etc/hosts root@node02:/etc

配置yum源

1
2
3
4
5
6
7
8
9
10
11
vim /etc/yum.repos.d/ceph.repo

[Ceph]
name=Ceph packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-mimic/el7/x86_64/
gpgcheck=0

[Ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirrors.aliyun.com/ceph/rpm-mimic/el7/noarch/
gpgcheck=0

创建普通用户并设置sudo免密

1
2
3
4
5
groupadd -g 3000 ceph
useradd -u 3000 -g ceph ceph
echo "ceph" | passwd --stdin ceph
echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
chmod 0440 /etc/sudoers.d/ceph

新建的用户创建ssh免密登录

在master节点执行：

1
2
3
4
5
su - ceph
ssh-keygen
ssh-copy-id ceph@master
ssh-copy-id ceph@node01
ssh-copy-id ceph@node02

安装软件

1
2
3
4
5
6
7
8
9
sudo su - root    # master要从ceph用户切换到root用户下
yum install ceph-deploy -y
yum -y install epel-release
yum install python-pip -y
yum install ceph ceph-osd ceph-mds ceph-mon ceph-radosgw -y

yum install ntp -y
systemctl start ntpd
systemctl enable ntpd

Tips：安装时间同步服务的目的是为了防止后续集群因为时间不同步导致健康状态从OK转变为WARN。

创建集群

在master节点执行：

1
2
3
4
5
6
7
8
9
10
11
su - ceph
mkdir cephcluster
cd cephcluster/
# 初始化创建ceph集群
ceph-deploy new --cluster-network 192.168.80.0/24 --public-network 192.168.80.0/24 master node01 node02
# 初始化monitor服务
ceph-deploy mon create-initial
# 配置信息拷贝到三台节点
ceph-deploy admin master node01 node02
sudo chown -R ceph:ceph /etc/ceph
chown -R ceph:ceph /etc/ceph    # 在其它节点执行

查看状态：

1
ceph -s

配置mgr服务

master上执行：

1
ceph-deploy mgr create master node01 node02

配置osd服务

master上执行：

1
2
3
ceph-deploy osd create --data /dev/sdb master
ceph-deploy osd create --data /dev/sdb node01
ceph-deploy osd create --data /dev/sdb node02

解决异常

ceph -s查看状态：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
cluster:
  id:     14450b7d-84ce-40c4-8a1e-46af50457fc6
  health: HEALTH_WARN
          clock skew detected on mon.node01

services:
  mon: 3 daemons, quorum master,node01,node02 (age 15m)
  mgr: master(active, since 41s), standbys: node02, node01
  osd: 3 osds: 3 up (since 10m), 3 in (since 10m)

data:
  pools:   0 pools, 0 pgs
  objects: 0 objects, 0 B
  usage:   3.0 GiB used, 597 GiB / 600 GiB avail
  pgs:

解决异常:

1
2
3
4
5
su - ceph
echo "mon clock drift allowed = 2" >> ~/cephcluster/ceph.conf
echo "mon clock drift warn backoff = 30" >> ~/cephcluster/ceph.conf
ceph-deploy --overwrite-conf config push master node01 node02
sudo systemctl restart ceph-mon.target

配置dashboard

master上执行：

1
2
3
4
5
6
yum -y install ceph-mgr-dashboard    # 三个节点都要执行安装操作
echo "mgr initial modules = dashboard" >> ~/cephcluster/ceph.conf
ceph-deploy --overwrite-conf config push master node01 node02
sudo systemctl restart ceph-mgr@master
ceph mgr module enable dashboard
ceph dashboard create-self-signed-cert

配置账户名密码：

1
ceph dashboard set-login-credentials admin ceph123

K8S部署ceph

POD使用CephFS做为持久数据卷

Ceph端创建CephFS pool

如下操作在ceph的mon或者admin节点 CephFS需要使用两个Pool来分别存储数据和元数据

1
2
3
4
5
6
7
8
9
1、如下操作在ceph的mon或者admin节点CephFS需要使用两个Pool来分别存储数据和元数据
ceph osd pool create fs_data 128
ceph osd pool create fs_metadata 128
ceph osd lspools
2、创建一个CephFS
ceph fs new cephfs fs_metadata fs_data
3、查看
# ceph fs ls
name: cephfs, metadata pool: fs_metadata, data pools: [fs_data ]

部署cephfs-provisioner

使用社区提供的cephfs-provisioner

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
# cat >external-storage-cephfs-provisioner.yaml<<EOF
apiVersion: v1
kind: Namespace
metadata:
   name: cephfs
   labels:
     name: cephfs
 ---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cephfs-provisioner
  namespace: kube-system
 ---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cephfs-provisioner
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["create", "get", "delete"]
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
 ---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
  namespace: kube-system
rules:
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
  - apiGroups: [""]
    resources: ["services"]
    resourceNames: ["kube-dns","coredns"]
    verbs: ["list", "get"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "create", "delete"]
  - apiGroups: ["policy"]
    resourceNames: ["cephfs-provisioner"]
    resources: ["podsecuritypolicies"]
    verbs: ["use"]
 ---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cephfs-provisioner
  namespace: kube-system
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cephfs-provisioner
subjects:
- kind: ServiceAccount
  name: cephfs-provisioner
 ---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cephfs-provisioner
subjects:
  - kind: ServiceAccount
    name: cephfs-provisioner
    namespace: kube-system
roleRef:
  kind: ClusterRole
  name: cephfs-provisioner
  apiGroup: rbac.authorization.k8s.io
 ---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cephfs-provisioner
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: cephfs-provisioner
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: cephfs-provisioner
    spec:
      containers:
      - name: cephfs-provisioner
        image: "quay.io/external_storage/cephfs-provisioner:latest"
        env:
        - name: PROVISIONER_NAME
          value: ceph.com/cephfs
        command:
        - "/usr/local/bin/cephfs-provisioner"
        args:
        - "-id=cephfs-provisioner-1"
        - "-disable-ceph-namespace-isolation=true"
      serviceAccount: cephfs-provisionerEOF
# kubectl apply -f external-storage-cephfs-provisioner.yaml
# kubectl get pod -n kube-system |grep cephfs
cephfs-provisioner-847468fc-5k8vx   1/1     Running   0          7m39s

配置secret

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
查看key 在ceph的mon或者admin节点
ceph auth get-key client.admin

创建 admin secret
# kubectl create secret generic ceph-secret --type="kubernetes.io/rbd" \--from-literal=key=AQCBrJ9eV/U5NBAAoDlM4gV3a+KNQDBOUqVxdw== \--namespace=kube-system

查看
# kubectl get secret ceph-secret -n kube-system -o yaml
apiVersion: v1
data:
  key: QVFDQnJKOWVWL1U1TkJBQW9EbE00Z1YzYStLTlFEQk9VcVZ4ZHc9PQ==
kind: Secret
metadata:
  creationTimestamp: "2020-06-08T08:17:09Z"
  name: ceph-secret
  namespace: kube-system
  resourceVersion: "42732"
  selfLink: /api/v1/namespaces/kube-system/secrets/ceph-secret
  uid: efec109a-17de-4f72-afd4-d126f4d4f8d6
type: kubernetes.io/rbd

配置 StorageClass

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# vim storageclass-cephfs.yaml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: dynamic-cephfs
provisioner: ceph.com/cephfs
parameters:
    monitors: 192.168.0.246:6789,192.168.0.247:6789,192.168.0.248:6789
    adminId: admin
    adminSecretName: ceph-secret
    adminSecretNamespace: "kube-system"
    claimRoot: /volumes/kubernetes
[root@k8s-master yaml]# kubectl apply -f storageclass-cephfs.yaml
storageclass.storage.k8s.io/dynamic-cephfs created[root@k8s-master yaml]# kubectl get scNAME               PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
dynamic-ceph-rdb   ceph.com/rbd      Delete          Immediate           false                  59m
dynamic-cephfs     ceph.com/cephfs   Delete          Immediate           false                  5s

创建pvc测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# cat >cephfs-pvc-test.yaml<<EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: cephfs-claim
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: dynamic-cephfs
  resources:
    requests:
      storage: 2GiEOF
# kubectl apply -f cephfs-pvc-test.yaml
persistentvolumeclaim/cephfs-claim created[root@k8s-master yaml]# kubectl get pv,pvcNAME                                                        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                  STORAGECLASS     REASON   AGE
persistentvolume/pvc-df8768d7-5111-4e14-a0cf-dd029b00469b   2Gi        RWO            Delete           Bound    default/cephfs-claim   dynamic-cephfs            12s
NAME                                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS     AGE
persistentvolumeclaim/cephfs-claim   Bound    pvc-df8768d7-5111-4e14-a0cf-dd029b00469b   2Gi        RWO            dynamic-cephfs   13s

创建 nginx pod 挂载测试

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# cat >nginx-pod.yaml<<EOF
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pod2
  labels:
    name: nginx-pod2
spec:
  containers:
  - name: nginx-pod2
    image: nginx
    ports:
    - name: web
      containerPort: 80
    volumeMounts:
    - name: cephfs
      mountPath: /usr/share/nginx/html
  volumes:
  - name: cephfs
    persistentVolumeClaim:
      claimName: cephfs-claimEOF
# kubectl apply -f nginx-pod.yaml[root@k8s-master yaml]# kubectl get pod -o wideNAME         READY   STATUS    RESTARTS   AGE   IP           NODE          NOMINATED NODE   READINESS GATES
nginx-pod2   1/1     Running   0          37s   10.244.2.5   k8s-master1   <none>           <none>
[root@k8s-master yaml]# kubectl exec -ti nginx-pod2 -- /bin/sh -c 'echo This is from CephFS!!! > /usr/share/nginx/html/index.html'
[root@k8s-master yaml]# curl 10.244.2.5
This is from CephFS!!!
###查看挂载信息[root@k8s-master yaml]# kubectl exec -ti nginx-pod2 -- /bin/sh -c 'df -h'