云存储服务
https://help.aliyun.com/zh/oss/support/overview-26?spm=a2c4g.11186623.help-menu-31815.d_6_2_0_0.3dd658fd2jFacL
对象存储OSS
https://help.aliyun.com/zh/oss/support/oss?spm=a2c4g.11186623.help-menu-31815.d_6_2_0_1.1ec12628eMPjv1
oss服务java sdk使用
详细用法请参考https://gitee.com/dexterleslie/demonstration/tree/master/demo-aliyun/oss/demo-java-sdk-aliyun-oss
sdk使用步骤：
- 登陆主账号配置
RAM子账号access key maven配置https://help.aliyun.com/document_detail/32009.html?spm=a2c4g.11186623.6.765.37aac06dXA9iJX- 入门
demo https://help.aliyun.com/document_detail/32011.html?spm=a2c4g.11186623.6.766.52ad6328whQ2Qj - 下载到本地文件
https://help.aliyun.com/document_detail/84824.html?spm=a2c4g.11186623.6.797.1e892ba81kJxfd
块存储
https://help.aliyun.com/zh/oss/support/block-storage?spm=a2c4g.11186623.0.0.3de54f1cmOaAmt#concept-2547939
文件存储NAS
https://help.aliyun.com/zh/oss/support/apsara-file-storage-nas?spm=a2c4g.11186623.0.0.3de54f1cmOaAmt#concept-2547938
创建NAS
https://help.aliyun.com/zh/nas/getting-started/quick-start?spm=a2c4g.11186623.help-menu-27516.d_1_1.38c770100RFybC&scm=20140722.H_312360._.OR_help-V_1
挂载NAS存储
通过控制台挂载NAS
ECS实例一键挂载NAS NFS协议文件系统https://help.aliyun.com/zh/nas/user-guide/mount-a-file-system-on-an-ecs-instance-by-using-the-nas-console
Linux系统手动挂载NFS协议文件系统
https://help.aliyun.com/zh/nas/user-guide/mount-an-nfs-file-system-on-a-linux-ecs-instance?spm=a2c4g.11186623.help-menu-27516.d_2_0_2_1_0.2475da29ovhtXL
容器挂载NAS
自建k8s通过内置的或者第三方nfs-subdir-external-provisioner挂载nfs
提示：经过实验证明阿里云
NAS支持此模式。
自建k8s通过CSI插件挂载nfs
https://help.aliyun.com/zh/nas/user-guide/mount-nas-by-manually-installing-the-csi-plug-in?spm=a2c4g.11186623.help-menu-27516.d_2_0_3_2_3.5eec2d5eaW6cjM&scm=20140722.H_2845656._.OR_help-V_1https://blog.csdn.net/ethnicitybeta/article/details/139061484https://blog.51cto.com/u_15121441/9904057https://segmentfault.com/a/1190000040363774?sort=newesthttps://blog.51cto.com/u_16236244/9893534https://blog.51cto.com/u_14269169/10064379https://zhuanlan.zhihu.com/p/389237922
deployment.yaml内容如下：
yaml
# rbac
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: csi-admin
namespace: kube-system
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: alicloud-csi-plugin
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "create", "list"]
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "update", "create", "delete", "patch"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: [""]
resources: ["persistentvolumeclaims/status"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["csinodes"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "watch", "list", "delete", "update", "create"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["csi.storage.k8s.io"]
resources: ["csinodeinfos"]
verbs: ["get", "list", "watch"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments"]
verbs: ["get", "list", "watch", "update", "patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotclasses"]
verbs: ["get", "list", "watch", "create"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents"]
verbs: ["create", "get", "list", "watch", "update", "delete"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots"]
verbs: ["get", "list", "watch", "update", "create"]
- apiGroups: ["apiextensions.k8s.io"]
resources: ["customresourcedefinitions"]
verbs: ["create", "list", "watch", "delete", "get", "update", "patch"]
- apiGroups: ["coordination.k8s.io"]
resources: ["leases"]
verbs: ["get", "create", "list", "watch", "delete", "update"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshotcontents/status"]
verbs: ["update"]
- apiGroups: ["storage.k8s.io"]
resources: ["volumeattachments/status"]
verbs: ["patch"]
- apiGroups: ["snapshot.storage.k8s.io"]
resources: ["volumesnapshots/status"]
verbs: ["update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["namespaces"]
verbs: ["get", "list"]
- apiGroups: [""]
resources: ["pods","pods/exec"]
verbs: ["create", "delete", "get", "post", "list", "watch", "patch", "udpate"]
- apiGroups: ["storage.alibabacloud.com"]
resources: ["rules"]
verbs: ["get"]
- apiGroups: ["storage.alibabacloud.com"]
resources: ["containernetworkfilesystems"]
verbs: ["get","list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: alicloud-csi-plugin
subjects:
- kind: ServiceAccount
name: csi-admin
namespace: kube-system
roleRef:
kind: ClusterRole
name: alicloud-csi-plugin
apiGroup: rbac.authorization.k8s.io
# driverinfo
---
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: nasplugin.csi.alibabacloud.com
spec:
attachRequired: false
podInfoOnMount: true
# node
---
kind: DaemonSet
apiVersion: apps/v1
metadata:
name: csi-plugin
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-plugin
template:
metadata:
labels:
app: csi-plugin
spec:
tolerations:
- operator: Exists
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
nodeSelector:
kubernetes.io/os: linux
serviceAccount: csi-admin
priorityClassName: system-node-critical
hostNetwork: true
hostPID: true
dnsPolicy: ClusterFirst
containers:
- name: nas-driver-registrar
image: registry.cn-beijing.aliyuncs.com/acs/csi-node-driver-registrar:v2.3.1-038aeb6-aliyun
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
cpu: 500m
memory: 1024Mi
args:
- "--v=5"
- "--csi-address=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock"
- "--kubelet-registration-path=/var/lib/kubelet/csi-plugins/nasplugin.csi.alibabacloud.com/csi.sock"
volumeMounts:
- name: kubelet-dir
mountPath: /var/lib/kubelet/
- name: registration-dir
mountPath: /registration
- name: csi-plugin
securityContext:
privileged: true
allowPrivilegeEscalation: true
image: registry.cn-beijing.aliyuncs.com/acs/csi-plugin:v1.24.5-39a3970-aliyun
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=2"
- "--driver=oss,nas,disk"
env:
- name: KUBE_NODE_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: spec.nodeName
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/csi-plugins/driverplugin.csi.alibabacloud.com-replace/csi.sock
- name: MAX_VOLUMES_PERNODE
value: "15"
- name: SERVICE_TYPE
value: "plugin"
- name: ACCESS_KEY_ID
value: "xxx"
- name: ACCESS_KEY_SECRET
value: "xxx"
- name: KUBELET_ROOT_DIR
value: "/var/lib/kubelet"
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 1024Mi
livenessProbe:
httpGet:
path: /healthz
port: healthz
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: /healthz
port: healthz
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 5
ports:
- name: healthz
containerPort: 11260
volumeMounts:
- name: kubelet-dir
mountPath: /var/lib/kubelet/
mountPropagation: "Bidirectional"
- name: etc
mountPath: /host/etc
- name: host-log
mountPath: /var/log/
- name: ossconnectordir
mountPath: /host/usr/
- name: container-dir
mountPath: /var/lib/container
mountPropagation: "Bidirectional"
- name: host-dev
mountPath: /dev
mountPropagation: "HostToContainer"
- mountPath: /host/var/run/
name: fuse-metrics-dir
volumes:
- name: fuse-metrics-dir
hostPath:
path: /var/run/
type: DirectoryOrCreate
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry
type: DirectoryOrCreate
- name: container-dir
hostPath:
path: /var/lib/container
type: DirectoryOrCreate
- name: kubelet-dir
hostPath:
path: /var/lib/kubelet
type: Directory
- name: host-dev
hostPath:
path: /dev
- name: host-log
hostPath:
path: /var/log/
- name: etc
hostPath:
path: /etc
- name: ossconnectordir
hostPath:
path: /usr/
updateStrategy:
rollingUpdate:
maxUnavailable: 30%
type: RollingUpdate
# controller
---
kind: Deployment
apiVersion: apps/v1
metadata:
name: csi-provisioner
namespace: kube-system
spec:
selector:
matchLabels:
app: csi-provisioner
strategy:
rollingUpdate:
maxSurge: 0
maxUnavailable: 1
type: RollingUpdate
replicas: 2
template:
metadata:
labels:
app: csi-provisioner
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 1
preference:
matchExpressions:
- key: node-role.kubernetes.io/master
operator: Exists
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: NotIn
values:
- virtual-kubelet
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchExpressions:
- key: app
operator: In
values:
- csi-provisioner
topologyKey: kubernetes.io/hostname
tolerations:
- effect: NoSchedule
operator: Exists
key: node-role.kubernetes.io/master
- effect: NoSchedule
operator: Exists
key: node.cloudprovider.kubernetes.io/uninitialized
serviceAccount: csi-admin
hostPID: true
priorityClassName: system-node-critical
containers:
- name: external-nas-provisioner
image: registry.cn-beijing.aliyuncs.com/acs/csi-provisioner:v3.0.0-080f01e64-aliyun
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
cpu: 500m
memory: 1024Mi
args:
- "--csi-address=$(ADDRESS)"
- "--volume-name-prefix=nas"
- "--timeout=150s"
- "--leader-election=true"
- "--retry-interval-start=500ms"
- "--default-fstype=nfs"
- "--v=5"
env:
- name: ADDRESS
value: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com/csi.sock
volumeMounts:
- name: nas-provisioner-dir
mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
- name: external-nas-resizer
image: registry.cn-beijing.aliyuncs.com/acs/csi-resizer:v1.3-ca84e84-aliyun
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
cpu: 500m
memory: 1024Mi
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election"
env:
- name: ADDRESS
value: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com/csi.sock
volumeMounts:
- name: nas-provisioner-dir
mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
- name: external-csi-snapshotter
image: registry.cn-beijing.aliyuncs.com/acs/csi-snapshotter:v4.0.0-a230d5b3-aliyun
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
cpu: 500m
memory: 1024Mi
args:
- "--v=5"
- "--csi-address=$(ADDRESS)"
- "--leader-election=true"
- "--extra-create-metadata=true"
env:
- name: ADDRESS
value: /csi/csi.sock
volumeMounts:
- name: disk-provisioner-dir
mountPath: /csi
- name: external-snapshot-controller
image: registry.cn-beijing.aliyuncs.com/acs/snapshot-controller:v4.0.0-a230d5b3-aliyun
resources:
requests:
cpu: 10m
memory: 16Mi
limits:
cpu: 500m
memory: 1024Mi
args:
- "--v=5"
- "--leader-election=true"
- name: csi-provisioner
securityContext:
privileged: true
image: registry.cn-beijing.aliyuncs.com/acs/csi-plugin:v1.24.5-39a3970-aliyun
args:
- "--endpoint=$(CSI_ENDPOINT)"
- "--v=2"
- "--driver=nas,disk,oss"
env:
- name: CSI_ENDPOINT
value: unix://var/lib/kubelet/csi-provisioner/driverplugin.csi.alibabacloud.com-replace/csi.sock
- name: MAX_VOLUMES_PERNODE
value: "15"
- name: SERVICE_TYPE
value: "provisioner"
- name: "CLUSTER_ID"
value: "abc"
- name: "ACCESS_KEY_ID"
value: "xxx"
- name: "ACCESS_KEY_SECRET"
value: "xxx"
livenessProbe:
httpGet:
path: /healthz
port: healthz
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 5
failureThreshold: 5
readinessProbe:
httpGet:
path: /healthz
port: healthz
initialDelaySeconds: 5
periodSeconds: 20
ports:
- name: healthz
containerPort: 11270
volumeMounts:
- name: host-log
mountPath: /var/log/
- name: disk-provisioner-dir
mountPath: /var/lib/kubelet/csi-provisioner/diskplugin.csi.alibabacloud.com
- name: nas-provisioner-dir
mountPath: /var/lib/kubelet/csi-provisioner/nasplugin.csi.alibabacloud.com
- name: oss-provisioner-dir
mountPath: /var/lib/kubelet/csi-provisioner/ossplugin.csi.alibabacloud.com
- mountPath: /var/addon
name: addon-token
readOnly: true
- mountPath: /mnt
mountPropagation: Bidirectional
name: host-dev
- mountPath: /host/etc
name: etc
resources:
limits:
cpu: 500m
memory: 1024Mi
requests:
cpu: 100m
memory: 128Mi
volumes:
- name: disk-provisioner-dir
emptyDir: {}
- name: nas-provisioner-dir
emptyDir: {}
- name: oss-provisioner-dir
emptyDir: {}
- name: host-log
hostPath:
path: /var/log/
- name: etc
hostPath:
path: /etc
type: ""
- name: host-dev
hostPath:
path: /mnt
type: ""
- name: addon-token
secret:
defaultMode: 420
optional: true
items:
- key: addon.token.config
path: token-config
secretName: addon.csi.token
# storageclass
---
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: nfs-client
mountOptions:
- nolock,tcp,noresvport
- vers=3
parameters:
volumeAs: subpath
server: "0f9xxx.cn-hangzhou.nas.aliyuncs.com:/data/"
provisioner: nasplugin.csi.alibabacloud.com
reclaimPolicy: Delete- 上面配置文件需要提供主帐号的
access key配置ACCESS_KEY_ID和ACCESS_KEY_SECRET，否则在创建pvc时报告关于access key错误。https://ram.console.aliyun.com/profile/access-keys?spm=5176.12818093_-1363046575.console-base_top-nav.dak.2e4616d08ix1CQ
部署CSI存储服务
bash
kubectl apply -f deployment.yaml查看部署情况
bash
kubectl -n kube-system get pods预期返回如下：
bash
NAME READY STATUS RESTARTS AGE
csi-plugin-5h6gq 4/4 Running 0 30m
csi-plugin-gjjp7 4/4 Running 0 31m
csi-provisioner-cfc45d84c-lzpvm 9/9 Running 0 31m
csi-provisioner-cfc45d84c-zczfn 9/9 Running 0 31m
......