kops
Kubernetes Operations (kops) - Production Grade K8s Installation, Upgrades, and Management
Installing kops (Binaries)
{
curl -Lo kops https://github.com/kubernetes/kops/releases/download/$(curl -s https://api.github.com/repos/kubernetes/kops/releases/latest | grep tag_name | cut -d '"' -f 4)/kops-linux-amd64
chmod +x ./kops
sudo mv ./kops /usr/local/bin/
}
Installing Other Dependencies
kubectl
sudo snap install kubectl --classic
AWS CLI
{
sudo apt update
sudo apt install python3-pip -y
pip3 install awscli --upgrade --user
}
Setup your environment
In a new terminal
aws configure
AWS Access Key ID [None]: AKIAQWNMG*********
AWS Secret Access Key [None]: GlTuDK6edXSg5MB4h8*****************
Default region name [None]: us-east-1
Default output format [None]: json
Setup IAM user
In order to build clusters within AWS we'll create a dedicated IAM user for kops.
{
aws iam create-group --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonRoute53FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/IAMFullAccess --group-name kops
aws iam attach-group-policy --policy-arn arn:aws:iam::aws:policy/AmazonVPCFullAccess --group-name kops
aws iam create-user --user-name kops
aws iam add-user-to-group --user-name kops --group-name kops
aws iam create-access-key --user-name kops
}
{
"Group": {
"Path": "/",
"GroupName": "kops",
"GroupId": "AGPAQWNMGOZOFGE63XYJX",
"Arn": "arn:aws:iam::048142644828:group/kops",
"CreateDate": "2019-06-24T07:28:22Z"
}
}
{
"User": {
"Path": "/",
"UserName": "kops",
"UserId": "AIDAQWNMGOZOFLQVLNW5T",
"Arn": "arn:aws:iam::048142644828:user/kops",
"CreateDate": "2019-06-24T07:28:26Z"
}
}
{
"AccessKey": {
"UserName": "kops",
"AccessKeyId": "AKIAQWNMGOZOBEDR64GV",
"Status": "Active",
"SecretAccessKey": "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",
"CreateDate": "2019-06-24T07:28:27Z"
}
}
You should record the SecretAccessKey and AccessKeyID in the returned JSON output, and then use them below:
aws configure
AWS Access Key ID [****************WK43]: AKIAQWNMGOZOBEDR64GV
AWS Secret Access Key [****************d36O]: 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
Default region name [us-east-1]:
Default output format [json]:
export AWS_ACCESS_KEY_ID=$(aws configure get aws_access_key_id)
export AWS_SECRET_ACCESS_KEY=$(aws configure get aws_secret_access_key)
Configure DNS
If you are using Kops 1.6.2 or later, then DNS configuration is optional. Instead, a gossip-based cluster can be easily created. The only requirement to trigger this is to have the cluster name end with .k8s.local. If a gossip-based cluster is created then you can skip this section.
Cluster State storage
In order to store the state of your cluster, and the representation of your cluster, we need to create a dedicated S3 bucket for kops to use. This bucket will become the source of truth for our cluster configuration.
We recommend keeping the creation of this bucket confined to us-east-1, otherwise more work will be required.
aws s3api create-bucket \
--bucket admatic-in-state-store \
--region us-east-1
{
"Location": "/admatic-in-state-store"
}
We STRONGLY recommend versioning your S3 bucket in case you ever need to revert or recover a previous state store.
aws s3api put-bucket-versioning \
--bucket admatic-in-state-store \
--versioning-configuration Status=Enabled
Creating your first cluster
Prepare local environment
For a gossip-based cluster, make sure the name ends with k8s.local.
export NAME=admatic-cluster.k8s.local
export KOPS_STATE_STORE=s3://admatic-in-state-store
You don’t have to use environmental variables here. You can always define the values using the
–nameand–stateflags later.
Create cluster configuration
kops create cluster \
--zones us-east-1a \
--node-count=4 \
${NAME}
I0624 07:35:31.170863 9032 create_cluster.go:519] Inferred --cloud=aws from zone "us-east-1a"
I0624 07:35:31.230414 9032 subnets.go:184] Assigned CIDR 172.20.32.0/19 to subnet us-east-1a
Previewing changes that will be made:
SSH public key must be specified when running with AWS (create with `kops create secret --name admatic-cluster.k8s.local sshpublickey admin -i ~/.ssh/id_rsa.pub`)
ssh-keygen -t rsa -C "adithya.j@admatic.in" -b 4096
Generating public/private rsa key pair.
Enter file in which to save the key (/home/ubuntu/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/ubuntu/.ssh/id_rsa.
Your public key has been saved in /home/ubuntu/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:OKTh5yOqbStcdNto5qnD/+nLjmEVF77MtCUtRiR4AqA adithya.j@admatic.in
The key's randomart image is:
+---[RSA 4096]----+
| .... ...+ |
|. o .+ o |
|E . +. B o |
| o = .B * |
| . + B.S= |
| . B.o |
|. o =o+ |
|.o.+.++.. |
|.++o+o+B. |
+----[SHA256]-----+
kops create secret --name admatic-cluster.k8s.local sshpublickey admin -i ~/.ssh/id_rsa.pub
kops create cluster \
--zones us-east-1a \
--node-count=4 \
${NAME}
cluster "admatic-cluster.k8s.local" already exists; use 'kops update cluster' to apply changes
kops update cluster ${NAME}
I0624 07:37:20.030998 9061 apply_cluster.go:559] Gossip DNS: skipping DNS validation
I0624 07:37:20.078866 9061 executor.go:103] Tasks: 0 done / 90 total; 42 can run
I0624 07:37:20.191021 9061 executor.go:103] Tasks: 42 done / 90 total; 24 can run
I0624 07:37:20.305033 9061 executor.go:103] Tasks: 66 done / 90 total; 20 can run
I0624 07:37:20.416020 9061 executor.go:103] Tasks: 86 done / 90 total; 3 can run
...
...
...
VPC/admatic-cluster.k8s.local
CIDR 172.20.0.0/16
EnableDNSHostnames true
EnableDNSSupport true
Shared false
Tags {Name: admatic-cluster.k8s.local, KubernetesCluster: admatic-cluster.k8s.local, kubernetes.io/cluster/admatic-cluster.k8s.local: owned}
VPCDHCPOptionsAssociation/admatic-cluster.k8s.local
VPC name:admatic-cluster.k8s.local
DHCPOptions name:admatic-cluster.k8s.local
Must specify --yes to apply changes
Customize Cluster Configuration
kops edit cluster ${NAME}
This opens your editor (as defined by $EDITOR) and allows you to edit the configuration. The configuration is loaded from the S3 bucket we created earlier, and automatically updated when we save and exit the editor.
Build the Cluster
kops update cluster ${NAME} --yes
I0624 07:38:37.772532 9068 apply_cluster.go:559] Gossip DNS: skipping DNS validation
I0624 07:38:37.995338 9068 executor.go:103] Tasks: 0 done / 90 total; 42 can run
I0624 07:38:38.850713 9068 vfs_castore.go:729] Issuing new certificate: "etcd-manager-ca-main"
I0624 07:38:38.976200 9068 vfs_castore.go:729] Issuing new certificate: "ca"
I0624 07:38:39.001199 9068 vfs_castore.go:729] Issuing new certificate: "etcd-peers-ca-main"
I0624 07:38:39.044428 9068 vfs_castore.go:729] Issuing new certificate: "etcd-peers-ca-events"
I0624 07:38:39.104139 9068 vfs_castore.go:729] Issuing new certificate: "etcd-clients-ca"
I0624 07:38:39.241927 9068 vfs_castore.go:729] Issuing new certificate: "apiserver-aggregator-ca"
I0624 07:38:39.262429 9068 vfs_castore.go:729] Issuing new certificate: "etcd-manager-ca-events"
I0624 07:38:39.467819 9068 executor.go:103] Tasks: 42 done / 90 total; 24 can run
I0624 07:38:40.109737 9068 vfs_castore.go:729] Issuing new certificate: "kops"
I0624 07:38:40.180521 9068 vfs_castore.go:729] Issuing new certificate: "apiserver-aggregator"
I0624 07:38:40.317427 9068 vfs_castore.go:729] Issuing new certificate: "kubelet-api"
I0624 07:38:40.524948 9068 vfs_castore.go:729] Issuing new certificate: "kube-scheduler"
I0624 07:38:40.581259 9068 vfs_castore.go:729] Issuing new certificate: "kubecfg"
I0624 07:38:40.592873 9068 vfs_castore.go:729] Issuing new certificate: "kubelet"
I0624 07:38:40.597943 9068 vfs_castore.go:729] Issuing new certificate: "apiserver-proxy-client"
I0624 07:38:40.683098 9068 vfs_castore.go:729] Issuing new certificate: "kube-proxy"
I0624 07:38:41.089027 9068 vfs_castore.go:729] Issuing new certificate: "kube-controller-manager"
I0624 07:38:41.307902 9068 executor.go:103] Tasks: 66 done / 90 total; 20 can run
I0624 07:38:41.510432 9068 launchconfiguration.go:364] waiting for IAM instance profile "masters.admatic-cluster.k8s.local" to be ready
I0624 07:38:41.521051 9068 launchconfiguration.go:364] waiting for IAM instance profile "nodes.admatic-cluster.k8s.local" to be ready
I0624 07:38:51.907313 9068 executor.go:103] Tasks: 86 done / 90 total; 3 can run
I0624 07:38:52.359216 9068 vfs_castore.go:729] Issuing new certificate: "master"
I0624 07:38:52.539059 9068 executor.go:103] Tasks: 89 done / 90 total; 1 can run
I0624 07:38:52.984303 9068 executor.go:103] Tasks: 90 done / 90 total; 0 can run
I0624 07:38:53.076692 9068 update_cluster.go:291] Exporting kubecfg for cluster
kops has set your kubectl context to admatic-cluster.k8s.local
Cluster is starting. It should be ready in a few minutes.
Suggestions:
* validate cluster: kops validate cluster
* list nodes: kubectl get nodes --show-labels
* ssh to the master: ssh -i ~/.ssh/id_rsa admin@api.admatic-cluster.k8s.local
* the admin user is specific to Debian. If not using Debian please use the appropriate user based on your OS.
* read about installing addons at: https://github.com/kubernetes/kops/blob/master/docs/addons.md.
This'll take a while. Once it finishes you'll have to wait longer while the booted instances finish downloading Kubernetes components and reach a "ready" state.
Use the Cluster
kops ships with a handy validation tool that can be ran to ensure your cluster is working as expected.
kops validate cluster
Using cluster from kubectl context: admatic-cluster.k8s.local
Validating cluster admatic-cluster.k8s.local
INSTANCE GROUPS
NAME ROLE MACHINETYPE MIN MAX SUBNETS
master-us-east-1a Master c4.large 1 1 us-east-1a
nodes Node t2.medium 4 4 us-east-1a
NODE STATUS
NAME ROLE READY
ip-172-20-34-81.ec2.internal node True
ip-172-20-49-236.ec2.internal node True
ip-172-20-59-238.ec2.internal master True
ip-172-20-59-244.ec2.internal node True
ip-172-20-61-58.ec2.internal node True
Your cluster admatic-cluster.k8s.local is ready
You can look at all the system components with the following command.
kubectl -n kube-system get po
NAME READY STATUS RESTARTS AGE
dns-controller-64b775df4d-hxjg5 1/1 Running 0 93s
etcd-manager-events-ip-172-20-59-238.ec2.internal 1/1 Running 0 53s
etcd-manager-main-ip-172-20-59-238.ec2.internal 1/1 Running 0 42s
kube-apiserver-ip-172-20-59-238.ec2.internal 1/1 Running 2 58s
kube-controller-manager-ip-172-20-59-238.ec2.internal 1/1 Running 0 48s
kube-dns-57dd96bb49-7psjj 3/3 Running 0 93s
kube-dns-57dd96bb49-9cr7c 2/3 Running 0 6s
kube-dns-autoscaler-867b9fd49d-kgh9p 1/1 Running 0 91s
kube-proxy-ip-172-20-49-236.ec2.internal 1/1 Running 0 7s
kube-proxy-ip-172-20-59-238.ec2.internal 1/1 Running 0 34s
kube-scheduler-ip-172-20-59-238.ec2.internal 1/1 Running 0 49s
Delete the Cluster
You can preview all of the AWS resources that will be destroyed when the cluster is deleted by issuing the following command.
kops delete cluster --name ${NAME}
TYPE NAME ID
autoscaling-config master-ap-south-1a.masters.admaticluster.k8s.local-20180618133110 master-ap-south-1a.masters.admaticluster.k8s.local-20180618133110
autoscaling-config nodes.admaticluster.k8s.local-20180618133110 nodes.admaticluster.k8s.local-20180618133110
autoscaling-group master-ap-south-1a.masters.admaticluster.k8s.local master-ap-south-1a.masters.admaticluster.k8s.local
autoscaling-group nodes.admaticluster.k8s.local nodes.admaticluster.k8s.local
dhcp-options admaticluster.k8s.local dopt-ee032f86
iam-instance-profile masters.admaticluster.k8s.local masters.admaticluster.k8s.local
iam-instance-profile nodes.admaticluster.k8s.local nodes.admaticluster.k8s.local
iam-role masters.admaticluster.k8s.local masters.admaticluster.k8s.local
iam-role nodes.admaticluster.k8s.local nodes.admaticluster.k8s.local
instance master-ap-south-1a.masters.admaticluster.k8s.local i-0ae15447219e54d9c
instance nodes.admaticluster.k8s.local i-065872d2b01501ffd
instance nodes.admaticluster.k8s.local i-0d5a9c5e7132b7c8c
internet-gateway admaticluster.k8s.local igw-f2db389a
keypair kubernetes.admaticluster.k8s.local-e4:cd:7d:02:b1:2f:f5:eb:ea:4f:c9:d8:fd:a0:ac:73 kubernetes.admaticluster.k8s.local-e4:cd:7d:02:b1:2f:f5:eb:ea:4f:c9:d8:fd:a0:ac:73
load-balancer api.admaticluster.k8s.local api-admaticluster-k8s-loc-vdk2oh
route-table admaticluster.k8s.local rtb-00d4d068
security-group api-elb.admaticluster.k8s.local sg-f52b319e
security-group masters.admaticluster.k8s.local sg-f3283298
security-group nodes.admaticluster.k8s.local sg-58283233
subnet ap-south-1a.admaticluster.k8s.local subnet-2f87f047
volume a.etcd-events.admaticluster.k8s.local vol-0d1917286a604eefb
volume a.etcd-main.admaticluster.k8s.local vol-00f01bcc365539df5
vpc admaticluster.k8s.local vpc-e6bdfd8e
When you are sure you want to delete your cluster, issue the delete command with the --yes flag.
kops delete cluster --name ${NAME} --yes
load-balancer:api-admaticluster-k8s-loc-vdk2oh ok
autoscaling-group:master-ap-south-1a.masters.admaticluster.k8s.local ok
instance:i-0ae15447219e54d9c ok
autoscaling-group:nodes.admaticluster.k8s.local ok
keypair:kubernetes.admaticluster.k8s.local-e4:cd:7d:02:b1:2f:f5:eb:ea:4f:c9:d8:fd:a0:ac:73 ok
instance:i-0d5a9c5e7132b7c8c ok
internet-gateway:igw-f2db389a still has dependencies, will retry
instance:i-065872d2b01501ffd ok
iam-instance-profile:masters.admaticluster.k8s.local ok
iam-instance-profile:nodes.admaticluster.k8s.local ok
iam-role:masters.admaticluster.k8s.local ok
iam-role:nodes.admaticluster.k8s.local ok
subnet:subnet-2f87f047 still has dependencies, will retry
autoscaling-config:nodes.admaticluster.k8s.local-20180618133110 ok
autoscaling-config:master-ap-south-1a.masters.admaticluster.k8s.local-20180618133110 ok
...
...
...
Deleted kubectl config for admaticluster.k8s.local
Deleted cluster: "admaticluster.k8s.local"
Other interesting modes
- Build a terraform model:
--target=terraformThe terraform model will be built in out/terraform - Build a Cloudformation model:
--target=cloudformationThe Cloudformation json file will be built inout/cloudformation - Specify the k8s build to run:
--kubernetes-version=1.2.2 - Run nodes in multiple zones:
--zones=us-east-1b,us-east-1c,us-east-1d - Run with a HA master:
--master-zones=us-east-1b,us-east-1c,us-east-1d - Specify the number of nodes:
--node-count=4 - Specify the node size:
--node-size=m4.large - Specify the master size:
--master-size=m4.large - Override the default DNS zone:
--dns-zone=<my.hosted.zone>