Hello, Cloud/Amazon EKS で Kubernetes の guestbook チュートリアルをやってみる

Posted on

Amazon EKS

Amazon EKS で Kubernetes のチュートリアル(guestbook)をやってみた記録です。既存の環境に組み込もうとするとIAMだったりVPCだったりハマりどころ満載でしたが動き始めると便利に扱うことができそうでした。

awscliをアップデートしておく

awscliが古いとeksを操作できないのでアップデートしておきます。

%sh
pip install --upgrade awscli
Requirement already up-to-date: awscli in /opt/conda/lib/python2.7/site-packages
Requirement already up-to-date: s3transfer<0.2.0,>=0.1.12 in /opt/conda/lib/python2.7/site-packages (from awscli)
Requirement already up-to-date: botocore==1.12.18 in /opt/conda/lib/python2.7/site-packages (from awscli)
Requirement already up-to-date: PyYAML<=3.13,>=3.10 in /opt/conda/lib/python2.7/site-packages (from awscli)
Requirement already up-to-date: rsa<=3.5.0,>=3.1.2 in /opt/conda/lib/python2.7/site-packages (from awscli)
Requirement already up-to-date: colorama<=0.3.9,>=0.2.5 in /opt/conda/lib/python2.7/site-packages (from awscli)
Requirement already up-to-date: docutils>=0.10 in /opt/conda/lib/python2.7/site-packages (from awscli)
Requirement already up-to-date: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in /opt/conda/lib/python2.7/site-packages (from s3transfer<0.2.0,>=0.1.12->awscli)
Requirement already up-to-date: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python2.7/site-packages (from botocore==1.12.18->awscli)
Requirement already up-to-date: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /opt/conda/lib/python2.7/site-packages (from botocore==1.12.18->awscli)
Requirement already up-to-date: urllib3<1.24,>=1.20 in /opt/conda/lib/python2.7/site-packages (from botocore==1.12.18->awscli)
Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
  Downloading https://files.pythonhosted.org/packages/d1/a1/7790cc85db38daa874f6a2e6308131b9953feb1367f2ae2d1123bb93a9f5/pyasn1-0.4.4-py2.py3-none-any.whl (72kB)
Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.18->awscli)
  Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
Installing collected packages: pyasn1, six
  Found existing installation: pyasn1 0.1.9
    DEPRECATION: Uninstalling a distutils installed project (pyasn1) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling pyasn1-0.1.9:
      Successfully uninstalled pyasn1-0.1.9
  Found existing installation: six 1.10.0
    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling six-1.10.0:
      Successfully uninstalled six-1.10.0
Successfully installed pyasn1-0.4.4 six-1.11.0
You are using pip version 9.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

下準備(IAM Role)

EKS用のIAM Roleを作成しておきます。

%sh
aws iam get-role --role-name hello-eks
{
    "Role": {
        "Description": "Allows EKS to manage clusters on your behalf.", 
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17", 
            "Statement": [
                {
                    "Action": "sts:AssumeRole", 
                    "Effect": "Allow", 
                    "Principal": {
                        "Service": "eks.amazonaws.com"
                    }
                }
            ]
        }, 
        "MaxSessionDuration": 3600, 
        "RoleId": "AROAICYPXU3SFVEBZSBDQ", 
        "CreateDate": "2018-09-30T10:43:59Z", 
        "RoleName": "hello-eks", 
        "Path": "/", 
        "Arn": "arn:aws:iam::845933287843:role/hello-eks"
    }
}

VPCの準備

今回は既存のVPCにEKS用のサブネットを切る形で導入してみました。EKSを動作させるには2つ以上のサブネットが必要になります

%sh
cat <<EOF > ./hello-eks-vpc.yaml
AWSTemplateFormatVersion: 2010-09-09

Parameters:
    Vpc:
        Type: String
        Default: vpc-fee0fb85
    InternetGateway:
        Type: String
        Default: igw-4739613f

Resources:
    RouteTable:
        Type: AWS::EC2::RouteTable
        Properties:
            VpcId: !Ref Vpc
            Tags:
            - Key: Name
              Value: public subnets for eks

    Route:
        Type: AWS::EC2::Route
        Properties:
            RouteTableId: !Ref RouteTable
            DestinationCidrBlock: 0.0.0.0/0
            GatewayId: !Ref InternetGateway
    
    PublicSubnet1:
        Type: AWS::EC2::Subnet
        Properties:
            AvailabilityZone:
                Fn::Select:
                - '0'
                - Fn::GetAZs: us-east-1
            CidrBlock: 10.0.10.0/24
            VpcId: !Ref Vpc
            Tags:
            - Key: Name
              Value: EKS-PublicSubnet1
    
    PublicSubnet2:
        Type: AWS::EC2::Subnet
        Properties:
            AvailabilityZone:
                Fn::Select:
                - '1'
                - Fn::GetAZs: us-east-1
            CidrBlock: 10.0.11.0/24
            VpcId: !Ref Vpc
            Tags:
            - Key: Name
              Value: EKS-PublicSubnet2
    
    PublicSubnet1RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            SubnetId: !Ref PublicSubnet1
            RouteTableId: !Ref RouteTable
    
    PublicSubnet2RouteTableAssociation:
        Type: AWS::EC2::SubnetRouteTableAssociation
        Properties:
            SubnetId: !Ref PublicSubnet2
            RouteTableId: !Ref RouteTable

    ControlPlaneSecurityGroup:
        Type: AWS::EC2::SecurityGroup
        Properties:
            GroupDescription: Cluster communication with worker nodes
            VpcId: vpc-fee0fb85

Outputs:
    SecurityGroup:
        Value: !Ref ControlPlaneSecurityGroup
EOF

aws cloudformation deploy \
    --stack-name hello-eks-vpc \
    --template-file ./hello-eks-vpc.yaml
Waiting for changeset to be created..
Waiting for stack create/update to complete
Successfully created/updated stack - hello-eks-vpc

クラスタを立ち上げたりするときに使うのでセキュリティグループとサブネットのIDを控えておきます。

%sh
aws cloudformation describe-stack-resources --stack-name hello-eks-vpc --query StackResources[].PhysicalResourceId
[
    "sg-0b9c339d18f49f74b", 
    "subnet-0b23128d40de826f0", 
    "rtbassoc-03894c3e5d0d2fd5b", 
    "subnet-09f154e7175f9f396", 
    "rtbassoc-01d609ab6a9c59582", 
    "hello-Route-1UYG7N4KIZRKK", 
    "rtb-0479468e156a1784d"
]

EKSクラスタの立ち上げはIAMロールとVPCの設定を記述すれば大丈夫です。

%sh
aws eks create-cluster \
    --name hello-eks \
    --role-arn arn:aws:iam::845933287843:role/hello-eks \
    --resources-vpc-config subnetIds=subnet-0b23128d40de826f0,subnet-09f154e7175f9f396,securityGroupIds=sg-0b9c339d18f49f74b
{
    "cluster": {
        "status": "CREATING", 
        "name": "hello-eks", 
        "certificateAuthority": {}, 
        "roleArn": "arn:aws:iam::845933287843:role/hello-eks", 
        "resourcesVpcConfig": {
            "subnetIds": [
                "subnet-0b23128d40de826f0", 
                "subnet-09f154e7175f9f396"
            ], 
            "vpcId": "vpc-fee0fb85", 
            "securityGroupIds": [
                "sg-0b9c339d18f49f74b"
            ]
        }, 
        "version": "1.10", 
        "arn": "arn:aws:eks:us-east-1:845933287843:cluster/hello-eks", 
        "platformVersion": "eks.2", 
        "createdAt": 1538961762.219
    }
}

クラスタが立ち上がるまで10分〜20分程度かかるので待っている間にkubectlコマンドなどをインストールしましょう。

%sh
# kubectl
curl -o kubectl https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/kubectl
curl -o kubectl.sha256 https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/kubectl.sha256
cat kubectl.sha256
openssl sha -sha256 kubectl
chmod +x kubectl
cp kubectl /usr/local/bin/

# aws-iam-authenticator: kubeconfigから指定
curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator
curl -o aws-iam-authenticator.sha256 https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator.sha256
cat aws-iam-authenticator.sha256
openssl sha -sha256 aws-iam-authenticator
chmod +x ./aws-iam-authenticator
cp aws-iam-authenticator /usr/local/bin/
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  2 51.6M    2 1545k    0     0  1049k      0  0:00:50  0:00:01  0:00:49 1048k
 11 51.6M   11 6093k    0     0  2475k      0  0:00:21  0:00:02  0:00:19 2474k
 21 51.6M   21 11.0M    0     0  3269k      0  0:00:16  0:00:03  0:00:13 3269k
 31 51.6M   31 16.3M    0     0  3765k      0  0:00:14  0:00:04  0:00:10 3764k
 42 51.6M   42 22.1M    0     0  4144k      0  0:00:12  0:00:05  0:00:07 4560k
 53 51.6M   53 27.6M    0     0  4376k      0  0:00:12  0:00:06  0:00:06 5358k
 64 51.6M   64 33.0M    0     0  4548k      0  0:00:11  0:00:07  0:00:04 5570k
 74 51.6M   74 38.5M    0     0  4667k      0  0:00:11  0:00:08  0:00:03 5631k
 85 51.6M   85 44.0M    0     0  4778k      0  0:00:11  0:00:09  0:00:02 5676k
 94 51.6M   94 48.6M    0     0  4772k      0  0:00:11  0:00:10  0:00:01 5464k
100 51.6M  100 51.6M    0     0  4794k      0  0:00:11  0:00:11 --:--:-- 5387k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    73  100    73    0     0    133      0 --:--:-- --:--:-- --:--:--   132
a624d08f7cae5e64aa73686c3b0fe7953a0733e20c4333b01635d1351fabfa2f kubectl
SHA256(kubectl)= a624d08f7cae5e64aa73686c3b0fe7953a0733e20c4333b01635d1351fabfa2f
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  8 25.1M    8 2118k    0     0  1802k      0  0:00:14  0:00:01  0:00:13 1801k
 33 25.1M   33 8663k    0     0  3984k      0  0:00:06  0:00:02  0:00:04 3983k
 61 25.1M   61 15.4M    0     0  5001k      0  0:00:05  0:00:03  0:00:02 5000k
 87 25.1M   87 21.9M    0     0  5370k      0  0:00:04  0:00:04 --:--:-- 5369k
100 25.1M  100 25.1M    0     0  5338k      0  0:00:04  0:00:04 --:--:-- 5568k
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    87  100    87    0     0    117      0 --:--:-- --:--:-- --:--:--   117
246f6d13b051bbfb12962edca074c8f67436930e84b2bec3a45a5d9242dc6f0c aws-iam-authenticator
SHA256(aws-iam-authenticator)= 246f6d13b051bbfb12962edca074c8f67436930e84b2bec3a45a5d9242dc6f0c

%sh
aws eks describe-cluster --name hello-eks --query cluster.status
"ACTIVE"

CREATINGACTIVE に変わったら食べてもいい。

%sh
aws eks describe-cluster --name hello-eks --query cluster.endpoint
"https://948358B29DBCDE3450C18739761A7828.yl4.us-east-1.eks.amazonaws.com"

kubectlの設定

KubernetesにEKSの認証情報を渡す必要があります。少し前までkubeconfigを自分で書く必要があったようですが、現在は awscli に update-kubeconfig というサブコマンドが入ったので、これを利用すれば .kube/config をよしなに生成することができました。

%sh
aws eks update-kubeconfig --name hello-eks 
Updated context arn:aws:eks:us-east-1:845933287843:cluster/hello-eks in /root/.kube/config

%sh
kubectl version --short
Client Version: v1.10.3
Server Version: v1.10.3-eks

Server Version に eks という接尾辞が入っていれば設定完了です。

%sh
kubectl get all
NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
service/kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   2m

クラスターも認識できています。

ワーカーノードを立ち上げる

ワーカー用のインスタンスを立ち上げます。クラスタと同様にVPCに関する設定を記述し、オートスケーリングの設定と使用するインスタンスの情報などを書いていきます。

%sh
aws eks describe-cluster --name hello-eks --query cluster.resourcesVpcConfig
{
    "subnetIds": [
        "subnet-0b23128d40de826f0", 
        "subnet-09f154e7175f9f396"
    ], 
    "vpcId": "vpc-fee0fb85", 
    "securityGroupIds": [
        "sg-0b9c339d18f49f74b"
    ]
}

%sh
aws cloudformation create-stack \
    --stack-name hello-eks-nodegroup \
    --template-body https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2018-08-30/amazon-eks-nodegroup.yaml \
    --parameters \
        ParameterKey=ClusterName,ParameterValue=hello-eks \
        ParameterKey=ClusterControlPlaneSecurityGroup,ParameterValue=sg-0b9c339d18f49f74b \
        ParameterKey=NodeGroupName,ParameterValue=hello-eks-nodegroup \
        ParameterKey=NodeAutoScalingGroupMinSize,ParameterValue=1 \
        ParameterKey=NodeAutoScalingGroupMaxSize,ParameterValue=2 \
        ParameterKey=NodeInstanceType,ParameterValue=t2.medium \
        ParameterKey=NodeImageId,ParameterValue=ami-0440e4f6b9713faf6 \
        ParameterKey=KeyName,ParameterValue=gateway \
        ParameterKey=VpcId,ParameterValue=vpc-fee0fb85 \
        ParameterKey=Subnets,ParameterValue='subnet-0b23128d40de826f0\,subnet-09f154e7175f9f396' \
    --capabilities CAPABILITY_IAM
{
    "StackId": "arn:aws:cloudformation:us-east-1:845933287843:stack/hello-eks-nodegroup/cb1ffbe0-ca9c-11e8-b555-500c28604cae"
}

ノードインスタンスの立ち上げも地味に時間が掛かるのでなるほどと言いながらドキュメントを眺めます。

%sh
aws cloudformation wait stack-create-complete --stack-name hello-eks-nodegroup
aws cloudformation describe-stacks --stack-name hello-eks-nodegroup --query Stacks[0].StackStatus
"CREATE_COMPLETE"

CREATE_COMPLETE になったら configmap を作成します。

%sh
aws cloudformation describe-stacks --stack-name hello-eks-nodegroup --query Stacks[0].Outputs
[
    {
        "Description": "The node instance role", 
        "OutputKey": "NodeInstanceRole", 
        "OutputValue": "arn:aws:iam::845933287843:role/hello-eks-nodegroup-NodeInstanceRole-1RUXQ4KQDNVF0"
    }
]

%sh
cat << EOF > configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::845933287843:role/hello-eks-nodegroup-NodeInstanceRole-1RUXQ4KQDNVF0
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
EOF

kubectl apply -f configmap.yaml
configmap "aws-auth" configured

設定が終わったら kubectl get nodes を実行してみましょう。

%sh
kubectl get nodes
NAME                        STATUS     ROLES     AGE       VERSION
ip-10-0-10-6.ec2.internal   NotReady   <none>    5s        v1.10.3
ip-10-0-11-6.ec2.internal   NotReady   <none>    4s        v1.10.3

guestbookを動かす

redisを立ち上げます

%sh
cat << EOF > redis-master-deployment.json
{
   "kind":"ReplicationController",
   "apiVersion":"v1",
   "metadata":{
      "name":"redis-master",
      "labels":{
         "app":"redis",
         "role":"master"
      }
   },
   "spec":{
      "replicas":1,
      "selector":{
         "app":"redis",
         "role":"master"
      },
      "template":{
         "metadata":{
            "labels":{
               "app":"redis",
               "role":"master"
            }
         },
         "spec":{
            "containers":[
               {
                  "name":"redis-master",
                  "image":"redis:2.8.23",
                  "ports":[
                     {
                        "name":"redis-server",
                        "containerPort":6379
                     }
                  ]
               }
            ]
         }
      }
   }
}
EOF

kubectl apply -f redis-master-deployment.json
replicationcontroller "redis-master" created

%sh
kubectl get pods
NAME                 READY     STATUS    RESTARTS   AGE
redis-master-d4gdq   1/1       Running   0          15s

podが立ち上がりました。kubectl exec を使うと pod でコマンドを実行することができます。

%sh
kubectl exec redis-master-d4gdq whoami
root

%sh
cat << EOF > redis-master-service.json
{
   "kind":"Service",
   "apiVersion":"v1",
   "metadata":{
      "name":"redis-master",
      "labels":{
         "app":"redis",
         "role":"master"
      }
   },
   "spec":{
      "ports": [
        {
          "port":6379,
          "targetPort":"redis-server"
        }
      ],
      "selector":{
         "app":"redis",
         "role":"master"
      }
   }
}
EOF

kubectl apply -f redis-master-service.json
service "redis-master" created

%sh
kubectl get services
NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
kubernetes     ClusterIP   172.20.0.1     <none>        443/TCP    29m
redis-master   ClusterIP   172.20.70.52   <none>        6379/TCP   9s

続いてredisのslaveを立ち上げます。

%sh
cat << EOF > redis-slave-deployment.json
{
   "kind":"ReplicationController",
   "apiVersion":"v1",
   "metadata":{
      "name":"redis-slave",
      "labels":{
         "app":"redis",
         "role":"slave"
      }
   },
   "spec":{
      "replicas":2,
      "selector":{
         "app":"redis",
         "role":"slave"
      },
      "template":{
         "metadata":{
            "labels":{
               "app":"redis",
               "role":"slave"
            }
         },
         "spec":{
            "containers":[
               {
                  "name":"redis-slave",
                  "image":"kubernetes/redis-slave:v2",
                  "ports":[
                     {
                        "name":"redis-server",
                        "containerPort":6379
                     }
                  ]
               }
            ]
         }
      }
   }
}
EOF

kubectl apply -f redis-slave-deployment.json
replicationcontroller "redis-slave" created

%sh
kubectl get pods
NAME                 READY     STATUS              RESTARTS   AGE
redis-master-d4gdq   1/1       Running             0          1m
redis-slave-ftswv    0/1       ContainerCreating   0          8s
redis-slave-tkls5    0/1       ContainerCreating   0          8s

%sh
cat << EOF > redis-slave-service.json
{
   "kind":"Service",
   "apiVersion":"v1",
   "metadata":{
      "name":"redis-slave",
      "labels":{
         "app":"redis",
         "role":"slave"
      }
   },
   "spec":{
      "ports": [
        {
          "port":6379,
          "targetPort":"redis-server"
        }
      ],
      "selector":{
         "app":"redis",
         "role":"slave"
      }
   }
}
EOF

kubectl apply -f redis-slave-service.json
service "redis-slave" created

%sh
kubectl get services
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
kubernetes     ClusterIP   172.20.0.1       <none>        443/TCP    31m
redis-master   ClusterIP   172.20.70.52     <none>        6379/TCP   2m
redis-slave    ClusterIP   172.20.213.112   <none>        6379/TCP   9s

Webアプリケーションもredisと同様に立ち上げていきます。

%sh
cat <<EOF > frontend-controller.json
{
   "kind":"ReplicationController",
   "apiVersion":"v1",
   "metadata":{
      "name":"guestbook",
      "labels":{
         "app":"guestbook"
      }
   },
   "spec":{
      "replicas":3,
      "selector":{
         "app":"guestbook"
      },
      "template":{
         "metadata":{
            "labels":{
               "app":"guestbook"
            }
         },
         "spec":{
            "containers":[
               {
                  "name":"guestbook",
                  "image":"k8s.gcr.io/guestbook:v3",
                  "ports":[
                     {
                        "name":"http-server",
                        "containerPort":3000
                     }
                  ]
               }
            ]
         }
      }
   }
}
EOF

kubectl apply -f frontend-controller.json
replicationcontroller "guestbook" created

%sh
cat << EOF > frontend-service.json
{
   "kind":"Service",
   "apiVersion":"v1",
   "metadata":{
      "name":"guestbook",
      "labels":{
         "app":"guestbook"
      }
   },
   "spec":{
      "ports": [
         {
           "port":3000,
           "targetPort":"http-server"
         }
      ],
      "selector":{
         "app":"guestbook"
      },
      "type": "LoadBalancer"
   }
}
EOF

kubectl apply -f frontend-service.json
service "guestbook" created

%sh
kubectl get services
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP        PORT(S)          AGE
guestbook      LoadBalancer   172.20.223.78    a894ee3c7ca9e...   3000:30068/TCP   2m
kubernetes     ClusterIP      172.20.0.1       <none>             443/TCP          36m
redis-master   ClusterIP      172.20.70.52     <none>             6379/TCP         7m
redis-slave    ClusterIP      172.20.213.112   <none>             6379/TCP         5m

EXTERNAL-IP が見切れているときは kubectl get services-o wide を渡します。

%sh
kubectl get services -o wide
NAME           TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)          AGE       SELECTOR
guestbook      LoadBalancer   172.20.223.78    a894ee3c7ca9e11e8a49b020fd078860-812237424.us-east-1.elb.amazonaws.com   3000:30068/TCP   2m        app=guestbook
kubernetes     ClusterIP      172.20.0.1       <none>                                                                   443/TCP          36m       <none>
redis-master   ClusterIP      172.20.70.52     <none>                                                                   6379/TCP         7m        app=redis,role=master
redis-slave    ClusterIP      172.20.213.112   <none>                                                                   6379/TCP         5m        app=redis,role=slave

接続してみましょう(ロードバランサーに設定が反映されるまで若干時間が掛かるのに注意しましょう)。

%sh
curl -s http://a894ee3c7ca9e11e8a49b020fd078860-812237424.us-east-1.elb.amazonaws.com:3000
<!DOCTYPE html>
<html lang="en">
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
    <meta charset="utf-8">
    <meta content="width=device-width" name="viewport">
    <link href="style.css" rel="stylesheet">
    <title>Guestbook</title>
  </head>
  <body>
    <div id="header">
      <h1>Guestbook</h1>
    </div>

    <div id="guestbook-entries">
      <p>Waiting for database connection...</p>
    </div>

    <div>
      <form id="guestbook-form">
        <input autocomplete="off" id="guestbook-entry-content" type="text">
        <a href="#" id="guestbook-submit">Submit</a>
      </form>
    </div>

    <div>
      <p><h2 id="guestbook-host-address"></h2></p>
      <p><a href="env">/env</a>
      <a href="info">/info</a></p>
    </div>
    <script src="//ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
    <script src="script.js"></script>
  </body>
</html>

ハマった点は

  • nodegroup作成時にCloudFormationのテンプレートURLを間違えていた-
    • 気をつけましょう… :-|
    • https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2018-08-30/amazon-eks-nodegroup.yaml を使うと動作しました
  • Service の EXTERNAL-IP が PENDING のまま変わらない
    • public subnetを一つしか割り当ててなかったのが原因のようです

おまけ: CloudFormationテンプレート(nodegroup)の差分

%sh
diff <(curl -s https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-06-05/amazon-eks-nodegroup.yaml) <(curl -s https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2018-08-30/amazon-eks-nodegroup.yaml)
echo
3c3
< Description: 'Amazon EKS - Node Group'
---
> Description: 'Amazon EKS - Node Group - Released 2018-08-30'
75c75
<     ConstraintDescription: must be a valid EC2 instance type
---
>     ConstraintDescription: Must be a valid EC2 instance type
86a87,91
>   NodeVolumeSize:
>     Type: Number
>     Description: Node volume size
>     Default: 20
> 
88c93,98
<     Description: The cluster name provided when the cluster was created.  If it is incorrect, nodes will not be able to join the cluster.
---
>     Description: The cluster name provided when the cluster was created. If it is incorrect, nodes will not be able to join the cluster.
>     Type: String
> 
>   BootstrapArguments:
>     Description: Arguments to pass to the bootstrap script. See files/bootstrap.sh in https://github.com/awslabs/amazon-eks-ami
>     Default: ""
107,219d116
< Mappings:
<   MaxPodsPerNode:
<     c4.large:
<       MaxPods: 29
<     c4.xlarge:
<       MaxPods: 58
<     c4.2xlarge:
<       MaxPods: 58
<     c4.4xlarge:
<       MaxPods: 234
<     c4.8xlarge:
<       MaxPods: 234
<     c5.large:
<       MaxPods: 29
<     c5.xlarge:
<       MaxPods: 58
<     c5.2xlarge:
<       MaxPods: 58
<     c5.4xlarge:
<       MaxPods: 234
<     c5.9xlarge:
<       MaxPods: 234
<     c5.18xlarge:
<       MaxPods: 737
<     i3.large:
<       MaxPods: 29
<     i3.xlarge:
<       MaxPods: 58
<     i3.2xlarge:
<       MaxPods: 58
<     i3.4xlarge:
<       MaxPods: 234
<     i3.8xlarge:
<       MaxPods: 234
<     i3.16xlarge:
<       MaxPods: 737
<     m3.medium:
<       MaxPods: 12
<     m3.large:
<       MaxPods: 29
<     m3.xlarge:
<       MaxPods: 58
<     m3.2xlarge:
<       MaxPods: 118
<     m4.large:
<       MaxPods: 20
<     m4.xlarge:
<       MaxPods: 58
<     m4.2xlarge:
<       MaxPods: 58
<     m4.4xlarge:
<       MaxPods: 234
<     m4.10xlarge:
<       MaxPods: 234
<     m5.large:
<       MaxPods: 29
<     m5.xlarge:
<       MaxPods: 58
<     m5.2xlarge:
<       MaxPods: 58
<     m5.4xlarge:
<       MaxPods: 234
<     m5.12xlarge:
<       MaxPods: 234
<     m5.24xlarge:
<       MaxPods: 737
<     p2.xlarge:
<       MaxPods: 58
<     p2.8xlarge:
<       MaxPods: 234
<     p2.16xlarge:
<       MaxPods: 234
<     p3.2xlarge:
<       MaxPods: 58
<     p3.8xlarge:
<       MaxPods: 234
<     p3.16xlarge:
<       MaxPods: 234
<     r3.xlarge:
<       MaxPods: 58
<     r3.2xlarge:
<       MaxPods: 58
<     r3.4xlarge:
<       MaxPods: 234
<     r3.8xlarge:
<       MaxPods: 234
<     r4.large:
<       MaxPods: 29
<     r4.xlarge:
<       MaxPods: 58
<     r4.2xlarge:
<       MaxPods: 58
<     r4.4xlarge:
<       MaxPods: 234
<     r4.8xlarge:
<       MaxPods: 234
<     r4.16xlarge:
<       MaxPods: 737
<     t2.small:
<       MaxPods: 8
<     t2.medium:
<       MaxPods: 17
<     t2.large:
<       MaxPods: 35
<     t2.xlarge:
<       MaxPods: 44
<     t2.2xlarge:
<       MaxPods: 44
<     x1.16xlarge:
<       MaxPods: 234
<     x1.32xlarge:
<       MaxPods: 234
< 
237a135
>           - NodeVolumeSize
238a137
>           - BootstrapArguments
315a215,236
>   NodeSecurityGroupFromControlPlaneOn443Ingress:
>     Type: AWS::EC2::SecurityGroupIngress
>     DependsOn: NodeSecurityGroup
>     Properties:
>       Description: Allow pods running extension API servers on port 443 to receive communication from cluster control plane
>       GroupId: !Ref NodeSecurityGroup
>       SourceSecurityGroupId: !Ref ClusterControlPlaneSecurityGroup
>       IpProtocol: tcp
>       FromPort: 443
>       ToPort: 443
> 
>   ControlPlaneEgressToNodeSecurityGroupOn443:
>     Type: AWS::EC2::SecurityGroupEgress
>     DependsOn: NodeSecurityGroup
>     Properties:
>       Description: Allow the cluster control plane to communicate with pods running extension API servers on port 443
>       GroupId: !Ref ClusterControlPlaneSecurityGroup
>       DestinationSecurityGroupId: !Ref NodeSecurityGroup
>       IpProtocol: tcp
>       FromPort: 443
>       ToPort: 443
> 
357a279,284
>       BlockDeviceMappings:
>         - DeviceName: /dev/xvda
>           Ebs:
>             VolumeSize: !Ref NodeVolumeSize
>             VolumeType: gp2
>             DeleteOnTermination: true
360,394c287,294
<           Fn::Join: [
<             "",
<             [
<               "#!/bin/bash -xe\n",
<               "CA_CERTIFICATE_DIRECTORY=/etc/kubernetes/pki", "\n",
<               "CA_CERTIFICATE_FILE_PATH=$CA_CERTIFICATE_DIRECTORY/ca.crt", "\n",
<               "MODEL_DIRECTORY_PATH=~/.aws/eks", "\n",
<               "MODEL_FILE_PATH=$MODEL_DIRECTORY_PATH/eks-2017-11-01.normal.json", "\n",
<               "mkdir -p $CA_CERTIFICATE_DIRECTORY", "\n",
<               "mkdir -p $MODEL_DIRECTORY_PATH", "\n",
<               "curl -o $MODEL_FILE_PATH https://s3-us-west-2.amazonaws.com/amazon-eks/1.10.3/2018-06-05/eks-2017-11-01.normal.json", "\n",
<               "aws configure add-model --service-model file://$MODEL_FILE_PATH --service-name eks", "\n",
<               "aws eks describe-cluster --region=", { Ref: "AWS::Region" }," --name=", { Ref: ClusterName }," --query 'cluster.{certificateAuthorityData: certificateAuthority.data, endpoint: endpoint}' > /tmp/describe_cluster_result.json", "\n",
<               "cat /tmp/describe_cluster_result.json | grep certificateAuthorityData | awk '{print $2}' | sed 's/[,\"]//g' | base64 -d >  $CA_CERTIFICATE_FILE_PATH", "\n",
<               "MASTER_ENDPOINT=$(cat /tmp/describe_cluster_result.json | grep endpoint | awk '{print $2}' | sed 's/[,\"]//g')", "\n",
<               "INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)", "\n",
<               "sed -i s,MASTER_ENDPOINT,$MASTER_ENDPOINT,g /var/lib/kubelet/kubeconfig", "\n",
<               "sed -i s,CLUSTER_NAME,", { Ref: ClusterName }, ",g /var/lib/kubelet/kubeconfig", "\n",
<               "sed -i s,REGION,", { Ref: "AWS::Region" }, ",g /etc/systemd/system/kubelet.service", "\n",
<               "sed -i s,MAX_PODS,", { "Fn::FindInMap": [ MaxPodsPerNode, { Ref: NodeInstanceType }, MaxPods ] }, ",g /etc/systemd/system/kubelet.service", "\n",
<               "sed -i s,MASTER_ENDPOINT,$MASTER_ENDPOINT,g /etc/systemd/system/kubelet.service", "\n",
<               "sed -i s,INTERNAL_IP,$INTERNAL_IP,g /etc/systemd/system/kubelet.service", "\n",
<               "DNS_CLUSTER_IP=10.100.0.10", "\n",
<               "if [[ $INTERNAL_IP == 10.* ]] ; then DNS_CLUSTER_IP=172.20.0.10; fi", "\n",
<               "sed -i s,DNS_CLUSTER_IP,$DNS_CLUSTER_IP,g  /etc/systemd/system/kubelet.service", "\n",
<               "sed -i s,CERTIFICATE_AUTHORITY_FILE,$CA_CERTIFICATE_FILE_PATH,g /var/lib/kubelet/kubeconfig" , "\n",
<               "sed -i s,CLIENT_CA_FILE,$CA_CERTIFICATE_FILE_PATH,g  /etc/systemd/system/kubelet.service" , "\n",
<               "systemctl daemon-reload", "\n",
<               "systemctl restart kubelet", "\n",
<               "/opt/aws/bin/cfn-signal -e $? ",
<               "         --stack ", { Ref: "AWS::StackName" },
<               "         --resource NodeGroup ",
<               "         --region ", { Ref: "AWS::Region" }, "\n"
<             ]
<           ]
---
>           !Sub |
>             #!/bin/bash
>             set -o xtrace
>             /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
>             /opt/aws/bin/cfn-signal --exit-code $? \
>                      --stack  ${AWS::StackName} \
>                      --resource NodeGroup  \
>                      --region ${AWS::Region}
400d299
< 

初期化の部分が変わっているようにみえる。

片付け

終わったら片付けましょう

%sh
aws cloudformation delete-stack --stack-name hello-eks-nodegroup
aws cloudformation delete-stack --stack-name hello-eks-vpc
aws eks delete-cluster --name hello-eks --query cluster.status
"DELETING"

// 雑感: Replication ControllerとDeploymentの違いがよく分かってない気がする :-|

%md