Amazon EKS で Kubernetes の guestbook チュートリアルをやってみる / Hello, Cloud

    Posted on 2018/10/07

    Amazon EKS

    Amazon EKS で Kubernetes のチュートリアル(guestbook)をやってみた記録です。既存の環境に組み込もうとするとIAMだったりVPCだったりハマりどころ満載でしたが動き始めると便利に扱うことができそうでした。

    awscliをアップデートしておく

    awscliが古いとeksを操作できないのでアップデートしておきます。

    %sh
    pip install --upgrade awscli
    Requirement already up-to-date: awscli in /opt/conda/lib/python2.7/site-packages
    Requirement already up-to-date: s3transfer<0.2.0,>=0.1.12 in /opt/conda/lib/python2.7/site-packages (from awscli)
    Requirement already up-to-date: botocore==1.12.18 in /opt/conda/lib/python2.7/site-packages (from awscli)
    Requirement already up-to-date: PyYAML<=3.13,>=3.10 in /opt/conda/lib/python2.7/site-packages (from awscli)
    Requirement already up-to-date: rsa<=3.5.0,>=3.1.2 in /opt/conda/lib/python2.7/site-packages (from awscli)
    Requirement already up-to-date: colorama<=0.3.9,>=0.2.5 in /opt/conda/lib/python2.7/site-packages (from awscli)
    Requirement already up-to-date: docutils>=0.10 in /opt/conda/lib/python2.7/site-packages (from awscli)
    Requirement already up-to-date: futures<4.0.0,>=2.2.0; python_version == "2.6" or python_version == "2.7" in /opt/conda/lib/python2.7/site-packages (from s3transfer<0.2.0,>=0.1.12->awscli)
    Requirement already up-to-date: jmespath<1.0.0,>=0.7.1 in /opt/conda/lib/python2.7/site-packages (from botocore==1.12.18->awscli)
    Requirement already up-to-date: python-dateutil<3.0.0,>=2.1; python_version >= "2.7" in /opt/conda/lib/python2.7/site-packages (from botocore==1.12.18->awscli)
    Requirement already up-to-date: urllib3<1.24,>=1.20 in /opt/conda/lib/python2.7/site-packages (from botocore==1.12.18->awscli)
    Collecting pyasn1>=0.1.3 (from rsa<=3.5.0,>=3.1.2->awscli)
      Downloading https://files.pythonhosted.org/packages/d1/a1/7790cc85db38daa874f6a2e6308131b9953feb1367f2ae2d1123bb93a9f5/pyasn1-0.4.4-py2.py3-none-any.whl (72kB)
    Collecting six>=1.5 (from python-dateutil<3.0.0,>=2.1; python_version >= "2.7"->botocore==1.12.18->awscli)
      Downloading https://files.pythonhosted.org/packages/67/4b/141a581104b1f6397bfa78ac9d43d8ad29a7ca43ea90a2d863fe3056e86a/six-1.11.0-py2.py3-none-any.whl
    Installing collected packages: pyasn1, six
      Found existing installation: pyasn1 0.1.9
        DEPRECATION: Uninstalling a distutils installed project (pyasn1) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
        Uninstalling pyasn1-0.1.9:
          Successfully uninstalled pyasn1-0.1.9
      Found existing installation: six 1.10.0
        DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
        Uninstalling six-1.10.0:
          Successfully uninstalled six-1.10.0
    Successfully installed pyasn1-0.4.4 six-1.11.0
    You are using pip version 9.0.1, however version 18.1 is available.
    You should consider upgrading via the 'pip install --upgrade pip' command.
    

    下準備(IAM Role)

    EKS用のIAM Roleを作成しておきます。

    %sh
    aws iam get-role --role-name hello-eks
    {
        "Role": {
            "Description": "Allows EKS to manage clusters on your behalf.", 
            "AssumeRolePolicyDocument": {
                "Version": "2012-10-17", 
                "Statement": [
                    {
                        "Action": "sts:AssumeRole", 
                        "Effect": "Allow", 
                        "Principal": {
                            "Service": "eks.amazonaws.com"
                        }
                    }
                ]
            }, 
            "MaxSessionDuration": 3600, 
            "RoleId": "AROAICYPXU3SFVEBZSBDQ", 
            "CreateDate": "2018-09-30T10:43:59Z", 
            "RoleName": "hello-eks", 
            "Path": "/", 
            "Arn": "arn:aws:iam::845933287843:role/hello-eks"
        }
    }
    

    VPCの準備

    今回は既存のVPCにEKS用のサブネットを切る形で導入してみました。EKSを動作させるには2つ以上のサブネットが必要になります

    %sh
    cat <<EOF > ./hello-eks-vpc.yaml
    AWSTemplateFormatVersion: 2010-09-09
    
    Parameters:
        Vpc:
            Type: String
            Default: vpc-fee0fb85
        InternetGateway:
            Type: String
            Default: igw-4739613f
    
    Resources:
        RouteTable:
            Type: AWS::EC2::RouteTable
            Properties:
                VpcId: !Ref Vpc
                Tags:
                - Key: Name
                  Value: public subnets for eks
    
        Route:
            Type: AWS::EC2::Route
            Properties:
                RouteTableId: !Ref RouteTable
                DestinationCidrBlock: 0.0.0.0/0
                GatewayId: !Ref InternetGateway
        
        PublicSubnet1:
            Type: AWS::EC2::Subnet
            Properties:
                AvailabilityZone:
                    Fn::Select:
                    - '0'
                    - Fn::GetAZs: us-east-1
                CidrBlock: 10.0.10.0/24
                VpcId: !Ref Vpc
                Tags:
                - Key: Name
                  Value: EKS-PublicSubnet1
        
        PublicSubnet2:
            Type: AWS::EC2::Subnet
            Properties:
                AvailabilityZone:
                    Fn::Select:
                    - '1'
                    - Fn::GetAZs: us-east-1
                CidrBlock: 10.0.11.0/24
                VpcId: !Ref Vpc
                Tags:
                - Key: Name
                  Value: EKS-PublicSubnet2
        
        PublicSubnet1RouteTableAssociation:
            Type: AWS::EC2::SubnetRouteTableAssociation
            Properties:
                SubnetId: !Ref PublicSubnet1
                RouteTableId: !Ref RouteTable
        
        PublicSubnet2RouteTableAssociation:
            Type: AWS::EC2::SubnetRouteTableAssociation
            Properties:
                SubnetId: !Ref PublicSubnet2
                RouteTableId: !Ref RouteTable
    
        ControlPlaneSecurityGroup:
            Type: AWS::EC2::SecurityGroup
            Properties:
                GroupDescription: Cluster communication with worker nodes
                VpcId: vpc-fee0fb85
    
    Outputs:
        SecurityGroup:
            Value: !Ref ControlPlaneSecurityGroup
    EOF
    
    aws cloudformation deploy \
        --stack-name hello-eks-vpc \
        --template-file ./hello-eks-vpc.yaml
    Waiting for changeset to be created..
    Waiting for stack create/update to complete
    Successfully created/updated stack - hello-eks-vpc
    

    クラスタを立ち上げたりするときに使うのでセキュリティグループとサブネットのIDを控えておきます。

    %sh
    aws cloudformation describe-stack-resources --stack-name hello-eks-vpc --query StackResources[].PhysicalResourceId
    [
        "sg-0b9c339d18f49f74b", 
        "subnet-0b23128d40de826f0", 
        "rtbassoc-03894c3e5d0d2fd5b", 
        "subnet-09f154e7175f9f396", 
        "rtbassoc-01d609ab6a9c59582", 
        "hello-Route-1UYG7N4KIZRKK", 
        "rtb-0479468e156a1784d"
    ]
    

    EKSクラスタの立ち上げはIAMロールとVPCの設定を記述すれば大丈夫です。

    %sh
    aws eks create-cluster \
        --name hello-eks \
        --role-arn arn:aws:iam::845933287843:role/hello-eks \
        --resources-vpc-config subnetIds=subnet-0b23128d40de826f0,subnet-09f154e7175f9f396,securityGroupIds=sg-0b9c339d18f49f74b
    {
        "cluster": {
            "status": "CREATING", 
            "name": "hello-eks", 
            "certificateAuthority": {}, 
            "roleArn": "arn:aws:iam::845933287843:role/hello-eks", 
            "resourcesVpcConfig": {
                "subnetIds": [
                    "subnet-0b23128d40de826f0", 
                    "subnet-09f154e7175f9f396"
                ], 
                "vpcId": "vpc-fee0fb85", 
                "securityGroupIds": [
                    "sg-0b9c339d18f49f74b"
                ]
            }, 
            "version": "1.10", 
            "arn": "arn:aws:eks:us-east-1:845933287843:cluster/hello-eks", 
            "platformVersion": "eks.2", 
            "createdAt": 1538961762.219
        }
    }
    

    クラスタが立ち上がるまで10分〜20分程度かかるので待っている間にkubectlコマンドなどをインストールしましょう。

    %sh
    # kubectl
    curl -o kubectl https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/kubectl
    curl -o kubectl.sha256 https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/kubectl.sha256
    cat kubectl.sha256
    openssl sha -sha256 kubectl
    chmod +x kubectl
    cp kubectl /usr/local/bin/
    
    # aws-iam-authenticator: kubeconfigから指定
    curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator
    curl -o aws-iam-authenticator.sha256 https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/linux/amd64/aws-iam-authenticator.sha256
    cat aws-iam-authenticator.sha256
    openssl sha -sha256 aws-iam-authenticator
    chmod +x ./aws-iam-authenticator
    cp aws-iam-authenticator /usr/local/bin/
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
      2 51.6M    2 1545k    0     0  1049k      0  0:00:50  0:00:01  0:00:49 1048k
     11 51.6M   11 6093k    0     0  2475k      0  0:00:21  0:00:02  0:00:19 2474k
     21 51.6M   21 11.0M    0     0  3269k      0  0:00:16  0:00:03  0:00:13 3269k
     31 51.6M   31 16.3M    0     0  3765k      0  0:00:14  0:00:04  0:00:10 3764k
     42 51.6M   42 22.1M    0     0  4144k      0  0:00:12  0:00:05  0:00:07 4560k
     53 51.6M   53 27.6M    0     0  4376k      0  0:00:12  0:00:06  0:00:06 5358k
     64 51.6M   64 33.0M    0     0  4548k      0  0:00:11  0:00:07  0:00:04 5570k
     74 51.6M   74 38.5M    0     0  4667k      0  0:00:11  0:00:08  0:00:03 5631k
     85 51.6M   85 44.0M    0     0  4778k      0  0:00:11  0:00:09  0:00:02 5676k
     94 51.6M   94 48.6M    0     0  4772k      0  0:00:11  0:00:10  0:00:01 5464k
    100 51.6M  100 51.6M    0     0  4794k      0  0:00:11  0:00:11 --:--:-- 5387k
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    100    73  100    73    0     0    133      0 --:--:-- --:--:-- --:--:--   132
    a624d08f7cae5e64aa73686c3b0fe7953a0733e20c4333b01635d1351fabfa2f kubectl
    SHA256(kubectl)= a624d08f7cae5e64aa73686c3b0fe7953a0733e20c4333b01635d1351fabfa2f
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
      8 25.1M    8 2118k    0     0  1802k      0  0:00:14  0:00:01  0:00:13 1801k
     33 25.1M   33 8663k    0     0  3984k      0  0:00:06  0:00:02  0:00:04 3983k
     61 25.1M   61 15.4M    0     0  5001k      0  0:00:05  0:00:03  0:00:02 5000k
     87 25.1M   87 21.9M    0     0  5370k      0  0:00:04  0:00:04 --:--:-- 5369k
    100 25.1M  100 25.1M    0     0  5338k      0  0:00:04  0:00:04 --:--:-- 5568k
      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
      0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
    100    87  100    87    0     0    117      0 --:--:-- --:--:-- --:--:--   117
    246f6d13b051bbfb12962edca074c8f67436930e84b2bec3a45a5d9242dc6f0c aws-iam-authenticator
    SHA256(aws-iam-authenticator)= 246f6d13b051bbfb12962edca074c8f67436930e84b2bec3a45a5d9242dc6f0c
    

    %sh
    aws eks describe-cluster --name hello-eks --query cluster.status
    "ACTIVE"
    

    CREATINGACTIVE に変わったら食べてもいい。

    %sh
    aws eks describe-cluster --name hello-eks --query cluster.endpoint
    "https://948358B29DBCDE3450C18739761A7828.yl4.us-east-1.eks.amazonaws.com"
    

    kubectlの設定

    KubernetesにEKSの認証情報を渡す必要があります。少し前までkubeconfigを自分で書く必要があったようですが、現在は awscli に update-kubeconfig というサブコマンドが入ったので、これを利用すれば .kube/config をよしなに生成することができました。

    %sh
    aws eks update-kubeconfig --name hello-eks 
    Updated context arn:aws:eks:us-east-1:845933287843:cluster/hello-eks in /root/.kube/config
    

    %sh
    kubectl version --short
    Client Version: v1.10.3
    Server Version: v1.10.3-eks
    

    Server Version に eks という接尾辞が入っていれば設定完了です。

    %sh
    kubectl get all
    NAME                 TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
    service/kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   2m
    

    クラスターも認識できています。

    ワーカーノードを立ち上げる

    ワーカー用のインスタンスを立ち上げます。クラスタと同様にVPCに関する設定を記述し、オートスケーリングの設定と使用するインスタンスの情報などを書いていきます。

    %sh
    aws eks describe-cluster --name hello-eks --query cluster.resourcesVpcConfig
    {
        "subnetIds": [
            "subnet-0b23128d40de826f0", 
            "subnet-09f154e7175f9f396"
        ], 
        "vpcId": "vpc-fee0fb85", 
        "securityGroupIds": [
            "sg-0b9c339d18f49f74b"
        ]
    }
    

    %sh
    aws cloudformation create-stack \
        --stack-name hello-eks-nodegroup \
        --template-body https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2018-08-30/amazon-eks-nodegroup.yaml \
        --parameters \
            ParameterKey=ClusterName,ParameterValue=hello-eks \
            ParameterKey=ClusterControlPlaneSecurityGroup,ParameterValue=sg-0b9c339d18f49f74b \
            ParameterKey=NodeGroupName,ParameterValue=hello-eks-nodegroup \
            ParameterKey=NodeAutoScalingGroupMinSize,ParameterValue=1 \
            ParameterKey=NodeAutoScalingGroupMaxSize,ParameterValue=2 \
            ParameterKey=NodeInstanceType,ParameterValue=t2.medium \
            ParameterKey=NodeImageId,ParameterValue=ami-0440e4f6b9713faf6 \
            ParameterKey=KeyName,ParameterValue=gateway \
            ParameterKey=VpcId,ParameterValue=vpc-fee0fb85 \
            ParameterKey=Subnets,ParameterValue='subnet-0b23128d40de826f0\,subnet-09f154e7175f9f396' \
        --capabilities CAPABILITY_IAM
    {
        "StackId": "arn:aws:cloudformation:us-east-1:845933287843:stack/hello-eks-nodegroup/cb1ffbe0-ca9c-11e8-b555-500c28604cae"
    }
    

    ノードインスタンスの立ち上げも地味に時間が掛かるのでなるほどと言いながらドキュメントを眺めます。

    %sh
    aws cloudformation wait stack-create-complete --stack-name hello-eks-nodegroup
    aws cloudformation describe-stacks --stack-name hello-eks-nodegroup --query Stacks[0].StackStatus
    "CREATE_COMPLETE"
    

    CREATE_COMPLETE になったら configmap を作成します。

    %sh
    aws cloudformation describe-stacks --stack-name hello-eks-nodegroup --query Stacks[0].Outputs
    [
        {
            "Description": "The node instance role", 
            "OutputKey": "NodeInstanceRole", 
            "OutputValue": "arn:aws:iam::845933287843:role/hello-eks-nodegroup-NodeInstanceRole-1RUXQ4KQDNVF0"
        }
    ]
    

    %sh
    cat << EOF > configmap.yaml
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: aws-auth
      namespace: kube-system
    data:
      mapRoles: |
        - rolearn: arn:aws:iam::845933287843:role/hello-eks-nodegroup-NodeInstanceRole-1RUXQ4KQDNVF0
          username: system:node:{{EC2PrivateDNSName}}
          groups:
            - system:bootstrappers
            - system:nodes
    EOF
    
    kubectl apply -f configmap.yaml
    configmap "aws-auth" configured
    

    設定が終わったら kubectl get nodes を実行してみましょう。

    %sh
    kubectl get nodes
    NAME                        STATUS     ROLES     AGE       VERSION
    ip-10-0-10-6.ec2.internal   NotReady   <none>    5s        v1.10.3
    ip-10-0-11-6.ec2.internal   NotReady   <none>    4s        v1.10.3
    

    guestbookを動かす

    redisを立ち上げます

    %sh
    cat << EOF > redis-master-deployment.json
    {
       "kind":"ReplicationController",
       "apiVersion":"v1",
       "metadata":{
          "name":"redis-master",
          "labels":{
             "app":"redis",
             "role":"master"
          }
       },
       "spec":{
          "replicas":1,
          "selector":{
             "app":"redis",
             "role":"master"
          },
          "template":{
             "metadata":{
                "labels":{
                   "app":"redis",
                   "role":"master"
                }
             },
             "spec":{
                "containers":[
                   {
                      "name":"redis-master",
                      "image":"redis:2.8.23",
                      "ports":[
                         {
                            "name":"redis-server",
                            "containerPort":6379
                         }
                      ]
                   }
                ]
             }
          }
       }
    }
    EOF
    
    kubectl apply -f redis-master-deployment.json
    replicationcontroller "redis-master" created
    

    %sh
    kubectl get pods
    NAME                 READY     STATUS    RESTARTS   AGE
    redis-master-d4gdq   1/1       Running   0          15s
    

    podが立ち上がりました。kubectl exec を使うと pod でコマンドを実行することができます。

    %sh
    kubectl exec redis-master-d4gdq whoami
    root
    

    %sh
    cat << EOF > redis-master-service.json
    {
       "kind":"Service",
       "apiVersion":"v1",
       "metadata":{
          "name":"redis-master",
          "labels":{
             "app":"redis",
             "role":"master"
          }
       },
       "spec":{
          "ports": [
            {
              "port":6379,
              "targetPort":"redis-server"
            }
          ],
          "selector":{
             "app":"redis",
             "role":"master"
          }
       }
    }
    EOF
    
    kubectl apply -f redis-master-service.json
    service "redis-master" created
    

    %sh
    kubectl get services
    NAME           TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
    kubernetes     ClusterIP   172.20.0.1     <none>        443/TCP    29m
    redis-master   ClusterIP   172.20.70.52   <none>        6379/TCP   9s
    

    続いてredisのslaveを立ち上げます。

    %sh
    cat << EOF > redis-slave-deployment.json
    {
       "kind":"ReplicationController",
       "apiVersion":"v1",
       "metadata":{
          "name":"redis-slave",
          "labels":{
             "app":"redis",
             "role":"slave"
          }
       },
       "spec":{
          "replicas":2,
          "selector":{
             "app":"redis",
             "role":"slave"
          },
          "template":{
             "metadata":{
                "labels":{
                   "app":"redis",
                   "role":"slave"
                }
             },
             "spec":{
                "containers":[
                   {
                      "name":"redis-slave",
                      "image":"kubernetes/redis-slave:v2",
                      "ports":[
                         {
                            "name":"redis-server",
                            "containerPort":6379
                         }
                      ]
                   }
                ]
             }
          }
       }
    }
    EOF
    
    kubectl apply -f redis-slave-deployment.json
    replicationcontroller "redis-slave" created
    

    %sh
    kubectl get pods
    NAME                 READY     STATUS              RESTARTS   AGE
    redis-master-d4gdq   1/1       Running             0          1m
    redis-slave-ftswv    0/1       ContainerCreating   0          8s
    redis-slave-tkls5    0/1       ContainerCreating   0          8s
    

    %sh
    cat << EOF > redis-slave-service.json
    {
       "kind":"Service",
       "apiVersion":"v1",
       "metadata":{
          "name":"redis-slave",
          "labels":{
             "app":"redis",
             "role":"slave"
          }
       },
       "spec":{
          "ports": [
            {
              "port":6379,
              "targetPort":"redis-server"
            }
          ],
          "selector":{
             "app":"redis",
             "role":"slave"
          }
       }
    }
    EOF
    
    kubectl apply -f redis-slave-service.json
    service "redis-slave" created
    

    %sh
    kubectl get services
    NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
    kubernetes     ClusterIP   172.20.0.1       <none>        443/TCP    31m
    redis-master   ClusterIP   172.20.70.52     <none>        6379/TCP   2m
    redis-slave    ClusterIP   172.20.213.112   <none>        6379/TCP   9s
    

    Webアプリケーションもredisと同様に立ち上げていきます。

    %sh
    cat <<EOF > frontend-controller.json
    {
       "kind":"ReplicationController",
       "apiVersion":"v1",
       "metadata":{
          "name":"guestbook",
          "labels":{
             "app":"guestbook"
          }
       },
       "spec":{
          "replicas":3,
          "selector":{
             "app":"guestbook"
          },
          "template":{
             "metadata":{
                "labels":{
                   "app":"guestbook"
                }
             },
             "spec":{
                "containers":[
                   {
                      "name":"guestbook",
                      "image":"k8s.gcr.io/guestbook:v3",
                      "ports":[
                         {
                            "name":"http-server",
                            "containerPort":3000
                         }
                      ]
                   }
                ]
             }
          }
       }
    }
    EOF
    
    kubectl apply -f frontend-controller.json
    replicationcontroller "guestbook" created
    

    %sh
    cat << EOF > frontend-service.json
    {
       "kind":"Service",
       "apiVersion":"v1",
       "metadata":{
          "name":"guestbook",
          "labels":{
             "app":"guestbook"
          }
       },
       "spec":{
          "ports": [
             {
               "port":3000,
               "targetPort":"http-server"
             }
          ],
          "selector":{
             "app":"guestbook"
          },
          "type": "LoadBalancer"
       }
    }
    EOF
    
    kubectl apply -f frontend-service.json
    service "guestbook" created
    

    %sh
    kubectl get services
    NAME           TYPE           CLUSTER-IP       EXTERNAL-IP        PORT(S)          AGE
    guestbook      LoadBalancer   172.20.223.78    a894ee3c7ca9e...   3000:30068/TCP   2m
    kubernetes     ClusterIP      172.20.0.1       <none>             443/TCP          36m
    redis-master   ClusterIP      172.20.70.52     <none>             6379/TCP         7m
    redis-slave    ClusterIP      172.20.213.112   <none>             6379/TCP         5m
    

    EXTERNAL-IP が見切れているときは kubectl get services-o wide を渡します。

    %sh
    kubectl get services -o wide
    NAME           TYPE           CLUSTER-IP       EXTERNAL-IP                                                              PORT(S)          AGE       SELECTOR
    guestbook      LoadBalancer   172.20.223.78    a894ee3c7ca9e11e8a49b020fd078860-812237424.us-east-1.elb.amazonaws.com   3000:30068/TCP   2m        app=guestbook
    kubernetes     ClusterIP      172.20.0.1       <none>                                                                   443/TCP          36m       <none>
    redis-master   ClusterIP      172.20.70.52     <none>                                                                   6379/TCP         7m        app=redis,role=master
    redis-slave    ClusterIP      172.20.213.112   <none>                                                                   6379/TCP         5m        app=redis,role=slave
    

    接続してみましょう(ロードバランサーに設定が反映されるまで若干時間が掛かるのに注意しましょう)。

    %sh
    curl -s http://a894ee3c7ca9e11e8a49b020fd078860-812237424.us-east-1.elb.amazonaws.com:3000
    <!DOCTYPE html>
    <html lang="en">
      <head>
        <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
        <meta charset="utf-8">
        <meta content="width=device-width" name="viewport">
        <link href="style.css" rel="stylesheet">
        <title>Guestbook</title>
      </head>
      <body>
        <div id="header">
          <h1>Guestbook</h1>
        </div>
    
        <div id="guestbook-entries">
          <p>Waiting for database connection...</p>
        </div>
    
        </div>
          <form id="guestbook-form">
            <input autocomplete="off" id="guestbook-entry-content" type="text">
            <a href="#" id="guestbook-submit">Submit</a>
          </form>
        </div>
    
        </div>
          <p><h2 id="guestbook-host-address"></h2></p>
          <p><a href="env">/env</a>
          <a href="info">/info</a></p>
        </div>
        <script src="//ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>
        <script src="script.js"></script>
      </body>
    </html>
    

    ハマった点は

    • nodegroup作成時にCloudFormationのテンプレートURLを間違えていた-
      • 気をつけましょう… :-|
      • https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2018-08-30/amazon-eks-nodegroup.yaml を使うと動作しました
    • Service の EXTERNAL-IP が PENDING のまま変わらない
      • public subnetを一つしか割り当ててなかったのが原因のようです

    おまけ: CloudFormationテンプレート(nodegroup)の差分

    %sh
    diff <(curl -s https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-06-05/amazon-eks-nodegroup.yaml) <(curl -s https://amazon-eks.s3-us-west-2.amazonaws.com/cloudformation/2018-08-30/amazon-eks-nodegroup.yaml)
    echo
    3c3
    < Description: 'Amazon EKS - Node Group'
    ---
    > Description: 'Amazon EKS - Node Group - Released 2018-08-30'
    75c75
    <     ConstraintDescription: must be a valid EC2 instance type
    ---
    >     ConstraintDescription: Must be a valid EC2 instance type
    86a87,91
    >   NodeVolumeSize:
    >     Type: Number
    >     Description: Node volume size
    >     Default: 20
    > 
    88c93,98
    <     Description: The cluster name provided when the cluster was created.  If it is incorrect, nodes will not be able to join the cluster.
    ---
    >     Description: The cluster name provided when the cluster was created. If it is incorrect, nodes will not be able to join the cluster.
    >     Type: String
    > 
    >   BootstrapArguments:
    >     Description: Arguments to pass to the bootstrap script. See files/bootstrap.sh in https://github.com/awslabs/amazon-eks-ami
    >     Default: ""
    107,219d116
    < Mappings:
    <   MaxPodsPerNode:
    <     c4.large:
    <       MaxPods: 29
    <     c4.xlarge:
    <       MaxPods: 58
    <     c4.2xlarge:
    <       MaxPods: 58
    <     c4.4xlarge:
    <       MaxPods: 234
    <     c4.8xlarge:
    <       MaxPods: 234
    <     c5.large:
    <       MaxPods: 29
    <     c5.xlarge:
    <       MaxPods: 58
    <     c5.2xlarge:
    <       MaxPods: 58
    <     c5.4xlarge:
    <       MaxPods: 234
    <     c5.9xlarge:
    <       MaxPods: 234
    <     c5.18xlarge:
    <       MaxPods: 737
    <     i3.large:
    <       MaxPods: 29
    <     i3.xlarge:
    <       MaxPods: 58
    <     i3.2xlarge:
    <       MaxPods: 58
    <     i3.4xlarge:
    <       MaxPods: 234
    <     i3.8xlarge:
    <       MaxPods: 234
    <     i3.16xlarge:
    <       MaxPods: 737
    <     m3.medium:
    <       MaxPods: 12
    <     m3.large:
    <       MaxPods: 29
    <     m3.xlarge:
    <       MaxPods: 58
    <     m3.2xlarge:
    <       MaxPods: 118
    <     m4.large:
    <       MaxPods: 20
    <     m4.xlarge:
    <       MaxPods: 58
    <     m4.2xlarge:
    <       MaxPods: 58
    <     m4.4xlarge:
    <       MaxPods: 234
    <     m4.10xlarge:
    <       MaxPods: 234
    <     m5.large:
    <       MaxPods: 29
    <     m5.xlarge:
    <       MaxPods: 58
    <     m5.2xlarge:
    <       MaxPods: 58
    <     m5.4xlarge:
    <       MaxPods: 234
    <     m5.12xlarge:
    <       MaxPods: 234
    <     m5.24xlarge:
    <       MaxPods: 737
    <     p2.xlarge:
    <       MaxPods: 58
    <     p2.8xlarge:
    <       MaxPods: 234
    <     p2.16xlarge:
    <       MaxPods: 234
    <     p3.2xlarge:
    <       MaxPods: 58
    <     p3.8xlarge:
    <       MaxPods: 234
    <     p3.16xlarge:
    <       MaxPods: 234
    <     r3.xlarge:
    <       MaxPods: 58
    <     r3.2xlarge:
    <       MaxPods: 58
    <     r3.4xlarge:
    <       MaxPods: 234
    <     r3.8xlarge:
    <       MaxPods: 234
    <     r4.large:
    <       MaxPods: 29
    <     r4.xlarge:
    <       MaxPods: 58
    <     r4.2xlarge:
    <       MaxPods: 58
    <     r4.4xlarge:
    <       MaxPods: 234
    <     r4.8xlarge:
    <       MaxPods: 234
    <     r4.16xlarge:
    <       MaxPods: 737
    <     t2.small:
    <       MaxPods: 8
    <     t2.medium:
    <       MaxPods: 17
    <     t2.large:
    <       MaxPods: 35
    <     t2.xlarge:
    <       MaxPods: 44
    <     t2.2xlarge:
    <       MaxPods: 44
    <     x1.16xlarge:
    <       MaxPods: 234
    <     x1.32xlarge:
    <       MaxPods: 234
    < 
    237a135
    >           - NodeVolumeSize
    238a137
    >           - BootstrapArguments
    315a215,236
    >   NodeSecurityGroupFromControlPlaneOn443Ingress:
    >     Type: AWS::EC2::SecurityGroupIngress
    >     DependsOn: NodeSecurityGroup
    >     Properties:
    >       Description: Allow pods running extension API servers on port 443 to receive communication from cluster control plane
    >       GroupId: !Ref NodeSecurityGroup
    >       SourceSecurityGroupId: !Ref ClusterControlPlaneSecurityGroup
    >       IpProtocol: tcp
    >       FromPort: 443
    >       ToPort: 443
    > 
    >   ControlPlaneEgressToNodeSecurityGroupOn443:
    >     Type: AWS::EC2::SecurityGroupEgress
    >     DependsOn: NodeSecurityGroup
    >     Properties:
    >       Description: Allow the cluster control plane to communicate with pods running extension API servers on port 443
    >       GroupId: !Ref ClusterControlPlaneSecurityGroup
    >       DestinationSecurityGroupId: !Ref NodeSecurityGroup
    >       IpProtocol: tcp
    >       FromPort: 443
    >       ToPort: 443
    > 
    357a279,284
    >       BlockDeviceMappings:
    >         - DeviceName: /dev/xvda
    >           Ebs:
    >             VolumeSize: !Ref NodeVolumeSize
    >             VolumeType: gp2
    >             DeleteOnTermination: true
    360,394c287,294
    <           Fn::Join: [
    <             "",
    <             [
    <               "#!/bin/bash -xe\n",
    <               "CA_CERTIFICATE_DIRECTORY=/etc/kubernetes/pki", "\n",
    <               "CA_CERTIFICATE_FILE_PATH=$CA_CERTIFICATE_DIRECTORY/ca.crt", "\n",
    <               "MODEL_DIRECTORY_PATH=~/.aws/eks", "\n",
    <               "MODEL_FILE_PATH=$MODEL_DIRECTORY_PATH/eks-2017-11-01.normal.json", "\n",
    <               "mkdir -p $CA_CERTIFICATE_DIRECTORY", "\n",
    <               "mkdir -p $MODEL_DIRECTORY_PATH", "\n",
    <               "curl -o $MODEL_FILE_PATH https://s3-us-west-2.amazonaws.com/amazon-eks/1.10.3/2018-06-05/eks-2017-11-01.normal.json", "\n",
    <               "aws configure add-model --service-model file://$MODEL_FILE_PATH --service-name eks", "\n",
    <               "aws eks describe-cluster --region=", { Ref: "AWS::Region" }," --name=", { Ref: ClusterName }," --query 'cluster.{certificateAuthorityData: certificateAuthority.data, endpoint: endpoint}' > /tmp/describe_cluster_result.json", "\n",
    <               "cat /tmp/describe_cluster_result.json | grep certificateAuthorityData | awk '{print $2}' | sed 's/[,\"]//g' | base64 -d >  $CA_CERTIFICATE_FILE_PATH", "\n",
    <               "MASTER_ENDPOINT=$(cat /tmp/describe_cluster_result.json | grep endpoint | awk '{print $2}' | sed 's/[,\"]//g')", "\n",
    <               "INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)", "\n",
    <               "sed -i s,MASTER_ENDPOINT,$MASTER_ENDPOINT,g /var/lib/kubelet/kubeconfig", "\n",
    <               "sed -i s,CLUSTER_NAME,", { Ref: ClusterName }, ",g /var/lib/kubelet/kubeconfig", "\n",
    <               "sed -i s,REGION,", { Ref: "AWS::Region" }, ",g /etc/systemd/system/kubelet.service", "\n",
    <               "sed -i s,MAX_PODS,", { "Fn::FindInMap": [ MaxPodsPerNode, { Ref: NodeInstanceType }, MaxPods ] }, ",g /etc/systemd/system/kubelet.service", "\n",
    <               "sed -i s,MASTER_ENDPOINT,$MASTER_ENDPOINT,g /etc/systemd/system/kubelet.service", "\n",
    <               "sed -i s,INTERNAL_IP,$INTERNAL_IP,g /etc/systemd/system/kubelet.service", "\n",
    <               "DNS_CLUSTER_IP=10.100.0.10", "\n",
    <               "if [[ $INTERNAL_IP == 10.* ]] ; then DNS_CLUSTER_IP=172.20.0.10; fi", "\n",
    <               "sed -i s,DNS_CLUSTER_IP,$DNS_CLUSTER_IP,g  /etc/systemd/system/kubelet.service", "\n",
    <               "sed -i s,CERTIFICATE_AUTHORITY_FILE,$CA_CERTIFICATE_FILE_PATH,g /var/lib/kubelet/kubeconfig" , "\n",
    <               "sed -i s,CLIENT_CA_FILE,$CA_CERTIFICATE_FILE_PATH,g  /etc/systemd/system/kubelet.service" , "\n",
    <               "systemctl daemon-reload", "\n",
    <               "systemctl restart kubelet", "\n",
    <               "/opt/aws/bin/cfn-signal -e $? ",
    <               "         --stack ", { Ref: "AWS::StackName" },
    <               "         --resource NodeGroup ",
    <               "         --region ", { Ref: "AWS::Region" }, "\n"
    <             ]
    <           ]
    ---
    >           !Sub |
    >             #!/bin/bash
    >             set -o xtrace
    >             /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
    >             /opt/aws/bin/cfn-signal --exit-code $? \
    >                      --stack  ${AWS::StackName} \
    >                      --resource NodeGroup  \
    >                      --region ${AWS::Region}
    400d299
    < 
    
    

    初期化の部分が変わっているようにみえる。

    片付け

    終わったら片付けましょう

    %sh
    aws cloudformation delete-stack --stack-name hello-eks-nodegroup
    aws cloudformation delete-stack --stack-name hello-eks-vpc
    aws eks delete-cluster --name hello-eks --query cluster.status
    "DELETING"
    

    // 雑感: Replication ControllerとDeploymentの違いがよく分かってない気がする :-|