Kubernetes学习笔记(3)：Java应用容器化 & 迁移到k8s

发表于 更新于 分类于 本文字数： 2.1k 阅读时长 ≈ 8 分钟

本文梳理不同Java应用如何迁移到k8s。

定时任务（Maven）

确定基础镜像

openjdk:8-jre-alpine

搞定服务运行的相关文件

查看Java源码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
package com.mooc.demo.cronjob;

import java.util.Random;

public class Main {

    public static void main(String args[]) {

        Random r = new Random();
        int time = r.nextInt(20)+10;
        System.out.println("I will working for "+time+" seconds!");

        try{
            Thread.sleep(time*1000);
        }catch (Exception e) {
            e.printStackTrace();
        }
        System.out.println("All work is done! Bye!");
    }
}

用Maven将应用打成jar包

1
mvn package

确保应用可以直接运行：

1
java -jar xxx.jar

或采用library方式运行：

1
java -cp xxx.jar com.mooc.xxx.Main

构建Dockerfile

1
2
3
4
5
FROM hub.wangfanggang.com/kubernetes/openjdk:8-jre-alpine

COPY target/cronjob-demo-1.0-SNAPSHOT.jar /cronjob-demo.jar

ENTRYPOINT ["java", "-cp", "/cronjob-demo.jar", "com.mooc.demo.cronjob.Main"]

编写k8s配置文件

cronjob.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cronjob-demo
spec:
  schedule: "*/1 * * * *"
  successfulJobsHistoryLimit: 3
  suspend: false
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        metadata:
          labels:
            app: cronjob-demo
        spec:
          restartPolicy: Never
          containers:
          - name: cronjob-demo
            image: hub.wangfanggang.com/kubernetes/cronjob:v1

部署到k8s

1
kubectl apply -f cronjob.yaml

检查服务是否正常运行

1
kubectl get cronjob

刚部署时ACTIVE=0,过一段时间后，cronjob被自动调度起来，ACTIVE=1

查看pod信息

1
kubectl get pods -o wide

可以看到cronjob服务被部署在了w2节点上，而且如我们预期，k8s保留了3次成功的pod信息。

切换到w2上检查docker是否正常运行了

1
docker ps -a | grep cronjob

通过 docker logs [container_id] 可以看到cronjob正常完成了。

如果发现pod的状态是ErrDockerPull，说明节点从镜像仓库下载镜像时出错了，切换到对应节点，用docker login hub.wangfanggang.com检查是否可以访问镜像仓库。

SpringBoot

将springboot打成jar包

DemoController.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
package com.mooc.demo.controller;

import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class DemoController {

    @RequestMapping("/hello")
    public String sayHello(@RequestParam String name) {

        return "Hello "+name+"! I'm springboot-web-demo controller!";

    }
}

用maven将springboot应用打包

1
mvn package

查看生成的jar文件

1
jar -tf springboot-web-demo-1.0-SNAPSHOT.jar

运行jar包，检查服务是否可以正常启动。

1
java -jar target/springboot-web-demo-1.0-SNAPSHOT.jar

打开浏览器，访问一下hello服务 http://localhost:8080/hello?name=kenny

可以看到服务正常返回结果了，说明我们的jar包没问题。

编写Dockerfile

1
2
3
4
5
FROM hub.wangfanggang.com/kubernetes/openjdk:8-jre-alpine

COPY target/springboot-web-demo-1.0-SNAPSHOT.jar /springboot-web.jar

ENTRYPOINT ["java", "-jar", "/springboot-web.jar"]

构建Docker镜像

1
docker build -t springboot-web:v1 .

测试Docker容器

1
docker run -it -p 8080:8080 springboot-web:v1

可以看到springboot服务可以正常启动。

编写k8s配置文件

这里的k8s服务发现策略是用ingress-nginx来做服务发现。

springboot-web.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
#deploy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: springboot-web-demo
spec:
  selector:
    matchLabels:
      app: springboot-web-demo
  replicas: 1
  template:
    metadata:
      labels:
        app: springboot-web-demo
    spec:
      containers:
      - name: springboot-web-demo
        image: hub.wangfanggang.com/kubernetes/springboot-web:v1
        ports:
        - containerPort: 8080
---
#service
apiVersion: v1
kind: Service
metadata:
  name: springboot-web-demo
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: springboot-web-demo
  type: ClusterIP

---
#ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: springboot-web-demo
spec:
  rules:
  - host: springboot.xyz.com
    http:
      paths:
      - path: /
        backend:
          serviceName: springboot-web-demo
          servicePort: 80

可以看到，配置文件里ingress下添加了跳转规则，凡是访问springboot.xyz.com的，都会自动跳转到springboot-web-demo这个服务上，且端口是80。

部署springboot服务到k8s

1
kubectl apply -f springboot-web.yaml

测试服务是否正常，访问http://springboot.xyz.com/hello?name=kwang

说明springboot服务已被k8s正常调度了。

传统Dubbo应用

Dubbo源码介绍

dubbo-demo-api模块定义的API接口DemoService.java

1
2
3
4
5
6
package com.mooc.demo.api;

public interface DemoService {
    String sayHello(String name);

}

dubbo-demo模块下实现API接口DemoServiceImpl.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
package com.mooc.demo.service;

import com.mooc.demo.api.DemoService;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

public class DemoServiceImpl implements DemoService {

    private static final Logger log = LoggerFactory.getLogger(DemoServiceImpl.class);

    public String sayHello(String name) {

        log.debug("dubbo say hello to : {}", name);

        return "Hello "+name;
    }

}

dubbo.properties

1
2
3
4
5
dubbo.application.name=demo
dubbo.registry.address=zookeeper://10.155.20.62:2181
dubbo.spring.config=classpath*:spring/provider.xml
dubbo.protocol.name=dubbo
dubbo.protocol.port=20880

start.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
#!/bin/bash

cd `dirname $0`
BIN_DIR=`pwd`
cd ..

DEPLOY_DIR=`pwd`
CONF_DIR=${DEPLOY_DIR}/conf

SERVER_NAME=`sed '/dubbo.application.name/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'`
SERVER_PORT=`sed '/dubbo.protocol.port/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'`

if [ -z "${SERVER_NAME}" ]; then
    echo "ERROR: can not found 'dubbo.application.name' config in 'dubbo.properties' !"
  exit 1
fi

PIDS=`ps  --no-heading -C java -f --width 1000 | grep "${CONF_DIR}" |awk '{print $2}'`
if [ -n "${PIDS}" ]; then
    echo "ERROR: The ${SERVER_NAME} already started!"
    echo "PID: ${PIDS}"
    exit 1
fi

if [ -n "${SERVER_PORT}" ]; then
  SERVER_PORT_COUNT=`netstat -ntl | grep ${SERVER_PORT} | wc -l`
  if [ ${SERVER_PORT_COUNT} -gt 0 ]; then
    echo "ERROR: The ${SERVER_NAME} port ${SERVER_PORT} already used!"
    exit 1
  fi
fi

LOGS_DIR=""
if [ -n "${LOGS_FILE}" ]; then
  LOGS_DIR=`dirname ${LOGS_FILE}`
else
  LOGS_DIR=${DEPLOY_DIR}/logs
fi
if [ ! -d ${LOGS_DIR} ]; then
  mkdir ${LOGS_DIR}
fi
STDOUT_FILE=${LOGS_DIR}/stdout.log

LIB_DIR=${DEPLOY_DIR}/lib
LIB_JARS=`ls ${LIB_DIR} | grep .jar | awk '{print "'${LIB_DIR}'/"$0}'|tr "\n" ":"`

JAVA_OPTS=" -Djava.awt.headless=true -Djava.net.preferIPv4Stack=true "
JAVA_DEBUG_OPTS=""
if [ "$1" = "debug" ]; then
    JAVA_DEBUG_OPTS=" -Xdebug -Xnoagent -Djava.compiler=NONE -Xrunjdwp:transport=dt_socket,address=8000,server=y,suspend=n "
fi

echo -e "Starting the ${SERVER_NAME} ...\c"

nohup ${JAVA_HOME}/bin/java -Dapp.name=${SERVER_NAME} ${JAVA_OPTS} ${JAVA_DEBUG_OPTS} ${JAVA_JMX_OPTS} -classpath ${CONF_DIR}:${LIB_JARS} com.alibaba.dubbo.container.Main >> ${STDOUT_FILE} 2>&1 &

PIDS=`ps  --no-heading -C java -f --width 1000 | grep "${DEPLOY_DIR}" | awk '{print $2}'`
echo "PID: ${PIDS}"
echo "STDOUT: ${STDOUT_FILE}"

stop.sh

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
#!/bin/bash

cd `dirname $0`
BIN_DIR=`pwd`
cd ..
DEPLOY_DIR=`pwd`
CONF_DIR=${DEPLOY_DIR}/conf

SERVER_NAME=`sed '/dubbo.application.name/!d;s/.*=//' conf/dubbo.properties | tr -d '\r'`

if [ -z "${SERVER_NAME}" ]; then
    echo "ERROR: can not found 'dubbo.application.name' config in 'dubbo.properties' !"
  exit 1
fi

PIDS=`ps -ef | grep java | grep "${CONF_DIR}" |awk '{print $2}'`
if [ -z "$PIDS" ]; then
    echo "ERROR: The ${SERVER_NAME} does not started!"
    exit 1
fi

echo -e "Stopping the ${SERVER_NAME} ...\c"
for PID in ${PIDS} ; do
    kill ${PID} > /dev/null 2>&1
done

COUNT=0
while [ ${COUNT} -lt 1 ]; do
    echo -e ".\c"
    sleep 1
    COUNT=1
    for PID in ${PIDS} ; do
        PID_EXIST=`ps -f -p ${PID} | grep java`
        if [ -n "${PID_EXIST}" ]; then
            COUNT=0
            break
        fi
    done
done

echo "OK!"
echo "PID: $PIDS"

由于没有用SpringBoot，需要自己来实现打包。

打包Dubbo应用

安装必要的API接口服务

1
2
3
cd dubbo-demo-api

mvn install

执行mvn package将应用打包。

1
2
3
cd dubbo-demo

mvn package

打包成功，查看打包好的文件是否包含了我们需要的所有文件。

1
tar -tf target/dubbo-demo-1.0-SNAPSHOT-assembly.tar.gz

测试打包文件，运行start.sh脚本启动应用。

1
2
3
4
5
cd target 

tar zxvf dubbo-demo-1.0-SNAPSHOT-assembly.tar.gz

sh bin/start.sh

测试dubbo服务是否正常。

编写Dockerfile

1
2
3
4
5
FROM hub.wangfanggang.com/kubernetes/openjdk:8-jre-alpine

COPY target/ROOT /ROOT

ENTRYPOINT ["sh", "/ROOT/bin/start.sh"]

注意：由于我们的start.sh脚本里是以后台启动的dubbo服务（nohup）,这样的话一旦脚本运行完毕，docker容器也会推出，所以需要把nohup去掉，让脚本在前台执行，避免docker容器自动推出。

构建Docker镜像

1
docker build -t dubbo:v1 .

测试Doker镜像

1
docker run -it dubbo:v1

确定k8s服务部署策略

采用hostNetwork方式部署dubbo服务，需要修改Dockerfile动态修改服务端口。

重新构建docker镜像。

将Docker镜像推送到镜像仓库上

1
2
3
docker tag dubbo:v1 hub.wangfanggang.com/kubernetes/dubbo:v1

docker push hub.wangfanggang.com/kubernetes/dubbo:v1

编写k8s配置文件

dubbo.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
#deploy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dubb-demo
spec:
  selector:
    matchLabels:
      app: dubb-demo
  replicas: 1
  template:
    metadata:
      labels:
        app: dubb-demo
    spec:
      hostNetwork: true
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
              - key: app
                operator: In
                values:
                - dubb-demo
            topologyKey: "kubernetes.io/hostname"
      containers:
      - name: dubb-demo
        image: hub.mooc.com/kubernetes/dubbo:v1
        ports:
        - containerPort: 20880

注意：这里的podAntiAffinity可以阻止服务被部署在同一个节点上，发生端口冲突。

在k8s配置文件里定义一个环境变量DUBBO_PORT, 对应我们刚才修改的start.sh里的变量。

部署k8s服务

1
kubectl apply -f dubbo.yaml

测试k8s服务

迁移传统的Web服务

查看Java源码

DemoController.java

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
package com.mooc.demo.controller;

import com.mooc.demo.api.DemoService;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.ResponseBody;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.context.ContextLoader;

@Controller
public class DemoController {

    private static final Logger log = LoggerFactory.getLogger(DemoController.class);

    @Autowired
    private DemoService demoService;

    @RequestMapping("/hello")
    @ResponseBody
    public String sayHello(@RequestParam String name) {

        log.debug("say hello to :{}", name);

        String message = demoService.sayHello(name);

        log.debug("dubbo result:{}", message);

        return message;

    }
}

applicationContext-service-config.xml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dubbo="http://code.alibabatech.com/schema/dubbo"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://code.alibabatech.com/schema/dubbo
        http://code.alibabatech.com/schema/dubbo/dubbo.xsd">

  <dubbo:application name="demo" />

  <dubbo:registry address="zookeeper://10.155.20.62:2181" />

  <dubbo:consumer retries="3" timeout="60000" />

</beans>

applicationContext.xml

1
2
3
4
5
6
7
8
9
10
11
12
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:dubbo="http://code.alibabatech.com/schema/dubbo"
       xsi:schemaLocation="http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://code.alibabatech.com/schema/dubbo
        http://code.alibabatech.com/schema/dubbo/dubbo.xsd"
       default-autowire="byName" default-lazy-init="false">

    <dubbo:reference id="demoService" interface="com.mooc.demo.api.DemoService" />

</beans>

确定基础镜像

因为是传统的Web服务，需要选择一个tomcat容器。

打包Java文件

1
mvn package

将打包文件解压到待安装目录中

1
2
3
4
5
6
7
8
cd target
mkdir ROOT
mv web-demo-1.0-SNAPSHOT.war ROOT/

cd ROOT
jar -xvf web-demo-1.0-SNAPSHOT.war

rm web-demo-1.0-SNAPSHOT.war

编写Dockerfile

1
2
3
4
5
6
7
FROM hub.wangfanggang.com/kubernetes/tomcat:8.0.51-alpine

COPY ROOT /usr/local/tomcat/webapps/ROOT

COPY dockerfiles/start.sh /usr/local/tomcat/bin/start.sh

ENTRYPOINT ["sh" , "/usr/local/tomcat/bin/start.sh"]

编写一个脚本文件，在tomcat启动后，执行一个tail命令，以便让脚本在容器中挂起。

start.sh

1
2
3
4
5
#!/bin/bash

sh /usr/local/tomcat/bin/startup.sh

tail -f /usr/local/tomcat/logs/catalina.out

构建Docker镜像

1
docker build -t web:v1 .

测试Docker镜像

1
docker run -it web:v1

将Docker镜像上传到镜像仓库

1
docker tag web:v1 hub.wangfanggang.com/kubernetes/web:v1

确定k8s服务发现策略

使用ingress-nginx

编写k8s配置文件

web.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
#deploy
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-demo
spec:
  selector:
    matchLabels:
      app: web-demo
  replicas: 1
  template:
    metadata:
      labels:
        app: web-demo
    spec:
      containers:
      - name: web-demo
        image: hub.mooc.com/kubernetes/web:v1
        ports:
        - containerPort: 8080
---
#service
apiVersion: v1
kind: Service
metadata:
  name: web-demo
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: web-demo
  type: ClusterIP

---
#ingress
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web-demo
spec:
  rules:
  - host: web.mooc.com
    http:
      paths:
      - path: /
        backend:
          serviceName: web-demo
          servicePort: 80

部署到k8s

1
kubectl apply -f web.yaml

测试k8s服务

访问 http://web.mooc.com/hello/name=小姐姐