学习 Druid(六):Druid on Kubernetes

更新至 Druid 0.15.1 版本

Docker 镜像

Apache Druid 官方提供了 all-in-one 的 Docker 镜像,然而并不适合部署到 Kubernetes 集群。

以一个容器对应一个进程的原则,自己制作了 Docker 镜像,已上传至 Docker Hub:

架构

Druid on Kubernetes 架构,如下图所示:

Druid on K8s

前提:已部署 Zookeeper 集群、作为 Metadata 的 MySQL、作为 DeepStorage 的 HDFS。

资源定义

配置

为了通过 druid.host 配置项访问 MiddleManager 和 Historical,需要使用 Jinja2 模板引擎在 InitContainer 里生成配置。

删除 conf/_common/common.runtime.properties 文件中 druid.host 配置项。

重命名 conf/data/middleManager 目录下 runtime.properties 为 runtime.properties.jinja2,重命名 conf/data/historical 目录下 runtime.properties 为 runtime.properties.jinja2,编辑 jinja2 模板文件,添加 druid.host 配置项:

druid.host={{ POD_NAME }}.{{ SERVICE_NAME }}  

创建 ConfigMap:

kubectl create configmap druid-common-cm \  
    --from-file=conf/_common/common.runtime.properties \
    --from-file=conf/_common/log4j2.xml \
    --from-file=conf/_common/core-site.xml \
    --from-file=conf/_common/hdfs-site.xml

kubectl create configmap druid-coordinator-overlord-cm \  
    --from-file=conf/master/coordinator-overlord/runtime.properties

kubectl create configmap druid-broker-cm \  
    --from-file=conf/query/broker/runtime.properties

kubectl create configmap druid-router-cm \  
    --from-file=conf/query/router/runtime.properties

kubectl create configmap druid-middlemanager-cm \  
    --from-file=conf/data/middleManager/runtime.properties.jinja2

kubectl create configmap druid-historical-cm \  
    --from-file=conf/data/historical/runtime.properties.jinja2

Coordinator Overlord

coordinator-orverlord.yaml

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: druid-coordinator-overlord-deploy
spec:  
  replicas: 1
  selector:
    matchLabels:
      app: druid-coordinator-overlord
  template:
    metadata:
      labels:
        app: druid-coordinator-overlord
    spec:
      containers:
        - name: druid-coordinator-overlord
          image: dyingbleed/druid-coordinator-overlord:0.15.1-incubating
          ports:
            - containerPort: 8081
          volumeMounts:
            - name: common
              mountPath: /etc/druid/_common
            - name: coordinator-overlord
              mountPath: /etc/druid/coordinator-overlord
      hostname: druid-coordinator-overlord
      volumes:
        - name: common
          configMap:
            name: druid-common-cm
            items:
              - key: common.runtime.properties
                path: common.runtime.properties
              - key: log4j2.xml
                path: log4j2.xml
        - name: coordinator-overlord
          configMap:
            name: druid-coordinator-overlord-cm
            items:
              - key: runtime.properties
                path: runtime.properties
---
apiVersion: v1  
kind: Service  
metadata:  
  name: druid-coordinator-overlord
spec:  
  selector:
    app: druid-coordinator-overlord
  ports:
    - port: 8081
      targetPort: 8081

Broker

broker.yaml

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: druid-broker-deploy
spec:  
  replicas: 1
  selector:
    matchLabels:
      app: druid-broker
  template:
    metadata:
      labels:
        app: druid-broker
    spec:
      containers:
        - name: druid-broker
          image: dyingbleed/druid-broker:0.15.1-incubating
          ports:
            - containerPort: 8082
          volumeMounts:
            - name: common
              mountPath: /etc/druid/_common
            - name: broker
              mountPath: /etc/druid/broker
      hostname: druid-broker
      volumes:
        - name: common
          configMap:
            name: druid-common-cm
            items:
              - key: common.runtime.properties
                path: common.runtime.properties
              - key: log4j2.xml
                path: log4j2.xml
        - name: broker
          configMap:
            name: druid-broker-cm
            items:
              - key: runtime.properties
                path: runtime.properties
---
apiVersion: v1  
kind: Service  
metadata:  
  name: druid-broker
spec:  
  selector:
    app: druid-broker
  ports:
    - port: 8082
      targetPort: 8082

Router

router.yaml

apiVersion: apps/v1  
kind: Deployment  
metadata:  
  name: druid-router-deploy
spec:  
  replicas: 1
  selector:
    matchLabels:
      app: druid-router
  template:
    metadata:
      labels:
        app: druid-router
    spec:
      containers:
        - name: druid-router
          image: dyingbleed/druid-router:0.15.1-incubating
          ports:
            - containerPort: 8888
          volumeMounts:
            - name: common
              mountPath: /etc/druid/_common
            - name: router
              mountPath: /etc/druid/router
      hostname: druid-router
      volumes:
        - name: common
          configMap:
            name: druid-common-cm
            items:
              - key: common.runtime.properties
                path: common.runtime.properties
              - key: log4j2.xml
                path: log4j2.xml
        - name: router
          configMap:
            name: druid-router-cm
            items:
              - key: runtime.properties
                path: runtime.properties
---
apiVersion: v1  
kind: Service  
metadata:  
  name: druid-router
spec:  
  selector:
    app: druid-router
  ports:
    - port: 8888
      targetPort: 8888

MiddleManager

middlemanager.yaml

apiVersion: apps/v1  
kind: StatefulSet  
metadata:  
  name: druid-middlemanager-sts
spec:  
  serviceName: druid-middlemanager
  replicas: 1
  selector:
    matchLabels:
      app: druid-middlemanager
  template:
    metadata:
      labels:
        app: druid-middlemanager
    spec:
      initContainers:
        - name: jinja2
          image: dyingbleed/jinja2
          command: [ "/bin/sh", "-c" ]
          args: 
            - "cat /opt/jinja2/runtime.properties.jinja2 | python /opt/render.py > /opt/output/runtime.properties"
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: SERVICE_NAME
              value: druid-middlemanager
          volumeMounts:
            - name: middlemanager
              mountPath: /opt/jinja2
            - name: share
              mountPath: /opt/output
      containers:
        - name: druid-middlemanager
          image: dyingbleed/druid-middlemanager:0.15.1-incubating
          ports:
            - containerPort: 8091
          volumeMounts:
            - name: common
              mountPath: /etc/druid/_common
            - name: share
              mountPath: /etc/druid/middleManager
            - name: hdfs
              mountPath: /etc/hadoop/conf
      volumes:
        - name: common
          configMap:
            name: druid-common-cm
            items:
              - key: common.runtime.properties
                path: common.runtime.properties
              - key: log4j2.xml
                path: log4j2.xml
        - name: middlemanager
          configMap:
            name: druid-middlemanager-cm
            items:
              - key: runtime.properties.jinja2
                path: runtime.properties.jinja2
        - name: hdfs
          configMap:
            name: druid-common-cm
            items:
              - key: core-site.xml
                path: core-site.xml
              - key: hdfs-site.xml
                path: hdfs-site.xml
        - name: share
          emptyDir: {}
---
apiVersion: v1  
kind: Service  
metadata:  
  name: druid-middlemanager
spec:  
  type: ClusterIP
  clusterIP: None
  selector:
    app: druid-middlemanager
  ports:
    - port: 8091
      targetPort: 8091

Historical

historical.yaml

apiVersion: apps/v1  
kind: StatefulSet  
metadata:  
  name: druid-historical-sts
spec:  
  serviceName: druid-historical
  replicas: 1
  selector:
    matchLabels:
      app: druid-historical
  template:
    metadata:
      labels:
        app: druid-historical
    spec:
      initContainers:
        - name: jinja2
          image: dyingbleed/jinja2
          command: [ "/bin/sh", "-c" ]
          args: 
            - "cat /opt/jinja2/runtime.properties.jinja2 | python /opt/render.py > /opt/output/runtime.properties"
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: SERVICE_NAME
              value: druid-historical
          volumeMounts:
            - name: historical
              mountPath: /opt/jinja2
            - name: share
              mountPath: /opt/output
      containers:
        - name: druid-historical
          image: dyingbleed/druid-historical:0.15.1-incubating
          ports:
            - containerPort: 8083
          volumeMounts:
            - name: common
              mountPath: /etc/druid/_common
            - name: share
              mountPath: /etc/druid/historical
            - name: hdfs
              mountPath: /etc/hadoop/conf
      volumes:
        - name: common
          configMap:
            name: druid-common-cm
            items:
              - key: common.runtime.properties
                path: common.runtime.properties
              - key: log4j2.xml
                path: log4j2.xml
        - name: historical
          configMap:
            name: druid-historical-cm
            items:
              - key: runtime.properties.jinja2
                path: runtime.properties.jinja2
        - name: hdfs
          configMap:
            name: druid-common-cm
            items:
              - key: core-site.xml
                path: core-site.xml
              - key: hdfs-site.xml
                path: hdfs-site.xml
        - name: share
          emptyDir: {}
---
apiVersion: v1  
kind: Service  
metadata:  
  name: druid-historical
spec:  
  type: ClusterIP
  clusterIP: None
  selector:
    app: druid-historical
  ports:
    - port: 8083
      targetPort: 8083

优化项:

  • 为容器配置 resources.limits,对 CPU 和内存资源进行限制;
  • 为容器配置 livenessProbe.httpGet,调用 /status/health 接口检测进程健康情况;
  • 推荐使用 PersistentVolumeClaim 为 Segment 数据申请 PersistentVolume 资源存储文件。