Running Twemproxy In Kubernetes

In this post, i would like share on how to setup twemproxy for redis cluster sharding running in kubernetes.

Setup
These are the tasks required:

Running twemproxy as a single replica deployment in kubernetes.
Running redis replicas as statefulsets with with in kubernetes.
Allowing other pods in kubernetes to access twemproxy.
Allowing twemproxy to connect to the redis’s pods.
Twemproxy
We can run twemproxy as a single deployment in kubernetes. We can use the following community image jgoodall/twemproxy

twemproxy’s deployment & service spec:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: twemproxy
  name: twemproxy
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: twemproxy
    spec:
      containers:
      - image: jgoodall/twemproxy:latest
        name: twemproxy
        ports:
        - containerPort: 63791
        - containerPort: 63792
        - containerPort: 6222
      restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: twemproxy
  name: twemproxy
spec:
  selector:
    app: twemproxy
  ports:
  - name: "63791"
    port: 63791
    targetPort: 63791
  - name: "63792"
    port: 63792
    targetPort: 63792
  - name: "6222"
    port: 6222
    targetPort: 6222

The configuration above will deploy twemproxy’s deployment and service. service in kubernetes allow pods to network with each other, this service will allow other pods to reach twemproxy.

twemproxy.yml:

local:
  listen: 0.0.0.0:63791
  hash: fnv1a_64
  hash_tag: "{}"
  distribution: ketama
  timeout: 500
  redis: true
  servers:
    - redis-0:6379:1
    - redis-1:6379:1
    - redis-2:6379:1

session:
  listen: 0.0.0.0:63792
  hash: fnv1a_64
  hash_tag: "{}"
  distribution: ketama
  auto_eject_hosts: true
  server_retry_timeout: 30000
  server_failure_limit: 3
  timeout: 500
  redis: true
  servers:
    - redis-0:6379:1
    - redis-1:6379:1
    - redis-2:6379:1
 

Above is twemproxy’s configuration which need to be mount as configmap volume in twemproxy’s deployment.

Redis Statefulset
Next, we need to create redis’s statefulset. Statefulset with 3 replicas will create the following pods redis-0, redis-1 and redis-2. Pods created by statefulset will be labeled as statefulset.kubernetes.io/pod-name: , we can use this as selector for the service. Unfortunably, we will still need to create 3 services to allow each pod to be addressable by twemproxy,

redis statefulset:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis
spec:
  selector:
    matchLabels:
      app: redis
  serviceName: "redis"
  replicas: 3
  template:
    metadata:
      labels:
        app: redis
    spec:
      terminationGracePeriodSeconds: 3
      containers:
      - name: redis
        image: redis:3.0.7
        ports:
        - containerPort: 6379
redis services:

apiVersion: v1
kind: Service
metadata:
  name: redis
  labels:
    app: redis
spec:
  ports:
  - port: 6379
    name: redis
  clusterIP: None
  selector:
    app: redis
---
apiVersion: v1
kind: Service
metadata:
  name: redis-1
  labels:
    app: redis-1
spec:
  ports:
  - port: 6379
    name: redis-1
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: redis-1
---
apiVersion: v1
kind: Service
metadata:
  name: redis-1
  labels:
    app: redis-1
spec:
  ports:
  - port: 6379
    name: redis-1
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: redis-1
---
apiVersion: v1
kind: Service
metadata:
  name: redis-2
  labels:
    app: redis-2
spec:
  ports:
  - port: 6379
    name: redis-2
  clusterIP: None
  selector:
    statefulset.kubernetes.io/pod-name: redis-2