Some background
Here at Kvanttori we prefer to self host the majority of our own internal services. We decided on using docker compose for quick and painless deployment & migration.
Wait, what about kubernetes?
While kubernetes is great for public-facing services that require zero downtime and need to be scalable, it is often overkill for smaller projects, for which a single node might very well be more than enough.
Thinking about our use case, we just want to run a handful of services with a maximum of 100 people using them at the same time. A single 10€ compute node at Hetzner for example would be more than enough to fit our needs, while also being simple to manage and migrate.
As for uptime, we really only use our internal services during standard working hours. This gives us the freedom to perform maintenance during weekends and evenings, which is why we don't really need high availability to ensure 99.999% uptime.
Tip #1: Backups, backups, backups
Backups are the heart of every good deployment - you should always be prepared for data corruption and nodes failing. The 3-2-1 backup principle is a good rule of thumb for ensuring you are backing up data correctly. It states that:
- You should always keep at least 3 copies of your data
- You should keep your data in at least 2 different formats
- At least 1 of your copies should be off-site
How we do it
One simple and reliable way we can tick off all three of these is by using an external service provider to store our backups. For example AWS S3. You could also use any other object-based storage provider such as Wasabi or even self-host one yourself by using something like min.io.
Our preferred and automated solution for this is to use the open-source docker-volume-backup container, which automatically backs up docker volumes to our object storage, while also ensuring data consistency by shutting down containers while they are being backed up.
Using docker-volume-backup
Using docker-volume-backup is very straight forward. You just need to add it to your compose file, and fill in the credentials for your storage provider. Here is an example on how you could back up services that uses two separate volumes to store data:
### Rest of your docker-compose.yml ###
backup:
# In production, it is advised to lock your image tag to a proper
# release version instead of using `latest`.
image: offen/docker-volume-backup:[current-version]
restart: always
environment:
# This signifies the number of days you want to keep
# your backups, you may omit this if you want to keep your
# backups indefinitely
BACKUP_RETENTION_DAYS: 30
AWS_S3_BUCKET_NAME: [aws-bucket-name]
AWS_ACCESS_KEY_ID: [aws-user-key-id]
AWS_SECRET_ACCESS_KEY: [aws-user-secret-key]
volumes:
# Here we bind our volumes into the /backup directory
# in the docker volume backup container, with read only access
# to ensure no harm can be done to the original files
- [docker-volume-1]:/backup/[docker-volume-1]:ro
- [docker-volume-2]:/backup/[docker-volume-2]:ro
# Mounting the Docker socket allows the script to stop and restart
# the container during backup. You can omit this if you don't want
# to stop the container.
- /var/run/docker.sock:/var/run/docker.sock:ro
Tip #2: Docker based networking
One of the superpowers of Docker is the ability to automatically create internal networks between containers and connect between them by simply using image names. Here is an example for how one could connect a service to a Mongo instance purely through docker networks, without relying on opening ports on the local machine:
version: '3'
services:
your-service:
image: your-service-image:latest
depends_on:
- mongo-db
mongo-db:
restart: always
image: mongo:latest
After this, you can access the mongo-db
container through the your-service
container with the connection string mongodb://mongo-db:27017
Why this works
Assuming your app is in a directory called "my-app", when you run docker compose up
, the following happens:
- A network called my-app_default is created.
- A container is created using your-service’s configuration. It joins the network my-app_default under the name your-service.
- A container is created using mongo-db’s configuration. It joins the network my-app_default under the name mongo-db.
If you are interested in reading up more about Docker networking in docker compose, you can refer to their documentation here to get a deeper understanding on how it works.
Tip #3: Mounting volumes directly on the network
Often when you want to run containers, especially ones processing data, you might want to access data from a network resource. Your first idea might be to mount the volume on your host, and then bind mount the mounted directory to your container. There is, however, a simpler and more elegant way to do this: mounting the network volumes directly in docker.
There are two major file protocols docker supports for volume mounting, NFS and CIFS/SAMBA.
NFS
Here's how one could host a nextcloud instance that stores files on a separate NAS running on 192.168.0.2 and has a exposed directory at /nextcloud/nextcloud_nextcloud
services:
app:
ports:
- 8085:80
image: nextcloud:apache
restart: always
volumes:
- nextcloud:/var/www/html
volumes:
nextcloud:
driver: local
driver_opts:
type: nfs
o: nfsvers=4,addr=192.168.0.2,rw
device: ':/nextcloud/nextcloud_nextcloud'
SAMBA/CIFS
Here's the same scenario, but instead with a SAMBA/CIFS share
services:
app:
ports:
- 8085:80
image: nextcloud:apache
restart: always
volumes:
- nextcloud:/var/www/html
volumes:
nextcloud:
driver: local
driver_opts:
type: cifs
o: addr=192.168.0.2,username=xxxxx,password=xxxxx,file_mode=0777,dir_mode=0777
device: '://192.168.0.2/nextcloud/nextcloud_nextcloud'
Tip #4: Automatically updating your images through watchtower
One problem many people often come across when developing their applications is how to automatically deploy them. Docker containers are a good way to run your applications, but you would have to set up a complex webhook or pipeline process to first copy the content you want on to the server, after which you want to re-start the containers. Setting all of this up by yourself could be quite daunting and would probably take some time. Luckily there is a better way.
Enter Watchtower
At its core, Watchtower works in quite a simple manner. It simply checks if there is an updated version of any image available, and if that is the case, it recreates the container with the new image. While this isn't really the best idea to do for external images, it is great for images you build yourself:
version: '3'
services:
watchtower:
environment:
# Makes it so that Watchtower only updates containers we explicitly tell it to
WATCHTOWER_MONITOR_ONLY: true
WATCHTOWER_POLL_INTERVAL: 1800 #Checks for new updates every half an hour
image: containrrr/watchtower
volumes:
- /var/run/docker.sock:/var/run/docker.sock
Next we configure our service, whilst remembering to add the label to enable Watchtower updates:
version: '3'
services:
yourservice:
image: yourservice:latest
labels:
- 'com.centurylinklabs.watchtower.enable=true'
That's it, from here on out, watchtower will periodically check for any new updates to your image and restart the container when a new version is detected.
Git set up
Now that we have Watchtower listening for changes to our image, we want to automatically build and deploy it when it updates. GitHub has a great tutorial on how to do just that using their actions, you can find it here. Similar solutions exist for pretty much all other CI/CD providers, since building and pushing images to registries is so common place.
Tip #5: Manage all routing through Traefik
Traefik is a container-first proxy, which makes it easy to route traffic to your containers securely. With Traefik you can define your routing configuration straight in the docker compose files for each service. This, with the addition of automated certificate management and renewal, makes Traefik an excellent addition to many deployments.
Setting up the Traefik container
The first step in setting up Traefik is to spin up the root container. Below is an example taken from the Traefik getting started docs for how to set up your root container. Please note that this is just an example for a minimal Traefik set up. In production use you would probably want to secure the web UI, add https redirection, and add automated SSL certificate generation.
version: '3'
services:
reverse-proxy:
# The official v2 Traefik docker image
image: traefik:v2.10
# Enables the web UI and tells Traefik to listen to docker
command: --api.insecure=true --providers.docker
ports:
# The HTTP port
- '80:80'
# The Web UI (enabled by --api.insecure=true)
- '8080:8080'
volumes:
# So that Traefik can listen to the Docker events
- /var/run/docker.sock:/var/run/docker.sock
Setting up a service container
After we have our Traefik container running, our next step it to add services it can route traffic to. Continuing with the getting started example given by Traefik, we can use the whoami docker container as a simple example for how one might configure an http service:
version: '3'
services:
whoami:
# A container that exposes an API to show its IP address
image: traefik/whoami
labels:
- 'traefik.http.routers.whoami.rule=Host(`whoami.docker.localhost`)'
Wrapping it up
In conclusion, by implementing these five strategies, you can elevate your Docker Compose deployments to the next level, empowering you to build robust, scalable, and efficient containerized applications with ease.