Deployment Guide

Last updated: 2025-06-12

Overview

Tenki Cloud uses GitOps with Flux for Kubernetes deployments. All deployments are triggered via Git commits and automatically reconciled by Flux.

Deployment Environments

Deployment Process

1. Local Development → Staging

# 1. Ensure tests pass
pnpm test
gotest

# 2. Build and push images
make docker-build
make docker-push TAG=staging-$(git rev-parse --short HEAD)

# 3. Update staging manifests
cd infra/flux/apps/staging
vim engine-deployment.yaml  # Update image tag
git add .
git commit -m "deploy: engine staging-abc123"
git push

# 4. Monitor deployment
kubectl --context=staging get pods -w
flux logs -f

2. Staging → Production

# 1. Create release PR
gh pr create --base main --title "Release v1.2.3"

# 2. After approval and merge, tag release
git checkout main
git pull
git tag -a v1.2.3 -m "Release v1.2.3"
git push origin v1.2.3

# 3. CI/CD builds and pushes production images

# 4. Update production manifests
cd infra/flux/apps/production
# Update image tags to v1.2.3
git commit -m "deploy: production v1.2.3"
git push

# 5. Monitor rollout
kubectl --context=production rollout status deployment/engine

Service-Specific Deployments

Backend Engine

# Build
cd backend
make build-engine

# Test
make test

# Docker image
docker build -t tenki/engine:$TAG .
docker push tenki/engine:$TAG

# Update manifest
kubectl set image deployment/engine engine=tenki/engine:$TAG

Frontend App

# Build
cd apps/app
pnpm build

# Docker image
docker build -t tenki/app:$TAG .
docker push tenki/app:$TAG

# Deploy
kubectl set image deployment/app app=tenki/app:$TAG

Database Migrations

# Always run migrations before deploying new code
kubectl exec -it deploy/engine -- /app/migrate up

# Verify migrations
kubectl exec -it postgres-0 -- psql -U postgres tenki -c "\dt"

Rollback Procedures

Quick Rollback (< 5 mins)

# 1. Rollback deployment
kubectl rollout undo deployment/engine

# 2. Verify rollback
kubectl rollout status deployment/engine
kubectl logs -l app=engine --tail=100

# 3. Rollback database if needed
kubectl exec -it deploy/engine -- /app/migrate down

GitOps Rollback

# 1. Revert commit in Git
git revert <commit-hash>
git push

# 2. Flux will automatically sync
flux reconcile source git flux-system

# 3. Monitor
watch flux get kustomizations

Health Checks

Pre-deployment

# Check cluster health
kubectl get nodes
kubectl top nodes

# Check dependencies
kubectl get pods -n default
kubectl get pvc

# Verify secrets
kubectl get secrets

During Deployment

# Watch rollout
kubectl rollout status deployment/engine -w

# Monitor pods
kubectl get pods -l app=engine -w

# Check logs
kubectl logs -f -l app=engine --tail=50

Post-deployment

# Smoke tests
curl https://api.tenki.cloud/health
curl https://app.tenki.cloud

# Check metrics
open https://grafana.tenki.cloud/d/deployment

# Run integration tests
cd backend && gotest-integration

Monitoring Deployments

Grafana Dashboards

• Deployment Dashboard
• Application Metrics
• Error Rates

Key Metrics to Watch

• Request rate changes
• Error rate spikes
• Response time increases
• CPU/Memory usage
• Database connections

Alerts

# Deployment alerts configured in Prometheus
- name: deployment_failed
  expr: kube_deployment_status_replicas_unavailable > 0
  for: 5m

- name: high_error_rate_after_deploy
  expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
  for: 5m

Blue-Green Deployments

For high-risk changes:

# 1. Deploy to green environment
kubectl apply -f engine-deployment-green.yaml

# 2. Test green environment
curl https://api-green.tenki.cloud/health

# 3. Switch traffic
kubectl patch service engine -p '{"spec":{"selector":{"version":"green"}}}'

# 4. Monitor
watch 'kubectl get pods -l app=engine'

# 5. If issues, switch back
kubectl patch service engine -p '{"spec":{"selector":{"version":"blue"}}}'

Deployment Checklist

Pre-deployment

• All tests passing
• Code reviewed and approved
• Database migrations tested
• Rollback plan prepared
• Team notified in Slack

Deployment

• Images built and pushed
• Manifests updated
• Deployment monitored
• Health checks passing
• Smoke tests completed

Post-deployment

• Metrics normal
• No error spikes
• Customer reports checked
• Documentation updated
• Deployment logged

Troubleshooting

Pod Won’t Start

kubectl describe pod <pod-name>
kubectl logs <pod-name> --previous
kubectl get events --sort-by='.lastTimestamp'

Image Pull Errors

# Check secret
kubectl get secret regcred -o yaml

# Re-create if needed
kubectl create secret docker-registry regcred \
  --docker-server=registry.tenki.cloud \
  --docker-username=$USER \
  --docker-password=$PASS

Configuration Issues

# Check ConfigMaps
kubectl get configmap
kubectl describe configmap engine-config

# Check Secrets
kubectl get secrets
kubectl describe secret engine-secrets

CI/CD Pipeline

Our GitHub Actions pipeline:

1. On PR: Run tests, build images, deploy to preview
2. On merge to main: Build, tag, push to registry
3. On tag: Build production images, create release

See .github/workflows/deploy.yml in the repository root