Posts on elijah's blog

ArgoCD in Production

Sun, 05 Apr 2026 00:00:00 +0000

Most kubernetes clusters go a certain way the first time around. You see something cool, you write some manifest files, and you perform a kubectl apply -f deployment.yaml. This works, is great for debugging, or making sure you got manifests correct. However this means you don’t have a source of truth, it’s harder to work with others trying to modify the same manifests, and you don’t always know if the manifests have been applied.

One solution to this is GitOps. This allows a git repository to act as a source of truth; constantly checking and updating any running configuration. There are two real choices, FluxCD and ArgoCD. FluxCD is a very minimal choice - no UI, no RBAC, and less community support. FluxCD is also the choice on Big Bang a Department of Defense repository. On the other hand ArgoCD is the opposite — built-in UI and RBAC and great community support.

Defining ArgoCD applications is pretty easy. Take a look at the manifest below:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: adguard # the name of the application in the ArgoCD UI
 namespace: argocd
spec:
 destination:
 server: https://kubernetes.default.svc # what cluster to deploy to
 project: default # the default project
 source:
 path: manifests/adguard/ # the relative file path
 repoURL: https://github.com/ej-east/lisa-cluster # target repo
 targetRevision: HEAD # this is the tag to pull, HEAD is latest
 syncPolicy:
 automated:
 prune: true # removes old resources 
 selfHeal: true # automatically detects and corrects drift

As you can see in ~20 lines you can define an Argo Application. One really cool thing you can do with Argo is create an app-of-apps setup. Defining one app, the root app, to look at a directory of ArgoCD apps and create them.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
 name: root-app
 namespace: argocd
 finalizers:
 - resources-finalizer.argocd.argoproj.io
spec:
 destination:
 namespace: argocd
 server: https://kubernetes.default.svc
 project: default
 source:
 path: argocd-applications/
 repoURL: https://github.com/ej-east/lisa-cluster
 targetRevision: HEAD
 syncPolicy:
 syncOptions:
 - CreateNamespace=true
 automated:
 prune: true
 selfHeal: true

One kubectl apply to bootstrap, git handles the rest.

This structure is what makes ArgoCD worth it for me. I can commit everything to git, not worry if what I have locally is running or in GitHub. I no longer have to manually run kubectl apply -f ., I no longer have to do ClickOps — I can do GitOps.

Secret Management in Kubernetes with Vault and ESO

Sun, 05 Apr 2026 00:00:00 +0000

When you make a secret in Kubernetes it’s not encrypted. It’s encoded in base64. This means anyone can decode your secret and log in as you. It also means you cannot commit your secret to a git repository unless you want others to have the ability to decode it.

The majority of secrets are meant to be secret. So this method is obviously not going to work for most production environments. Instead External Secrets Operator (ESO) exists. This dynamically pulls secrets from a provider, think of Hashicorp Vault, AWS Secrets Manager, or Google Cloud Secrets Manager, and dynamically creates a Kubernetes secret from it. This allows you to share your manifest without having to worry about accidentally leaking any passwords.

Another method is Sealed Secrets this works with asymmetric encryption - the public key is used to encrypt the secret and the private key, stored on the cluster, is used to decrypt it. This also solves the problem of secret management. Only the people who need to read it can. However there are a few issues with this. First passwords sometimes need to rotate, and most people don’t want the additional complexity of a CronJob to update the sealed secret. Secondly you’re still committing passwords to git. Which is just bad repository hygiene. Thirdly it’s not very production ready. Let’s say you get hacked, the pod has a ServiceAccount and the hacker is able to pull the decryption key, and now every sealed secret in the cluster is compromised.

For these reasons I went with the first option, External Secrets. There are lots of provider options for External Secrets, 1Password, Keeper Security, even ngrok. I went with Hashicorp Vault. This does have some added complexities but it was nice to have it run on cluster as I’m trying to have next to no costs for my homelab.

It’s relatively easy to pull secrets from Hashicorp Vault.

Secrets like LISA

First create your secret!

vault kv put secret/my-app/config \
 username="placeholder" \
 password="placeholder" \
 api_key="placeholder"

Then create your access policy

# policy.hcl
path "secret/data/my-app/config" {
 capabilities = ["read"]
}

vault policy write my-app-read policy.hcl

Then configure Vault’s Kubernetes auth

vault auth enable kubernetes # you only have to do this once

vault write auth/kubernetes/config \  # you only have to do this once
 kubernetes_host="https://kubernetes.default.svc:443" 

vault write auth/kubernetes/role/my-app-role \ # do this for each secret 
 bound_service_account_names="my-app-sa" \
 bound_service_account_namespaces="my-app" \
 policies="my-app-read" \
 ttl="1h"

Then create your SecretStore

apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
 name: vault-backend
 namespace: my-app
spec:
 provider:
 vault:
 server: "http://hashicorp-vault.hashicorp-vault.svc.cluster.local:8200"
 path: "secret"
 version: "v2"
 auth:
 kubernetes:
 mountPath: "kubernetes"
 role: "my-app-role"
 serviceAccountRef:
 name: "my-app-sa"

And pull the secret!

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
 name: my-app-secrets
 namespace: my-app
spec:
 refreshInterval: "1h"
 secretStoreRef:
 name: vault-backend
 kind: SecretStore
 target:
 name: my-app-user-password
 creationPolicy: Owner
 data:
 - secretKey: username
 remoteRef:
 key: my-app/config
 property: username
 - secretKey: password
 remoteRef:
 key: my-app/config
 property: password

Congratulations! You just did secret management the production way. ESO automatically refreshes the secret every hour, so if you rotate it in Vault, the pod picks it up. No redeploys required.

Securing traffic to the World Wide Web

Sun, 05 Apr 2026 00:00:00 +0000

There are lots of ways to host a website securely. Originally, I hosted my website on AWS. I used a static S3 site with Route53 for DNS and Cloudfront as a Content Delivery Network(CDN). This worked really well but at the end of the day was static. I couldn’t host services like Authentik.

The solution is simple, get my current cluster on the internet, but doing that securely is hard. My goals: no ongoing cost, end-to-end encryption, and no open ports on my home router.

cloudflared a Cloudflare tunnel client is a great option. It is free, it can run as a pod on my cluster, and I already use Cloudflare to manage my domain. So why not it? Simple - Cloudflare can see the plain text traffic that goes through. This is a giant no-no and breaks one of my requirements. This would be fine for static things but for things like a Wiki or Authentik [anything that has a password] it wouldn’t be secure.

For clarification, Cloudflare isn’t doing anything malicious. Their tunnels work by terminating TLS at their edge. Your traffic is encrypted between the request and Cloudflare, and encrypted again between Cloudflare and your server - but in between, Cloudflare can and does see everything in plain text. For a static blog that’s fine. For anything with authentication, session tokens, or sensitive data, you’re trusting a third party with every request. I didn’t want that.

The solution was to build my own tunnel. I spun up a free-tier Oracle Cloud VM, set up WireGuard between it and my cluster, and configured iptables to forward traffic through. The Oracle VM is a dumb proxy - just forwarding the encrypted traffic. I cover the full setup, config, and iptables rules in How This Blog Is Hosted.

This setup means I own every piece of the path. No vendor sees my traffic in plain text. No Cloudflare outage takes me down. No policy change locks me out. If I want to move domains tomorrow, I update a DNS record and everything still works. The tradeoff is I’m responsible for uptime and patching - but that’s the point.

Why an IdP is a Necessity

Sun, 05 Apr 2026 00:00:00 +0000

Most homelab setups end the same way. You spin up service after service, set up a username and unique [or worse shared] password. Before you know it you’ve got 15 different logins written down in some notepad, and if you want to add anyone else you’ve got no way to centrally manage access.

On my old cluster this was a major problem of mine. I do come from an IT background and actually have experience with a solution called Okta, an Identity Provider (IdP). Instead of each app having its own username and password they all defer to the IdP. It solves a simple problem, one login for everywhere.

At first I was looking at Keycloak, it’s a great choice, it’s even used by the Department of Defense. For me at least it felt rather bulky and not something I wanted to continue forward with. So I kept looking and found Authentik.

Getting Authentik running was straightforward - there is a pre-existing helm chart and it just needs a PostgreSQL database and a Redis instance, both of which I was already running in the cluster. It handles OIDC, SAML, LDAP, and even proxy connections. That last one is worth mentioning because it lets you put auth in front of apps that have zero native SSO support. You just point the proxy provider at the service and suddenly it’s behind a login page.

For each service that I want to have SSO I create a provider in Authentik. This generates a corresponding client ID and client secret. Then you configure the service to use Authentik as its OIDC/SAML provider. The pattern is almost always the same, create a provider in Authentik, configure the application, done. Most applications do take it a bit further with RBAC; you create groups in Authentik that get mapped to certain permissions. For example assigning a user to the sso_grafana_viewer group would grant them access to view dashboards but not edit them. RBAC is by far one of my favorite features of an IdP.

The real payoff is when something needs to change. Instead of having to go to each application and review what permissions each user has I can just look in Authentik. I can see everything from what one user can do to what a group of users can do. One action, access everywhere. If I need to revoke it, I can just disable the account in one place and they’re immediately locked out. No hunting, no panic.

An IdP isn’t a nice to have feature anymore. It’s the difference from fighting access management to controlling it. Once you centralize identity, everything else gets simpler.

How This Blog Is Hosted

Sat, 04 Apr 2026 00:00:00 +0000

A homelab is what you make it. You can have the best hardware, fancy GPUs, and tons of memory and still just be running Docker Compose. Does that work? Yes. But it’s not how I host things on my homelab. It’s not about the resources of the nodes, it’s about what you do with them.

So what makes my homelab different? Hardware-wise not much. I run a 3-node K3s cluster - one master, two workers, each with 16GB of RAM and 512GB of storage. I’ve been using K3s since day one. Nothing crazy. K3s is what makes Kubernetes feel like home for me. While it is good to be comfortable with the tools you use, I am planning on switching to Talos.

I used to have an older version of my homelab. It taught me a lot about the do’s and don’ts. When I restarted I gave myself a few rules:

Always try to deploy things in the most secure way possible
No opening ports on the router
GitOps over everything else

Each of these shaped how the whole thing is put together. The security one will always be ongoing; there’s always new vulnerabilities and each app has something to lock down. But the other two directly dictated the architecture.

No Open Ports

This is the rule that feels the strangest. If I don’t open any ports on my router, how does traffic actually reach my cluster?

The answer is a VPN tunnel. I bought a domain, the one you’re on right now, cheesedipper.com, and spun up a small instance in Oracle Cloud using terraform. Oracle has a genuinely good free tier that gives you a VM with a public IP for nothing. It felt like the obvious choice.

From there I set up WireGuard between the Oracle instance and my cluster. WireGuard is fast, simple, and the config is small enough to actually understand. The Oracle instance acts as a public-facing entry point. When traffic hits it on port 80 or 443, iptables forwards that traffic through the WireGuard tunnel to my cluster, where Traefik picks it up and routes it to the right service.

Here’s what the WireGuard config looks like on the Oracle instance:

[Interface]
Address = 10.200.0.1/24
ListenPort = 51820
PrivateKey = <privatekey>

# Enable forwarding and set up NAT rules when the tunnel comes up
PostUp = sysctl -w net.ipv4.ip_forward=1
PostUp = iptables -t nat -A PREROUTING -i enp0s6 -p tcp --dport 80 -j DNAT --to-destination 10.200.0.2:80
PostUp = iptables -t nat -A PREROUTING -i enp0s6 -p tcp --dport 443 -j DNAT --to-destination 10.200.0.2:443
PostUp = iptables -t nat -A POSTROUTING -o wg0 -j MASQUERADE
PostUp = iptables -A FORWARD -i enp0s6 -o wg0 -p tcp --dport 80 -j ACCEPT
PostUp = iptables -A FORWARD -i enp0s6 -o wg0 -p tcp --dport 443 -j ACCEPT
PostUp = iptables -A FORWARD -i wg0 -o enp0s6 -m state --state RELATED,ESTABLISHED -j ACCEPT

# Clean up when the tunnel goes down
PostDown = iptables -t nat -D PREROUTING -i enp0s6 -p tcp --dport 80 -j DNAT --to-destination 10.200.0.2:80
PostDown = iptables -t nat -D PREROUTING -i enp0s6 -p tcp --dport 443 -j DNAT --to-destination 10.200.0.2:443
PostDown = iptables -t nat -D POSTROUTING -o wg0 -j MASQUERADE
PostDown = iptables -D FORWARD -i enp0s6 -o wg0 -p tcp --dport 80 -j ACCEPT
PostDown = iptables -D FORWARD -i enp0s6 -o wg0 -p tcp --dport 443 -j ACCEPT
PostDown = iptables -D FORWARD -i wg0 -o enp0s6 -m state --state RELATED,ESTABLISHED -j ACCEPT

# My cluster node
[Peer]
PublicKey = <publickey>
AllowedIPs = 10.200.0.2/32

When a request comes in on the Oracle VM’s network interface (enp0s6) on port 80 or 443, the PREROUTING rule rewrites the destination to 10.200.0.2 — which is my cluster node on the other side of the WireGuard tunnel. The MASQUERADE rule makes sure return traffic knows how to get back. The FORWARD rules allow the traffic to pass between interfaces, and the RELATED,ESTABLISHED rule lets response traffic flow back out.

The PostDown rules are just the cleanup - they remove everything when WireGuard shuts down so you don’t end up with stale iptables rules.

My home IP is never exposed. DNS points to the Oracle VM, and the Oracle VM forwards everything to my cluster through an encrypted tunnel. From the outside, it looks like everything is hosted in Oracle Cloud.

TLS Certificates with cert-manager

With traffic flowing through the tunnel, the next problem was TLS. I need HTTPS, don’t want to rely on Cloudflare, and don’t want to deal with certificates manually.

I use cert-manager with Let’s Encrypt. The important choice here is the challenge type. There are two options: HTTP-01 and DNS-01. HTTP-01 requires Let’s Encrypt to reach your server on port 80 to verify you own the domain. DNS-01 verifies ownership by checking a DNS TXT record.

I went with DNS-01 for a simple reason - it felt the easiest. My DNS is managed through Cloudflare, and cert-manager has a built-in Cloudflare solver. I give cert-manager a Cloudflare API token, it creates the TXT record, Let’s Encrypt verifies it, and the certificate gets issued. No extra ports, no special ingress rules for the challenge. The whole thing happens through the Cloudflare API, which means cert-manager doesn’t need any inbound traffic at all to get a certificate.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
 name: letsencrypt-prod
 namespace: kube-system
spec:
 acme:
 server: https://acme-v02.api.letsencrypt.org/directory
 email: ej@cheesedipper.com 
 privateKeySecretRef:
 name: letsencrypt-prod-account-key 
 solvers:
 - dns01:
 cloudflare:
 apiTokenSecretRef:
 name: cloudflare-api-token
 key: api-token

Once the cert is issued, cert-manager stores it as a Kubernetes secret that Traefik picks up automatically. Certificates renew on their own, so TLS is one of those things I set up once and can safely forget.

GitOps with ArgoCD

The third rule - GitOps over everything - is the one that changed how I think about managing a cluster.

I went with ArgoCD over Flux because of the UI, the RBAC model, and the workflow in general. My philosophy: everything that runs on my cluster is defined in a Git repo. ArgoCD watches that repo and makes sure the cluster matches what’s in Git. If I want to deploy something, I push a commit. If I want to roll something back, I revert a commit. I only run kubectl apply against the cluster when testing, to make sure the working code is always in git.

I use the App of Apps pattern. There’s a root ArgoCD Application that points to a directory of other Application manifests. Each of those points to either a Helm chart or a set of plain Kubernetes manifests for a specific app. When I want to add something new to the cluster, I add an Application manifest to the repo and ArgoCD picks it up.

Some apps use Helm charts - things like cert-manager, gatekeeper, and ArgoCD itself where the upstream chart handles the complexity. Other apps that I’ve written or that are simpler just use plain manifests. The mix works fine. ArgoCD doesn’t care how the manifests are generated, it just syncs them.

The thing I like most about this setup is that my cluster is reproducible. If something goes wrong, the Git repo is the source of truth. I could wipe the cluster, reinstall K3s, point ArgoCD at the repo, and everything comes back.

How It All Fits Together

When you loaded this page, your browser resolved cheesedipper.com to the Oracle VM’s public IP. The request hit the VM on port 443, iptables forwarded it through the WireGuard tunnel to my cluster, and Traefik routed it to the right service. The response came back through the same tunnel. Your browser has no idea it’s talking to a server in my house.

Everything running on the cluster is defined in Git. Certs renew through the Cloudflare API. My home IP never shows up anywhere public.

Is it overkill for a blog? Probably. But the blog isn’t really the point; showcasing secure, scalable infrastructure is.