Skip to content

feat(helm): add kubernetes local-dev environment#1158

Open
TaylorMutch wants to merge 5 commits intomainfrom
kube-support/local-dev/tmutch
Open

feat(helm): add kubernetes local-dev environment#1158
TaylorMutch wants to merge 5 commits intomainfrom
kube-support/local-dev/tmutch

Conversation

@TaylorMutch
Copy link
Copy Markdown
Collaborator

@TaylorMutch TaylorMutch commented May 4, 2026

Summary

Adds a local Kubernetes development environment for OpenShell using k3d (Docker-backed k3s), Skaffold, and the existing Helm chart. Includes optional Envoy Gateway (GRPCRoute / Gateway API), Keycloak OIDC, and cert-manager PKI add-ons.

Related Issue

Changes

  • Add `helm:k3s:*` mise tasks (`create`, `delete`, `start`, `stop`, `status`) backed by `tasks/scripts/helm-k3s-local.sh` — cluster name derived from git branch for per-worktree isolation
  • Add `helm:skaffold:*` mise tasks (`dev`, `run`, `delete`, `diagnose`) for iterative and one-shot deploys
  • Add `helm:gateway:apply` task to activate Envoy Gateway routing after deploy
  • Add `keycloak:k8s:*` mise tasks (`setup`, `teardown`) backed by `tasks/scripts/keycloak-k8s-setup.sh`
  • Add Skaffold config (`deploy/helm/openshell/skaffold.yaml`) with Envoy Gateway, Keycloak, and cert-manager as opt-in valuesFiles
  • Add `pkiInitJob` Helm hook — generates mTLS CA + server/client certs via an alpine/openssl Job on pre-install/pre-upgrade (default PKI path)
  • Add cert-manager PKI support (`deploy/helm/openshell/templates/cert-manager-pki.yaml`, `values-cert-manager.yaml`) as an alternative to `pkiInitJob` — creates namespaced Issuer + CA + server/client Certificates; mutual exclusion with `pkiInitJob` enforced at template render time
  • Add `sshHandshakeSecret` Helm hook for SSH handshake init
  • Add `gateway.yaml` and `grpcroute.yaml` Helm templates for Envoy Gateway integration
  • Add `values-gateway.yaml`, `values-keycloak.yaml`, `values-skaffold.yaml`, `values-cert-manager.yaml` overlay files
  • Add `deploy/kube/manifests/envoy-gateway-openshell.yaml` GatewayClass manifest
  • Add `helm-dev-environment` agent skill documenting the full cluster lifecycle, TLS toggle, and mTLS port-forward setup
  • Remove stale `scripts/bin/k9s` and `scripts/bin/kubectl` wrapper scripts (replaced by mise-managed tools)
  • Add `k3d` and update `k9s` in `mise.toml`

Testing

  • `mise run pre-commit` passes
  • Cluster create → deploy → `sandbox list` verified with `pkiInitJob` (plaintext and mTLS)
  • Cluster create → deploy → `sandbox list` + `sandbox create` verified with cert-manager PKI over mTLS port-forward
  • Unit tests added/updated
  • E2E tests added/updated (if applicable)

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)
  • Architecture docs updated (if applicable)

@TaylorMutch TaylorMutch requested a review from a team as a code owner May 4, 2026 22:18
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 4, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

* Add support for grpcRoute from Kubernetes Gateway API spec
* Add pkiInitJob to initialize mTLS resources
* Add sshHandshake init job
* Test integration with Envoy Gateway
* Add keycloak integration testing with Skaffold
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 4, 2026

@TaylorMutch TaylorMutch force-pushed the kube-support/local-dev/tmutch branch from 38dda79 to a4ec87d Compare May 4, 2026 22:18
… setup

Add a TLS behaviour section explaining that values-skaffold.yaml disables
TLS by default, and a port-forward connection guide covering both plaintext
and mTLS modes with the exact commands to extract client certs from the
cluster PKI secret.
Re-add the openshell.issuerSelfSigned helper, the mutual-exclusion guard
in pki-hook.yaml, and the certManager condition in the statefulset volume
mount. Add server.disableTls: false to values-cert-manager.yaml so the
overlay correctly overrides the skaffold dev default. Tested end-to-end
with cert-manager issuing mTLS certs and sandbox create over port-forward.
@TaylorMutch
Copy link
Copy Markdown
Collaborator Author

/ok to test 5fac9ca


The gateway Service uses ClusterIP. Access is via Envoy Gateway (port `8080`) or `kubectl port-forward`.

### TLS behaviour
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎩 so fancy

K8S_CA=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
API=https://kubernetes.default.svc

# Idempotency: skip if server TLS secret already exists
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to check if the client TLS secret also exists? I feel like I've hit this before with k3s usage, where a previously cleanup failed and I only get left with one half of the secret pair.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes we should - good catch!

### Connecting via port-forward

```bash
KUBECONFIG=kubeconfig kubectl port-forward -n openshell svc/openshell 8080:8080
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would this kube port-forward on 8080 collide with the LB port on 8080?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so - I'll take another pass and check on this and see if I can come up with a better solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants