Apr 29, 2025
Apr 29, 2025
On the customer success (CS) team, part of our work is anticipating our customers' needs from ngrok and making sure we can address those needs effectively. One way we're doing that with the ngrok Kubernetes Operator is by working with Jon Stacks on our infrastructure team to build out a customer testing framework for the Operator using GitHub Actions.
Why GitHub Actions? Our day-to-day work in CS is not administering Kubernetes clusters, so we didn't want to get too deep in the weeds of setting up a cluster on one of the major cloud providers. But we also didn't want to fiddle with one-off manual tests with Minikube since we need to test dozens or hundreds of ngrok Kubernetes Operator resources at once.
Deploying a Kubernetes cluster in a GitHub Action let us abstract away much of the cluster setup and management so we could iterate with the ngrok Operator quickly and in a version-controlled way. We've learned a ton along the way, and are sharing an overview of this project in case you need for something similar for the CI pipelines you use to ship apps and APIs to Kubernetes.
Every GitHub Action needs a YAML workflow file to define what the action will do. Here's a condensed version of ours:
---
steps:
- name: Checkout repository
uses: actions/checkout@v4
- name: Setup go
uses: actions/setup-go@v5
with:
go-version: "1.23.5"
- name: Start k8s cluster
uses: engineerd/setup-kind@v0.6.2
- name: Install Helm
uses: azure/setup-helm@v4
- name: Install ngrok Helm chart
run: |
helm repo add ngrok https://charts.ngrok.com
- name: Deploy the ngrok-operator Helm chart
run: |
helm install ngrok-operator ngrok/ngrok-operator \
--create-namespace \
--namespace ngrok-operator \
--version 0.18.1 \
--set credentials.apiKey=${{ secrets.NGROK_API_KEY }} \
--set credentials.authtoken=${{ secrets.NGROK_AUTHTOKEN }}
- name: Set ngrok URL
run: |
NGROK_URL=${{ vars.NGROK_URL }} envsubst < ${{ github.event.inputs.plan }} > tmp && mv tmp ${{ github.event.inputs.plan }}
env:
NGROK_URL: ${{ vars.NGROK_URL }}
- name: Run Go script (app Helm install, etc.)
run: go run ./... ${{ github.event.inputs.plan }}
env:
JOB_NUMBER: ${{ matrix.job-number }}
- name: Operator cleanup
run: sleep 30In human words as opposed to computer words, the steps we've defined in this workflow are:
ngrok-operator Helm chart.Now that you have the high level overview, let's dig in a bit.
The main branch of our repository that we deploy into a Github Action looks like this:
.github/
|-- workflow.yaml
charts/app
|-- templates/
|-- values.yaml
|-- Chart.yaml
|-- .helmignore
plans/
|-- agent-endpoints-small-w-traffic-policy.yaml
main.goSo at the top level, we have the GitHub directory where our workflow file is defined, as well as a Helm chart, a directory of test plans that we've written, and a Go script.
Much of the heavy lifting happens in the Go script.
The script takes in a plan file, parses it, and uses the parsed values to deploy a Helm chart based on those values.
Here's a snippet of our main() function:
...
// Read and parse the plan file
contents, err := os.ReadFile(planFile)
if err != nil {
logger.Error("Failed to read file", "err", err)
os.Exit(1)
}
cfg := &Config{}
if err := yaml.Unmarshal(contents, cfg); err != nil {
logger.Error("Failed to unmarshal YAML", "err", err)
os.Exit(1)
}
ctx, cancel := context.WithTimeout(context.Background(), 15*time.Minute)
defer cancel()
g, gctx := errgroup.WithContext(ctx)
// Run the plan defined in the file, including deploying the Helm chart and running the defined tests
for idx, plan := range cfg.Plans {
planLogger := logger.With("plan", plan.Name)
g.Go(func() error { return runPlan(gctx, planLogger, plan, idx+1) })
}
g.Wait()
...For each Helm chart defined in a plan file, there is also a scenario we want to test, like: Is the app we deployed via Helm chart available at the ngrok URL we expect? Is the HTTP status code what we expect? And how long does it take for our ngrok endpoint become available?
Here's what part of a test plan file in the plans/ directory looks like:
plans:
- name: "Test Plan 1"
install:
helm:
chart: ./charts/app
release-name: game2048
namespace: default
set:
ngrokEndpoints.url: https://$NGROK_URL:443
ingress.host: $NGROK_URL
values-files:
- "./charts/app/values/disable-ingress.yaml"
- "./charts/app/values/enable-aep-crds.yaml"
- "./charts/app/values/enable-aep-crd-traffic-policy.yaml"
tests:
- wait-url-ready:
url: https://$NGROK_URL
retries: 5
expected-status-code: 302Each test plan file like the above can include one test plan or one thousand. Running these test plans with GitHub Actions means we can run the action with a specific test plan or all of them at once. This setup also gives us the flexibility to run the same tests against different Kubernetes APIs—Gateway API, Ingress, or regular CRDs—to make sure the ngrok Operator behaves the way we expect with all of them.
We simplified this repo to deploy one endpoint to one ngrok URL so any ngrok account type from free to pay-as-you-go can try this walkthrough—but if you want to scale up this framework to test multiple endpoints and domains, you'll want to dynamically generate your endpoint hostnames instead of hard-coding a single hostname.
Now that you have a sense of how the pieces of this project fit together, you can hop over to our GitHub repository to clone it and run it yourself. All you have to do to get started is set a few environment variables.
Check out the README for detailed instructions.
You ran the Action. It has a green checkmark next to it. What does that mean? What actually happened?
Within your Action run, the main logs you're interested in are under the Run Go script (app Helm install, etc.) step. This is where you can see what test plan is being installed:
time=2025-04-29T15:09:08.904Z level=DEBUG msg="Installing chart" plan="Test Plan 1" chart=./charts/app release-name=game2048-1 namespace=default args="[install game2048-1 ./charts/app --namespace default --wait --timeout 15m --atomic --set fullnameOverride=app-2048-1 --set ingress.host=exciting-needlessly-eel.ngrok-free.app --set ngrokEndpoints.url=https://exciting-needlessly-eel.ngrok-free.app:443 --values ./charts/app/values/disable-ingress.yaml --values ./charts/app/values/enable-aep-crds.yaml --values ./charts/app/values/enable-aep-crd-traffic-policy.yaml]"The next important log line is where you see that you successfully fetched from our endpoint URL:
time=2025-04-29T15:09:25.379Z level=INFO msg="Successfully fetched URL" plan="Test Plan 1" url=https://exciting-needlessly-eel.ngrok-free.app attempt=2 successes=1Since we got a successful fetch, the Go script will start spinning down resources so your Kubernetes cluster can be fully cleaned up:
time=2025-04-29T15:10:25.438Z level=DEBUG msg="Uninstalling chart" plan="Test Plan 1" chart=./charts/app release-name=game2048-1 namespace=defaultThis repository is meant to be a small self-contained example of how we're using the ngrok Operator with GitHub Actions internally. We're using ngrok CRDs directly to create an agent endpoint with a simple Traffic Policy rule.
If you wanted to tweak this, you could use a Kubernetes Ingress resource instead of CRDs, or you could try the Kubernetes Gateway API.
You could edit the Traffic Policy rules to do something else like send a custom-response action or use the redirect action to point traffic to a different URL—see our Traffic Policy actions for more ideas.
If your ngrok plan allows for internal bindings and additional domains, you could edit the existing test plan file, plans/agent-endpoints-with-traffic-policy.yaml, to have more than one test. Or you could write new plans to make your CI pipelines even more robust, like:
matrix strategy.Just keep in mind that containers used in these jobs are ephemeral and short-lived, so this works best with automated testing—not so much deploy previews or anything you want to keep around for a while.
This was a dense one! If you have questions or run into any issues setting up this project, don't hesitate to reach out to us at support@ngrok.com. Here's the repo link again: https://github.com/ngrok/ngrok-operator-gh-action-blog
Have fun!