Getting Started

Troubleshooting

Diagnose and fix common AIOStack installation and runtime issues.

Tip:k9s is a terminal UI for Kubernetes that makes it much easier to browse pods, stream logs, and inspect resources without memorizing kubectl commands. If you're going to be debugging AIOStack (or any Kubernetes workload), it's worth installing before you start.

Quick status check

Run these first to get a picture of what's happening:

Terminal

# Overall pod status
kubectl get pods -n aiostack

# All AIOStack resources
kubectl get all -n aiostack

# Recent events (errors surface here first)
kubectl get events -n aiostack --sort-by='.lastTimestamp'

Pods not starting

Pods stuck in `Pending`

The node doesn't have enough resources or there's no node that satisfies the DaemonSet's scheduling requirements.

Terminal

# Describe the stuck pod for scheduling details
kubectl describe pod -n aiostack -l app.kubernetes.io/name=observer

# Check node capacity and allocatable resources
kubectl describe nodes | grep -A 5 "Allocated resources"

# Check if any taints are blocking scheduling
kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

Pods stuck in `CrashLoopBackOff`

The container is starting and immediately exiting. Check logs from the current and previous container:

Terminal

# Current logs
kubectl logs -n aiostack -l app.kubernetes.io/name=observer --tail=100

# Logs from the previous (crashed) container
kubectl logs -n aiostack -l app.kubernetes.io/name=observer --previous --tail=100

# Same for the outpost
kubectl logs -n aiostack -l app.kubernetes.io/name=outpost --tail=100
kubectl logs -n aiostack -l app.kubernetes.io/name=outpost --previous --tail=100

Pods stuck in `ImagePullBackOff`

The cluster cannot pull the AIOStack container image.

Terminal

kubectl describe pod -n aiostack -l app.kubernetes.io/name=observer | grep -A 10 "Events:"

Check that your cluster nodes have outbound internet access to pull images. If you're in an air-gapped environment, contact support@aurva.io.

eBPF not working

Kernel version too old

AIOStack requires Linux kernel 5.15+. Verify on each node:

Terminal

# Check kernel version on all nodes
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kernelVersion}{"\n"}{end}'

If any node is below 5.15, AIOStack's observer will not be able to attach eBPF probes on that node.

eBPF probes failing to attach

Terminal

# Look for eBPF-related errors in observer logs
kubectl logs -n aiostack -l app.kubernetes.io/name=observer --tail=200 | grep -i "ebpf\|probe\|kprobe\|uprobe\|permission\|denied"

The observer requires CAP_BPF and CAP_SYS_ADMIN capabilities. If your cluster has a restrictive PodSecurityPolicy or OPA/Gatekeeper rules blocking privileged containers, the probes will fail to load. Check with your cluster admin.

No data appearing in the console

Observer is running but nothing shows up

Terminal

# Confirm observer pods are running on all nodes
kubectl get pods -n aiostack -o wide

# Check observer is seeing network traffic
kubectl logs -n aiostack -l app.kubernetes.io/name=observer --tail=200 | grep -i "connection\|trace\|event"

# Check outpost is connected to the control plane
kubectl logs -n aiostack -l app.kubernetes.io/name=outpost --tail=200 | grep -i "connected\|commander\|error\|failed"

Outpost cannot reach the control plane

The outpost communicates with hq.aurva.ai:443. Verify outbound connectivity from the cluster:

Terminal

# Run a one-off pod to test connectivity
kubectl run -it --rm curl-test --image=curlimages/curl --restart=Never -n aiostack -- \
  curl -v https://hq.aurva.ai

If the connection fails, check your cluster's egress network policies and firewall rules to ensure hq.aurva.ai:443 is reachable.

Wrong or missing Company ID

Terminal

# Check the Company ID set in the outpost env
kubectl get pod -n aiostack -l app.kubernetes.io/name=outpost -o jsonpath='{.items[0].spec.containers[0].env}' | \
  python3 -m json.tool | grep -A 1 "COMPANY_ID"

If the value is empty or wrong, reinstall with the correct COMPANY_ID from app.aurva.ai.

IAM / cloud inventory not working

IAM roles, RDS, or S3 not appearing

The outpost's cloud service account is either not configured or the role doesn't have the required permissions.

Terminal

# Check if IAM role annotation is set on the service account (AWS)
kubectl get serviceaccount aiostack-outpost-sa -n aiostack -o yaml | grep "eks.amazonaws.com"

# Check outpost logs for permission errors
kubectl logs -n aiostack -l app.kubernetes.io/name=outpost --tail=200 | grep -i "iam\|permission\|unauthorized\|forbidden\|rds\|s3"

For AWS, verify the IRSA trust policy references the correct OIDC issuer and namespace:

Terminal

# Get your cluster's OIDC issuer
aws eks describe-cluster --name <CLUSTER_NAME> --region <REGION> \
  --query "cluster.identity.oidc.issuer" --output text

For GCP, verify the Workload Identity binding:

Terminal

gcloud iam service-accounts get-iam-policy \
  aiostack-outpost@<PROJECT_ID>.iam.gserviceaccount.com

Helm installation issues

Helm release already exists

Terminal

# List existing releases
helm list -n aiostack

# Upgrade instead of install
helm upgrade myaiostack aiostack/aiostack --namespace aiostack \
  --set outpost.env[3].name=COMPANY_ID \
  --set outpost.env[3].value=<YOUR_COMPANY_ID>

# Or uninstall and reinstall
helm uninstall myaiostack -n aiostack

Helm repo not found or chart unavailable

Terminal

# Re-add the repo and update
helm repo remove aiostack
helm repo add aiostack https://charts.aurva.ai/
helm repo update

# Confirm chart is available
helm search repo aiostack/aiostack

Values not taking effect

--set with commas in values (e.g. SKIP_NAMESPACES) requires escaping:

Terminal

# Commas must be escaped with a backslash
--set "outpost.env[0].value=kube-system\,aiostack\,monitoring"

Namespace issues

Namespace already exists with a conflicting installation

Terminal

# Check what's already running in the namespace
kubectl get all -n aiostack

# Check for existing Helm releases in that namespace
helm list -n aiostack

Use a different release name or namespace if there's a conflict:

Terminal

helm install myaiostack-2 aiostack/aiostack --namespace aiostack-2 --create-namespace \
  --set outpost.env[3].name=COMPANY_ID \
  --set outpost.env[3].value=<YOUR_COMPANY_ID>

Watching logs in real time

Terminal

# Stream observer logs
kubectl logs -n aiostack -l app.kubernetes.io/name=observer -f

# Stream outpost logs
kubectl logs -n aiostack -l app.kubernetes.io/name=outpost -f

# Watch pod status changes
kubectl get pods -n aiostack -w

# All containers across all pods
kubectl logs -n aiostack -l app.kubernetes.io/instance=myaiostack --all-containers --tail=50

Still stuck?

GitHub Issues — search existing issues or open a new one
support@aurva.io — include your Kubernetes version, cloud provider, and the output of kubectl get pods -n aiostack and relevant logs

Edit this page

Upgrades

How to upgrade AIOStack to a newer version using image tags or a rolling restart.

Uninstall

Remove AIOStack from your Kubernetes cluster using the one-line uninstaller or manually with Helm.