migrate yaml auth keys.sh
migrate-yaml-auth-keys.sh
Migrates YAML-managed ingestion auth keys (Kubernetes secret + ingester ConfigMap) to the UI-managed (Postgres-backed) form by calling config-mgmt-service's GraphQL API.
What this script does:
1. Reads tokens from the auth secret (default: kfuse-auth-ingest).
Each data. is a base64-encoded token.
2. Reads authKeyAdditionalLabels from the ingester ConfigMap (default: ingester, key config.yaml).
Display names are lowercased and trimmed to match the ingester's Viper-loaded semantics.
3. Joins each token with its label entry (by display name) and calls createAuthKeyLabel against config-mgmt-service. The token is reused as-is so existing agents keep working.
4. Skips display names that already have a UI-managed entry (idempotent re-runs).
What this script does NOT do:
- It does not delete the Kubernetes secret or remove authKeyAdditionalLabels from custom_values.yaml. Do that manually after verifying ingestion is healthy.
- It does not restart config-mgmt-service. Do that after removing the YAML config so it stops surfacing the now-deleted yaml entries.
Prerequisites:
- kubectl configured for the target cluster.
- jq, base64, curl.
- python3 with PyYAML installed (pip3 install PyYAML if missing).
- config-mgmt-service port-forwarded: kubectl port-forward -n svc/config-mgmt-service 8080:8080
Usage:
./migrate-yaml-auth-keys.sh -n [options]
Options:
-n, --namespace NAME Kloudfuse namespace (default: kfuse)
--secret NAME Auth secret (default: kfuse-auth-ingest)
--configmap NAME Ingester ConfigMap (default: ingester)
--configmap-key KEY Key inside the ConfigMap (default: config.yaml)
--port PORT Local port for config-mgmt-service (default: 8080)
--admin-email EMAIL Email recorded as updatedBy (default: yaml-migration@kloudfuse.local)
--token-name NAME Name for the imported token (default: yaml-imported)
--dry-run Print what would be created and exit
-h, --help Show this help
set -euo pipefail
NAMESPACE="kfuse" SECRET="kfuse-auth-ingest" CONFIGMAP="ingester" CONFIGMAP_KEY="config.yaml" PORT="8080" ADMIN_EMAIL="yaml-migration@kloudfuse.local" TOKEN_NAME="yaml-imported" DRY_RUN=0
usage() { sed -n '/^# Usage:/,/^# -h,/p' "$0" | sed 's/^# {0,1}//'; exit 0; }
while [[ $# -gt 0 ]]; do
case "$1" in
-n|--namespace) NAMESPACE="$2"; shift 2 ;;
--secret) SECRET="$2"; shift 2 ;;
--configmap) CONFIGMAP="$2"; shift 2 ;;
--configmap-key) CONFIGMAP_KEY="$2"; shift 2 ;;
--port) PORT="$2"; shift 2 ;;
--admin-email) ADMIN_EMAIL="$2"; shift 2 ;;
--token-name) TOKEN_NAME="$2"; shift 2 ;;
--dry-run) DRY_RUN=1; shift ;;
-h|--help) usage ;;
*) echo "Unknown argument: $1" >&2; exit 2 ;;
esac
done
GQL_URL="http://localhost:${PORT}/config/graph/api"
red() { printf '\033[31m%s\033[0m\n' "$"; } green() { printf '\033[32m%s\033[0m\n' "$"; } yellow() { printf '\033[33m%s\033[0m\n' "$"; } bold() { printf '\033[1m%s\033[0m\n' "$"; }
---------------------------------------------------------------------------
Read tokens from the secret. Each .data. is a base64-encoded token. We collapse displayName to lowercase + trimmed, matching the loader.
Output: TSV "displayName\tplaintextToken" (one per line).
---------------------------------------------------------------------------
read_secret_tokens() { local raw if ! raw=$(kubectl get secret "$SECRET" -n "$NAMESPACE" -o json 2>/dev/null); then echo "ERROR: cannot read secret $SECRET in namespace $NAMESPACE" >&2 exit 1 fi echo "$raw" | python3 -c ' import json, sys, base64 doc = json.load(sys.stdin) data = doc.get("data") or {} for name, b64 in data.items(): name_clean = name.strip().lower() if not name_clean: continue try: token = base64.b64decode(b64).decode("utf-8").strip() except Exception as e: print(f"WARN: skipping {name}: cannot base64-decode ({e})", file=sys.stderr) continue if not token: continue print(f"{name_clean}\t{token}") ' }
---------------------------------------------------------------------------
Read authKeyAdditionalLabels from the ConfigMap. We accept either top-level (matching how config-mgmt-service mounts it) or nested under ingester.config (matching how helm/custom_values.yaml is typically structured).
Output: JSON object keyed by lowercased displayName, value is an array of {name, value} entries. Empty {} if no labels.
---------------------------------------------------------------------------
read_configmap_labels() { local raw if ! raw=$(kubectl get cm "$CONFIGMAP" -n "$NAMESPACE" -o json 2>/dev/null); then echo "WARN: cannot read configmap $CONFIGMAP in namespace $NAMESPACE — proceeding with no labels" >&2 echo "{}" return fi local content content=$(echo "$raw" | jq -r --arg k "$CONFIGMAP_KEY" '.data[$k] // empty') if [[ -z "$content" ]]; then echo "WARN: configmap $CONFIGMAP has no key $CONFIGMAP_KEY — proceeding with no labels" >&2 echo "{}" return fi echo "$content" | python3 -c ' import sys, json, yaml doc = yaml.safe_load(sys.stdin) or {} candidates = [ doc.get("authKeyAdditionalLabels"), (doc.get("ingester") or {}).get("config", {}).get("authKeyAdditionalLabels") if isinstance(doc.get("ingester"), dict) else None, (doc.get("config") or {}).get("authKeyAdditionalLabels") if isinstance(doc.get("config"), dict) else None, ] labels = next((c for c in candidates if isinstance(c, dict)), {}) or {} out = {} for name, entries in labels.items(): name_clean = (name or "").strip().lower() if not name_clean or not isinstance(entries, list): continue cleaned = [] for e in entries: if not isinstance(e, dict): continue n = str(e.get("name", "")).strip() if not n: continue cleaned.append({"name": n, "value": str(e.get("value", ""))}) out[name_clean] = cleaned print(json.dumps(out)) ' }
---------------------------------------------------------------------------
GraphQL helpers. RBAC headers spoof an admin caller.
---------------------------------------------------------------------------
gql() {
local query="$1" vars="${2:-{}}"
local payload
payload=$(python3 -c '
import json, sys
print(json.dumps({"query": sys.argv[1], "variables": json.loads(sys.argv[2])}))
' "$query" "$vars")
curl -sS -X POST "$GQL_URL"
-H 'Content-Type: application/json'
-H 'x-auth-request-role: admin'
-H "x-auth-request-email: ${ADMIN_EMAIL}"
-d "$payload"
}
list_existing_display_names() { local body body=$(gql 'query{ listAuthKeyLabels(limit: 1000){ items{ id displayName source } total } }' '{}') echo "$body" | python3 -c ' import sys, json doc = json.load(sys.stdin) errs = doc.get("errors") or [] if errs: sys.stderr.write("GraphQL errors: " + json.dumps(errs) + "\n") sys.exit(1) resp = ((doc.get("data") or {}).get("listAuthKeyLabels") or {}) items = resp.get("items") or [] for i in items:
Skip yaml-sourced rows — those are exactly what the migration is supposed to shadow.
Only treat UI-managed entries (source != "yaml") as "already exists" so the migration is not blocked by its own input.
if (i.get("source") or "").strip().lower() == "yaml": continue print((i.get("displayName") or "").strip().lower()) ' }
create_label() { local display_name="$1" token="$2" labels_json="$3" local vars vars=$(python3 -c ' import json, sys display_name, token_name, token, labels = sys.argv[1], sys.argv[2], sys.argv[3], json.loads(sys.argv[4]) print(json.dumps({"input": { "displayName": display_name, "tokenName": token_name, "token": token, "labels": labels, }})) ' "$display_name" "$TOKEN_NAME" "$token" "$labels_json") gql 'mutation($input: CreateAuthKeyLabelInput!){ createAuthKeyLabel(input: $input){ id displayName } }' "$vars" }
---------------------------------------------------------------------------
Main
---------------------------------------------------------------------------
bold "==> Reading YAML auth keys from cluster" echo " namespace : $NAMESPACE" echo " secret : $SECRET" echo " configmap : $CONFIGMAP (key=$CONFIGMAP_KEY)" echo " target : $GQL_URL" echo " dry-run : $( [[ $DRY_RUN -eq 1 ]] && echo yes || echo no )" echo
TOKENS_TSV=$(read_secret_tokens) LABELS_JSON=$(read_configmap_labels)
if [[ -z "$TOKENS_TSV" ]]; then yellow "No tokens found in secret. Nothing to migrate." exit 0 fi
bold "==> Discovered yaml entries:" echo "$TOKENS_TSV" | awk -F'\t' '{ print " " $1 }' echo
Surface label-only entries (declared in authKeyAdditionalLabels but without a matching token in the auth secret).
The script can't migrate them — without a token there's nothing to authenticate — but operators should
know these will be dropped when authKeyAdditionalLabels is cleaned up.
ORPHAN_LABELS=$(LABELS_JSON_ENV="$LABELS_JSON" TOKENS_TSV_ENV="$TOKENS_TSV" python3 -c ' import json, os labels = json.loads(os.environ.get("LABELS_JSON_ENV") or "{}") or {} tokens = set() for line in (os.environ.get("TOKENS_TSV_ENV") or "").splitlines(): if not line.strip(): continue tokens.add(line.split("\t", 1)[0].strip().lower()) for name in labels: if name.strip().lower() not in tokens: print(name) ')
if [[ -n "$ORPHAN_LABELS" ]]; then bold "==> Orphan label entries (declared in authKeyAdditionalLabels but no matching token in secret):" yellow " These entries cannot authenticate ingest traffic and will not be migrated." echo "$ORPHAN_LABELS" | sed 's/^/ /' echo fi
Always query existing UI-managed entries — dry-run also needs this so it can accurately predict which entries would be skipped on a real run.
bold "==> Listing existing UI-managed entries" if ! EXISTING=$(list_existing_display_names 2>&1); then red "Failed to query existing labels — is config-mgmt-service reachable on :$PORT?" echo "$EXISTING" >&2 exit 1 fi if [[ -n "$EXISTING" ]]; then echo "$EXISTING" | sed 's/^/ /' else echo " (none)" fi echo
CREATED=0 SKIPPED=0 FAILED=0
bold "==> Migrating" while IFS=$'\t' read -r display_name token; do [[ -z "$display_name" ]] && continue labels=$(echo "$LABELS_JSON" | jq -c --arg k "$display_name" '.[$k] // []')
if echo "$EXISTING" | grep -Fxq "$display_name"; then if [[ $DRY_RUN -eq 1 ]]; then yellow " [dry-skip] $display_name — already exists in UI" else yellow " [skip] $display_name — already exists in UI" fi SKIPPED=$((SKIPPED + 1)) continue fi
if [[ $DRY_RUN -eq 1 ]]; then label_count=$(echo "$labels" | jq 'length') green " [dry] $display_name (labels=$label_count, token=${token:0:8}…)" continue fi
resp=$(create_label "$display_name" "$token" "$labels" || true) err=$(echo "$resp" | python3 -c ' import sys, json try: d = json.load(sys.stdin) except Exception as e: print(f"unparseable response: {e}") sys.exit(0) errs = d.get("errors") or [] if errs: print(errs[0].get("message", "unknown error")) ' 2>/dev/null || echo "parse error")
if [[ -n "$err" ]]; then red " [fail] $display_name — $err" FAILED=$((FAILED + 1)) else green " [create] $display_name" CREATED=$((CREATED + 1)) fi done <<< "$TOKENS_TSV"
echo bold "==> Summary" echo " created : $CREATED" echo " skipped : $SKIPPED" echo " failed : $FAILED"
if [[ $DRY_RUN -eq 0 && $FAILED -eq 0 && $CREATED -gt 0 ]]; then echo bold "Next steps:" echo " 1. Verify in the UI (Admin > Settings > Auth key labels) that all" echo " YAML entries now have a matching UI-managed entry that" echo " 'shadows' the YAML row." echo " 2. Verify ingestion is still healthy." echo " 3. Delete the secret and remove authKeyAdditionalLabels from" echo " custom_values.yaml, then restart config-mgmt-service:" echo " kubectl delete secret $SECRET -n $NAMESPACE" echo " kubectl rollout restart deploy/config-mgmt-service -n $NAMESPACE" fi
[[ $FAILED -eq 0 ]] || exit 1