Known Issues :: Kloudfuse Docs

Known Issues

We are aware of some issues you may encounter in Kloudfuse and their causes.

Installation

helm registry login failure

The helm registry login command may fail.

Resolution

Replace helm registry login with docker login:

cat token.json | docker login -u _json_key --password-stdin https://us-east1-docker.pkg.dev

Review helm values

We configured the default Kloudfuse helm chart for a single-node cluster install, without a deepstore. It may be difficult to understand what these are.

Resolution

Run the following command:

helm show values oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version <VERSION.NUM.BER> (1)


1	`version`: Use the version number of the most current Kloudfuse release; the pattern looks like `3.0.0`, `3.1.3`, and so on. See Version documentation.

As your configuration changes, you can add customizations by editing the custom_values.yaml file.

To view your helm values at any time, run the command on your cluster:

helm show values <Your Kloudfuse Installation IP or address>/kfuse-helm/kfuse --version <VERSION.NUM.BER> (1)

Networking

Kloudfuse cannot be reached from an external host

External IP, host, or DNS is unable to access Kfuse:

curl http://EXTERNAL_IP
curl: (28) Failed to connect to XX.XX.XX.XX port 80 after 129551 ms: Connection timed out
curl https://EXTERNAL_IP --insecure
curl: (28) Failed to connect to XX.XX.XX.XX port 443 after 129551 ms: Connection timed out

Resolution

Ensure that the security group or firewall policy for the Kubernetes cluster, node, and VPC endpoint allows external incoming traffic.

Ingress-NGINX drops packet

The ingress-NGINX logs the error client intended to send too large body.

2023/03/06 05:38:22 [error] 43#43: *128072996 client intended to send too large body: 1097442 bytes, client: XXXX, server: _, request: "POST /ingester/v1/fluent_bit HTTP/1.1", host: "XXXX"

Resolution

The default request body size is 1M. Configure Ingress-NGINX to accept larger request body size; include the following specification in the custom-values.yml file:

ingress-nginx:
  controller:
    config:
      proxy-body-size: <REPLACE THE BODY SIZE HERE, e.g., 8m. Setting to 0 will disable all limits.>

Kafka

Increase Kafka partition replication factor

In some scenarios, you have to increase the Kafka partition replication factor.

Resolution

In the custom-values.yaml file, update the global.kafkaTopics section with new replicationFactor. This does not change the configuration of a deployed cluster. It is necessary for tracking if you use yaml to re-deploy in a fresh installation.
If necessary, resize Kafka Persistent Disk.

By default, Kloudfuse uses 10GB retention size per partition, set in the kafka.logRetentionBytes variable.

Ensure that the persistent disk size has enough capacity.

Use this formula:

(NP)(RF)(LRB)NKB(NP)(RF)(LRB)NKB

where:

NP is the number of partitions
RF is the ReplicationFactor across all topics defined in global.kafkaTopics
LRB is the logRetentionBytes variable
NKB is the number of Kafka brokers

If you must increase the size of the persistent disk; see Increase existing PVC size and Resize PVC on Azure.
To start increasing the ReplicationFactor, log in to the Kafka Pod:

   kubectl exec -ti -n kfuse kafka-broker-0 -- bash

Unset the JMX_PORT:
```
unset JMX_PORT
```
Get the list of configured topics; you can get the list from global.kafkaTopics:
```
/opt/bitnami/kafka/bin/kafka-topics.sh --bootstrap-server :9092 --list
```

Create the topics.json file, and save it in the /bitnami/kafka directory:

cat > /bitnami/kafka/topics.json
{
  "version": 1,
  "topics": [\
    { "topic": "kf_events_topic" },\
    { "topic": "kf_logs_metric_topic" },\
    { "topic": "kf_logs_topic" },\
    { "topic": "kf_metrics_topic" },\
    { "topic": "kf_traces_errors_topic" },\
    { "topic": "kf_traces_metric_topic" },\
    { "topic": "kf_traces_topic" },\
    { "topic": "logs_ingest_topic" }\
  ]
}

Get into the Kafka Zookeeper pod.

kubectl exec -ti -n kfuse kafka-zookeeper-0 -- bash

Get the broker ids.

Get List of Active Brokers

/opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181 (1)
ls /brokers/ids (2)

Get the current partition assignment.

The --broker-list should match the number of current configured brokers.

This command sets up 3 Kafka brokers:

/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --generate --topics-to-move-json-file /bitnami/kafka/topics.json  --broker-list <Broker IDs> (1)

Copy the Current partition replica assignment to a file.
Create a copy of the partition replica assignment file, and modify the replicas and log_dirs fields.
For each replicas field, add N number of brokers, depending on the desired replicationFactor.

The log_dirs field must match. For each new broker that you add in the replicas field, add an "any" item in the log_dirs field.

Balance the replicas across all brokers.

Save the new assignment files in the /bitnami/kafka directory:

cat > /bitnami/kafka/topics.assignment.json
<PASTE THE new assignments here>

Run the reassignment:

/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --execute --reassignment-json-file /bitnami/kafka/topics.assignment.json

Verify the reassignment:

/opt/bitnami/kafka/bin/kafka-reassign-partitions.sh --bootstrap-server :9092 --verify --reas

Pinot

Pinot server realtime pods in crash loop back off

Container logs shows the following JFR initialization errors:

jdk.jfr.internal.dcmd.DCmdException: Could not use /var/pinot/server/data/jfr as repository. Unable to create JFR repository directory using base location (/var/pinot/server/data/jfr)Error occurred during initialization of VM
Failure when starting JFR on_create_vm_2

Pinot server realtime disk usage is at 100%.

Resolution

Restart the Pinot server offline:

kubectl rollout restart -n kfuse statefulset pinot-server-offline

Edit pinot-server-realtime sts to set BALLOON_DISK env variable to false.
Wait for pinot server realtime to start, and to complete moving segments to offline servers.
Edit pinot-server-realtime sts to set BALLOON_DISK env variable to true.

Pinot Deepstore access issues

Pinot-related jobs are stuck in a crash loop back-off. For example, kfuse-set-tag-hook, pinot-metrics-table-creation, and similar errors.
Pinot controller logs a Deepstore access-related exception.

Resolution

Configure Deepstore on Pinot.

GCP
- AWS S3

Ensure that the secret has correct access to the cloud storage bucket.

pinot:
    deepStore:
      enabled: true
      type: "s3"
      useSecret: true
      createSecret: true
      dataDir: "s3://[REPLACE BUCKET HERE]/kfuse/controller/data"
      s3:
        region: "YOUR REGION"
        accessKey: "YOUR AWS ACCESS KEY"
        secretKey: "YOUR AWS SECRET KEY"

If Pinot has the correct access credentials to the Deepstore, then the configured bucket creates a directory that matches the dataDir.

Rehydration of segments from Deepstore

As we decommission older Kloudfuse installations and deploy new ones, segments from the old installation can be loaded into the new installation, if:

The Deepstore location for the new installation has a different path from the old installation.
Pinot servers on the new installation have permissions to read from the old deep store location.

Resolution

Use this script to monitor rehydration.

Point the Pinot controller port to the new location:

kubectl port-forward pinot-controller-0 -n kfuse 9000:9000

For each table (kf_metrics, kf_logs, kf_traces, kf_traces_errors, kf_events) run this command:

curl -X POST --fail -H "Content-Type: application/json" -H "TABLE_TYPE:REALTIME" -H "UPLOAD_TYPE:BATCH" -H "DOWNLOAD_URI:<OLD DEEPSTORE PATH>/controller/data/<TABLE NAME>" -v "http://localhost:9000/v2/segments?tableName=<TABLE NAME>&tableType=REALTIME&enableParallelPushProtection=false&allowRefresh=false"

To prevent data loss, do not delete the older Deepstore folder. The new Kloudfuse installation downloads the segments from older deeptstore locations, but still has a reference to it.
Retention on the new cluster does not reset. Instead, Kloudfuse computes it from the time when the data was initially ingested into the cluster (older installation).

Getting ideal state and external view for segments from Pinot controller

To see these, follow these steps:

Ensure that pinot-controller-0 pod is running and fully up:
```
kubectl get pods
```

Enable port-forward for pinot-controller:

kubectl port-forward pinot-controller-0 9000:9000

Dump the ideal state and external view for segments:

curl "http://localhost:9000/tables/<tableName>/idealstate" | jq > ideal_state.json 2>&1
curl "http://localhost:9000/tables/<tableName>/externalview" | jq > external_state.json 2>&1

Replace <tableName> with one of the following, depending on the stream type:
- Metrics: kf_metrics_REALTIME
- Events: kf_events_REALTIME
- Logs: kf_logs_REALTIME
- Traces: kf_traces_REALTIME

Realtime usage continuously increasing

The pinot-server-realtime persistent volume usage keeps increasing when there is a disconnect in segment movement.

Resolution

Restart the pinot-realtime and pinot-offline servers:

kubectl rollout restart sts pinot-server-offline pinot-server-realtime

If PV usage already reached 100% and cannot be restarted gracefully, you must increase the PVC size of pinot-realtime PVCs by approximately 10% to accommodate the increased requirements, and then restart the pinot-server offline and pinot-server realtime.

Storage

Increase existing PVC size

In some scenarios, you have to increase the size of PVC.

Resolution

Run the resize_pvc.sh script from our customer/scripts/ directory.
Ensure that the Helm values.yaml file reflects the updated disk size.

For example, to increase the size of Kafka stateful PVCs to 100GB in Kfuse namespace, run the script:

sh resize_pvc.sh kafka 100Gi kfuse

Resize PVC on Azure

In some scenarios, you have to increase the size of PVC. On Azure, you must detach the PremiumV2_LRS disk before resizing.

Resolution

Cordon all nodes:
```
kubectl cordon <NODE>
```
Delete the statefulset:
```
kubectl sts <STATEFULSET>
```
In Azure Portal, verify that the disk is in unattached state.

Patch all PVCs to the desired size:

kubectl patch pvc <PVC> --patch '{"spec": {"resources": {"requests": {"storage": "'<SIZE>'" }}}}'

Remove the cordon from the node:
```
kubectl uncordon <NODE>
```
Update custom_values.yaml with disk size for the statefulset disk.

[Optional] Run helm upgrade on kfuse using the updated custom_values.yaml file:

helm upgrade --install -n kfuse kfuse <source_location> --version <VERSION.NUM.BER>  -f custom_values.yaml (1)

Fluent-Bit Agent

Duplicate logs in Kloudfuse stack

When using the Fluent-Bit agent, Kloudfuse stack may show duplicate logs with the same timestamp and log event.

Resolution

To help diagnose this issue, add a randomly-generated number or string as part of the Fluent-Bit record.

[FILTER]
      Name lua
      Match *
      Call append_rand_number
      Code function append_rand_number(tag, timestamp, record) math.randomseed(os.clock()*100000000000); new_record = record; new_record["rand_id"] = tostring(math.random(1, 1000000000)); return 1, timestamp, new_record end

Increase the buffer size by adding Buffer_Chunk_Size and Buffer_Max_Size to the configuration of eachtail plugin.

[INPUT]
       Name              tail
       Path              <file_path_to_tail>
       Tag               <tag>
       Buffer_Chunk_Size 1M
       Buffer_Max_Size   8M

Datadog Agent

Kube_cluster_name label does not appear in Kloudfuse stack

The Kube_cluster_name label does not show in Kloudfuse stack when MELT data ingested from Datadog agent is missing the kube_cluster_name label.

Resolution

Perform a rollout restart of the Datadog agent daemonset:

kubectl rollout restart daemonset datadog-agent

Access denied when creating an alert or contact point

A non-admin (SSO) user may get one of these permission errors when creating an alert or a contact point:

Resolution

Log in as an admin user.
Create a contact point or an alert.

UI

Step size differences in charts between Kloudfuse and Grafana UIs

When rendering charts in Kloudfuse UI, we determine the rollup time intervals based on the overall timeframe that the chart renders.

Resolution

None

Additional resources

AWS

InvalidClientTokenId Error

You receive the following error:

Resolution

Several AWS regions are not enabled by default, and this causes the error. To enable your region, and fix the issue, follow the recommendations and steps in the AWS documentation to Enable or disable AWS Regions in your account.

RBAC

Inconsistent RBAC filters on telemetry streams and signals

The APM Services interface combines data from trace streams with RED data from metric streams. When the RBAC policies conflict at the stream level, the APM Services interface does not display some of the expected information.

User unable to suppress (mute) an alert they created

A user may not be able to suppress (mute) an alert that they previously created because of insufficient permission level they have for the folder where they saved the alert.

FuseQL

Advanced Search Alerts fail for Algorithmic Operators

Resolution: We are working to provide a programmatic solution to this issue in an upcoming release.

Alerts

Unable to Create the First Suppress Schedule

Issue fixed in Kloudfuse 3.2.4, see Fixed Issues: Cannot Create First Alert Suppress Schedule.

Resolution: Create an initial suppress schedule using the Grafana interface. Subsequently, create new suppress schedules using the Kloudfuse UI.