AWS Terraform Integration with Kloudfuse :: Kloudfuse Docs

AWS Terraform Integration with Kloudfuse

Build the required infrastructure with Terraform to deploy Kloudfuse in AWS.

See the Terraform documentation for Get Started — AWS for further details.

Pre-Requisites

Enable these artifacts to configure S3 as a Terraform backend:

For details, see Terraform Configuration.

Terraform Configuration

You must change the backend.tf file in each component folder, according to the created resources in S3 and dynamodb. Note that the state key is unique for each component; for a VPC, it can be vpc.tfstate.

Example Terraform Configuration

terraform {
  backend "s3"{
    bucket                 = "bucket-name"
    region                 = "us-west-2"
    key                    = "<resource>.tfstate"
    dynamodb_table         = "dynamodb-table-name"
  }
}

Terraform Variables

You must modify the variables in variables.tf for every deployed AWS component.

Configure Infrastructure for Kloudfuse

Choose the method for using Terraform to configure the infrastructure:

Configuration for AWS Accounts without Infrastructure

For AWS account without provisioned infrastructure, complete these steps:

  1. Provision VPC: Create the VPC using the terraform files for Kloudfuse, from the kloudfuse/terraform/netwworking/ directory.

    Be sure to modify the variables in the variables.tf file to reflect the instance requirements for VPC CIDR, Public subnet CIDR, VPC name, and so on.

  2. Provision S3: Ensure that the Kloudfuse datastore Pinot has access to an S3 bucket for long-term storage. See Terraform resources for creating the appropriate S3 bucket in the terraform/s3/ directory.

  3. Provision EKS: After creating and provisioning the VPC, use the Terraform resources in the /terraform/eks/ directory to provision the EKS cluster.

    Use the terraform_remote_state block in main.tf ` file to retrieve VPC S3 Backend values for public subnet IDs.

    You must modify the variables in the variables.tf file. need to be modified; refer to Terraform Variables. Be sure to add the proper ARN for cluster users in the principal_* variables.

  4. Provision Route53: Create a public hosted zone for the ACM certificate. Use the Terraform files in the terraform/route53/ directory.

  5. Provision ACM: Create certificate to enable SSL/TLS. Use the Terraform files in the terraform/acm/ directory.

Configuration for AWS Accounts with non-EKS VPC Infrastructure

For AWS accounts with provisioned VPC that run a non-EKS cluster, you must add EKS, S3, ACM, and Route53.
Because VPC is created without using Terraform files supplied by Kloudfuse, you must provide the public subnet IDs in the variables.tf file and use them in the main.tf file.

  1. Provision S3: Ensure that the Kloudfuse datastore Pinot has access to an S3 bucket for long-term storage. See Terraform resources for creating the appropriate S3 bucket in the terraform/s3/ directory.

  2. Provision EKS: After creating and provisioning the VPC, use the Terraform resources in the /terraform/eks/ directory to provision the EKS cluster.

    Use the terraform_remote_state block in main.tf ` file to retrieve VPC S3 Backend values for public subnet IDs.

    You must modify the variables in the variables.tf file. need to be modified; refer to Terraform Variables. Be sure to add the proper ARN for cluster users in the principal_* variables.

  3. Provision Route53: Create a public hosted zone for the ACM certificate. Use the Terraform files in the terraform/route53/ directory.

  4. Provision ACM: Create certificate to enable SSL/TLS. Use the Terraform files in the terraform/acm/ directory.

Configuration for AWS Accounts with EKS Cluster

For AWS accounts with an existing EKS cluster (shared or dedicated), you must only deploy Kloudfuse. These are the possible deployment options:

Terraform and Helm

  1. Copy the Kloudfuse token.json and your ~/.kube/config (as kubeconfig in destination) files in the /prekfuse folder.

  2. Update the config in the kubeconfig for the EKS Cluster where you plan to deploy Kloudfuse.

  3. Run the Kloudfuse pre-requisite Terraform files for setting up the cluster using the files in the terraform/prekfuse/ directory.

    This also creates the namespace.

  4. Login to the Kloudfuse Helm registry.

  5. Create a secret so the services can pull their Helm charts.

  6. Run the Helm upgrade command:

    helm upgrade --install -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse --version 2.7.4  -f custom_values.yaml

Kubectl Commands and Helm

  1. Login to Kloudfuse Helm registry using the token.json that Kloudfuse sent to your email address.

  2. Create a secret for Helm, to pull the required docker images.

    kubectl create ns kfuse
kubectl config set-context --current --namespace=kfuse
kubectl create secret docker-registry kfuse-image-pull-credentials \
           --namespace='kfuse' --docker-server 'us.gcr.io' --docker-username _json_key \
           --docker-email 'container-registry@mvp-demo-301906.iam.gserviceaccount.com' \
           --docker-password='"$(cat token.json)"'

  3. Run the following command to install the Kloudfuse chart.

    helm upgrade \
   --install -n kfuse kfuse oci://us-east1-docker.pkg.dev/mvp-demo-301906/kfuse-helm/kfuse
   --version <VERSION.NUM.BER>  -f custom_values.yaml

1 Use the most current Kloudfuse version number.