Terraform

The process of setting up a working Managed kdb environment manually can take some time - especially if you are new to AWS. To aid this process we have a Terraform deployment option which should allow you to boot TorQ in Managed kdb Insights in a few simple commands.

This Terraform script can be used to deploy an entire environment from scratch. This will include:

creating and uploading data to S3 buckets with required policies
creating IAM roles
creating network and transit gateway
as well as deploying clusters.

It is split into two modules, one for the environment and one for the clusters. This makes the directory more organised and cluster deployments easier to manage. The cluster module is still dependent on the environment module as it will import some variables from here that are needed for cluster creation.

Terraform Module Diagram

This Terraform setup is designed to deploy and manage a Managed kdb Insights environment running a TorQ bundle.

Prerequisites

Ensure you have followed the standard prerequisite steps to ensure you have the latest versions of code.
Have the latest version of the AWS CLI installed.
Have the latest version of Terraform installed.
Configure the AWS CLI to your AWS account.
Create a KMS key in the region where you intend to set up your environment. You will also need to edit the key policy to grant FinSpace permissions.
Note that FinSpace environments are limited to one per region. Make sure you don't already have an environment set up in the same region.
This instruction refers to Linux and would only work under the Linux environment.

Resource Link

For detailed Terraform deployment instructions, refer to TorQ in Finspace Deployment / Terraform.

How to Use - Initial Deployment

New user please continue and follow this section - Users with existing infrastructure, please skip to our existing infrastructure section.

(Optional) If you have an HDB you want to migrate to FinSpace, replace the dummy HDB in TorQ-Amazon-FinSpace-Starter-Pack/hdb.
Move into the TorQ-Amazon-FinSpace-Starter-Pack/terraform-deployment/deployments directory; this will be the Terraform working directory from which you should run all terraform commands.
Modify variables inside the terraform.tfvars file, such as region name, environment name, database name. You can modify it by replacing the variable name inside of "Name". For example, For the variable on role-name, you can change the variable name by replacing "finspace-role".
(Optional) If you have changed the database name from the default finspace-database to any other names, please also edit the env.q file, changing the database name to the new variable that you have set in line 19.
Run aws configure in the terminal to set up your access key and secret key from your AWS account. This is needed to connect to your account and use the Terraform deployment. Check our resource link for more instructions on how to find your access key and secret key Prerequisites.
From your Terraform working directory which is TorQ-Amazon-FinSpace-Starter-Pack/terraform-deployment/deployments, run terraform init.
If initialized without error, run terraform plan. This will show all resources set to be created or destroyed by Terraform.
Run terraform apply to execute this plan. The initial deployment can take approximately 45 minutes, and connection losses can cause errors with deployment, so it's a good idea to run this in nohup. (Using nohup might lead to a higher cost of operating the codes if you are using Terraform from a cloud environment.) Example nohup run: nohup terraform apply -auto-approve > terraform_apply.log 2>&1 &.

You can now skip ahead to our Managing Your Infrastructure section

How to Use - Deploying With Terraform For Users With Existing Infrastructure

For users with existing infrastructure in their AWS account who would like to reuse the same resources for their TorQ in Finspace bundle, you can use import blocks in Terraform. This functionality allows you to import existing infrastructure resources into Terraform, bringing them under Terraform's management. The import block records that Terraform imported the resource and did not create it. After importing, you can optionally remove import blocks from your configuration or leave them as a record of the resource's origin.

Once imported, Terraform tracks the resource in your state file. You can then manage the imported resource like any other, updating its attributes and destroying it as part of a standard resource lifecycle.

Move into the deployments directory, and you'll see an imports.tf file (currently empty). This imports.tf file is automatically run before Terraform applies any changes to the structure, importing existing structures from your AWS to the deployment system.

Terraform Import Block Syntax

import {
  to = aws_instance.example
  id = "i-abcd1234"
}

resource "aws_instance" "example" {
  name = "hashi"
  # (other resource arguments...)
}

The above import block defines an import of the AWS instance with the ID "i-abcd1234" into the aws_instance.example resource in the root module.

The import block has the following arguments:

Argument	Description	Example
`to`	The instance address this resource will have in your state file.	`to = aws_instance.example`
resourse (e.g. `id` or `name`)	A string with the import information of the resource.	Example 1. `id = "i-abcd1234"` Example 2. `name = "aws/vendedlogs/finspace/myclustername"`
`provider` (optional)	An optional custom resource provider. If you do not set the provider argument, Terraform attempts to import from the default provider.	See The Resource provider Meta-Argument for details.

The import block's ID/name argument can be a literal string of your resource's import ID, or an expression that evaluates to a string. Terraform needs this detail to locate the resource you want to import.

The import ID/name must be known at plan time for planning to succeed. If the value of id/name is only known after apply, terraform plan will fail with an error.

The identifier you use for a resource's import ID/name is resource-specific. You can find the required ID/name in the provider documentation for the resource you wish to import.

Terraform import block Template

We have created a Terraform import block template in terraform-deployment/importtemplate.md. In this template, you can select the needed import block and paste it into the imports.tf file within the terraform-deployment/deployments/imports.tf directory. Remember to change the ID to the referring ID of your existing infrastructure.

Managing Your Infrastructure

Once your environment is up and running, you can use this configuration to manage it:

Code Updates: If you make any code changes in TorQ or TorQ-Amazon-FinSpace-Starter-Pack and want to apply these to your clusters, rezip these directories and run the Terraform deployment again. This will recreate clusters with the updated code.
Cluster Config: If you want to make changes to a cluster's config settings (e.g., node size of the RDB), update this in clusters/rdb.tf and run Terraform again. The RDB will be recreated with the new node size.
Delete/Create Clusters: Clusters can be deleted or created individually or all at once from the terraform.tfvars file. To delete a cluster, set its count to 0. To delete all clusters, set create-clusters to 0.

Basic Commands in Terraform

terraform init - Prepare your working directory for other commands
terraform validate - Check whether the configuration is valid
terraform plan - Show changes required by the current configuration
terraform apply - Create or update infrastructure
terraform destroy - Destroy previously-created infrastructure
For more commands in Terraform, please visit Terraform Command

Terraform State Management

Terraform maintains a state file that tracks the state of the deployed infrastructure. This state file is crucial for Terraform to understand what resources have been created and to make changes to them. To ensure proper state management:

Always store your state files securely, as they may contain sensitive information.
Consider using remote state storage, such as Amazon S3, to keep your state files safe and accessible from multiple locations.
Avoid manual changes to resources managed by Terraform, as this can lead to inconsistencies between the actual infrastructure and Terraform's state.

List of AWS Structures that will be created with our Terraform deployment

module.clusters.aws_finspace_kx_cluster.discovery-cluster[0]
module.clusters.aws_finspace_kx_cluster.feed-cluster[0]
module.clusters.aws_finspace_kx_cluster.gateway-cluster[0]
module.clusters.aws_finspace_kx_cluster.hdb-cluster[0]
module.clusters.aws_finspace_kx_cluster.rdb-cluster[0]
module.environment.data.aws_iam_policy_document.iam-policy
module.environment.data.aws_iam_policy_document.s3-code-policy
module.environment.data.aws_iam_policy_document.s3-data-policy
module.environment.aws_ec2_transit_gateway.test
module.environment.aws_finspace_kx_database.database
module.environment.aws_finspace_kx_environment.environment
module.environment.aws_finspace_kx_user.finspace-user
module.environment.aws_iam_policy.finspace-policy
module.environment.aws_iam_role.finspace-test-role
module.environment.aws_iam_role_policy_attachment.policy_attachment
module.environment.aws_s3_bucket.finspace-code-bucket
module.environment.aws_s3_bucket.finspace-data-bucket
module.environment.aws_s3_bucket_policy.code-policy
module.environment.aws_s3_bucket_policy.data-policy
module.environment.aws_s3_bucket_public_access_block.code_bucket
module.environment.aws_s3_bucket_public_access_block.data_bucket
module.environment.aws_s3_bucket_versioning.versioning
module.environment.null_resource.create_changeset
module.environment.null_resource.upload_hdb
module.network.aws_internet_gateway.finspace-igw
module.network.aws_route.finspace-route
module.network.aws_route_table.finspace-route-table
module.network.aws_security_group.finspace-security-group
module.network.aws_route_table_association.subnet-assocations[0]
module.network.aws_route_table_association.subnet-assocations[1]
module.network.aws_route_table_association.subnet-assocations[2]
module.network.aws_subnet.finspace-subnets[0]
module.network.aws_subnet.finspace-subnets[1]
module.network.aws_subnet.finspace-subnets[2]
module.network.aws_subnet.finspace-subnets[3]
module.network.aws_vpc.finspace-vpc

References and Documentation

For more in-depth information and documentation, explore the following resources:

These resources provide detailed information about Terraform and AWS services, best practices, and advanced configurations.