Skip to content

Use ansible/terraform modules over go/exec #8

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
k8s-ci-robot merged 1 commit into from
Sep 3, 2025
Merged

Use ansible/terraform modules over go/exec #8

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion kubetest2-tf/data/config.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ variable "kubeconfig_path" {

variable "workers_count" {
description = "Number of workers in the cluster"
default = 1
default = "1"
}

variable "bootstrap_token" {
Expand Down
6 changes: 3 additions & 3 deletions kubetest2-tf/data/k8s-ansible/roles/install-k8s/templates/kubeadm.yaml.j2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ bootstrapTokens:
- signing
- authentication
localAPIEndpoint:
bindPort: {{ apiserver_port }}
bindPort: {{ apiserver_port | int }}
nodeRegistration:
{% if (runtime is defined) and 'containerd' == runtime %}
criSocket: "unix:///run/containerd/containerd.sock"
Expand All @@ -35,7 +35,7 @@ networking:
dnsDomain: "cluster.local"
kubernetesVersion: "{{ release_marker }}"
{% if (loadbalancer is defined) and '' != loadbalancer %}
controlPlaneEndpoint: "{{ loadbalancer }}:{{ apiserver_port }}"
controlPlaneEndpoint: "{{ loadbalancer }}:{{ apiserver_port | int }}"
{% endif %}
apiServer:
{% if (feature_gates is defined) and (runtime_config is defined) %}
Expand Down Expand Up @@ -91,7 +91,7 @@ kind: JoinConfiguration
caCertPath: /etc/kubernetes/pki/ca.crt
discovery:
bootstrapToken:
apiServerEndpoint: :{{ apiserver_port }}
apiServerEndpoint: :{{ apiserver_port | int }}
token: {{ bootstrap_token }}
unsafeSkipCAVerification: true
tlsBootstrapToken: {{ bootstrap_token }}
Expand Down
105 changes: 51 additions & 54 deletions kubetest2-tf/deployer/deployer.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ import (
"encoding/json"
goflag "flag"
"fmt"
"maps"
"net"
"net/url"
"os"
Expand All @@ -19,11 +20,12 @@ import (
"github.com/octago/sflags/gen/gpflag"
"github.com/spf13/pflag"

"k8s.io/client-go/tools/clientcmd"
"k8s.io/klog/v2"
"sigs.k8s.io/kubetest2/pkg/artifacts"
"sigs.k8s.io/kubetest2/pkg/types"

"k8s.io/client-go/tools/clientcmd"
"k8s.io/klog/v2"

"sigs.k8s.io/provider-ibmcloud-test-infra/kubetest2-tf/deployer/options"
"sigs.k8s.io/provider-ibmcloud-test-infra/kubetest2-tf/pkg/ansible"
"sigs.k8s.io/provider-ibmcloud-test-infra/kubetest2-tf/pkg/build"
Expand Down Expand Up @@ -76,10 +78,10 @@ type deployer struct {

RepoRoot string `desc:"The path to the root of the local kubernetes repo. Necessary to call certain scripts. Defaults to the current directory. If operating in legacy mode, this should be set to the local kubernetes/kubernetes repo."`
IgnoreClusterDir bool `desc:"Ignore the cluster folder if exists"`
AutoApprove bool `desc:"Terraform Auto Approve"`
AutoApprove bool `desc:"Auto-approve the deployment of infrastructure through terraform" flag:",deprecated"`
Copy link
Contributor Author

@kishen-v kishen-v Mar 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since auto-approved is no longer supported, setting/unsetting will not have any effect on the execution.

Copy link
Member

@mkumatag mkumatag Aug 28, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

interesting to see that they are always setting the auto-approve in the code - https://github.com/hashicorp/terraform-exec/blob/7ae12dd2176573ec55c9b5c84542d1b466efce54/tfexec/apply.go#L176

Maybe for later: explore whether we can influence the community to support this option, as it would be helpful for a dry‑run test to see what changes are occurring before the actual run.

Copy link
Contributor Author

@kishen-v kishen-v Sep 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Created an issue on the terraform exec repo just to get an opinion!

RetryOnTfFailure int `desc:"Retry on Terraform Apply Failure"`
BreakKubetestOnUpfail bool `desc:"Breaks kubetest2 when up fails"`
Playbook string `desc:"name of ansible playbook to be run"`
Playbook string `desc:"Name of ansible playbook to be run"`
ExtraVars map[string]string `desc:"Passes extra-vars to ansible playbook, enter a string of key=value pairs"`
SetKubeconfig bool `desc:"Flag to set kubeconfig"`
TargetProvider string `desc:"provider value to be used(powervs, vpc)"`
Expand All @@ -96,7 +98,7 @@ func (d *deployer) init() error {
}

func (d *deployer) initialize() error {
fmt.Println("Check if package dependencies are installed in the environment")
klog.Info("Check if package dependencies are installed in the environment")
if d.commonOptions.ShouldBuild() {
if err := d.verifyBuildFlags(); err != nil {
return fmt.Errorf("init failed to check build flags: %s", err)
Expand Down Expand Up @@ -151,17 +153,16 @@ func New(opts types.Options) (types.Deployer, *pflag.FlagSet) {
}
klog.InitFlags(nil)
flagSet.AddGoFlagSet(goflag.CommandLine)
fs := bindFlags(d)
fs := bindFlags()
flagSet.AddFlagSet(fs)
return d, flagSet
}

func bindFlags(d *deployer) *pflag.FlagSet {
func bindFlags() *pflag.FlagSet {
flags := pflag.NewFlagSet(Name, pflag.ContinueOnError)
common.CommonProvider.BindFlags(flags)
vpc.VPCProvider.BindFlags(flags)
powervs.PowerVSProvider.BindFlags(flags)

return flags
}

Expand All @@ -179,9 +180,8 @@ func (d *deployer) Up() error {
if err != nil {
return fmt.Errorf("failed to dumpconfig to: %s and err: %+v", d.tmpDir, err)
}

for i := 0; i <= d.RetryOnTfFailure; i++ {
path, err := terraform.Apply(d.tmpDir, d.TargetProvider, d.AutoApprove)
path, err := terraform.Apply(d.tmpDir, d.TargetProvider)
op, oerr := terraform.Output(d.tmpDir, d.TargetProvider)
if err != nil {
if i == d.RetryOnTfFailure {
Expand All @@ -201,24 +201,29 @@ func (d *deployer) Up() error {
}
}
inventory := AnsibleInventory{}
tfMetaOutput, err := terraform.Output(d.tmpDir, d.TargetProvider)
if err != nil {
return err
}
var tfOutput map[string][]interface{}
data, err := json.Marshal(tfMetaOutput)
if err != nil {
return fmt.Errorf("error while marshaling data %v", err)
}
if err := json.Unmarshal(data, &tfOutput); err != nil {
return fmt.Errorf("error while unmarshaling data %v", err)
}
for _, machineType := range []string{"Masters", "Workers"} {
var tmp []interface{}
op, err := terraform.Output(d.tmpDir, d.TargetProvider, "-json", strings.ToLower(machineType))

if err != nil {
return fmt.Errorf("terraform.Output failed: %v", err)
}
klog.Infof("%s: %s", strings.ToLower(machineType), op)
err = json.Unmarshal([]byte(op), &tmp)
if err != nil {
return fmt.Errorf("failed to unmarshal: %v", err)
}
for index := range tmp {
inventory.addMachine(machineType, tmp[index].(string))
d.machineIPs = append(d.machineIPs, tmp[index].(string))
if machineIps, ok := tfOutput[strings.ToLower(machineType)]; !ok {
return fmt.Errorf("error while unmarshaling machine IPs from terraform output")
} else {
for _, machineIp := range machineIps {
inventory.addMachine(machineType, machineIp.(string))
d.machineIPs = append(d.machineIPs, machineIp.(string))
}
}
}
klog.Infof("inventory: %v", inventory)
klog.Infof("Kubernetes cluster node inventory: %+v", inventory)
t := template.New("Ansible inventory file")

t, err = t.Parse(inventoryTemplate)
Expand All @@ -228,54 +233,46 @@ func (d *deployer) Up() error {

inventoryFile, err := os.Create(filepath.Join(d.tmpDir, "hosts"))
if err != nil {
klog.Errorf("Error while creating a file: %v", err)
return fmt.Errorf("failed to create inventory file: %v", err)
return fmt.Errorf("failed to create ansible inventory file: %v", err)
}

err = t.Execute(inventoryFile, inventory)
if err != nil {
return fmt.Errorf("template execute failed: %v", err)
if err = t.Execute(inventoryFile, inventory); err != nil {
return fmt.Errorf("ansible inventory file templatation failed: %v", err)
}

common.CommonProvider.ExtraCerts = strings.Join(inventory.Masters, ",")

commonJSON, err := json.Marshal(common.CommonProvider)
ansibleParams, err := json.Marshal(common.CommonProvider)
if err != nil {
return fmt.Errorf("failed to marshal provider into JSON: %v", err)
}
klog.Infof("commonJSON: %v", string(commonJSON))
//Unmarshalling commonJSON into map to add extra-vars
final := map[string]interface{}{}
json.Unmarshal(commonJSON, &final)
//Iterating through extra-vars and adding them to map
for k := range d.ExtraVars {
final[k] = d.ExtraVars[k]
}
//Marshalling back the map to JSON
finalJSON, err := json.Marshal(final)
if err != nil {
return fmt.Errorf("failed to marshal provider into JSON: %v", err)
klog.Infof("Ansible params exposed under groupvars: %v", string(ansibleParams))
// Unmarshalling commonJSON into map to add extra-vars
combinedAnsibleVars := map[string]string{}
if err = json.Unmarshal(ansibleParams, &combinedAnsibleVars); err != nil {
return fmt.Errorf("failed to unmarshal ansible group variables: %v", err)
}
klog.Infof("finalJSON with extra vars: %v", string(finalJSON))

exitcode, err := ansible.Playbook(d.tmpDir, filepath.Join(d.tmpDir, "hosts"), string(finalJSON), d.Playbook)
if err != nil {
return fmt.Errorf("failed to run ansible playbook: %v\n with exit code: %d", err, exitcode)
// Add-in the extra-vars set to the final set.
maps.Insert(combinedAnsibleVars, maps.All(d.ExtraVars))
klog.Infof("Updated ansible variables with extra vars: %+v", combinedAnsibleVars)
if err = ansible.Playbook(d.tmpDir, filepath.Join(d.tmpDir, "hosts"), d.Playbook, combinedAnsibleVars); err != nil {
return fmt.Errorf("failed to run ansible playbook: %v", err)
}

if d.SetKubeconfig {
if err = setKubeconfig(inventory.Masters[0]); err != nil {
if err := setKubeconfig(inventory.Masters[0]); err != nil {
return fmt.Errorf("failed to setKubeconfig: %v", err)
}
fmt.Printf("KUBECONFIG set to: %s\n", os.Getenv("KUBECONFIG"))
klog.Infof("KUBECONFIG set to: %s", os.Getenv("KUBECONFIG"))
}

if isUp, err := d.IsUp(); err != nil {
klog.Warningf("failed to check if cluster is up: %v", err)
} else if isUp {
klog.V(1).Infof("cluster reported as up")
klog.V(1).Info("cluster reported as up")
} else {
klog.Errorf("cluster reported as down")
klog.Error("cluster reported as down")
}

klog.Infof("Dumping cluster info..")
Expand All @@ -294,7 +291,7 @@ func setKubeconfig(host string) error {

config, err := clientcmd.LoadFromFile(common.CommonProvider.KubeconfigPath)
if err != nil {
klog.Error("failed to load the kubeconfig file")
klog.Errorf("failed to load the kubeconfig file. error: %v", err)
}
for i := range config.Clusters {
surl, err := url.Parse(config.Clusters[i].Server)
Expand All @@ -313,7 +310,7 @@ func setKubeconfig(host string) error {
if err != nil {
return fmt.Errorf("failed to create absolute path for the kubeconfig file: %v", err)
}
if err = os.Setenv("KUBECONFIG", kubecfgAbsPath); err != nil {
if err := os.Setenv("KUBECONFIG", kubecfgAbsPath); err != nil {
return fmt.Errorf("failed to set the KUBECONFIG environment variable")
}
return nil
Expand All @@ -324,7 +321,7 @@ func (d *deployer) Down() error {
if err := d.init(); err != nil {
return fmt.Errorf("down failed to init: %s", err)
}
err := terraform.Destroy(d.tmpDir, d.TargetProvider, d.AutoApprove)
err := terraform.Destroy(d.tmpDir, d.TargetProvider)
if err != nil {
if common.CommonProvider.IgnoreDestroy {
klog.Infof("terraform.Destroy failed: %v", err)
Expand Down
15 changes: 13 additions & 2 deletions kubetest2-tf/go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,9 @@ toolchain go1.24.1

require (
github.com/IBM/ibm-cos-sdk-go v1.12.1
github.com/apenella/go-ansible/v2 v2.1.1
github.com/hashicorp/terraform-exec v0.22.0
github.com/octago/sflags v0.3.1
github.com/pkg/errors v0.9.1
github.com/spf13/pflag v1.0.6
k8s.io/client-go v0.32.2
k8s.io/cluster-bootstrap v0.32.2
Expand Down Expand Up @@ -54,6 +55,9 @@ require (
github.com/alibabacloud-go/tea-utils v1.4.5 // indirect
github.com/alibabacloud-go/tea-xml v1.1.3 // indirect
github.com/aliyun/credentials-go v1.3.2 // indirect
github.com/apenella/go-common-utils/data v0.0.0-20220913191136-86daaa87e7df // indirect
github.com/apenella/go-common-utils/error v0.0.0-20220913191136-86daaa87e7df // indirect
github.com/apparentlymart/go-textseg/v15 v15.0.0 // indirect
github.com/asaskevich/govalidator v0.0.0-20230301143203-a9d515a09cc2 // indirect
github.com/aws/aws-sdk-go-v2 v1.30.3 // indirect
github.com/aws/aws-sdk-go-v2/config v1.27.27 // indirect
Expand Down Expand Up @@ -149,7 +153,9 @@ require (
github.com/gorilla/mux v1.8.1 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/hashicorp/go-retryablehttp v0.7.7 // indirect
github.com/hashicorp/go-version v1.7.0 // indirect
github.com/hashicorp/hcl v1.0.1-vault-5 // indirect
github.com/hashicorp/terraform-json v0.24.0 // indirect
github.com/in-toto/in-toto-golang v0.9.0 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
github.com/jbenet/go-context v0.0.0-20150711004518-d14ea06fba99 // indirect
Expand Down Expand Up @@ -187,6 +193,8 @@ require (
github.com/pborman/uuid v1.2.1 // indirect
github.com/pelletier/go-toml/v2 v2.1.1 // indirect
github.com/pjbgf/sha1cd v0.3.0 // indirect
github.com/pkg/errors v0.9.1 // indirect
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
github.com/power-devops/perfstat v0.0.0-20221212215047-62379fc7944b // indirect
github.com/prometheus/client_golang v1.20.2 // indirect
github.com/prometheus/client_model v0.6.1 // indirect
Expand Down Expand Up @@ -218,6 +226,8 @@ require (
github.com/spf13/cobra v1.8.1 // indirect
github.com/spf13/viper v1.18.2 // indirect
github.com/spiffe/go-spiffe/v2 v2.2.0 // indirect
github.com/stretchr/objx v0.5.2 // indirect
github.com/stretchr/testify v1.10.0 // indirect
github.com/subosito/gotenv v1.6.0 // indirect
github.com/syndtr/goleveldb v1.0.1-0.20220721030215-126854af5e6d // indirect
github.com/tchap/go-patricia/v2 v2.3.1 // indirect
Expand All @@ -234,6 +244,7 @@ require (
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/yashtewari/glob-intersection v0.2.0 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zclconf/go-cty v1.16.1 // indirect
github.com/zeebo/errs v1.3.0 // indirect
gitlab.alpinelinux.org/alpine/go v0.9.0 // indirect
go.mongodb.org/mongo-driver v1.14.0 // indirect
Expand All @@ -249,7 +260,7 @@ require (
go.uber.org/zap v1.27.0 // indirect
golang.org/x/crypto v0.36.0 // indirect
golang.org/x/exp v0.0.0-20240719175910-8a7402abbf56 // indirect
golang.org/x/mod v0.21.0 // indirect
golang.org/x/mod v0.22.0 // indirect
golang.org/x/net v0.38.0 // indirect
golang.org/x/oauth2 v0.27.0 // indirect
golang.org/x/sync v0.12.0 // indirect
Expand Down
Loading
Loading