Insecure Harbor registry with Tanzu Kubernetes Grid+ on vSphere

August 6, 2020 By Corey Dinkens

In searching for ways to use an ‘insecure’ registry with Tanzu Kubernetes Grid, I found a post by William Lam explaining how. I wanted to build on this and figure out what was needed to deploy a customized cluster with tkg-cli, and skip deploying the kind cluster first.

The files located in .tkg/bom/ are the key to this, as they are the templates that tkg-cli uses to bootstrap kind, and deploy the TKG cluster. With the same technique, you can perform additional customization as needed; possibly more on that in the future.

Step 1.

Locate and open the following files (should be in home dir) in editor of choice:

  • .tkg/providers/infrastructure-vsphere/v0.6.4/cluster-template-dev.yaml
  • .tkg/providers/infrastructure-vsphere/v0.6.4/cluster-template-prod.yaml

Step 2.

Insert the customized files block below the last line of the preKubeadmCommands section:

    - hostname "{{ ds.meta_data.hostname }}"
    - echo "::1         ipv6-localhost ipv6-loopback" >/etc/hosts
    - echo "   localhost" >>/etc/hosts
    - echo "   {{ ds.meta_data.hostname }}" >>/etc/hosts
    - echo "{{ ds.meta_data.hostname }}" >/etc/hostname
      - path: /etc/containerd/config.toml
        content: |
          version = 2
              sandbox_image = ""
                default_runtime_name = "runc"
                    runtime_type = "io.containerd.runc.v2"
                    runtime_type = "io.containerd.runc.v2"
                  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]
                    endpoint = ["http://<fqdn or registry IP>:80"]

Step 3.

Locate and open:

  • .tkg/bom/bom-1.1.2+vmware.1.yaml
    (edit the file that matches the version of tkg being used or version of k8s being deployed. I am using tkg 1.1.2)

Locate the kubeadmConfigSpec section, and insert your customized kindKubeadmConfigSpec immediately after:

  kind: ClusterConfiguration
  kubernetesVersion: v1.18.3+vmware.1
      dataDir: /var/lib/etcd
      imageTag: v3.4.3_vmware.5
    type: CoreDNS
    imageTag: v1.6.7_vmware.1
- 'kind: Cluster'
- 'apiVersion:'
- 'containerdConfigPatches:'
- '- |-'
- '  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."<fqdn or registry IP>:80"]'
- '    endpoint = ["https://<fqdn or registry IP>:80"]'
- 'kubeadmConfigPatches:'
- '- |'
- '  apiVersion:'
- '  kind: ClusterConfiguration'
- '  imageRepository:'
- '  etcd:'
- '    local:'
- '      imageRepository:'
- '      imageTag: v3.4.3_vmware.5'
- '  dns:'
- '    type: CoreDNS'
- '    imageRepository:'
- '    imageTag: v1.6.7_vmware.1'

Step 4.

Verify config output by doing a dry run:

tkg create cluster –plan {dev|prod|custom} –dry-run

Step 5.

If the dry run succeeded, deploy a TKG cluster using:

tkg create cluster –plan {dev|prod|custom}

(In case you were wondering: creating custom plans)

You can verify that the above steps have successfully patched the node by SSHing into a node, and

 cat /etc/containerd/config.toml
You should see the containerd configuration that you added above


William Lam’s post on TKG + Insecure registry:

Erick aka Gubi on custom TKG plans:

Versions Used

vSphere 6.7u3kubectl: 1.18
TKG (Tanzu Kubernetes Grid+): 1.1.2Kubernetes: 1.18.3
VIC (vSphere Intergrated Containers): 1.5.5tkg-cli: 1.1.2