How can I include vendor charts (using the helm3 mixin) in an exported bundle for an air-gapped install?

[Steve973]

I may have originally mis-filed this as an issue (of the question subtype) but this is probably a better place for it.

What is your question?
I want to include vendor charts when I export a bundle for an air-gapped install. Aren't charts supposed to be included, since air-gapped systems do not have network connectivity?

What have you tried already?
I am using the helm3 mixin in my porter manifest, and I build my bundle and export it as a tgz file. The build output shows the addition of the helm repos that are defined in my manifest, but it does not show the charts being included.

Where have you looked already to figure this out?
I have read all of the porter documentation, and the relevant parts multiple times. I have read the helm3 mixin documentation at the gitlab repo for that mixin, but nothing mentions the download and inclusion of vendor charts.

Was the existing documentation unclear or had gaps?
Nothing mentions vendor charts, specifically. Most documentation includes charts that exist in a place relative to the porter manifest.

Additional Notes
I have filed an issue at the helm3 mixin MChorfa/porter-helm3#56 and I have pushed my porter manifest and the shell script that I am using to build and export my bundle (so that it is repeatable, and so that anyone can see the exact steps that I am taking to do everything with this bundle) to github. You can see my repo here.

Am I doing something wrong, or am I misunderstanding something? If anyone needs any additional information, please let me know. Thanks in advance!

[kichristensen]

To the best of my knowlegde the only way to do vendoring it to download the Helm charts locally, next to porter.yaml, before running porter build, and the it the helm3 mixin refer to the downloaded tgz file instead of an externally chart reference.
The reason is that the mixin is never invoked during porter build, it is only added to the invocation image.

[kichristensen]

Ahh yes, adding the repos is done during porter build (generating the invocation image). The reason is that it is part of the mixin configuration, https://github.com/Steve973/porter-example-app-stack/blob/f02c5c607596642f22bbe15a60c167b18c2cc198/porter.yaml#L24-L30.
But the configuration says nothing about what charts that should be install later on when you call, e.g., porter install.

[Steve973]

Ok, but shouldn't an air-gapped install contain everything? That is a use case that I could make excellent use of in my enterprise. If it works well, I think that many devops engineers will love it! So, I want to come up with a good example for them to demonstrate how cool porter is. At the moment, I am attempting to learn all of this by doing this experiment and deploying on a k3s cluster on my personal development machine.

For the time being, I can use a shell script to download the manifests, but I (fairly strongly) believe that the helm3 mixin should probably ensure that any referenced charts are available. If there's a reference to a relative path to a chart, I'd think that it should verify that. If the chart is going to come from a repository, then it seems reasonable that it should be pulled. Of course, there are use cases that software supports, and other use cases that it does not support. But the porter documentation specifically shows the air-gapped install as a use case. Therefore, it may be helpful to include something in the documentation about the limitations of things (e.g., mixins) that will not work for the air-gapped use case.

While we are also talking about documentation, I tried to follow the instructions for setting up Porter for KinD to make use of my newly-installed k3s cluster. The cluster is at https://localhost:6443, but even though I created credentials for k3s that points to my kube config file:

{
    "schemaType": "CredentialSet",
    "schemaVersion": "1.0.1",
    "name": "k3s_creds",
    "credentials": [
        {
            "name": "kubeconfig",
            "source": {
                "path": "/home/steve/.kube/config"
            }
        }
    ]
}

it still tries to connect via http://localhost:8080, or it at least checks by trying http://localhost:8080/version.

I am willing to contribute and even help with the documentation if something that I'm attempting to do might be useful to others. My specific use case is to author a bundle on a computer that is connected to the internet, and then to upload it to an enclave where I can provide the bundle to people who want to run the bundled software on their cluster. Do you think that we can make this happen with my example project?

[kichristensen]

If I follow the guide Connect to KinD everything works for me. I haven't tried to run it with k3s.
Did you apply the credentials multiple times? Or only once after configure the kube config correctly.

[Steve973]

I tried switching to minikube, and when the helm mixin runs, it keeps trying to reach the cluster at http://localhost:8080, and so it always fails. I have ~/.kube/config set up correctly, but that doesn't help, particularly when the helm3 mixin is trying to run. Did I miss something in the documentation?

[kichristensen]

Could you try look at the kube config file inside the invocation image?
After you run porter build you should get a new image in Docker. In you change the entrypoint to /bin/bash you should be able check the contents of the kubeconfig file and see if it contains the expected cluster address.
Running something like this docker run -it --entrypoint /bin/bash localhost:32769/porter-hello:porter-93f8fd90e0a57dba2ed53a680cfdd294 would get you into the container image, of course the image name will be different.

[Steve973]

It's going to be whatever the default is, right? I haven't configured anything in my Porter manifest, because I wouldn't want to decide something for the target system. So, how can I help Porter, or the bundle, know the kubernetes URL on the target system?

…

On Fri, Apr 19, 2024, 6:05 PM Kim Christensen ***@***.***> wrote: Could you try look at the kube config file inside the invocation image? After you run porter build you should get a new image in Docker. In you change the entrypoint to /bin/bash you should be able check the contents of the kubeconfig file and see if it contains the expected cluster address. Running something like this docker run -it --entrypoint /bin/bash localhost:32769/porter-hello:porter-93f8fd90e0a57dba2ed53a680cfdd294 would get you into the container image, of course the image name will be different. — Reply to this email directly, view it on GitHub <#3081 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AD5JDW6V2MBZOMKBOLB3JXLY6GIL3AVCNFSM6AAAAABGGFBSY2VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TCNZRGI3TC> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Steve973]

Since it's trying to reach kubernetes at the wrong URL, I am pretty confident that it is not seeing (or maybe not even attempting to read) the ~/.kube/config file at all. My question continues to be, "How do I tell porter, at install time, what my kube config is?" I haven't seen this discussed in the documentation, other than applying a credentials file, which I have attempted to apply:

{
    "schemaType": "CredentialSet",
    "schemaVersion": "1.0.1",
    "name": "mk",
    "credentials": [
        {
            "name": "kubeconfig",
            "source": {
                "path": "/home/steve/.kube/config"
            }
        }
    ]
}

I wonder how other people are making this work...

[kichristensen]

Credentials are resolve during porter install, not during build. The credential set tells porter where to find the kubeconfig when it should be resolved. During porter install you instruct porter to use the credential set using -c.
when installing the bundle in an air gapped environment or on another machine, you create a new credential set with the path to the correct kubeconfig available on the machine running the installation

[Steve973]

That is what I have been trying. Here's my kube config, minified, but you can see the URL:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://127.0.0.1:6443
  name: rancher-desktop
contexts:
- context:
    cluster: rancher-desktop
    user: rancher-desktop
  name: rancher-desktop
current-context: rancher-desktop
kind: Config
preferences: {}
users:
- name: rancher-desktop
  user:
    client-certificate-data: DATA+OMITTED
    client-key-data: DATA+OMITTED

Now, when I try to install with the credentials I've applied to the distribution:

porter install -c mk example-app-stack                                                                                                                                                                                                                                                                                        

executing install action from example-app-stack (installation: /example-app-stack)
Install Redpanda from the vendor chart
/usr/local/bin/helm3 /usr/local/bin/helm3 upgrade --install redpanda ./charts/redpanda --atomic --create-namespace --set global.name=redpanda --set namespace=$redpanda-namespace
Error: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080: connect: connection refused
Error: exit status 1
err: exit status 1
1 error occurred:
        * mixin execution failed: package command failed /cnab/app/cnab/app/mixins/helm3/runtimes/helm3-runtime install
Error: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080: connect: connection refused
Error: exit status 1

1 error occurred:
        * mixin execution failed: package command failed /cnab/app/cnab/app/mixins/helm3/runtimes/helm3-runtime install
Error: Kubernetes cluster unreachable: Get "http://localhost:8080/version": dial tcp 127.0.0.1:8080: connect: connection refused
Error: exit status 1

1 error occurred:
        * container exit code: 1, message: <nil>

It seems like I'm doing everything that the documentation says, but this is not working.

[kichristensen]

I have an idea on what is going on here. In the repo you linked, you don't have any credentials defined. Is the repo updated, or are you missing the credential definition in the bundle? An example of how it looks can be found in the helm3 mixin repo. It seem to be that Helm will return the error you are getting when the kubeconfig is not available.

[Steve973]

Thanks for that. It got me a little bit farther. Now it is trying to connect to https://127.0.0.1:6443 which is the correct address from my kube config file. The connection is still refused, but I'm moving in the right direction. I'll try more tomorrow!

[Steve973]

@kichristensen could you please have a look at the repo? I pushed my updates with things working much better. I am running into an issue where it tries to install redpanda and then it times out, and I am not able to tell why.

[Steve973]

Hello. I finally had some time to re-run this with the debug verbosity set, and this is the output:

Resolving parameter sources...
Resolving parameter source porter-state
Injected installation /example-app-stack output porter-state as parameter porter-state
Selected default plugin {"plugin-key": "host"}
Connecting to plugin    {"plugin-command": "/home/steve/.porter/porter plugin run secrets.porter.host"}
starting plugin
plugin started
waiting for RPC address
using plugin
plugin address
Using runtime driver docker

saving action install for /example-app-stack installation with status running
executing install action from example-app-stack (installation: /example-app-stack)
Install cert manager for redpanda from the vendor chart
/usr/local/bin/helm3 /usr/local/bin/helm3 upgrade --install cert-manager ./charts/cert-manager --namespace cert-manager --atomic --create-namespace --set global.name=cert-manager --set installCRDs=true
Release "cert-manager" has been upgraded. Happy Helming!
NAME: cert-manager
LAST DEPLOYED: Thu May  2 22:45:04 2024
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
cert-manager v1.14.4 has been deployed successfully!

In order to begin issuing certificates, you will need to set up a ClusterIssuer
or Issuer resource (for example, by creating a 'letsencrypt-staging' issuer).

More information on the different types of issuers and how to configure them
can be found in our documentation:

https://cert-manager.io/docs/configuration/

For information on how to configure cert-manager to automatically provision
Certificates for Ingress resources, take a look at the `ingress-shim`
documentation:

https://cert-manager.io/docs/usage/ingress/
Install Redpanda from the vendor chart
/usr/local/bin/helm3 /usr/local/bin/helm3 upgrade --install redpanda ./charts/redpanda --namespace redpanda --atomic --create-namespace --set global.name=redpanda
Release "redpanda" does not exist. Installing it now.
Error: release redpanda failed, and has been uninstalled due to atomic being set: client rate limiter Wait returned an error: context deadline exceeded
err: exit status 1
Error: exit status 1
package command failed /cnab/app/cnab/app/mixins/helm3/runtimes/helm3-runtime install
Error: release redpanda failed, and has been uninstalled due to atomic being set: client rate limiter Wait returned an error: context deadline exceeded
Error: exit status 1

Collecting bundle outputs...
1 error occurred:
        * mixin execution failed: package command failed /cnab/app/cnab/app/mixins/helm3/runtimes/helm3-runtime install
Error: release redpanda failed, and has been uninstalled due to atomic being set: client rate limiter Wait returned an error: context deadline exceeded
Error: exit status 1



1 error occurred:
        * mixin execution failed: package command failed /cnab/app/cnab/app/mixins/helm3/runtimes/helm3-runtime install
Error: release redpanda failed, and has been uninstalled due to atomic being set: client rate limiter Wait returned an error: context deadline exceeded
Error: exit status 1



1 error occurred:
        * container exit code: 1, message: <nil>


received EOF, stopping recv loop
plugin process exited
plugin exited
received EOF, stopping recv loop
plugin process exited
plugin exited

At the point where it says that it is installing Redpanda, it sits there for about 5 minutes before the output continues and declares the failure. It was able to pull and install the cert manager, so I am not sure what, exactly, is going wrong.

[kichristensen]

It looks like a Helm issue. Found the following similar issue for Helm, helm/helm#12154

[Steve973]

I got a bit farther. At the moment, I think that it is mostly a matter of doing what these specific helm charts need.

[kichristensen]

👍 Sounds good otherwise please don't hesitate to reach out

[Steve973]

The guy who maintains the helm3 plugin said he'd look into having it download the charts. So that'll be cool, too.