ARM based nodes are significantly cheaper than amd64 based ones. It's worthwhile us looking at how JX deploys to these node architectures.
I accidently tried to deploy a new Azure based cluster to arm based nodes and most things deployed okay aside from lighthouse and a couple of other docker/chart based repos of ours. I'm going to continue trying to deploy this and fix issues as I go.
Seems that most of the cli based tools already seem setup for this, it's mostly getting charts/docker images to work.
Roadblocks
Hit an issue with cross-arch building where any RUN commands exectuted in the in the dockerfiles have to be exectuted on the host machine, resulting in:
error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: exec format error
Most of our existing packages that build for amd64/arm64 get around this by either not having any RUN commands in the dockerfiles or by using github actions. For some packages, e.g. Lighthouse:Webhooks we need git to be part of the image so we cannot avoid the RUN command, and personally I'm not a fan of github actions as we should be dog fooding imo.
Seems the only solutions to this are either to:
- Run arm64 based node on the jx build cluster
I don't think its feasible for us to completely switch to arm64 nodes as we will still always want to compile to amd64 as well - so we'd have the same situation but reversed. So the only way to do this would be to run two types of nodes and run a build step for each arch on the respective node. I'm not sure whether you can add nodeSelectors etc to steps in tekton so not sure how feasible this route is.
- Enable QEMU on our existing nodes to emulate arm64
Not sure if our current nodes support this, would probably need some changes to the terraform base repos. Generally not a fan of this as I'd prefer for us to use the native arch to avoid weird cross-compliation issues.
Non-functioning Default JX Deployments
ARM based nodes are significantly cheaper than amd64 based ones. It's worthwhile us looking at how JX deploys to these node architectures.
I accidently tried to deploy a new Azure based cluster to arm based nodes and most things deployed okay aside from lighthouse and a couple of other docker/chart based repos of ours. I'm going to continue trying to deploy this and fix issues as I go.
Seems that most of the cli based tools already seem setup for this, it's mostly getting charts/docker images to work.
Roadblocks
Hit an issue with cross-arch building where any
RUNcommands exectuted in the in the dockerfiles have to be exectuted on the host machine, resulting in:Most of our existing packages that build for amd64/arm64 get around this by either not having any
RUNcommands in the dockerfiles or by using github actions. For some packages, e.g. Lighthouse:Webhooks we need git to be part of the image so we cannot avoid theRUNcommand, and personally I'm not a fan of github actions as we should be dog fooding imo.Seems the only solutions to this are either to:
I don't think its feasible for us to completely switch to arm64 nodes as we will still always want to compile to amd64 as well - so we'd have the same situation but reversed. So the only way to do this would be to run two types of nodes and run a build step for each arch on the respective node. I'm not sure whether you can add
nodeSelectorsetc to steps in tekton so not sure how feasible this route is.Not sure if our current nodes support this, would probably need some changes to the terraform base repos. Generally not a fan of this as I'd prefer for us to use the native arch to avoid weird cross-compliation issues.
Non-functioning Default JX Deployments