mirror of
https://github.com/kubernetes-sigs/kubespray.git
synced 2025-12-13 21:34:40 +03:00
Removed equinix provider (#12229)
This commit is contained in:
Anshuman Agarwala
committed by
GitHub
GitHub
parent
175babc4df
commit
63cdf87915
@@ -32,7 +32,6 @@ terraform_validate:
|
||||
matrix:
|
||||
- PROVIDER:
|
||||
- openstack
|
||||
- equinix
|
||||
- aws
|
||||
- exoscale
|
||||
- hetzner
|
||||
|
||||
@@ -1,246 +0,0 @@
|
||||
# Kubernetes on Equinix Metal with Terraform
|
||||
|
||||
Provision a Kubernetes cluster with [Terraform](https://www.terraform.io) on
|
||||
[Equinix Metal](https://metal.equinix.com) ([formerly Packet](https://blog.equinix.com/blog/2020/10/06/equinix-metal-metal-and-more/)).
|
||||
|
||||
## Status
|
||||
|
||||
This will install a Kubernetes cluster on Equinix Metal. It should work in all locations and on most server types.
|
||||
|
||||
## Approach
|
||||
|
||||
The terraform configuration inspects variables found in
|
||||
[variables.tf](variables.tf) to create resources in your Equinix Metal project.
|
||||
There is a [python script](../terraform.py) that reads the generated`.tfstate`
|
||||
file to generate a dynamic inventory that is consumed by [cluster.yml](../../../cluster.yml)
|
||||
to actually install Kubernetes with Kubespray.
|
||||
|
||||
### Kubernetes Nodes
|
||||
|
||||
You can create many different kubernetes topologies by setting the number of
|
||||
different classes of hosts.
|
||||
|
||||
- Master nodes with etcd: `number_of_k8s_masters` variable
|
||||
- Master nodes without etcd: `number_of_k8s_masters_no_etcd` variable
|
||||
- Standalone etcd hosts: `number_of_etcd` variable
|
||||
- Kubernetes worker nodes: `number_of_k8s_nodes` variable
|
||||
|
||||
Note that the Ansible script will report an invalid configuration if you wind up
|
||||
with an *even number* of etcd instances since that is not a valid configuration. This
|
||||
restriction includes standalone etcd nodes that are deployed in a cluster along with
|
||||
master nodes with etcd replicas. As an example, if you have three master nodes with
|
||||
etcd replicas and three standalone etcd nodes, the script will fail since there are
|
||||
now six total etcd replicas.
|
||||
|
||||
## Requirements
|
||||
|
||||
- [Install Terraform](https://www.terraform.io/intro/getting-started/install.html)
|
||||
- [Install Ansible dependencies](/docs/ansible/ansible.md#installing-ansible)
|
||||
- Account with Equinix Metal
|
||||
- An SSH key pair
|
||||
|
||||
## SSH Key Setup
|
||||
|
||||
An SSH keypair is required so Ansible can access the newly provisioned nodes (Equinix Metal hosts). By default, the public SSH key defined in cluster.tfvars will be installed in authorized_key on the newly provisioned nodes (~/.ssh/id_rsa.pub). Terraform will upload this public key and then it will be distributed out to all the nodes. If you have already set this public key in Equinix Metal (i.e. via the portal), then set the public keyfile name in cluster.tfvars to blank to prevent the duplicate key from being uploaded which will cause an error.
|
||||
|
||||
If you don't already have a keypair generated (~/.ssh/id_rsa and ~/.ssh/id_rsa.pub), then a new keypair can be generated with the command:
|
||||
|
||||
```ShellSession
|
||||
ssh-keygen -f ~/.ssh/id_rsa
|
||||
```
|
||||
|
||||
## Terraform
|
||||
|
||||
Terraform will be used to provision all of the Equinix Metal resources with base software as appropriate.
|
||||
|
||||
### Configuration
|
||||
|
||||
#### Inventory files
|
||||
|
||||
Create an inventory directory for your cluster by copying the existing sample and linking the `hosts` script (used to build the inventory based on Terraform state):
|
||||
|
||||
```ShellSession
|
||||
cp -LRp contrib/terraform/equinix/sample-inventory inventory/$CLUSTER
|
||||
cd inventory/$CLUSTER
|
||||
ln -s ../../contrib/terraform/equinix/hosts
|
||||
```
|
||||
|
||||
This will be the base for subsequent Terraform commands.
|
||||
|
||||
#### Equinix Metal API access
|
||||
|
||||
Your Equinix Metal API key must be available in the `METAL_AUTH_TOKEN` environment variable.
|
||||
This key is typically stored outside of the code repo since it is considered secret.
|
||||
If someone gets this key, they can startup/shutdown hosts in your project!
|
||||
|
||||
For more information on how to generate an API key or find your project ID, please see
|
||||
[Accounts Index](https://metal.equinix.com/developers/docs/accounts/).
|
||||
|
||||
The Equinix Metal Project ID associated with the key will be set later in `cluster.tfvars`.
|
||||
|
||||
For more information about the API, please see [Equinix Metal API](https://metal.equinix.com/developers/api/).
|
||||
|
||||
For more information about terraform provider authentication, please see [the equinix provider documentation](https://registry.terraform.io/providers/equinix/equinix/latest/docs).
|
||||
|
||||
Example:
|
||||
|
||||
```ShellSession
|
||||
export METAL_AUTH_TOKEN="Example-API-Token"
|
||||
```
|
||||
|
||||
Note that to deploy several clusters within the same project you need to use [terraform workspace](https://www.terraform.io/docs/state/workspaces.html#using-workspaces).
|
||||
|
||||
#### Cluster variables
|
||||
|
||||
The construction of the cluster is driven by values found in
|
||||
[variables.tf](variables.tf).
|
||||
|
||||
For your cluster, edit `inventory/$CLUSTER/cluster.tfvars`.
|
||||
|
||||
The `cluster_name` is used to set a tag on each server deployed as part of this cluster.
|
||||
This helps when identifying which hosts are associated with each cluster.
|
||||
|
||||
While the defaults in variables.tf will successfully deploy a cluster, it is recommended to set the following values:
|
||||
|
||||
- cluster_name = the name of the inventory directory created above as $CLUSTER
|
||||
- equinix_metal_project_id = the Equinix Metal Project ID associated with the Equinix Metal API token above
|
||||
|
||||
#### Enable localhost access
|
||||
|
||||
Kubespray will pull down a Kubernetes configuration file to access this cluster by enabling the
|
||||
`kubeconfig_localhost: true` in the Kubespray configuration.
|
||||
|
||||
Edit `inventory/$CLUSTER/group_vars/k8s_cluster/k8s_cluster.yml` and comment back in the following line and change from `false` to `true`:
|
||||
`\# kubeconfig_localhost: false`
|
||||
becomes:
|
||||
`kubeconfig_localhost: true`
|
||||
|
||||
Once the Kubespray playbooks are run, a Kubernetes configuration file will be written to the local host at `inventory/$CLUSTER/artifacts/admin.conf`
|
||||
|
||||
#### Terraform state files
|
||||
|
||||
In the cluster's inventory folder, the following files might be created (either by Terraform
|
||||
or manually), to prevent you from pushing them accidentally they are in a
|
||||
`.gitignore` file in the `contrib/terraform/equinix` directory :
|
||||
|
||||
- `.terraform`
|
||||
- `.tfvars`
|
||||
- `.tfstate`
|
||||
- `.tfstate.backup`
|
||||
- `.lock.hcl`
|
||||
|
||||
You can still add them manually if you want to.
|
||||
|
||||
### Initialization
|
||||
|
||||
Before Terraform can operate on your cluster you need to install the required
|
||||
plugins. This is accomplished as follows:
|
||||
|
||||
```ShellSession
|
||||
cd inventory/$CLUSTER
|
||||
terraform -chdir=../../contrib/terraform/metal init -var-file=cluster.tfvars
|
||||
```
|
||||
|
||||
This should finish fairly quickly telling you Terraform has successfully initialized and loaded necessary modules.
|
||||
|
||||
### Provisioning cluster
|
||||
|
||||
You can apply the Terraform configuration to your cluster with the following command
|
||||
issued from your cluster's inventory directory (`inventory/$CLUSTER`):
|
||||
|
||||
```ShellSession
|
||||
terraform -chdir=../../contrib/terraform/equinix apply -var-file=cluster.tfvars
|
||||
export ANSIBLE_HOST_KEY_CHECKING=False
|
||||
ansible-playbook -i hosts ../../cluster.yml
|
||||
```
|
||||
|
||||
### Destroying cluster
|
||||
|
||||
You can destroy your new cluster with the following command issued from the cluster's inventory directory:
|
||||
|
||||
```ShellSession
|
||||
terraform -chdir=../../contrib/terraform/equinix destroy -var-file=cluster.tfvars
|
||||
```
|
||||
|
||||
If you've started the Ansible run, it may also be a good idea to do some manual cleanup:
|
||||
|
||||
- Remove SSH keys from the destroyed cluster from your `~/.ssh/known_hosts` file
|
||||
- Clean up any temporary cache files: `rm /tmp/$CLUSTER-*`
|
||||
|
||||
### Debugging
|
||||
|
||||
You can enable debugging output from Terraform by setting `TF_LOG` to `DEBUG` before running the Terraform command.
|
||||
|
||||
## Ansible
|
||||
|
||||
### Node access
|
||||
|
||||
#### SSH
|
||||
|
||||
Ensure your local ssh-agent is running and your ssh key has been added. This
|
||||
step is required by the terraform provisioner:
|
||||
|
||||
```ShellSession
|
||||
eval $(ssh-agent -s)
|
||||
ssh-add ~/.ssh/id_rsa
|
||||
```
|
||||
|
||||
If you have deployed and destroyed a previous iteration of your cluster, you will need to clear out any stale keys from your SSH "known hosts" file ( `~/.ssh/known_hosts`).
|
||||
|
||||
#### Test access
|
||||
|
||||
Make sure you can connect to the hosts. Note that Flatcar Container Linux by Kinvolk will have a state `FAILED` due to Python not being present. This is okay, because Python will be installed during bootstrapping, so long as the hosts are not `UNREACHABLE`.
|
||||
|
||||
```ShellSession
|
||||
$ ansible -i inventory/$CLUSTER/hosts -m ping all
|
||||
example-k8s_node-1 | SUCCESS => {
|
||||
"changed": false,
|
||||
"ping": "pong"
|
||||
}
|
||||
example-etcd-1 | SUCCESS => {
|
||||
"changed": false,
|
||||
"ping": "pong"
|
||||
}
|
||||
example-k8s-master-1 | SUCCESS => {
|
||||
"changed": false,
|
||||
"ping": "pong"
|
||||
}
|
||||
```
|
||||
|
||||
If it fails try to connect manually via SSH. It could be something as simple as a stale host key.
|
||||
|
||||
### Deploy Kubernetes
|
||||
|
||||
```ShellSession
|
||||
ansible-playbook --become -i inventory/$CLUSTER/hosts cluster.yml
|
||||
```
|
||||
|
||||
This will take some time as there are many tasks to run.
|
||||
|
||||
## Kubernetes
|
||||
|
||||
### Set up kubectl
|
||||
|
||||
- [Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/) on the localhost.
|
||||
- Verify that Kubectl runs correctly
|
||||
|
||||
```ShellSession
|
||||
kubectl version
|
||||
```
|
||||
|
||||
- Verify that the Kubernetes configuration file has been copied over
|
||||
|
||||
```ShellSession
|
||||
cat inventory/alpha/$CLUSTER/admin.conf
|
||||
```
|
||||
|
||||
- Verify that all the nodes are running correctly.
|
||||
|
||||
```ShellSession
|
||||
kubectl version
|
||||
kubectl --kubeconfig=inventory/$CLUSTER/artifacts/admin.conf get nodes
|
||||
```
|
||||
|
||||
## What's next
|
||||
|
||||
Try out your new Kubernetes cluster with the [Hello Kubernetes service](https://kubernetes.io/docs/tasks/access-application-cluster/service-access-application-cluster/).
|
||||
@@ -1 +0,0 @@
|
||||
../terraform.py
|
||||
@@ -1,57 +0,0 @@
|
||||
resource "equinix_metal_ssh_key" "k8s" {
|
||||
count = var.public_key_path != "" ? 1 : 0
|
||||
name = "kubernetes-${var.cluster_name}"
|
||||
public_key = chomp(file(var.public_key_path))
|
||||
}
|
||||
|
||||
resource "equinix_metal_device" "k8s_master" {
|
||||
depends_on = [equinix_metal_ssh_key.k8s]
|
||||
|
||||
count = var.number_of_k8s_masters
|
||||
hostname = "${var.cluster_name}-k8s-master-${count.index + 1}"
|
||||
plan = var.plan_k8s_masters
|
||||
metro = var.metro
|
||||
operating_system = var.operating_system
|
||||
billing_cycle = var.billing_cycle
|
||||
project_id = var.equinix_metal_project_id
|
||||
tags = ["cluster-${var.cluster_name}", "k8s_cluster", "kube_control_plane", "etcd", "kube_node"]
|
||||
}
|
||||
|
||||
resource "equinix_metal_device" "k8s_master_no_etcd" {
|
||||
depends_on = [equinix_metal_ssh_key.k8s]
|
||||
|
||||
count = var.number_of_k8s_masters_no_etcd
|
||||
hostname = "${var.cluster_name}-k8s-master-${count.index + 1}"
|
||||
plan = var.plan_k8s_masters_no_etcd
|
||||
metro = var.metro
|
||||
operating_system = var.operating_system
|
||||
billing_cycle = var.billing_cycle
|
||||
project_id = var.equinix_metal_project_id
|
||||
tags = ["cluster-${var.cluster_name}", "k8s_cluster", "kube_control_plane"]
|
||||
}
|
||||
|
||||
resource "equinix_metal_device" "k8s_etcd" {
|
||||
depends_on = [equinix_metal_ssh_key.k8s]
|
||||
|
||||
count = var.number_of_etcd
|
||||
hostname = "${var.cluster_name}-etcd-${count.index + 1}"
|
||||
plan = var.plan_etcd
|
||||
metro = var.metro
|
||||
operating_system = var.operating_system
|
||||
billing_cycle = var.billing_cycle
|
||||
project_id = var.equinix_metal_project_id
|
||||
tags = ["cluster-${var.cluster_name}", "etcd"]
|
||||
}
|
||||
|
||||
resource "equinix_metal_device" "k8s_node" {
|
||||
depends_on = [equinix_metal_ssh_key.k8s]
|
||||
|
||||
count = var.number_of_k8s_nodes
|
||||
hostname = "${var.cluster_name}-k8s-node-${count.index + 1}"
|
||||
plan = var.plan_k8s_nodes
|
||||
metro = var.metro
|
||||
operating_system = var.operating_system
|
||||
billing_cycle = var.billing_cycle
|
||||
project_id = var.equinix_metal_project_id
|
||||
tags = ["cluster-${var.cluster_name}", "k8s_cluster", "kube_node"]
|
||||
}
|
||||
@@ -1,15 +0,0 @@
|
||||
output "k8s_masters" {
|
||||
value = equinix_metal_device.k8s_master.*.access_public_ipv4
|
||||
}
|
||||
|
||||
output "k8s_masters_no_etc" {
|
||||
value = equinix_metal_device.k8s_master_no_etcd.*.access_public_ipv4
|
||||
}
|
||||
|
||||
output "k8s_etcds" {
|
||||
value = equinix_metal_device.k8s_etcd.*.access_public_ipv4
|
||||
}
|
||||
|
||||
output "k8s_nodes" {
|
||||
value = equinix_metal_device.k8s_node.*.access_public_ipv4
|
||||
}
|
||||
@@ -1,17 +0,0 @@
|
||||
terraform {
|
||||
required_version = ">= 1.0.0"
|
||||
|
||||
provider_meta "equinix" {
|
||||
module_name = "kubespray"
|
||||
}
|
||||
required_providers {
|
||||
equinix = {
|
||||
source = "equinix/equinix"
|
||||
version = "1.24.0"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Configure the Equinix Metal Provider
|
||||
provider "equinix" {
|
||||
}
|
||||
@@ -1,35 +0,0 @@
|
||||
# your Kubernetes cluster name here
|
||||
cluster_name = "mycluster"
|
||||
|
||||
# Your Equinix Metal project ID. See https://metal.equinix.com/developers/docs/accounts/
|
||||
equinix_metal_project_id = "Example-Project-Id"
|
||||
|
||||
# The public SSH key to be uploaded into authorized_keys in bare metal Equinix Metal nodes provisioned
|
||||
# leave this value blank if the public key is already setup in the Equinix Metal project
|
||||
# Terraform will complain if the public key is setup in Equinix Metal
|
||||
public_key_path = "~/.ssh/id_rsa.pub"
|
||||
|
||||
# Equinix interconnected bare metal across our global metros.
|
||||
metro = "da"
|
||||
|
||||
# operating_system
|
||||
operating_system = "ubuntu_22_04"
|
||||
|
||||
# standalone etcds
|
||||
number_of_etcd = 0
|
||||
|
||||
plan_etcd = "t1.small.x86"
|
||||
|
||||
# masters
|
||||
number_of_k8s_masters = 1
|
||||
|
||||
number_of_k8s_masters_no_etcd = 0
|
||||
|
||||
plan_k8s_masters = "t1.small.x86"
|
||||
|
||||
plan_k8s_masters_no_etcd = "t1.small.x86"
|
||||
|
||||
# nodes
|
||||
number_of_k8s_nodes = 2
|
||||
|
||||
plan_k8s_nodes = "t1.small.x86"
|
||||
@@ -1 +0,0 @@
|
||||
../../../../inventory/sample/group_vars
|
||||
@@ -1,56 +0,0 @@
|
||||
variable "cluster_name" {
|
||||
default = "kubespray"
|
||||
}
|
||||
|
||||
variable "equinix_metal_project_id" {
|
||||
description = "Your Equinix Metal project ID. See https://metal.equinix.com/developers/docs/accounts/"
|
||||
}
|
||||
|
||||
variable "operating_system" {
|
||||
default = "ubuntu_22_04"
|
||||
}
|
||||
|
||||
variable "public_key_path" {
|
||||
description = "The path of the ssh pub key"
|
||||
default = "~/.ssh/id_rsa.pub"
|
||||
}
|
||||
|
||||
variable "billing_cycle" {
|
||||
default = "hourly"
|
||||
}
|
||||
|
||||
variable "metro" {
|
||||
default = "da"
|
||||
}
|
||||
|
||||
variable "plan_k8s_masters" {
|
||||
default = "c3.small.x86"
|
||||
}
|
||||
|
||||
variable "plan_k8s_masters_no_etcd" {
|
||||
default = "c3.small.x86"
|
||||
}
|
||||
|
||||
variable "plan_etcd" {
|
||||
default = "c3.small.x86"
|
||||
}
|
||||
|
||||
variable "plan_k8s_nodes" {
|
||||
default = "c3.medium.x86"
|
||||
}
|
||||
|
||||
variable "number_of_k8s_masters" {
|
||||
default = 1
|
||||
}
|
||||
|
||||
variable "number_of_k8s_masters_no_etcd" {
|
||||
default = 0
|
||||
}
|
||||
|
||||
variable "number_of_etcd" {
|
||||
default = 0
|
||||
}
|
||||
|
||||
variable "number_of_k8s_nodes" {
|
||||
default = 1
|
||||
}
|
||||
1
docs/_sidebar.md
generated
1
docs/_sidebar.md
generated
@@ -23,7 +23,6 @@
|
||||
* [Aws](/docs/cloud_providers/aws.md)
|
||||
* [Azure](/docs/cloud_providers/azure.md)
|
||||
* [Cloud](/docs/cloud_providers/cloud.md)
|
||||
* [Equinix-metal](/docs/cloud_providers/equinix-metal.md)
|
||||
* CNI
|
||||
* [Calico](/docs/CNI/calico.md)
|
||||
* [Cilium](/docs/CNI/cilium.md)
|
||||
|
||||
@@ -1,100 +0,0 @@
|
||||
# Equinix Metal
|
||||
|
||||
Kubespray provides support for bare metal deployments using the [Equinix Metal](http://metal.equinix.com).
|
||||
Deploying upon bare metal allows Kubernetes to run at locations where an existing public or private cloud might not exist such
|
||||
as cell tower, edge collocated installations. The deployment mechanism used by Kubespray for Equinix Metal is similar to that used for
|
||||
AWS and OpenStack clouds (notably using Terraform to deploy the infrastructure). Terraform uses the Equinix Metal provider plugin
|
||||
to provision and configure hosts which are then used by the Kubespray Ansible playbooks. The Ansible inventory is generated
|
||||
dynamically from the Terraform state file.
|
||||
|
||||
## Local Host Configuration
|
||||
|
||||
To perform this installation, you will need a localhost to run Terraform/Ansible (laptop, VM, etc) and an account with Equinix Metal.
|
||||
In this example, we are provisioning a m1.large CentOS7 OpenStack VM as the localhost for the Kubernetes installation.
|
||||
You'll need Ansible, Git, and PIP.
|
||||
|
||||
```bash
|
||||
sudo yum install epel-release
|
||||
sudo yum install ansible
|
||||
sudo yum install git
|
||||
sudo yum install python-pip
|
||||
```
|
||||
|
||||
## Playbook SSH Key
|
||||
|
||||
An SSH key is needed by Kubespray/Ansible to run the playbooks.
|
||||
This key is installed into the bare metal hosts during the Terraform deployment.
|
||||
You can generate a key new key or use an existing one.
|
||||
|
||||
```bash
|
||||
ssh-keygen -f ~/.ssh/id_rsa
|
||||
```
|
||||
|
||||
## Install Terraform
|
||||
|
||||
Terraform is required to deploy the bare metal infrastructure. The steps below are for installing on CentOS 7.
|
||||
[More terraform installation options are available.](https://learn.hashicorp.com/terraform/getting-started/install.html)
|
||||
|
||||
Grab the latest version of Terraform and install it.
|
||||
|
||||
```bash
|
||||
echo "https://releases.hashicorp.com/terraform/$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')/terraform_$(curl -s https://checkpoint-api.hashicorp.com/v1/check/terraform | jq -r -M '.current_version')_linux_amd64.zip"
|
||||
sudo yum install unzip
|
||||
sudo unzip terraform_0.14.10_linux_amd64.zip -d /usr/local/bin/
|
||||
```
|
||||
|
||||
## Download Kubespray
|
||||
|
||||
Pull over Kubespray and setup any required libraries.
|
||||
|
||||
```bash
|
||||
git clone https://github.com/kubernetes-sigs/kubespray
|
||||
cd kubespray
|
||||
```
|
||||
|
||||
## Install Ansible
|
||||
|
||||
Install Ansible according to [Ansible installation guide](/docs/ansible/ansible.md#installing-ansible)
|
||||
|
||||
## Cluster Definition
|
||||
|
||||
In this example, a new cluster called "alpha" will be created.
|
||||
|
||||
```bash
|
||||
cp -LRp contrib/terraform/packet/sample-inventory inventory/alpha
|
||||
cd inventory/alpha/
|
||||
ln -s ../../contrib/terraform/packet/hosts
|
||||
```
|
||||
|
||||
Details about the cluster, such as the name, as well as the authentication tokens and project ID
|
||||
for Equinix Metal need to be defined. To find these values see [Equinix Metal API Accounts](https://metal.equinix.com/developers/docs/accounts/).
|
||||
|
||||
```bash
|
||||
vi cluster.tfvars
|
||||
```
|
||||
|
||||
* cluster_name = alpha
|
||||
* packet_project_id = ABCDEFGHIJKLMNOPQRSTUVWXYZ123456
|
||||
* public_key_path = 12345678-90AB-CDEF-GHIJ-KLMNOPQRSTUV
|
||||
|
||||
## Deploy Bare Metal Hosts
|
||||
|
||||
Initializing Terraform will pull down any necessary plugins/providers.
|
||||
|
||||
```bash
|
||||
terraform init ../../contrib/terraform/packet/
|
||||
```
|
||||
|
||||
Run Terraform to deploy the hardware.
|
||||
|
||||
```bash
|
||||
terraform apply -var-file=cluster.tfvars ../../contrib/terraform/packet
|
||||
```
|
||||
|
||||
## Run Kubespray Playbooks
|
||||
|
||||
With the bare metal infrastructure deployed, Kubespray can now install Kubernetes and setup the cluster.
|
||||
|
||||
```bash
|
||||
ansible-playbook --become -i inventory/alpha/hosts cluster.yml
|
||||
```
|
||||
Reference in New Issue
Block a user