Compare commits

..

23 Commits

Author SHA1 Message Date
k8s-infra-cherrypick-robot
5fb85dc8a5 Add rbac for calico kube-controllers to access services (#12831)
Co-authored-by: Lawik974 <loic97429@gmail.com>
2026-01-02 21:00:35 +05:30
Max Gautier
84d8746b41 Patch versions updates (#12800)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-12-20 00:26:31 -08:00
k8s-infra-cherrypick-robot
8181d8c688 Upgrade cilium from 1.18.4 to 1.18.5 (#12804)
Signed-off-by: Ali Afsharzadeh <afsharzadeh8@gmail.com>
Co-authored-by: Ali Afsharzadeh <afsharzadeh8@gmail.com>
2025-12-19 07:44:34 -08:00
ChengHao Yang
c4c3205a71 Releng: galaxy version to 2.29.2 (#12786)
Signed-off-by: ChengHao Yang <17496418+tico88612@users.noreply.github.com>
2025-12-11 19:47:30 -08:00
ChengHao Yang
0c6a29553f Patch versions updates (#12782)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-12-11 00:55:31 -08:00
Max Gautier
2375fae1c2 Patch versions updates (#12763)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-12-04 06:39:01 -08:00
Max Gautier
55f7b7f54c Patch versions updates (#12744)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-11-25 00:52:36 -08:00
k8s-infra-cherrypick-robot
dbca6a7757 [release-2.29] CI: enable unsafe_show_logs == true by default (#12728)
* CI: enable unsafe_show_logs == true by default

* Deduplicate defaults vars (unsafe_show_logs)

---------

Co-authored-by: Max Gautier <mg@max.gautier.name>
2025-11-19 23:32:02 -08:00
k8s-infra-cherrypick-robot
c5c43619a7 Upgrade cilium from 1.18.3 to 1.18.4 (#12725)
Signed-off-by: Ali Afsharzadeh <afsharzadeh8@gmail.com>
Co-authored-by: Ali Afsharzadeh <afsharzadeh8@gmail.com>
2025-11-18 20:09:59 -08:00
Max Gautier
584b0a4036 Patch versions updates (#12719)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-11-18 05:15:39 -08:00
k8s-infra-cherrypick-robot
084c2be8b9 CI: use a dedicated disk for releases (#12721)
This should make 'no space left on device' problems easier to handle

Use /tmp/releases as local_release_dir CI created machine, while keeping
the same folder on the runner (needed for gitlab-ci runner pods)

Co-authored-by: Max Gautier <mg@max.gautier.name>
2025-11-18 03:29:39 -08:00
k8s-infra-cherrypick-robot
932025fbd6 Let containerd create storage / state dir (#12722)
Containerd manages by itself, so there is no need to override it and
change permissions.

Co-authored-by: Max Gautier <mg@max.gautier.name>
2025-11-18 03:09:38 -08:00
k8s-infra-cherrypick-robot
a04592de18 Adjust hubble export values for cilium 1.18 schema change (#12718)
Signed-off-by: Ali Afsharzadeh <afsharzadeh8@gmail.com>
Co-authored-by: Ali Afsharzadeh <afsharzadeh8@gmail.com>
2025-11-18 00:41:42 -08:00
k8s-infra-cherrypick-robot
d8b9288b27 [release-2.29] CI: Try a full ssh connection on hosts instead of only checking the port (#12711)
* CI: Try a full ssh connection on hosts instead of only checking the port

If we only try the port, we can try to connect in the playbook which is
executed next even though the managed node has not yet completed it's
boot-up sequence ("System is booting up. Unprivileged users are not
permitted to log in yet. Please come back later. For technical details,
see pam_nologin(8).")

This does not account for python-less hosts, but we don't use those in
CI anyway (for now, at least).

* CI: Remove connection method override when creating VMs

This prevented wait_for_connection to work correctly by hijacking the
connection to localhost, thus bypassing the connection check.

---------

Co-authored-by: Max Gautier <mg@max.gautier.name>
2025-11-15 12:37:38 -08:00
Max Gautier
cbdd7cf3a7 update pre-commit hooks (#12706) 2025-11-14 22:41:40 -08:00
k8s-infra-cherrypick-robot
3c0cff983d fix(cilium):correct loadBalancer.mode rendering in values.yaml (#12705)
Co-authored-by: Anurag Ojha <aojharaj2004@gmail.com>
2025-11-14 07:01:40 -08:00
k8s-infra-cherrypick-robot
e5a1f68a2c Update Calico apiserver RBAC for Kubernetes 1.33+ (#12695)
Add missing RBAC permissions for Calico apiserver to function correctly
with Kubernetes 1.33+

Changes:

1. Add K8s 1.33 ValidatingAdmissionPolicy resources to calico-webhook-reader
   - validatingadmissionpolicies
   - validatingadmissionpolicybindings

Kubernetes 1.33 introduced ValidatingAdmissionPolicy resources (KEP-3488)
that require explicit RBAC permissions. Without these changes, Calico
apiserver on k8s 1.33+ will not work and needless errors are logged

Co-authored-by: rickerc <chris.ricker@gmail.com>
2025-11-14 04:49:38 -08:00
k8s-infra-cherrypick-robot
fe566df651 Fix the (upgrade/remove_node) + collection test cases (#12687)
The 'old' playbook and the collection use '-' and '_' as separator,
which breaks the logic in scripts/testcases_run.sh.

Add aliases using the old schemes to make the test work and avoid
breaking anything.

Both '-' and '_' variants will be deleted once we switch to supporting
collection only.

Co-authored-by: Max Gautier <mg@max.gautier.name>
2025-11-10 06:46:57 -08:00
k8s-infra-cherrypick-robot
59b3c686a8 [release-2.29] Remove etcd member by peerURLs (#12685)
* Remove etcd member by peerURLs

The way to obtain the IP of a particular member is convoluted and depend
on multiple variables. The match is also textual and it's not clear
against what we're matching

It's also broken for etcd member which are not also Kubernetes nodes,
because the "Lookup node IP in kubernetes" task will fail and abort the
play.

Instead, match against 'peerURLs', which does not need new variable, and
use json output.

* Add testcase for etcd removal on external etcd

* do not merge

* fixup! Remove etcd member by peerURLs

* fixup! Remove etcd member by peerURLs

---------

Co-authored-by: Max Gautier <mg@max.gautier.name>
2025-11-10 05:48:56 -08:00
Ali Afsharzadeh
4b970baa5a [release-2.29] Upgrade cilium from 1.18.2 to 1.18.3 (#12679) 2025-11-09 06:00:52 -08:00
ChengHao Yang
a15fcb729b Patch versions updates (#12646)
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2025-11-03 02:19:36 -08:00
k8s-infra-cherrypick-robot
9a9e33dc9f fix(calico): Add missed rbac verb for hostendpoints (#12644)
Signed-off-by: Meza <meza-xyz@proton.me>
Co-authored-by: Meza <meza-xyz@proton.me>
2025-10-24 01:05:34 -07:00
ChengHao Yang
d9f188c39c [release-2.29] Releng: galaxy version to 2.29.1 (#12645)
Signed-off-by: ChengHao Yang <17496418+tico88612@users.noreply.github.com>
2025-10-24 00:41:36 -07:00
203 changed files with 3988 additions and 1158 deletions

View File

@@ -1,4 +1,5 @@
---
parseable: true
skip_list:
# see https://docs.ansible.com/ansible-lint/rules/default_rules.html for a list of all default rules
@@ -33,8 +34,6 @@ skip_list:
# Disable run-once check with free strategy
# (Disabled in June 2023 after ansible upgrade; FIXME)
- 'run-once[task]'
- 'jinja[spacing]'
exclude_paths:
# Generated files
- tests/files/custom_cni/cilium.yaml

View File

@@ -13,16 +13,16 @@ jobs:
issues: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8
- name: Parse issue form
uses: stefanbuck/github-issue-parser@10dcc54158ba4c137713d9d69d70a2da63b6bda3
uses: stefanbuck/github-issue-parser@2ea9b35a8c584529ed00891a8f7e41dc46d0441e
id: issue-parser
with:
template-path: .github/ISSUE_TEMPLATE/bug-report.yaml
- name: Set labels based on OS field
uses: redhat-plumbers-in-action/advanced-issue-labeler@b80ae64e3e156e9c111b075bfa04b295d54e8e2e
uses: redhat-plumbers-in-action/advanced-issue-labeler@e38e6809c5420d038eed380d49ee9a6ca7c92dbf
with:
issue-form: ${{ steps.issue-parser.outputs.jsonString }}
section: os

View File

@@ -13,14 +13,14 @@ jobs:
outputs:
branches: ${{ steps.get-branches.outputs.data }}
steps:
- uses: octokit/graphql-action@ddde8ebb2493e79f390e6449c725c21663a67505
- uses: octokit/graphql-action@8ad880e4d437783ea2ab17010324de1075228110
id: get-branches
with:
query: |
query get_release_branches($owner:String!, $name:String!) {
repository(owner:$owner, name:$name) {
refs(refPrefix: "refs/heads/",
first: 3,
first: 1, # TODO increment once we have release branch with the new checksums format
query: "release-",
orderBy: {
field: ALPHABETICAL,

View File

@@ -11,7 +11,7 @@ jobs:
update-patch-versions:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
- uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8
with:
ref: ${{ inputs.branch }}
- uses: actions/setup-python@v6
@@ -22,14 +22,14 @@ jobs:
- run: update-hashes
env:
API_KEY: ${{ secrets.GITHUB_TOKEN }}
- uses: actions/cache@v5
- uses: actions/cache@v4
with:
key: pre-commit-hook-propagate
path: |
~/.cache/pre-commit
- run: pre-commit run --all-files propagate-ansible-variables
continue-on-error: true
- uses: peter-evans/create-pull-request@c0f553fe549906ede9cf27b5156039d195d2ece0
- uses: peter-evans/create-pull-request@271a8d0340265f705b14b6d32b9829c1cb33d45e
with:
commit-message: Patch versions updates
title: Patch versions updates - ${{ inputs.branch }}

View File

@@ -41,34 +41,20 @@ pr:
- debian12-cilium
- debian13-cilium
- fedora39-kube-router
- fedora41-kube-router
- fedora42-calico
- openeuler24-calico
- rockylinux9-cilium
- rockylinux10-cilium
- ubuntu22-calico-all-in-one
- ubuntu22-calico-all-in-one-upgrade
- ubuntu24-calico-etcd-datastore
- ubuntu24-calico-all-in-one-hardening
- ubuntu24-cilium-sep
- ubuntu24-crio-scale
- ubuntu24-crio-upgrade
- ubuntu24-flannel-collection
- ubuntu24-kube-router-sep
- ubuntu24-kube-router-svc-proxy
- ubuntu24-ha-separate-etcd
- flatcar4081-calico
- fedora40-flannel-crio-collection-scale
# This is for flakey test so they don't disrupt the PR worklflow too much.
# Jobs here MUST have a open issue so we don't lose sight of them
pr-flakey:
extends: pr
retry: 1
parallel:
matrix:
- TESTCASE:
- flatcar4081-calico # https://github.com/kubernetes-sigs/kubespray/issues/12309
- openeuler24-calico # https://github.com/kubernetes-sigs/kubespray/issues/12877
# The ubuntu24-calico-all-in-one jobs are meant as early stages to prevent running the full CI if something is horribly broken
ubuntu24-calico-all-in-one:
stage: deploy-part1
@@ -104,8 +90,6 @@ pr_full:
- debian12-custom-cni-helm
- fedora39-calico-swap-selinux
- fedora39-crio
- fedora41-calico-swap-selinux
- fedora41-crio
- ubuntu24-calico-ha-wireguard
- ubuntu24-flannel-ha
- ubuntu24-flannel-ha-once
@@ -143,7 +127,6 @@ pr_extended:
- debian12-docker
- debian13-calico
- rockylinux9-calico
- rockylinux10-calico
- ubuntu22-all-in-one-docker
- ubuntu24-all-in-one-docker
- ubuntu24-calico-all-in-one
@@ -165,7 +148,6 @@ periodic:
- debian12-cilium-svc-proxy
- fedora39-calico-selinux
- fedora40-docker-calico
- fedora41-calico-selinux
- ubuntu24-calico-etcd-kubeadm-upgrade-ha
- ubuntu24-calico-ha-recover
- ubuntu24-calico-ha-recover-noquorum

View File

@@ -37,6 +37,7 @@ terraform_validate:
- hetzner
- vsphere
- upcloud
- nifcloud
.terraform_apply:
extends: .terraform_install
@@ -88,10 +89,11 @@ tf-elastx_cleanup:
- ./scripts/openstack-cleanup/main.py
allow_failure: true
tf-elastx_ubuntu24-calico:
tf-elastx_ubuntu20-calico:
extends: .terraform_apply
stage: deploy-part1
when: on_success
allow_failure: true
variables:
<<: *elastx_variables
PROVIDER: openstack
@@ -114,5 +116,5 @@ tf-elastx_ubuntu24-calico:
TF_VAR_az_list_node: '["sto1"]'
TF_VAR_flavor_k8s_master: 3f73fc93-ec61-4808-88df-2580d94c1a9b # v1-standard-2
TF_VAR_flavor_k8s_node: 3f73fc93-ec61-4808-88df-2580d94c1a9b # v1-standard-2
TF_VAR_image: ubuntu-24.04-server-latest
TF_VAR_image: ubuntu-20.04-server-latest
TF_VAR_k8s_allowed_remote_ips: '["0.0.0.0/0"]'

View File

@@ -36,7 +36,7 @@ vagrant:
policy: pull-push # TODO: change to "pull" when not on main
stage: deploy-extended
rules:
- if: $PR_LABELS =~ /.*ci-full.*/
- if: $PR_LABELS =~ /.*(ci-extended|ci-full).*/
when: on_success
- if: $CI_PIPELINE_SOURCE == "schedule" && $CI_PIPELINE_SCHEDULE_DESCRIPTION == "daily-ci"
when: on_success

View File

@@ -1,7 +1,7 @@
# syntax=docker/dockerfile:1
# Use immutable image tags rather than mutable tags (like ubuntu:24.04)
FROM ubuntu:noble-20260113@sha256:cd1dba651b3080c3686ecf4e3c4220f026b521fb76978881737d24f200828b2b
# Use immutable image tags rather than mutable tags (like ubuntu:22.04)
FROM ubuntu:22.04@sha256:149d67e29f765f4db62aa52161009e99e389544e25a8f43c8c89d4a445a7ca37
# Some tools like yamllint need this
# Pip needs this as well at the moment to install ansible
@@ -29,14 +29,14 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
RUN --mount=type=bind,source=requirements.txt,target=requirements.txt \
--mount=type=cache,sharing=locked,id=pipcache,mode=0777,target=/root/.cache/pip \
pip install --break-system-packages --no-compile --no-cache-dir -r requirements.txt \
pip install --no-compile --no-cache-dir -r requirements.txt \
&& find /usr -type d -name '*__pycache__' -prune -exec rm -rf {} \;
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
RUN OS_ARCHITECTURE=$(dpkg --print-architecture) \
&& curl -L "https://dl.k8s.io/release/v1.35.0/bin/linux/${OS_ARCHITECTURE}/kubectl" -o /usr/local/bin/kubectl \
&& echo "$(curl -L "https://dl.k8s.io/release/v1.35.0/bin/linux/${OS_ARCHITECTURE}/kubectl.sha256")" /usr/local/bin/kubectl | sha256sum --check \
&& curl -L "https://dl.k8s.io/release/v1.33.7/bin/linux/${OS_ARCHITECTURE}/kubectl" -o /usr/local/bin/kubectl \
&& echo "$(curl -L "https://dl.k8s.io/release/v1.33.7/bin/linux/${OS_ARCHITECTURE}/kubectl.sha256")" /usr/local/bin/kubectl | sha256sum --check \
&& chmod a+x /usr/local/bin/kubectl
COPY *.yml ./

View File

@@ -22,7 +22,7 @@ Ensure you have installed Docker then
```ShellSession
docker run --rm -it --mount type=bind,source="$(pwd)"/inventory/sample,dst=/inventory \
--mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa \
quay.io/kubespray/kubespray:v2.30.0 bash
quay.io/kubespray/kubespray:v2.29.0 bash
# Inside the container you may now run the kubespray playbooks:
ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml
```
@@ -89,13 +89,13 @@ vagrant up
- **Flatcar Container Linux by Kinvolk**
- **Debian** Bookworm, Bullseye, Trixie
- **Ubuntu** 22.04, 24.04
- **CentOS Stream / RHEL** [9, 10](docs/operating_systems/rhel.md#rhel-8)
- **Fedora** 39, 40, 41, 42
- **CentOS/RHEL** [8, 9](docs/operating_systems/rhel.md#rhel-8)
- **Fedora** 39, 40
- **Fedora CoreOS** (see [fcos Note](docs/operating_systems/fcos.md))
- **openSUSE** Leap 15.x/Tumbleweed
- **Oracle Linux** [9, 10](docs/operating_systems/rhel.md#rhel-8)
- **Alma Linux** [9, 10](docs/operating_systems/rhel.md#rhel-8)
- **Rocky Linux** [9, 10](docs/operating_systems/rhel.md#rhel-8) (experimental in 10: see [Rocky Linux 10 notes](docs/operating_systems/rhel.md#rocky-linux-10))
- **Oracle Linux** [8, 9](docs/operating_systems/rhel.md#rhel-8)
- **Alma Linux** [8, 9](docs/operating_systems/rhel.md#rhel-8)
- **Rocky Linux** [8, 9](docs/operating_systems/rhel.md#rhel-8)
- **Kylin Linux Advanced Server V10** (experimental: see [kylin linux notes](docs/operating_systems/kylinlinux.md))
- **Amazon Linux 2** (experimental: see [amazon linux notes](docs/operating_systems/amazonlinux.md))
- **UOS Linux** (experimental: see [uos linux notes](docs/operating_systems/uoslinux.md))
@@ -111,23 +111,24 @@ Note:
<!-- BEGIN ANSIBLE MANAGED BLOCK -->
- Core
- [kubernetes](https://github.com/kubernetes/kubernetes) 1.35.0
- [kubernetes](https://github.com/kubernetes/kubernetes) 1.33.7
- [etcd](https://github.com/etcd-io/etcd) 3.5.26
- [docker](https://www.docker.com/) 28.3
- [containerd](https://containerd.io/) 2.2.1
- [cri-o](http://cri-o.io/) 1.35.0 (experimental: see [CRI-O Note](docs/CRI/cri-o.md). Only on fedora, ubuntu and centos based OS)
- [containerd](https://containerd.io/) 2.1.6
- [cri-o](http://cri-o.io/) 1.33.7 (experimental: see [CRI-O Note](docs/CRI/cri-o.md). Only on fedora, ubuntu and centos based OS)
- Network Plugin
- [cni-plugins](https://github.com/containernetworking/plugins) 1.8.0
- [calico](https://github.com/projectcalico/calico) 3.30.6
- [cilium](https://github.com/cilium/cilium) 1.18.6
- [calico](https://github.com/projectcalico/calico) 3.30.5
- [cilium](https://github.com/cilium/cilium) 1.18.5
- [flannel](https://github.com/flannel-io/flannel) 0.27.3
- [kube-ovn](https://github.com/alauda/kube-ovn) 1.12.21
- [kube-router](https://github.com/cloudnativelabs/kube-router) 2.1.1
- [multus](https://github.com/k8snetworkplumbingwg/multus-cni) 4.2.2
- [kube-vip](https://github.com/kube-vip/kube-vip) 1.0.3
- [kube-vip](https://github.com/kube-vip/kube-vip) 0.8.0
- Application
- [cert-manager](https://github.com/jetstack/cert-manager) 1.15.3
- [coredns](https://github.com/coredns/coredns) 1.12.4
- [coredns](https://github.com/coredns/coredns) 1.12.0
- [ingress-nginx](https://github.com/kubernetes/ingress-nginx) 1.13.3
- [argocd](https://argoproj.github.io/) 2.14.5
- [helm](https://helm.sh/) 3.18.4
- [metallb](https://metallb.universe.tf/) 0.13.9
@@ -201,6 +202,8 @@ See also [Network checker](docs/advanced/netcheck.md).
## Ingress Plugins
- [nginx](https://kubernetes.github.io/ingress-nginx): the NGINX Ingress Controller.
- [metallb](docs/ingress/metallb.md): the MetalLB bare-metal service LoadBalancer provider.
## Community docs and resources

View File

@@ -15,7 +15,7 @@ The Kubespray Project is released on an as-needed basis. The process is as follo
1. The release issue is closed
1. An announcement email is sent to `dev@kubernetes.io` with the subject `[ANNOUNCE] Kubespray $VERSION is released`
1. The topic of the #kubespray channel is updated with `vX.Y.Z is released! | ...`
1. Create/Update Issue for upgrading kubernetes and [k8s-conformance](https://github.com/cncf/k8s-conformance)
1. Create/Update Issue for upgradeing kubernetes and [k8s-conformance](https://github.com/cncf/k8s-conformance)
## Major/minor releases and milestones

3
Vagrantfile vendored
View File

@@ -35,9 +35,6 @@ SUPPORTED_OS = {
"fedora40" => {box: "fedora/40-cloud-base", user: "vagrant"},
"fedora39-arm64" => {box: "bento/fedora-39-arm64", user: "vagrant"},
"fedora40-arm64" => {box: "bento/fedora-40", user: "vagrant"},
"fedora41" => {box: "fedora/41-cloud-base", user: "vagrant"},
"fedora42" => {box: "fedora/42-cloud-base", user: "vagrant"},
"fedora41-bento" => {box: "bento/fedora-41", user: "vagrant"},
"opensuse" => {box: "opensuse/Leap-15.6.x86_64", user: "vagrant"},
"opensuse-tumbleweed" => {box: "opensuse/Tumbleweed.x86_64", user: "vagrant"},
"oraclelinux" => {box: "generic/oracle7", user: "vagrant"},

View File

@@ -1,9 +0,0 @@
#!/bin/bash -eux
# Install collection from source assuming dependencies are present.
# Run in SemaphoreUI this bash script can install Kubespray from the repo
NAMESPACE=kubernetes_sigs
COLLECTION=kubespray
MY_VER=$(grep '^version:' galaxy.yml|cut -d: -f2|sed 's/ //')
ansible-galaxy collection build --force --output-path .
ansible-galaxy collection install --offline --force $NAMESPACE-$COLLECTION-$MY_VER.tar.gz

View File

@@ -20,6 +20,7 @@ function create_container_image_tar() {
kubectl describe cronjobs,jobs,pods --all-namespaces | grep " Image:" | awk '{print $2}' | sort | uniq > "${IMAGES}"
# NOTE: etcd and pause cannot be seen as pods.
# The pause image is used for --pod-infra-container-image option of kubelet.
kubectl cluster-info dump | grep -E "quay.io/coreos/etcd:|registry.k8s.io/pause:" | sed s@\"@@g >> "${IMAGES}"
else
echo "Getting images from file \"${IMAGES_FROM_FILE}\""

View File

@@ -51,7 +51,7 @@ To generate kubespray inventory based on the terraform state file you can run th
You should now have a inventory file named `inventory.ini` that you can use with kubespray, e.g.
```bash
ansible-playbook -i contrib/terraform/gcp/inventory.ini cluster.yml -b -v
ansible-playbook -i contrib/terraform/gcs/inventory.ini cluster.yml -b -v
```
## Variables

5
contrib/terraform/nifcloud/.gitignore vendored Normal file
View File

@@ -0,0 +1,5 @@
*.tfstate*
.terraform.lock.hcl
.terraform
sample-inventory/inventory.ini

View File

@@ -0,0 +1,138 @@
# Kubernetes on NIFCLOUD with Terraform
Provision a Kubernetes cluster on [NIFCLOUD](https://pfs.nifcloud.com/) using Terraform and Kubespray
## Overview
The setup looks like following
```text
Kubernetes cluster
+----------------------------+
+---------------+ | +--------------------+ |
| | | | +--------------------+ |
| API server LB +---------> | | | |
| | | | | Control Plane/etcd | |
+---------------+ | | | node(s) | |
| +-+ | |
| +--------------------+ |
| ^ |
| | |
| v |
| +--------------------+ |
| | +--------------------+ |
| | | | |
| | | Worker | |
| | | node(s) | |
| +-+ | |
| +--------------------+ |
+----------------------------+
```
## Requirements
* Terraform 1.3.7
## Quickstart
### Export Variables
* Your NIFCLOUD credentials:
```bash
export NIFCLOUD_ACCESS_KEY_ID=<YOUR ACCESS KEY>
export NIFCLOUD_SECRET_ACCESS_KEY=<YOUR SECRET ACCESS KEY>
```
* The SSH KEY used to connect to the instance:
* FYI: [Cloud Help(SSH Key)](https://pfs.nifcloud.com/help/ssh.htm)
```bash
export TF_VAR_SSHKEY_NAME=<YOUR SSHKEY NAME>
```
* The IP address to connect to bastion server:
```bash
export TF_VAR_working_instance_ip=$(curl ifconfig.me)
```
### Create The Infrastructure
* Run terraform:
```bash
terraform init
terraform apply -var-file ./sample-inventory/cluster.tfvars
```
### Setup The Kubernetes
* Generate cluster configuration file:
```bash
./generate-inventory.sh > sample-inventory/inventory.ini
```
* Export Variables:
```bash
BASTION_IP=$(terraform output -json | jq -r '.kubernetes_cluster.value.bastion_info | to_entries[].value.public_ip')
API_LB_IP=$(terraform output -json | jq -r '.kubernetes_cluster.value.control_plane_lb')
CP01_IP=$(terraform output -json | jq -r '.kubernetes_cluster.value.control_plane_info | to_entries[0].value.private_ip')
export ANSIBLE_SSH_ARGS="-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ProxyCommand=\"ssh root@${BASTION_IP} -W %h:%p\""
```
* Set ssh-agent"
```bash
eval `ssh-agent`
ssh-add <THE PATH TO YOUR SSH KEY>
```
* Run cluster.yml playbook:
```bash
cd ./../../../
ansible-playbook -i contrib/terraform/nifcloud/inventory/inventory.ini cluster.yml
```
### Connecting to Kubernetes
* [Install kubectl](https://kubernetes.io/docs/tasks/tools/) on the localhost
* Fetching kubeconfig file:
```bash
mkdir -p ~/.kube
scp -o ProxyCommand="ssh root@${BASTION_IP} -W %h:%p" root@${CP01_IP}:/etc/kubernetes/admin.conf ~/.kube/config
```
* Rewrite /etc/hosts
```bash
sudo echo "${API_LB_IP} lb-apiserver.kubernetes.local" >> /etc/hosts
```
* Run kubectl
```bash
kubectl get node
```
## Variables
* `region`: Region where to run the cluster
* `az`: Availability zone where to run the cluster
* `private_ip_bn`: Private ip address of bastion server
* `private_network_cidr`: Subnet of private network
* `instances_cp`: Machine to provision as Control Plane. Key of this object will be used as part of the machine' name
* `private_ip`: private ip address of machine
* `instances_wk`: Machine to provision as Worker Node. Key of this object will be used as part of the machine' name
* `private_ip`: private ip address of machine
* `instance_key_name`: The key name of the Key Pair to use for the instance
* `instance_type_bn`: The instance type of bastion server
* `instance_type_wk`: The instance type of worker node
* `instance_type_cp`: The instance type of control plane
* `image_name`: OS image used for the instance
* `working_instance_ip`: The IP address to connect to bastion server
* `accounting_type`: Accounting type. (1: monthly, 2: pay per use)

View File

@@ -0,0 +1,64 @@
#!/bin/bash
#
# Generates a inventory file based on the terraform output.
# After provisioning a cluster, simply run this command and supply the terraform state file
# Default state file is terraform.tfstate
#
set -e
TF_OUT=$(terraform output -json)
CONTROL_PLANES=$(jq -r '.kubernetes_cluster.value.control_plane_info | to_entries[]' <(echo "${TF_OUT}"))
WORKERS=$(jq -r '.kubernetes_cluster.value.worker_info | to_entries[]' <(echo "${TF_OUT}"))
mapfile -t CONTROL_PLANE_NAMES < <(jq -r '.key' <(echo "${CONTROL_PLANES}"))
mapfile -t WORKER_NAMES < <(jq -r '.key' <(echo "${WORKERS}"))
API_LB=$(jq -r '.kubernetes_cluster.value.control_plane_lb' <(echo "${TF_OUT}"))
echo "[all]"
# Generate control plane hosts
i=1
for name in "${CONTROL_PLANE_NAMES[@]}"; do
private_ip=$(jq -r '. | select( .key=='"\"${name}\""' ) | .value.private_ip' <(echo "${CONTROL_PLANES}"))
echo "${name} ansible_user=root ansible_host=${private_ip} access_ip=${private_ip} ip=${private_ip} etcd_member_name=etcd${i}"
i=$(( i + 1 ))
done
# Generate worker hosts
for name in "${WORKER_NAMES[@]}"; do
private_ip=$(jq -r '. | select( .key=='"\"${name}\""' ) | .value.private_ip' <(echo "${WORKERS}"))
echo "${name} ansible_user=root ansible_host=${private_ip} access_ip=${private_ip} ip=${private_ip}"
done
API_LB=$(jq -r '.kubernetes_cluster.value.control_plane_lb' <(echo "${TF_OUT}"))
echo ""
echo "[all:vars]"
echo "upstream_dns_servers=['8.8.8.8','8.8.4.4']"
echo "loadbalancer_apiserver={'address':'${API_LB}','port':'6443'}"
echo ""
echo "[kube_control_plane]"
for name in "${CONTROL_PLANE_NAMES[@]}"; do
echo "${name}"
done
echo ""
echo "[etcd]"
for name in "${CONTROL_PLANE_NAMES[@]}"; do
echo "${name}"
done
echo ""
echo "[kube_node]"
for name in "${WORKER_NAMES[@]}"; do
echo "${name}"
done
echo ""
echo "[k8s_cluster:children]"
echo "kube_control_plane"
echo "kube_node"

View File

@@ -0,0 +1,36 @@
provider "nifcloud" {
region = var.region
}
module "kubernetes_cluster" {
source = "./modules/kubernetes-cluster"
availability_zone = var.az
prefix = "dev"
private_network_cidr = var.private_network_cidr
instance_key_name = var.instance_key_name
instances_cp = var.instances_cp
instances_wk = var.instances_wk
image_name = var.image_name
instance_type_bn = var.instance_type_bn
instance_type_cp = var.instance_type_cp
instance_type_wk = var.instance_type_wk
private_ip_bn = var.private_ip_bn
additional_lb_filter = [var.working_instance_ip]
}
resource "nifcloud_security_group_rule" "ssh_from_bastion" {
security_group_names = [
module.kubernetes_cluster.security_group_name.bastion
]
type = "IN"
from_port = 22
to_port = 22
protocol = "TCP"
cidr_ip = var.working_instance_ip
}

View File

@@ -0,0 +1,301 @@
#################################################
##
## Local variables
##
locals {
# e.g. east-11 is 11
az_num = reverse(split("-", var.availability_zone))[0]
# e.g. east-11 is e11
az_short_name = "${substr(reverse(split("-", var.availability_zone))[1], 0, 1)}${local.az_num}"
# Port used by the protocol
port_ssh = 22
port_kubectl = 6443
port_kubelet = 10250
# calico: https://docs.tigera.io/calico/latest/getting-started/kubernetes/requirements#network-requirements
port_bgp = 179
port_vxlan = 4789
port_etcd = 2379
}
#################################################
##
## General
##
# data
data "nifcloud_image" "this" {
image_name = var.image_name
}
# private lan
resource "nifcloud_private_lan" "this" {
private_lan_name = "${var.prefix}lan"
availability_zone = var.availability_zone
cidr_block = var.private_network_cidr
accounting_type = var.accounting_type
}
#################################################
##
## Bastion
##
resource "nifcloud_security_group" "bn" {
group_name = "${var.prefix}bn"
description = "${var.prefix} bastion"
availability_zone = var.availability_zone
}
resource "nifcloud_instance" "bn" {
instance_id = "${local.az_short_name}${var.prefix}bn01"
security_group = nifcloud_security_group.bn.group_name
instance_type = var.instance_type_bn
user_data = templatefile("${path.module}/templates/userdata.tftpl", {
private_ip_address = var.private_ip_bn
ssh_port = local.port_ssh
hostname = "${local.az_short_name}${var.prefix}bn01"
})
availability_zone = var.availability_zone
accounting_type = var.accounting_type
image_id = data.nifcloud_image.this.image_id
key_name = var.instance_key_name
network_interface {
network_id = "net-COMMON_GLOBAL"
}
network_interface {
network_id = nifcloud_private_lan.this.network_id
ip_address = "static"
}
# The image_id changes when the OS image type is demoted from standard to public.
lifecycle {
ignore_changes = [
image_id,
user_data,
]
}
}
#################################################
##
## Control Plane
##
resource "nifcloud_security_group" "cp" {
group_name = "${var.prefix}cp"
description = "${var.prefix} control plane"
availability_zone = var.availability_zone
}
resource "nifcloud_instance" "cp" {
for_each = var.instances_cp
instance_id = "${local.az_short_name}${var.prefix}${each.key}"
security_group = nifcloud_security_group.cp.group_name
instance_type = var.instance_type_cp
user_data = templatefile("${path.module}/templates/userdata.tftpl", {
private_ip_address = each.value.private_ip
ssh_port = local.port_ssh
hostname = "${local.az_short_name}${var.prefix}${each.key}"
})
availability_zone = var.availability_zone
accounting_type = var.accounting_type
image_id = data.nifcloud_image.this.image_id
key_name = var.instance_key_name
network_interface {
network_id = "net-COMMON_GLOBAL"
}
network_interface {
network_id = nifcloud_private_lan.this.network_id
ip_address = "static"
}
# The image_id changes when the OS image type is demoted from standard to public.
lifecycle {
ignore_changes = [
image_id,
user_data,
]
}
}
resource "nifcloud_load_balancer" "this" {
load_balancer_name = "${local.az_short_name}${var.prefix}cp"
accounting_type = var.accounting_type
balancing_type = 1 // Round-Robin
load_balancer_port = local.port_kubectl
instance_port = local.port_kubectl
instances = [for v in nifcloud_instance.cp : v.instance_id]
filter = concat(
[for k, v in nifcloud_instance.cp : v.public_ip],
[for k, v in nifcloud_instance.wk : v.public_ip],
var.additional_lb_filter,
)
filter_type = 1 // Allow
}
#################################################
##
## Worker
##
resource "nifcloud_security_group" "wk" {
group_name = "${var.prefix}wk"
description = "${var.prefix} worker"
availability_zone = var.availability_zone
}
resource "nifcloud_instance" "wk" {
for_each = var.instances_wk
instance_id = "${local.az_short_name}${var.prefix}${each.key}"
security_group = nifcloud_security_group.wk.group_name
instance_type = var.instance_type_wk
user_data = templatefile("${path.module}/templates/userdata.tftpl", {
private_ip_address = each.value.private_ip
ssh_port = local.port_ssh
hostname = "${local.az_short_name}${var.prefix}${each.key}"
})
availability_zone = var.availability_zone
accounting_type = var.accounting_type
image_id = data.nifcloud_image.this.image_id
key_name = var.instance_key_name
network_interface {
network_id = "net-COMMON_GLOBAL"
}
network_interface {
network_id = nifcloud_private_lan.this.network_id
ip_address = "static"
}
# The image_id changes when the OS image type is demoted from standard to public.
lifecycle {
ignore_changes = [
image_id,
user_data,
]
}
}
#################################################
##
## Security Group Rule: Kubernetes
##
# ssh
resource "nifcloud_security_group_rule" "ssh_from_bastion" {
security_group_names = [
nifcloud_security_group.wk.group_name,
nifcloud_security_group.cp.group_name,
]
type = "IN"
from_port = local.port_ssh
to_port = local.port_ssh
protocol = "TCP"
source_security_group_name = nifcloud_security_group.bn.group_name
}
# kubectl
resource "nifcloud_security_group_rule" "kubectl_from_worker" {
security_group_names = [
nifcloud_security_group.cp.group_name,
]
type = "IN"
from_port = local.port_kubectl
to_port = local.port_kubectl
protocol = "TCP"
source_security_group_name = nifcloud_security_group.wk.group_name
}
# kubelet
resource "nifcloud_security_group_rule" "kubelet_from_worker" {
security_group_names = [
nifcloud_security_group.cp.group_name,
]
type = "IN"
from_port = local.port_kubelet
to_port = local.port_kubelet
protocol = "TCP"
source_security_group_name = nifcloud_security_group.wk.group_name
}
resource "nifcloud_security_group_rule" "kubelet_from_control_plane" {
security_group_names = [
nifcloud_security_group.wk.group_name,
]
type = "IN"
from_port = local.port_kubelet
to_port = local.port_kubelet
protocol = "TCP"
source_security_group_name = nifcloud_security_group.cp.group_name
}
#################################################
##
## Security Group Rule: calico
##
# vslan
resource "nifcloud_security_group_rule" "vxlan_from_control_plane" {
security_group_names = [
nifcloud_security_group.wk.group_name,
]
type = "IN"
from_port = local.port_vxlan
to_port = local.port_vxlan
protocol = "UDP"
source_security_group_name = nifcloud_security_group.cp.group_name
}
resource "nifcloud_security_group_rule" "vxlan_from_worker" {
security_group_names = [
nifcloud_security_group.cp.group_name,
]
type = "IN"
from_port = local.port_vxlan
to_port = local.port_vxlan
protocol = "UDP"
source_security_group_name = nifcloud_security_group.wk.group_name
}
# bgp
resource "nifcloud_security_group_rule" "bgp_from_control_plane" {
security_group_names = [
nifcloud_security_group.wk.group_name,
]
type = "IN"
from_port = local.port_bgp
to_port = local.port_bgp
protocol = "TCP"
source_security_group_name = nifcloud_security_group.cp.group_name
}
resource "nifcloud_security_group_rule" "bgp_from_worker" {
security_group_names = [
nifcloud_security_group.cp.group_name,
]
type = "IN"
from_port = local.port_bgp
to_port = local.port_bgp
protocol = "TCP"
source_security_group_name = nifcloud_security_group.wk.group_name
}
# etcd
resource "nifcloud_security_group_rule" "etcd_from_worker" {
security_group_names = [
nifcloud_security_group.cp.group_name,
]
type = "IN"
from_port = local.port_etcd
to_port = local.port_etcd
protocol = "TCP"
source_security_group_name = nifcloud_security_group.wk.group_name
}

View File

@@ -0,0 +1,48 @@
output "control_plane_lb" {
description = "The DNS name of LB for control plane"
value = nifcloud_load_balancer.this.dns_name
}
output "security_group_name" {
description = "The security group used in the cluster"
value = {
bastion = nifcloud_security_group.bn.group_name,
control_plane = nifcloud_security_group.cp.group_name,
worker = nifcloud_security_group.wk.group_name,
}
}
output "private_network_id" {
description = "The private network used in the cluster"
value = nifcloud_private_lan.this.id
}
output "bastion_info" {
description = "The basion information in cluster"
value = { (nifcloud_instance.bn.instance_id) : {
instance_id = nifcloud_instance.bn.instance_id,
unique_id = nifcloud_instance.bn.unique_id,
private_ip = nifcloud_instance.bn.private_ip,
public_ip = nifcloud_instance.bn.public_ip,
} }
}
output "worker_info" {
description = "The worker information in cluster"
value = { for v in nifcloud_instance.wk : v.instance_id => {
instance_id = v.instance_id,
unique_id = v.unique_id,
private_ip = v.private_ip,
public_ip = v.public_ip,
} }
}
output "control_plane_info" {
description = "The control plane information in cluster"
value = { for v in nifcloud_instance.cp : v.instance_id => {
instance_id = v.instance_id,
unique_id = v.unique_id,
private_ip = v.private_ip,
public_ip = v.public_ip,
} }
}

View File

@@ -0,0 +1,45 @@
#!/bin/bash
#################################################
##
## IP Address
##
configure_private_ip_address () {
cat << EOS > /etc/netplan/01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
ens192:
dhcp4: yes
dhcp6: yes
dhcp-identifier: mac
ens224:
dhcp4: no
dhcp6: no
addresses: [${private_ip_address}]
EOS
netplan apply
}
configure_private_ip_address
#################################################
##
## SSH
##
configure_ssh_port () {
sed -i 's/^#*Port [0-9]*/Port ${ssh_port}/' /etc/ssh/sshd_config
}
configure_ssh_port
#################################################
##
## Hostname
##
hostnamectl set-hostname ${hostname}
#################################################
##
## Disable swap files genereated by systemd-gpt-auto-generator
##
systemctl mask "dev-sda3.swap"

View File

@@ -0,0 +1,9 @@
terraform {
required_version = ">=1.3.7"
required_providers {
nifcloud = {
source = "nifcloud/nifcloud"
version = ">= 1.8.0, < 2.0.0"
}
}
}

View File

@@ -0,0 +1,81 @@
variable "availability_zone" {
description = "The availability zone"
type = string
}
variable "prefix" {
description = "The prefix for the entire cluster"
type = string
validation {
condition = length(var.prefix) <= 5
error_message = "Must be a less than 5 character long."
}
}
variable "private_network_cidr" {
description = "The subnet of private network"
type = string
validation {
condition = can(cidrnetmask(var.private_network_cidr))
error_message = "Must be a valid IPv4 CIDR block address."
}
}
variable "private_ip_bn" {
description = "Private IP of bastion server"
type = string
}
variable "instances_cp" {
type = map(object({
private_ip = string
}))
}
variable "instances_wk" {
type = map(object({
private_ip = string
}))
}
variable "instance_key_name" {
description = "The key name of the Key Pair to use for the instance"
type = string
}
variable "instance_type_bn" {
description = "The instance type of bastion server"
type = string
}
variable "instance_type_wk" {
description = "The instance type of worker"
type = string
}
variable "instance_type_cp" {
description = "The instance type of control plane"
type = string
}
variable "image_name" {
description = "The name of image"
type = string
}
variable "additional_lb_filter" {
description = "Additional LB filter"
type = list(string)
}
variable "accounting_type" {
type = string
default = "1"
validation {
condition = anytrue([
var.accounting_type == "1", // Monthly
var.accounting_type == "2", // Pay per use
])
error_message = "Must be a 1 or 2."
}
}

View File

@@ -0,0 +1,3 @@
output "kubernetes_cluster" {
value = module.kubernetes_cluster
}

View File

@@ -0,0 +1,22 @@
region = "jp-west-1"
az = "west-11"
instance_key_name = "deployerkey"
instance_type_bn = "e-medium"
instance_type_cp = "e-medium"
instance_type_wk = "e-medium"
private_network_cidr = "192.168.30.0/24"
instances_cp = {
"cp01" : { private_ip : "192.168.30.11/24" }
"cp02" : { private_ip : "192.168.30.12/24" }
"cp03" : { private_ip : "192.168.30.13/24" }
}
instances_wk = {
"wk01" : { private_ip : "192.168.30.21/24" }
"wk02" : { private_ip : "192.168.30.22/24" }
}
private_ip_bn = "192.168.30.10/24"
image_name = "Ubuntu Server 22.04 LTS"

View File

@@ -0,0 +1 @@
../../../../inventory/sample/group_vars

View File

@@ -0,0 +1,9 @@
terraform {
required_version = ">=1.3.7"
required_providers {
nifcloud = {
source = "nifcloud/nifcloud"
version = "1.8.0"
}
}
}

View File

@@ -0,0 +1,77 @@
variable "region" {
description = "The region"
type = string
}
variable "az" {
description = "The availability zone"
type = string
}
variable "private_ip_bn" {
description = "Private IP of bastion server"
type = string
}
variable "private_network_cidr" {
description = "The subnet of private network"
type = string
validation {
condition = can(cidrnetmask(var.private_network_cidr))
error_message = "Must be a valid IPv4 CIDR block address."
}
}
variable "instances_cp" {
type = map(object({
private_ip = string
}))
}
variable "instances_wk" {
type = map(object({
private_ip = string
}))
}
variable "instance_key_name" {
description = "The key name of the Key Pair to use for the instance"
type = string
}
variable "instance_type_bn" {
description = "The instance type of bastion server"
type = string
}
variable "instance_type_wk" {
description = "The instance type of worker"
type = string
}
variable "instance_type_cp" {
description = "The instance type of control plane"
type = string
}
variable "image_name" {
description = "The name of image"
type = string
}
variable "working_instance_ip" {
description = "The IP address to connect to bastion server."
type = string
}
variable "accounting_type" {
type = string
default = "2"
validation {
condition = anytrue([
var.accounting_type == "1", // Monthly
var.accounting_type == "2", // Pay per use
])
error_message = "Must be a 1 or 2."
}
}

View File

@@ -281,9 +281,9 @@ For your cluster, edit `inventory/$CLUSTER/cluster.tfvars`.
|`k8s_allowed_remote_ips_ipv6` | List of IPv6 CIDR allowed to initiate a SSH connection, empty by default |
|`k8s_allowed_egress_ipv6_ips` | List of IPv6 CIDRs allowed for egress traffic, `["::/0"]` by default |
|`worker_allowed_ports` | List of ports to open on worker nodes, `[{ "protocol" = "tcp", "port_range_min" = 30000, "port_range_max" = 32767, "remote_ip_prefix" = "0.0.0.0/0"}]` by default |
|`worker_allowed_ports_ipv6` | List of ports to open on worker nodes for IPv6 CIDR blocks, `[{ "protocol" = "tcp", "port_range_min" = 30000, "port_range_max" = 32767, "remote_ip_prefix" = "::/0"}, { "protocol" = "ipv6-icmp", "port_range_min" = 0, "port_range_max" = 0, "remote_ip_prefix" = "::/0"}]` by default |
|`worker_allowed_ports_ipv6` | List of ports to open on worker nodes for IPv6 CIDR blocks, `[{ "protocol" = "tcp", "port_range_min" = 30000, "port_range_max" = 32767, "remote_ip_prefix" = "::/0"}]` by default |
|`master_allowed_ports` | List of ports to open on master nodes, expected format is `[{ "protocol" = "tcp", "port_range_min" = 443, "port_range_max" = 443, "remote_ip_prefix" = "0.0.0.0/0"}]`, empty by default |
|`master_allowed_ports_ipv6` | List of ports to open on master nodes for IPv6 CIDR blocks, `[{ "protocol" = "ipv6-icmp", "port_range_min" = 0, "port_range_max" = 0, "remote_ip_prefix" = "::/0"}]` by default |
|`master_allowed_ports_ipv6` | List of ports to open on master nodes for IPv6 CIDR blocks, expected format is `[{ "protocol" = "tcp", "port_range_min" = 443, "port_range_max" = 443, "remote_ip_prefix" = "::/0"}]`, empty by default |
|`node_root_volume_size_in_gb` | Size of the root volume for nodes, 0 to use ephemeral storage |
|`master_root_volume_size_in_gb` | Size of the root volume for masters, 0 to use ephemeral storage |
|`master_volume_type` | Volume type of the root volume for control_plane, 'Default' by default |

View File

@@ -1006,7 +1006,7 @@ resource "openstack_compute_instance_v2" "glusterfs_node_no_floating_ip" {
name = "${var.cluster_name}-gfs-node-nf-${count.index + 1}"
count = var.number_of_gfs_nodes_no_floating_ip
availability_zone = element(var.az_list, count.index)
image_id = var.gfs_root_volume_size_in_gb == 0 ? local.image_to_use_gfs : null
image_name = var.gfs_root_volume_size_in_gb == 0 ? local.image_to_use_gfs : null
flavor_id = var.flavor_gfs_node
key_pair = openstack_compute_keypair_v2.k8s.name
@@ -1078,7 +1078,7 @@ resource "openstack_networking_floatingip_associate_v2" "k8s_nodes" {
port_id = openstack_networking_port_v2.k8s_nodes_port[each.key].id
}
resource "openstack_blockstorage_volume_v3" "glusterfs_volume" {
resource "openstack_blockstorage_volume_v2" "glusterfs_volume" {
name = "${var.cluster_name}-glusterfs_volume-${count.index + 1}"
count = var.gfs_root_volume_size_in_gb == 0 ? var.number_of_gfs_nodes_no_floating_ip : 0
description = "Non-ephemeral volume for GlusterFS"
@@ -1088,5 +1088,5 @@ resource "openstack_blockstorage_volume_v3" "glusterfs_volume" {
resource "openstack_compute_volume_attach_v2" "glusterfs_volume" {
count = var.gfs_root_volume_size_in_gb == 0 ? var.number_of_gfs_nodes_no_floating_ip : 0
instance_id = element(openstack_compute_instance_v2.glusterfs_node_no_floating_ip.*.id, count.index)
volume_id = element(openstack_blockstorage_volume_v3.glusterfs_volume.*.id, count.index)
volume_id = element(openstack_blockstorage_volume_v2.glusterfs_volume.*.id, count.index)
}

View File

@@ -271,14 +271,7 @@ variable "master_allowed_ports" {
variable "master_allowed_ports_ipv6" {
type = list(any)
default = [
{
"protocol" = "ipv6-icmp"
"port_range_min" = 0
"port_range_max" = 0
"remote_ip_prefix" = "::/0"
},
]
default = []
}
variable "worker_allowed_ports" {
@@ -304,12 +297,6 @@ variable "worker_allowed_ports_ipv6" {
"port_range_max" = 32767
"remote_ip_prefix" = "::/0"
},
{
"protocol" = "ipv6-icmp"
"port_range_min" = 0
"port_range_max" = 0
"remote_ip_prefix" = "::/0"
},
]
}

View File

@@ -3,7 +3,7 @@ terraform {
required_providers {
upcloud = {
source = "UpCloudLtd/upcloud"
version = "~>5.29.1"
version = "~>5.9.0"
}
}
required_version = ">= 0.13"

View File

@@ -3,7 +3,7 @@ terraform {
required_providers {
upcloud = {
source = "UpCloudLtd/upcloud"
version = "~>5.29.1"
version = "~>5.9.0"
}
}
required_version = ">= 0.13"

View File

@@ -1,13 +1,5 @@
# Cilium
## Unprivileged agent configuration
By default, Cilium is installed with `securityContext.privileged: false`. You need to set the `kube_owner` variable to `root` in the inventory:
```yml
kube_owner: root
```
## IP Address Management (IPAM)
IP Address Management (IPAM) is responsible for the allocation and management of IP addresses used by network endpoints (container and others) managed by Cilium. The default mode is "Cluster Scope".
@@ -245,7 +237,7 @@ cilium_operator_extra_volume_mounts:
## Choose Cilium version
```yml
cilium_version: "1.18.6"
cilium_version: "1.18.5"
```
## Add variable to config

View File

@@ -32,7 +32,7 @@ add `kube_proxy_masquerade_all: true` in `group_vars/all/all.yml`
* Disable nodelocaldns
The nodelocal dns IP is not reachable.
The nodelocal dns IP is not reacheable.
Disable it in `sample/group_vars/k8s_cluster/k8s_cluster.yml`

View File

@@ -65,8 +65,9 @@ In kubespray, the default runtime name is "runc", and it can be configured with
containerd_runc_runtime:
name: runc
type: "io.containerd.runc.v2"
engine: ""
root: ""
options:
Root: ""
SystemdCgroup: "false"
BinaryName: /usr/local/bin/my-runc
base_runtime_spec: cri-base.json

View File

@@ -80,7 +80,7 @@ The `crio_remap_enable` configures the `/etc/subuid` and `/etc/subgid` files to
By default, 16M uids and gids are reserved for user namespaces (256 pods * 65536 uids/gids) at the end of the uid/gid space.
The `crio_default_capabilities` configure the default containers capabilities for the crio.
Defaults capabilities are:
Defaults capabilties are:
```yaml
crio_default_capabilities:

2
docs/_sidebar.md generated
View File

@@ -6,6 +6,7 @@
* [Downloads](/docs/advanced/downloads.md)
* [Gcp-lb](/docs/advanced/gcp-lb.md)
* [Kubernetes-reliability](/docs/advanced/kubernetes-reliability.md)
* [Mitogen](/docs/advanced/mitogen.md)
* [Netcheck](/docs/advanced/netcheck.md)
* [Ntp](/docs/advanced/ntp.md)
* [Proxy](/docs/advanced/proxy.md)
@@ -57,6 +58,7 @@
* [Setting-up-your-first-cluster](/docs/getting_started/setting-up-your-first-cluster.md)
* Ingress
* [Alb Ingress Controller](/docs/ingress/alb_ingress_controller.md)
* [Ingress Nginx](/docs/ingress/ingress_nginx.md)
* [Kube-vip](/docs/ingress/kube-vip.md)
* [Metallb](/docs/ingress/metallb.md)
* Operating Systems

View File

@@ -6,7 +6,7 @@
- [Create New TLS Root CA Certificate and Key](#create-new-tls-root-ca-certificate-and-key)
- [Install Cloudflare PKI/TLS `cfssl` Toolkit.](#install-cloudflare-pkitls-cfssl-toolkit)
- [Create Root Certificate Authority (CA) Configuration File](#create-root-certificate-authority-ca-configuration-file)
- [Create Certificate Signing Request (CSR) Configuration File](#create-certificate-signing-request-csr-configuration-file)
- [Create Certficate Signing Request (CSR) Configuration File](#create-certficate-signing-request-csr-configuration-file)
- [Create TLS Root CA Certificate and Key](#create-tls-root-ca-certificate-and-key)
Cert-Manager is a native Kubernetes certificate management controller. It can help with issuing certificates from a variety of sources, such as Lets Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, or self signed. It will ensure certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry.
@@ -30,7 +30,14 @@ If you don't have a TLS Root CA certificate and key available, you can create th
A common use-case for cert-manager is requesting TLS signed certificates to secure your ingress resources. This can be done by simply adding annotations to your Ingress resources and cert-manager will facilitate creating the Certificate resource for you. A small sub-component of cert-manager, ingress-shim, is responsible for this.
For example, if you're using the Traefik ingress controller, you can secure the Prometheus ingress by adding the annotation `cert-manager.io/cluster-issuer: ca-issuer` and the `spec.tls` section to the `Ingress` resource definition.
To enable the Nginx Ingress controller as part of your Kubespray deployment, simply edit your K8s cluster addons inventory e.g. `inventory\sample\group_vars\k8s_cluster\addons.yml` and set `ingress_nginx_enabled` to true.
```ini
# Nginx ingress controller deployment
ingress_nginx_enabled: true
```
For example, if you're using the Nginx ingress controller, you can secure the Prometheus ingress by adding the annotation `cert-manager.io/cluster-issuer: ca-issuer` and the `spec.tls` section to the `Ingress` resource definition.
```yaml
apiVersion: networking.k8s.io/v1
@@ -41,9 +48,9 @@ metadata:
labels:
prometheus: k8s
annotations:
kubernetes.io/ingress.class: "nginx"
cert-manager.io/cluster-issuer: ca-issuer
spec:
ingressClassName: "traefik"
tls:
- hosts:
- prometheus.example.com
@@ -65,8 +72,8 @@ Once deployed to your K8s cluster, every 3 months cert-manager will automaticall
Please consult the official upstream documentation:
- [cert-manager Ingress Usage](https://cert-manager.io/usage/ingress/)
- [cert-manager Ingress Tutorial](https://cert-manager.io/tutorials/acme/ingress/#step-3-assign-a-dns-name)
- [cert-manager Ingress Usage](https://cert-manager.io/v1.5-docs/usage/ingress/)
- [cert-manager Ingress Tutorial](https://cert-manager.io/v1.5-docs/tutorials/acme/ingress/#step-3-assign-a-dns-name)
### ACME
@@ -74,12 +81,12 @@ The ACME Issuer type represents a single account registered with the Automated C
Certificates issued by public ACME servers are typically trusted by clients computers by default. This means that, for example, visiting a website that is backed by an ACME certificate issued for that URL, will be trusted by default by most clients web browsers. ACME certificates are typically free.
- [ACME Configuration](https://cert-manager.io/docs/configuration/acme/)
- [ACME HTTP Validation](https://cert-manager.io/docs/tutorials/acme/http-validation/)
- [HTTP01 Challenges](https://cert-manager.io/docs/configuration/acme/http01/)
- [ACME DNS Validation](https://cert-manager.io/docs/tutorials/acme/dns-validation/)
- [DNS01 Challenges](https://cert-manager.io/docs/configuration/acme/dns01/)
- [ACME FAQ](https://cert-manager.io/docs/troubleshooting/acme/)
- [ACME Configuration](https://cert-manager.io/v1.5-docs/configuration/acme/)
- [ACME HTTP Validation](https://cert-manager.io/v1.5-docs/tutorials/acme/http-validation/)
- [HTTP01 Challenges](https://cert-manager.io/v1.5-docs/configuration/acme/http01/)
- [ACME DNS Validation](https://cert-manager.io/v1.5-docs/tutorials/acme/dns-validation/)
- [DNS01 Challenges](https://cert-manager.io/v1.5-docs/configuration/acme/dns01/)
- [ACME FAQ](https://cert-manager.io/v1.5-docs/faq/acme/)
#### ACME With An Internal Certificate Authority
@@ -127,7 +134,7 @@ $ cat > ca-config.json <<EOF
EOF
```
#### Create Certificate Signing Request (CSR) Configuration File
#### Create Certficate Signing Request (CSR) Configuration File
The TLS certificate `names` details can be updated to your own specific requirements.

View File

@@ -1,4 +1,4 @@
# GCP Load Balancers for type=LoadBalancer of Kubernetes Services
# GCP Load Balancers for type=LoadBalacer of Kubernetes Services
> **Removed**: Since v1.31 (the Kubespray counterpart is v2.27), Kubernetes no longer supports `cloud_provider`. (except external cloud provider)

30
docs/advanced/mitogen.md Normal file
View File

@@ -0,0 +1,30 @@
# Mitogen
*Warning:* Mitogen support is now deprecated in kubespray due to upstream not releasing an updated version to support ansible 4.x (ansible-base 2.11.x) and above. The CI support has been stripped for mitogen and we are no longer validating any support or regressions for it. The supporting mitogen install playbook and integration documentation will be removed in a later version.
[Mitogen for Ansible](https://mitogen.networkgenomics.com/ansible_detailed.html) allow a 1.25x - 7x speedup and a CPU usage reduction of at least 2x, depending on network conditions, modules executed, and time already spent by targets on useful work. Mitogen cannot improve a module once it is executing, it can only ensure the module executes as quickly as possible.
## Install
```ShellSession
ansible-playbook contrib/mitogen/mitogen.yml
```
The above playbook sets the ansible `strategy` and `strategy_plugins` in `ansible.cfg` but you can also enable them if you use your own `ansible.cfg` by setting the environment varialbles:
```ShellSession
export ANSIBLE_STRATEGY=mitogen_linear
export ANSIBLE_STRATEGY_PLUGINS=plugins/mitogen/ansible_mitogen/plugins/strategy
```
... or `ansible.cfg` setup:
```ini
[defaults]
strategy_plugins = plugins/mitogen/ansible_mitogen/plugins/strategy
strategy=mitogen_linear
```
## Limitation
If you are experiencing problems, please see the [documentation](https://mitogen.networkgenomics.com/ansible_detailed.html#noteworthy-differences).

View File

@@ -30,9 +30,9 @@ If the latest version supported according to pip is 6.7.0 it means you are runni
Based on the table below and the available python version for your ansible host you should choose the appropriate ansible version to use with kubespray.
| Ansible Version | Python Version |
|-------------------|----------------|
| >=2.18.0, <2.19.0 | 3.11-3.13 |
| Ansible Version | Python Version |
|-----------------|----------------|
| >= 2.17.3 | 3.10-3.12 |
## Customize Ansible vars
@@ -42,10 +42,13 @@ Kubespray expects users to use one of the following variables sources for settin
|----------------------------------------|------------------------------------------------------------------------------|
| inventory vars | |
| - **inventory group_vars** | most used |
| - inventory host_vars | host specific vars overrides, group_vars is usually more practical |
| - inventory host_vars | host specifc vars overrides, group_vars is usually more practical |
| **extra vars** (always win precedence) | override with ``ansible-playbook -e @foo.yml`` |
> Extra vars are best used to override kubespray internal variables, for instances, roles/vars/. Those vars are usually **not expected** (by Kubespray developers) to be modified by end users, and not part of Kubespray interface. Thus they can change, disappear, or break stuff unexpectedly.
[!IMPORTANT]
Extra vars are best used to override kubespray internal variables, for instances, roles/vars/.
Those vars are usually **not expected** (by Kubespray developers) to be modified by end users, and not part of Kubespray
interface. Thus they can change, disappear, or break stuff unexpectedly.
## Ansible tags
@@ -78,6 +81,7 @@ The following tags are defined in playbooks:
| crio | Configuring crio container engine for hosts |
| crun | Configuring crun runtime |
| csi-driver | Configuring csi driver |
| dashboard | Installing and configuring the Kubernetes Dashboard |
| dns | Remove dns entries when resetting |
| docker | Configuring docker engine runtime for hosts |
| download | Fetching container images to a delegate host |
@@ -118,7 +122,7 @@ The following tags are defined in playbooks:
| metrics_server | Configuring metrics_server |
| netchecker | Installing netchecker K8s app |
| network | Configuring networking plugins for K8s |
| mounts | Umount kubelet dirs when resetting |
| mounts | Umount kubelet dirs when reseting |
| multus | Network plugin multus |
| nginx | Configuring LB for kube-apiserver instances |
| node | Configuring K8s minion (compute) node role |
@@ -177,13 +181,17 @@ ansible-playbook -i inventory/sample/hosts.ini cluster.yml \
Note: use `--tags` and `--skip-tags` wisely and only if you're 100% sure what you're doing.
## Mitogen
Mitogen support is deprecated, please see [mitogen related docs](/docs/advanced/mitogen.md) for usage and reasons for deprecation.
## Troubleshooting Ansible issues
Having the wrong version of ansible, ansible collections or python dependencies can cause issue.
In particular, Kubespray ship custom modules which Ansible needs to find, for which you should specify [ANSIBLE_LIBRARY](https://docs.ansible.com/ansible/latest/dev_guide/developing_locally.html#adding-a-module-or-plugin-outside-of-a-collection)
In particular, Kubespray ship custom modules which Ansible needs to find, for which you should specify [ANSIBLE_LIBRAY](https://docs.ansible.com/ansible/latest/dev_guide/developing_locally.html#adding-a-module-or-plugin-outside-of-a-collection)
```ShellSession
export ANSIBLE_LIBRARY=<kubespray_dir>/library`
export ANSIBLE_LIBRAY=<kubespray_dir>/library`
```
A simple way to ensure you get all the correct version of Ansible is to use
@@ -192,11 +200,11 @@ You will then need to use [bind mounts](https://docs.docker.com/storage/bind-mou
to access the inventory and SSH key in the container, like this:
```ShellSession
git checkout v2.30.0
docker pull quay.io/kubespray/kubespray:v2.30.0
git checkout v2.29.0
docker pull quay.io/kubespray/kubespray:v2.29.0
docker run --rm -it --mount type=bind,source="$(pwd)"/inventory/sample,dst=/inventory \
--mount type=bind,source="${HOME}"/.ssh/id_rsa,dst=/root/.ssh/id_rsa \
quay.io/kubespray/kubespray:v2.30.0 bash
quay.io/kubespray/kubespray:v2.29.0 bash
# Inside the container you may now run the kubespray playbooks:
ansible-playbook -i /inventory/inventory.ini --private-key /root/.ssh/id_rsa cluster.yml
```

View File

@@ -6,7 +6,7 @@ See [.gitlab-ci.yml](/.gitlab-ci.yml) and the included files for an overview.
## Runners
Kubespray has 2 types of GitLab runners, both deployed on the Kubespray CI cluster (hosted on Oracle Cloud Infrastructure):
Kubespray has 2 types of GitLab runners, both deployed on the Kubespray CI cluster (hosted on Oracle Cloud Infrastucture):
- pods: use the [gitlab-ci kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes/)
- vagrant: custom executor running in pods with access to the libvirt socket on the nodes
@@ -145,6 +145,7 @@ upstream_dns_servers:
- 1.0.0.1
# Extensions
ingress_nginx_enabled: True
helm_enabled: True
cert_manager_enabled: True
metrics_server_enabled: True
@@ -155,7 +156,7 @@ kube_feature_gates:
- "NodeSwap=True"
```
## Additional files
## Aditional files
This section documents additional files used to complete a deployment of the kubespray CI, these files sit on the control-plane node and assume a working kubernetes cluster.

View File

@@ -13,12 +13,10 @@ debian12 | :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: |
debian13 | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
fedora39 | :white_check_mark: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: |
fedora40 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora41 | :white_check_mark: | :x: | :x: | :x: | :x: | :white_check_mark: | :x: |
fedora42 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
flatcar4081 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
openeuler24 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
rockylinux10 | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
rockylinux9 | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
ubuntu20 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu22 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu24 | :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: | :x: |
@@ -33,14 +31,12 @@ debian12 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
debian13 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora39 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora40 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora41 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora42 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
flatcar4081 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
openeuler24 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
rockylinux10 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
rockylinux9 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu20 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu22 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu24 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu24 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
## docker
@@ -53,11 +49,9 @@ debian12 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
debian13 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora39 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora40 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora41 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
fedora42 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
flatcar4081 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
openeuler24 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
rockylinux10 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
rockylinux9 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu20 | :x: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu22 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |
ubuntu24 | :white_check_mark: | :x: | :x: | :x: | :x: | :x: | :x: |

View File

@@ -83,6 +83,32 @@ authentication. One can get a kubeconfig from kube_control_plane hosts
For more information on kubeconfig and accessing a Kubernetes cluster, refer to
the Kubernetes [documentation](https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/).
## Accessing Kubernetes Dashboard
Supported version is kubernetes-dashboard v2.0.x :
- Login option : token/kubeconfig by default
- Deployed by default in "kube-system" namespace, can be overridden with `dashboard_namespace: kubernetes-dashboard` in inventory,
- Only serves over https
Access is described in [dashboard docs](https://github.com/kubernetes/dashboard/tree/master/docs/user/accessing-dashboard). With kubespray's default deployment in kube-system namespace, instead of kubernetes-dashboard :
- Proxy URL is <http://localhost:8001/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/#/login>
- kubectl commands must be run with "-n kube-system"
Accessing through Ingress is highly recommended. For proxy access, please note that proxy must listen to [localhost](https://github.com/kubernetes/dashboard/issues/692#issuecomment-220492484) (`proxy --address="x.x.x.x"` will not work)
For token authentication, guide to create Service Account is provided in [dashboard sample user](https://github.com/kubernetes/dashboard/blob/master/docs/user/access-control/creating-sample-user.md) doc. Still take care of default namespace.
Access can also by achieved via ssh tunnel on a control plane :
```bash
# localhost:8081 will be sent to control-plane-1's own localhost:8081
ssh -L8001:localhost:8001 user@control-plane-1
sudo -i
kubectl proxy
```
## Accessing Kubernetes API
The main client of Kubernetes is `kubectl`. It is installed on each kube_control_plane

View File

@@ -0,0 +1,203 @@
# Installation Guide
## Contents
- [Prerequisite Generic Deployment Command](#prerequisite-generic-deployment-command)
- [Provider Specific Steps](#provider-specific-steps)
- [Docker for Mac](#docker-for-mac)
- [minikube](#minikube)
- [AWS](#aws)
- [GCE - GKE](#gce-gke)
- [Azure](#azure)
- [Bare-metal](#bare-metal)
- [Verify installation](#verify-installation)
- [Detect installed version](#detect-installed-version)
- [Using Helm](#using-helm)
## Prerequisite Generic Deployment Command
!!! attention
The default configuration watches Ingress object from *all the namespaces*.
To change this behavior use the flag `--watch-namespace` to limit the scope to a particular namespace.
!!! warning
If multiple Ingresses define different paths for the same host, the ingress controller will merge the definitions.
!!! attention
If you're using GKE you need to initialize your user as a cluster-admin with the following command:
```console
kubectl create clusterrolebinding cluster-admin-binding \
--clusterrole cluster-admin \
--user $(gcloud config get-value account)
```
The following **Mandatory Command** is required for all deployments except for AWS. See below for the AWS version.
```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.13.3/deploy/static/provider/cloud/deploy.yaml
```
### Provider Specific Steps
There are cloud provider specific yaml files.
#### Docker for Mac
Kubernetes is available in Docker for Mac (from [version 18.06.0-ce](https://docs.docker.com/docker-for-mac/release-notes/#stable-releases-of-2018))
First you need to [enable kubernetes](https://docs.docker.com/docker-for-mac/#kubernetes).
Then you have to create a service:
```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
```
#### minikube
For standard usage:
```console
minikube addons enable ingress
```
For development:
1. Disable the ingress addon:
```console
minikube addons disable ingress
```
1. Execute `make dev-env`
1. Confirm the `nginx-ingress-controller` deployment exists:
```console
$ kubectl get pods -n ingress-nginx
NAME READY STATUS RESTARTS AGE
default-http-backend-66b447d9cf-rrlf9 1/1 Running 0 12s
nginx-ingress-controller-fdcdcd6dd-vvpgs 1/1 Running 0 11s
```
#### AWS
In AWS we use an Elastic Load Balancer (ELB) to expose the NGINX Ingress controller behind a Service of `Type=LoadBalancer`.
Since Kubernetes v1.9.0 it is possible to use a classic load balancer (ELB) or network load balancer (NLB)
Please check the [elastic load balancing AWS details page](https://aws.amazon.com/elasticloadbalancing/details/)
##### Elastic Load Balancer - ELB
This setup requires to choose in which layer (L4 or L7) we want to configure the Load Balancer:
- [Layer 4](https://en.wikipedia.org/wiki/OSI_model#Layer_4:_Transport_Layer): Use an Network Load Balancer (NLB) with TCP as the listener protocol for ports 80 and 443.
- [Layer 7](https://en.wikipedia.org/wiki/OSI_model#Layer_7:_Application_Layer): Use an Elastic Load Balancer (ELB) with HTTP as the listener protocol for port 80 and terminate TLS in the ELB
For L4:
```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/deploy.yaml
```
For L7:
Change the value of `service.beta.kubernetes.io/aws-load-balancer-ssl-cert` in the file `provider/aws/deploy-tls-termination.yaml` replacing the dummy id with a valid one. The dummy value is `"arn:aws:acm:us-west-2:XXXXXXXX:certificate/XXXXXX-XXXXXXX-XXXXXXX-XXXXXXXX"`
Check that no change is necessary with regards to the ELB idle timeout. In some scenarios, users may want to modify the ELB idle timeout, so please check the [ELB Idle Timeouts section](#elb-idle-timeouts) for additional information. If a change is required, users will need to update the value of `service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` in `provider/aws/deploy-tls-termination.yaml`
Then execute:
```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/deploy-tls-termination.yaml
```
This example creates an ELB with just two listeners, one in port 80 and another in port 443
![Listeners](https://github.com/kubernetes/ingress-nginx/blob/main/docs/images/elb-l7-listener.png)
##### ELB Idle Timeouts
In some scenarios users will need to modify the value of the ELB idle timeout.
Users need to ensure the idle timeout is less than the [keepalive_timeout](http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) that is configured for NGINX.
By default NGINX `keepalive_timeout` is set to `75s`.
The default ELB idle timeout will work for most scenarios, unless the NGINX [keepalive_timeout](http://nginx.org/en/docs/http/ngx_http_core_module.html#keepalive_timeout) has been modified,
in which case `service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout` will need to be modified to ensure it is less than the `keepalive_timeout` the user has configured.
*Please Note: An idle timeout of `3600s` is recommended when using WebSockets.*
More information with regards to idle timeouts for your Load Balancer can be found in the [official AWS documentation](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/config-idle-timeout.html).
##### Network Load Balancer (NLB)
This type of load balancer is supported since v1.10.0 as an ALPHA feature.
```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/aws/service-nlb.yaml
```
#### GCE-GKE
```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
```
**Important Note:** proxy protocol is not supported in GCE/GKE
#### Azure
```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud-generic.yaml
```
#### Bare-metal
Using [NodePort](https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport):
```console
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/baremetal/deploy.yaml
```
!!! tip
For extended notes regarding deployments on bare-metal, see [Bare-metal considerations](https://github.com/kubernetes/ingress-nginx/blob/main/docs/deploy/baremetal.md).
### Verify installation
To check if the ingress controller pods have started, run the following command:
```console
kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch
```
Once the operator pods are running, you can cancel the above command by typing `Ctrl+C`.
Now, you are ready to create your first ingress.
### Detect installed version
To detect which version of the ingress controller is running, exec into the pod and run `nginx-ingress-controller version` command.
```console
POD_NAMESPACE=ingress-nginx
POD_NAME=$(kubectl get pods -n $POD_NAMESPACE -l app.kubernetes.io/component=controller -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -n $POD_NAMESPACE -- /nginx-ingress-controller --version
```
## Using Helm
NGINX Ingress controller can be installed via [Helm](https://helm.sh/) using the chart [ingress-nginx/ingress-nginx](https://kubernetes.github.io/ingress-nginx).
Official documentation is [here](https://kubernetes.github.io/ingress-nginx/deploy/#using-helm)
To install the chart with the release name `my-nginx`:
```console
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install my-nginx ingress-nginx/ingress-nginx
```
Detect installed version:
```console
POD_NAME=$(kubectl get pods -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].metadata.name}')
kubectl exec -it $POD_NAME -- /nginx-ingress-controller --version
```

View File

@@ -21,12 +21,6 @@ metallb_enabled: true
metallb_speaker_enabled: true
```
By default, MetalLB resources are deployed into the `metallb-system` namespace. You can override this namespace using a variable.
```yaml
metallb_namespace: woodenlb-system
```
By default only the MetalLB BGP speaker is allowed to run on control plane nodes. If you have a single node cluster or a cluster where control plane are also worker nodes you may need to enable tolerations for the MetalLB controller:
```yaml
@@ -41,7 +35,7 @@ metallb_config:
effect: "NoSchedule"
```
If you'd like to set additional nodeSelector and tolerations values, you can do so in the following fashion:
If you'd like to set additional nodeSelector and tolerations values, you can do so in the following fasion:
```yaml
metallb_config:

View File

@@ -37,12 +37,4 @@ If you have containers that are using iptables in the host network namespace (`h
you need to ensure they are using iptables-nft.
An example how k8s do the autodetection can be found [in this PR](https://github.com/kubernetes/kubernetes/pull/82966)
The kernel version is lower than the kubernetes 1.32 system validation, please refer to the [kernel requirements](../operations/kernel-requirements.md).
## Rocky Linux 10
(Experimental in Kubespray CI)
The official Rocky Linux 10 cloud image does not include `kernel-module-extra`. Both Kube Proxy and CNI rely on this package, and since it relates to kernel version compatibility (which may require VM reboots, etc.), we haven't found an ideal solution.
However, some users report that it doesn't affect them (minimal version). Therefore, the Kubespray CI Rocky Linux 10 image is built by Kubespray maintainers using `diskimage-builder`. For detailed methods, please refer to [the comments](https://github.com/kubernetes-sigs/kubespray/pull/12355#issuecomment-3705400093).
The kernel version is lower than the kubenretes 1.32 system validation, please refer to the [kernel requirements](../operations/kernel-requirements.md).

View File

@@ -100,6 +100,8 @@ kubelet_make_iptables_util_chains: true
kubelet_feature_gates: ["RotateKubeletServerCertificate=true"]
kubelet_seccomp_default: true
kubelet_systemd_hardening: true
# To disable kubelet's staticPodPath (for nodes that don't use static pods like worker nodes)
kubelet_static_pod_path: ""
# In case you have multiple interfaces in your
# control plane nodes and you want to specify the right
# IP addresses, kubelet_secure_addresses allows you

View File

@@ -11,7 +11,7 @@ kubeadm_ignore_preflight_errors:
The Kernel Version Matrixs:
| OS Version | Kernel Version | Kernel >=4.19 |
| OS Verion | Kernel Verion | Kernel >=4.19 |
|--- | --- | --- |
| RHEL 9 | 5.14 | :white_check_mark: |
| RHEL 8 | 4.18 | :x: |

View File

@@ -31,8 +31,6 @@ That's it.
Append the new host to the inventory and run `cluster.yml`. You can NOT use `scale.yml` for that.
**Note:** When adding new control plane nodes, always append them to the end of the `kube_control_plane` group in your inventory. Adding control plane nodes in the first position is not supported and will cause the playbook to fail.
### 2) Restart kube-system/nginx-proxy
In all hosts, restart nginx-proxy pod. This pod is a local proxy for the apiserver. Kubespray will update its static config, but it needs to be restarted in order to reload.

View File

@@ -85,7 +85,7 @@ crictl_download_url: "{{ files_repo }}/kubernetes/cri-tools/crictl-v{{ crictl_ve
# If using Calico
calicoctl_download_url: "{{ files_repo }}/kubernetes/calico/v{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
# If using Calico with kdd
calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/raw/v{{ calico_version }}/manifests/crds.yaml"
calico_crds_download_url: "{{ files_repo }}/kubernetes/calico/v{{ calico_version }}.tar.gz"
# Containerd
containerd_download_url: "{{ files_repo }}/containerd-{{ containerd_version }}-linux-{{ image_arch }}.tar.gz"
runc_download_url: "{{ files_repo }}/runc.{{ image_arch }}"

View File

@@ -2,7 +2,7 @@
namespace: kubernetes_sigs
description: Deploy a production ready Kubernetes cluster
name: kubespray
version: 2.31.0
version: 2.29.2
readme: README.md
authors:
- The Kubespray maintainers (https://kubernetes.slack.com/channels/kubespray)

View File

@@ -38,7 +38,6 @@
loadSidebar: 'docs/_sidebar.md',
repo: 'https://github.com/kubernetes-sigs/kubespray',
auto2top: true,
noCompileLinks: ['.*\.ini'],
logo: '/logo/logo-clear.png'
}
</script>

View File

@@ -11,15 +11,15 @@
# containerd_runc_runtime:
# name: runc
# type: "io.containerd.runc.v2"
# options:
# Root: ""
# engine: ""
# root: ""
# containerd_additional_runtimes:
# Example for Kata Containers as additional runtime:
# - name: kata
# type: "io.containerd.kata.v2"
# options:
# Root: ""
# engine: ""
# root: ""
# containerd_grpc_max_recv_message_size: 16777216
# containerd_grpc_max_send_message_size: 16777216

View File

@@ -44,7 +44,7 @@
# [Optional] Calico: If using Calico network plugin
# calicoctl_download_url: "{{ files_repo }}/github.com/projectcalico/calico/releases/download/v{{ calico_ctl_version }}/calicoctl-linux-{{ image_arch }}"
# [Optional] Calico with kdd: If using Calico network plugin with kdd datastore
# calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/raw/v{{ calico_version }}/manifests/crds.yaml"
# calico_crds_download_url: "{{ files_repo }}/github.com/projectcalico/calico/archive/v{{ calico_version }}.tar.gz"
# [Optional] Cilium: If using Cilium network plugin
# ciliumcli_download_url: "{{ files_repo }}/github.com/cilium/cilium-cli/releases/download/v{{ cilium_cli_version }}/cilium-linux-{{ image_arch }}.tar.gz"

View File

@@ -1,4 +1,8 @@
---
# Kubernetes dashboard
# RBAC required. see docs/getting-started.md for access details.
# dashboard_enabled: false
# Helm deployment
helm_enabled: false
@@ -63,6 +67,39 @@ local_volume_provisioner_enabled: false
# Gateway API CRDs
gateway_api_enabled: false
# Nginx ingress controller deployment
ingress_nginx_enabled: false
# ingress_nginx_host_network: false
# ingress_nginx_service_type: LoadBalancer
# ingress_nginx_service_annotations:
# example.io/loadbalancerIPs: 1.2.3.4
# ingress_nginx_service_nodeport_http: 30080
# ingress_nginx_service_nodeport_https: 30081
ingress_publish_status_address: ""
# ingress_nginx_nodeselector:
# kubernetes.io/os: "linux"
# ingress_nginx_tolerations:
# - key: "node-role.kubernetes.io/control-plane"
# operator: "Equal"
# value: ""
# effect: "NoSchedule"
# ingress_nginx_namespace: "ingress-nginx"
# ingress_nginx_insecure_port: 80
# ingress_nginx_secure_port: 443
# ingress_nginx_configmap:
# map-hash-bucket-size: "128"
# ssl-protocols: "TLSv1.2 TLSv1.3"
# ingress_nginx_configmap_tcp_services:
# 9000: "default/example-go:8080"
# ingress_nginx_configmap_udp_services:
# 53: "kube-system/coredns:53"
# ingress_nginx_extra_args:
# - --default-ssl-certificate=default/foo-tls
# ingress_nginx_termination_grace_period_seconds: 300
# ingress_nginx_class: nginx
# ingress_nginx_without_class: true
# ingress_nginx_default: false
# ALB ingress controller deployment
ingress_alb_enabled: false
# alb_ingress_aws_region: "us-east-1"

View File

@@ -22,8 +22,7 @@ local_release_dir: "/tmp/releases"
# Random shifts for retrying failed ops like pushing/downloading
retry_stagger: 5
# This is the user that owns the cluster installation.
# Note: cilium needs to set kube_owner to root https://kubespray.io/#/docs/CNI/cilium?id=unprivileged-agent-configuration
# This is the user that owns tha cluster installation.
kube_owner: kube
# This is the group that the cert creation scripts chgrp the

View File

@@ -56,8 +56,8 @@ cilium_l2announcements: false
#
# Only effective when monitor aggregation is set to "medium" or higher.
# cilium_monitor_aggregation_flags: "all"
# Kube Proxy Replacement mode (true/false)
# cilium_kube_proxy_replacement: false
# Kube Proxy Replacement mode (strict/partial)
# cilium_kube_proxy_replacement: partial
# If upgrading from Cilium < 1.5, you may want to override some of these options
# to prevent service disruptions. See also:

View File

@@ -1,2 +1,2 @@
---
requires_ansible: ">=2.18.0,<2.19.0"
requires_ansible: ">=2.17.3"

View File

@@ -1,5 +1,5 @@
# Use immutable image tags rather than mutable tags (like ubuntu:24.04)
FROM ubuntu:noble-20260113@sha256:cd1dba651b3080c3686ecf4e3c4220f026b521fb76978881737d24f200828b2b
# Use immutable image tags rather than mutable tags (like ubuntu:22.04)
FROM ubuntu:jammy-20230308
# Some tools like yamllint need this
# Pip needs this as well at the moment to install ansible
# (and potentially other packages)
@@ -27,14 +27,14 @@ RUN apt update -q \
ca-certificates \
curl \
gnupg2 \
software-properties-common \
unzip \
libvirt-clients \
qemu-utils \
qemu-kvm \
dnsmasq \
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc \
&& echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}") stable" | tee /etc/apt/sources.list.d/docker.list \
&& curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add - \
&& add-apt-repository "deb [arch=$(dpkg --print-architecture)] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \
&& apt update -q \
&& apt install --no-install-recommends -yq docker-ce \
&& apt autoremove -yqq --purge && apt clean && rm -rf /var/lib/apt/lists/* /var/log/*
@@ -44,10 +44,11 @@ ADD ./requirements.txt /kubespray/requirements.txt
ADD ./tests/requirements.txt /kubespray/tests/requirements.txt
RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1 \
&& pip install --break-system-packages --ignore-installed --no-compile --no-cache-dir pip -U \
&& pip install --break-system-packages --no-compile --no-cache-dir -r tests/requirements.txt \
&& curl -L https://dl.k8s.io/release/v1.35.0/bin/linux/$(dpkg --print-architecture)/kubectl -o /usr/local/bin/kubectl \
&& echo $(curl -L https://dl.k8s.io/release/v1.35.0/bin/linux/$(dpkg --print-architecture)/kubectl.sha256) /usr/local/bin/kubectl | sha256sum --check \
&& pip install --no-compile --no-cache-dir pip -U \
&& pip install --no-compile --no-cache-dir -r tests/requirements.txt \
&& pip install --no-compile --no-cache-dir -r requirements.txt \
&& curl -L https://dl.k8s.io/release/v1.33.7/bin/linux/$(dpkg --print-architecture)/kubectl -o /usr/local/bin/kubectl \
&& echo $(curl -L https://dl.k8s.io/release/v1.33.7/bin/linux/$(dpkg --print-architecture)/kubectl.sha256) /usr/local/bin/kubectl | sha256sum --check \
&& chmod a+x /usr/local/bin/kubectl \
# Install Vagrant
&& curl -LO https://releases.hashicorp.com/vagrant/${VAGRANT_VERSION}/vagrant_${VAGRANT_VERSION}-1_$(dpkg --print-architecture).deb \
@@ -55,5 +56,5 @@ RUN update-alternatives --install /usr/bin/python python /usr/bin/python3 1 \
&& rm vagrant_${VAGRANT_VERSION}-1_$(dpkg --print-architecture).deb \
&& vagrant plugin install vagrant-libvirt \
# Install Kubernetes collections
&& pip install --break-system-packages --no-compile --no-cache-dir kubernetes \
&& pip install --no-compile --no-cache-dir kubernetes \
&& ansible-galaxy collection install kubernetes.core

View File

@@ -5,8 +5,8 @@
become: false
run_once: true
vars:
minimal_ansible_version: 2.18.0
maximal_ansible_version: 2.19.0
minimal_ansible_version: 2.17.3
maximal_ansible_version: 2.18.0
tags: always
tasks:
- name: "Check {{ minimal_ansible_version }} <= Ansible version < {{ maximal_ansible_version }}"

View File

@@ -55,7 +55,7 @@
- { role: kubernetes-apps/kubelet-csr-approver, tags: kubelet-csr-approver }
- { role: container-engine, tags: "container-engine", when: deploy_container_engine }
- { role: kubernetes/node, tags: node }
- { role: kubernetes/control-plane, tags: control-plane, upgrade_cluster_setup: true }
- { role: kubernetes/control-plane, tags: master, upgrade_cluster_setup: true }
- { role: kubernetes/client, tags: client }
- { role: kubernetes/node-label, tags: node-label }
- { role: kubernetes/node-taint, tags: node-taint }
@@ -100,7 +100,7 @@
environment: "{{ proxy_disable_env }}"
roles:
- { role: kubespray_defaults }
- { role: win_nodes/kubernetes_patch, tags: ["control-plane", "win_nodes"] }
- { role: win_nodes/kubernetes_patch, tags: ["master", "win_nodes"] }
- name: Install Calico Route Reflector
hosts: calico_rr

View File

@@ -1,7 +1,7 @@
ansible==11.13.0
ansible==10.7.0
# Needed for community.crypto module
cryptography==46.0.4
cryptography==46.0.2
# Needed for jinja2 json_query templating
jmespath==1.1.0
jmespath==1.0.1
# Needed for ansible.utils.ipaddr
netaddr==1.3.0

View File

@@ -9,8 +9,6 @@ platforms:
vm_memory: 512
provisioner:
name: ansible
env:
ANSIBLE_ROLES_PATH: ../../../
config_options:
defaults:
callbacks_enabled: profile_tasks

View File

@@ -9,8 +9,6 @@ platforms:
vm_memory: 512
provisioner:
name: ansible
env:
ANSIBLE_ROLES_PATH: ../../../
config_options:
defaults:
callbacks_enabled: profile_tasks

View File

@@ -21,8 +21,6 @@ platforms:
vm_memory: 512
provisioner:
name: ansible
env:
ANSIBLE_ROLES_PATH: ../../../
config_options:
defaults:
callbacks_enabled: profile_tasks

View File

@@ -13,9 +13,10 @@ containerd_snapshotter: "overlayfs"
containerd_runc_runtime:
name: runc
type: "io.containerd.runc.v2"
engine: ""
root: ""
base_runtime_spec: cri-base.json
options:
Root: ""
SystemdCgroup: "{{ containerd_use_systemd_cgroup | ternary('true', 'false') }}"
BinaryName: "{{ bin_dir }}/runc"
@@ -23,8 +24,8 @@ containerd_additional_runtimes: []
# Example for Kata Containers as additional runtime:
# - name: kata
# type: "io.containerd.kata.v2"
# options:
# Root: ""
# engine: ""
# root: ""
containerd_base_runtime_spec_rlimit_nofile: 65535
@@ -35,8 +36,8 @@ containerd_default_base_runtime_spec_patch:
hard: "{{ containerd_base_runtime_spec_rlimit_nofile }}"
soft: "{{ containerd_base_runtime_spec_rlimit_nofile }}"
# Only for containerd < 2.1; discard unpacked layers to save disk space
# https://github.com/containerd/containerd/blob/release/2.1/docs/cri/config.md#image-pull-configuration-since-containerd-v21
# Can help reduce disk usage
# https://github.com/containerd/containerd/discussions/6295
containerd_discard_unpacked_layers: true
containerd_base_runtime_specs:

View File

@@ -52,6 +52,8 @@ oom_score = {{ containerd_oom_score }}
{% for runtime in [containerd_runc_runtime] + containerd_additional_runtimes %}
[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.{{ runtime.name }}]
runtime_type = "{{ runtime.type }}"
runtime_engine = "{{ runtime.engine }}"
runtime_root = "{{ runtime.root }}"
{% if runtime.base_runtime_spec is defined %}
base_runtime_spec = "{{ containerd_cfg_dir }}/{{ runtime.base_runtime_spec }}"
{% endif %}
@@ -76,9 +78,7 @@ oom_score = {{ containerd_oom_score }}
[plugins."io.containerd.cri.v1.images"]
snapshotter = "{{ containerd_snapshotter }}"
{% if containerd_discard_unpacked_layers and containerd_version is version('2.1.0', '<') %}
discard_unpacked_layers = {{ containerd_discard_unpacked_layers | lower }}
{% endif %}
image_pull_progress_timeout = "{{ containerd_image_pull_progress_timeout }}"
[plugins."io.containerd.cri.v1.images".pinned_images]
sandbox = "{{ pod_infra_image_repo }}:{{ pod_infra_image_tag }}"

View File

@@ -6,6 +6,12 @@
masked: false
listen: Restart and enable cri-dockerd
- name: Cri-dockerd | restart docker.service
service:
name: docker.service
state: restarted
listen: Restart and enable cri-dockerd
- name: Cri-dockerd | reload cri-dockerd.socket
service:
name: cri-dockerd.socket

View File

@@ -25,8 +25,6 @@ provisioner:
group_vars:
all:
become: true
k8s_cluster:
container_manager: docker
playbooks:
create: ../../../../../tests/cloud_playbooks/create-kubevirt.yml
prepare: ../../../molecule/prepare.yml

View File

@@ -32,8 +32,6 @@ crio_registry_auth: []
crio_seccomp_profile: ""
crio_selinux: "{{ (preinstall_selinux_state == 'enforcing') | lower }}"
crio_signature_policy: "{% if ansible_os_family == 'ClearLinux' %}/usr/share/defaults/crio/policy.json{% endif %}"
# Set the pull progress timeout
crio_pull_progress_timeout: "10s"
# Override system default for storage driver
# crio_storage_driver: "overlay"

View File

@@ -2,6 +2,8 @@
- name: Converge
hosts: all
become: true
vars:
container_manager: crio
roles:
- role: kubespray_defaults
- role: container-engine/cri-o

View File

@@ -41,10 +41,6 @@ provisioner:
defaults:
callbacks_enabled: profile_tasks
timeout: 120
inventory:
group_vars:
k8s_cluster:
container_manager: crio
playbooks:
create: ../../../../../tests/cloud_playbooks/create-kubevirt.yml
prepare: ../../../molecule/prepare.yml

View File

@@ -2,6 +2,7 @@
- name: Test CRI-O cri
import_playbook: ../../../molecule/test_cri.yml
vars:
container_manager: crio
cri_socket: unix:///var/run/crio/crio.sock
cri_name: cri-o
- name: Test running a container with crun

View File

@@ -1,16 +1,16 @@
{% if crio_registry_auth is defined and crio_registry_auth|length %}
{
"auths": {
{% for reg in crio_registry_auth %}
"auths": {
"{{ reg.registry }}": {
"auth": "{{ (reg.username + ':' + reg.password) | string | b64encode }}"
{% if not loop.last %}
},
{% else %}
}
{% if not loop.last %}
},
{% else %}
}
{% endif %}
{% endfor %}
}
}
{% else %}
{}

View File

@@ -348,12 +348,6 @@ signature_policy = "{{ crio_signature_policy }}"
# ignore; the latter will ignore volumes entirely.
image_volumes = "mkdir"
# The timeout for an image pull to make progress until the pull operation gets
# canceled. This value will be also used for calculating the pull progress interval
# to pull_progress_timeout / 10. Can be set to 0 to disable the timeout as well as
# the progress output.
pull_progress_timeout = "{{ crio_pull_progress_timeout }}"
# The crio.network table containers settings pertaining to the management of
# CNI plugins.
[crio.network]

View File

@@ -0,0 +1,22 @@
---
- name: Crictl | Download crictl
include_tasks: "../../../download/tasks/download_file.yml"
vars:
download: "{{ download_defaults | combine(downloads.crictl) }}"
- name: Install crictl config
template:
src: crictl.yaml.j2
dest: /etc/crictl.yaml
owner: root
mode: "0644"
- name: Copy crictl binary from download dir
copy:
src: "{{ local_release_dir }}/crictl"
dest: "{{ bin_dir }}/crictl"
mode: "0755"
remote_src: true
notify:
- Get crictl completion
- Install crictl completion

View File

@@ -1,22 +1,3 @@
---
- name: Crictl | Download crictl
include_tasks: "../../../download/tasks/download_file.yml"
vars:
download: "{{ download_defaults | combine(downloads.crictl) }}"
- name: Install crictl config
template:
src: crictl.yaml.j2
dest: /etc/crictl.yaml
owner: root
mode: "0644"
- name: Copy crictl binary from download dir
copy:
src: "{{ local_release_dir }}/crictl"
dest: "{{ bin_dir }}/crictl"
mode: "0755"
remote_src: true
notify:
- Get crictl completion
- Install crictl completion
- name: Install crictl
include_tasks: crictl.yml

View File

@@ -55,7 +55,7 @@
register: keyserver_task_result
until: keyserver_task_result is succeeded
retries: 4
delay: "{{ retry_stagger }}"
delay: "{{ retry_stagger | d(3) }}"
with_items: "{{ docker_repo_key_info.repo_keys }}"
environment: "{{ proxy_env }}"
when: ansible_pkg_mgr == 'apt'
@@ -128,7 +128,7 @@
register: docker_task_result
until: docker_task_result is succeeded
retries: 4
delay: "{{ retry_stagger }}"
delay: "{{ retry_stagger | d(3) }}"
notify: Restart docker
when:
- not ansible_os_family in ["Flatcar", "Flatcar Container Linux by Kinvolk"]

View File

@@ -30,7 +30,7 @@ LimitCORE=infinity
TimeoutStartSec=1min
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=10
StartLimitBurst=3
StartLimitInterval=60s
# Set the cgroup slice of the service so that kube reserved takes effect
{% if kube_reserved is defined and kube_reserved|bool %}

View File

@@ -21,11 +21,6 @@ provisioner:
defaults:
callbacks_enabled: profile_tasks
timeout: 120
inventory:
group_vars:
k8s_cluster:
gvisor_enabled: true
container_manager: containerd
playbooks:
create: ../../../../../tests/cloud_playbooks/create-kubevirt.yml
prepare: ../../../molecule/prepare.yml

View File

@@ -12,20 +12,11 @@
is_ostree: "{{ ostree.stat.exists }}"
- name: Runc | Uninstall runc package managed by package manager
package:
name: "{{ runc_package_name }}"
state: absent
when:
- not is_ostree
- ansible_distribution != "Flatcar Container Linux by Kinvolk"
- ansible_distribution != "Flatcar"
block:
- name: Runc | Remove package
package:
name: "{{ runc_package_name }}"
state: absent
- name: Runc | Remove orphaned binary
file:
path: /usr/bin/runc
state: absent
when: runc_bin_dir != "/usr/bin"
- not (is_ostree or (ansible_distribution == "Flatcar Container Linux by Kinvolk") or (ansible_distribution == "Flatcar"))
- name: Runc | Download runc binary
include_tasks: "../../../download/tasks/download_file.yml"
@@ -38,3 +29,10 @@
dest: "{{ runc_bin_dir }}/runc"
mode: "0755"
remote_src: true
- name: Runc | Remove orphaned binary
file:
path: /usr/bin/runc
state: absent
when: runc_bin_dir != "/usr/bin"
ignore_errors: true # noqa ignore-errors

View File

@@ -21,11 +21,6 @@ provisioner:
defaults:
callbacks_enabled: profile_tasks
timeout: 120
inventory:
group_vars:
k8s_cluster:
youki_enabled: true
container_manager: crio
playbooks:
create: ../../../../../tests/cloud_playbooks/create-kubevirt.yml
prepare: ../../../molecule/prepare.yml

View File

@@ -5,7 +5,8 @@
group: "{{ etcd_cert_group }}"
state: directory
owner: "{{ etcd_owner }}"
mode: "0700"
mode: "{{ etcd_cert_dir_mode }}"
recurse: true
- name: "Gen_certs | create etcd script dir (on {{ groups['etcd'][0] }})"
file:
@@ -144,6 +145,15 @@
- ('k8s_cluster' in group_names) and
sync_certs | default(false) and inventory_hostname not in groups['etcd']
- name: Gen_certs | check certificate permissions
file:
path: "{{ etcd_cert_dir }}"
group: "{{ etcd_cert_group }}"
state: directory
owner: "{{ etcd_owner }}"
mode: "{{ etcd_cert_dir_mode }}"
recurse: true
# This is a hack around the fact kubeadm expect the same certs path on all kube_control_plane
# TODO: fix certs generation to have the same file everywhere
# OR work with kubeadm on node-specific config

View File

@@ -32,16 +32,23 @@ DNS.{{ counter["dns"] }} = {{ hostvars[host]['etcd_access_address'] }}{{ increme
{# This will always expand to inventory_hostname, which can be a completely arbitrary name, that etcd will not know or care about, hence this line is (probably) redundant. #}
DNS.{{ counter["dns"] }} = {{ host }}{{ increment(counter, 'dns') }}
{% endfor %}
{% if apiserver_loadbalancer_domain_name is defined %}
DNS.{{ counter["dns"] }} = {{ apiserver_loadbalancer_domain_name }}{{ increment(counter, 'dns') }}
{% endif %}
{% for etcd_alt_name in etcd_cert_alt_names %}
DNS.{{ counter["dns"] }} = {{ etcd_alt_name }}{{ increment(counter, 'dns') }}
{% endfor %}
{% for host in groups['etcd'] %}
{% for address in hostvars[host]['main_access_ips'] %}
IP.{{ counter["ip"] }} = {{ address }}{{ increment(counter, 'ip') }}
{% endfor %}
{% for address in hostvars[host]['main_ips'] %}
IP.{{ counter["ip"] }} = {{ address }}{{ increment(counter, 'ip') }}
{% endfor %}
{% if hostvars[host]['access_ip'] is defined %}
IP.{{ counter["ip"] }} = {{ hostvars[host]['access_ip'] }}{{ increment(counter, 'ip') }}
{% endif %}
{% if hostvars[host]['access_ip6'] is defined %}
IP.{{ counter["ip"] }} = {{ hostvars[host]['access_ip6'] }}{{ increment(counter, 'ip') }}
{% endif %}
{% if ipv6_stack %}
IP.{{ counter["ip"] }} = {{ hostvars[host]['ip6'] | default(hostvars[host]['fallback_ip6']) }}{{ increment(counter, 'ip') }}
{% endif %}
IP.{{ counter["ip"] }} = {{ hostvars[host]['main_ip'] }}{{ increment(counter, 'ip') }}
{% endfor %}
{% for cert_alt_ip in etcd_cert_alt_ips %}
IP.{{ counter["ip"] }} = {{ cert_alt_ip }}{{ increment(counter, 'ip') }}

View File

@@ -18,6 +18,7 @@ etcd_backup_retention_count: -1
force_etcd_cert_refresh: true
etcd_config_dir: /etc/ssl/etcd
etcd_cert_dir: "{{ etcd_config_dir }}/ssl"
etcd_cert_dir_mode: "0700"
etcd_cert_group: root
# Note: This does not set up DNS entries. It simply adds the following DNS
# entries to the certificate

View File

@@ -11,7 +11,6 @@ dns_nodes_per_replica: 16
dns_cores_per_replica: 256
dns_prevent_single_point_failure: "{{ 'true' if dns_min_replicas | int > 1 else 'false' }}"
enable_coredns_reverse_dns_lookups: true
coredns_svc_name: "coredns"
coredns_ordinal_suffix: ""
# dns_extra_tolerations: [{effect: NoSchedule, operator: "Exists"}]
coredns_affinity:
@@ -119,5 +118,29 @@ netchecker_agent_log_level: 5
netchecker_server_log_level: 5
netchecker_etcd_log_level: info
# Dashboard
dashboard_replicas: 1
# Namespace for dashboard
dashboard_namespace: kube-system
# Limits for dashboard
dashboard_cpu_limit: 100m
dashboard_memory_limit: 256M
dashboard_cpu_requests: 50m
dashboard_memory_requests: 64M
# Set dashboard_use_custom_certs to true if overriding dashboard_certs_secret_name with a secret that
# contains dashboard_tls_key_file and dashboard_tls_cert_file instead of using the initContainer provisioned certs
dashboard_use_custom_certs: false
dashboard_certs_secret_name: kubernetes-dashboard-certs
dashboard_tls_key_file: dashboard.key
dashboard_tls_cert_file: dashboard.crt
dashboard_master_toleration: true
# Override dashboard default settings
dashboard_token_ttl: 900
dashboard_skip_login: false
# Policy Controllers
# policy_controller_extra_tolerations: [{effect: NoSchedule, operator: "Exists"}]

View File

@@ -109,3 +109,15 @@
- netchecker-server-clusterrolebinding.yml.j2
- netchecker-server-deployment.yml.j2
- netchecker-server-svc.yml.j2
- name: Kubernetes Apps | Dashboard
command:
cmd: "{{ kubectl_apply_stdin }}"
stdin: "{{ lookup('template', 'dashboard.yml.j2') }}"
delegate_to: "{{ groups['kube_control_plane'][0] }}"
run_once: true
vars:
k8s_namespace: "{{ dashboard_namespace }}"
when: dashboard_enabled
tags:
- dashboard

View File

@@ -2,7 +2,7 @@
apiVersion: v1
kind: Service
metadata:
name: {{ coredns_svc_name }}{{ coredns_ordinal_suffix }}
name: coredns{{ coredns_ordinal_suffix }}
namespace: kube-system
labels:
k8s-app: kube-dns{{ coredns_ordinal_suffix }}

View File

@@ -0,0 +1,323 @@
# Copyright 2017 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Configuration to deploy release version of the Dashboard UI compatible with
# Kubernetes 1.8.
#
# Example usage: kubectl create -f <this_file>
{% if k8s_namespace != 'kube-system' %}
---
apiVersion: v1
kind: Namespace
metadata:
name: {{ k8s_namespace }}
labels:
name: {{ k8s_namespace }}
{% endif %}
---
# ------------------- Dashboard Secrets ------------------- #
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-certs
type: Opaque
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-csrf
type: Opaque
data:
csrf: ""
---
apiVersion: v1
kind: Secret
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-key-holder
type: Opaque
---
# ------------------- Dashboard ConfigMap ------------------- #
kind: ConfigMap
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard-settings
---
# ------------------- Dashboard Service Account ------------------- #
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
---
# ------------------- Dashboard Role & Role Binding ------------------- #
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Dashboard to get, update and delete Dashboard exclusive secrets.
- apiGroups: [""]
resources: ["secrets"]
resourceNames: ["kubernetes-dashboard-key-holder", "kubernetes-dashboard-certs", "kubernetes-dashboard-csrf"]
verbs: ["get", "update", "delete"]
# Allow Dashboard to get and update 'kubernetes-dashboard-settings' config map.
- apiGroups: [""]
resources: ["configmaps"]
resourceNames: ["kubernetes-dashboard-settings"]
verbs: ["get", "update"]
# Allow Dashboard to get metrics.
- apiGroups: [""]
resources: ["services"]
resourceNames: ["heapster", "dashboard-metrics-scraper"]
verbs: ["proxy"]
- apiGroups: [""]
resources: ["services/proxy"]
resourceNames: ["heapster", "http:heapster:", "https:heapster:", "dashboard-metrics-scraper", "http:dashboard-metrics-scraper"]
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: {{ k8s_namespace }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: kubernetes-dashboard
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: kubernetes-dashboard
subjects:
- kind: ServiceAccount
name: kubernetes-dashboard
namespace: {{ k8s_namespace }}
---
# ------------------- Dashboard Deployment ------------------- #
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
spec:
replicas: {{ dashboard_replicas }}
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-dashboard
template:
metadata:
labels:
k8s-app: kubernetes-dashboard
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
priorityClassName: system-cluster-critical
containers:
- name: kubernetes-dashboard
image: {{ dashboard_image_repo }}:{{ dashboard_image_tag }}
imagePullPolicy: {{ k8s_image_pull_policy }}
resources:
limits:
cpu: {{ dashboard_cpu_limit }}
memory: {{ dashboard_memory_limit }}
requests:
cpu: {{ dashboard_cpu_requests }}
memory: {{ dashboard_memory_requests }}
ports:
- containerPort: 8443
protocol: TCP
args:
- --namespace={{ k8s_namespace }}
{% if dashboard_use_custom_certs %}
- --tls-key-file={{ dashboard_tls_key_file }}
- --tls-cert-file={{ dashboard_tls_cert_file }}
{% else %}
- --auto-generate-certificates
{% endif %}
{% if dashboard_skip_login %}
- --enable-skip-login
{% endif %}
- --authentication-mode=token
# Uncomment the following line to manually specify Kubernetes API server Host
# If not specified, Dashboard will attempt to auto discover the API server and connect
# to it. Uncomment only if the default does not work.
# - --apiserver-host=http://my-address:port
- --token-ttl={{ dashboard_token_ttl }}
volumeMounts:
- name: kubernetes-dashboard-certs
mountPath: /certs
# Create on-disk volume to store exec logs
- mountPath: /tmp
name: tmp-volume
livenessProbe:
httpGet:
scheme: HTTPS
path: /
port: 8443
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumes:
- name: kubernetes-dashboard-certs
secret:
secretName: {{ dashboard_certs_secret_name }}
- name: tmp-volume
emptyDir: {}
serviceAccountName: kubernetes-dashboard
{% if dashboard_master_toleration %}
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
{% endif %}
---
# ------------------- Dashboard Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
spec:
ports:
- port: 443
targetPort: 8443
selector:
k8s-app: kubernetes-dashboard
---
# ------------------- Metrics Scraper Service Account ------------------- #
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
labels:
k8s-app: kubernetes-dashboard
name: kubernetes-dashboard
rules:
# Allow Metrics Scraper to get metrics from the Metrics server
- apiGroups: ["metrics.k8s.io"]
resources: ["pods", "nodes"]
verbs: ["get", "list", "watch"]
---
# ------------------- Metrics Scraper Service ------------------- #
kind: Service
apiVersion: v1
metadata:
labels:
k8s-app: kubernetes-metrics-scraper
name: dashboard-metrics-scraper
spec:
ports:
- port: 8000
targetPort: 8000
selector:
k8s-app: kubernetes-metrics-scraper
---
# ------------------- Metrics Scraper Deployment ------------------- #
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
k8s-app: kubernetes-metrics-scraper
name: kubernetes-metrics-scraper
spec:
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
k8s-app: kubernetes-metrics-scraper
template:
metadata:
labels:
k8s-app: kubernetes-metrics-scraper
spec:
securityContext:
seccompProfile:
type: RuntimeDefault
priorityClassName: system-cluster-critical
containers:
- name: kubernetes-metrics-scraper
image: {{ dashboard_metrics_scraper_repo }}:{{ dashboard_metrics_scraper_tag }}
ports:
- containerPort: 8000
protocol: TCP
livenessProbe:
httpGet:
scheme: HTTP
path: /
port: 8000
initialDelaySeconds: 30
timeoutSeconds: 30
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
runAsUser: 1001
runAsGroup: 2001
volumeMounts:
- mountPath: /tmp
name: tmp-volume
serviceAccountName: kubernetes-dashboard
volumes:
- name: tmp-volume
emptyDir: {}
{% if dashboard_master_toleration %}
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
{% endif %}

View File

@@ -1,5 +1,6 @@
---
gateway_api_enabled: false
gateway_api_version: 1.2.1
# `gateway_api_channel` default is "standard".
# "standard" release channel includes all resources that have graduated to GA or beta, including GatewayClass, Gateway, HTTPRoute, and ReferenceGrant.

View File

@@ -21,7 +21,7 @@ external_openstack_cacert: "{{ lookup('env', 'OS_CACERT') }}"
## arg1: "value1"
## arg2: "value2"
external_openstack_cloud_controller_extra_args: {}
external_openstack_cloud_controller_image_tag: "v1.35.0"
external_openstack_cloud_controller_image_tag: "v1.32.0"
external_openstack_cloud_controller_bind_address: 127.0.0.1
external_openstack_cloud_controller_dns_policy: ClusterFirst

View File

@@ -8,4 +8,3 @@ local_path_provisioner_is_default_storageclass: "true"
local_path_provisioner_debug: false
local_path_provisioner_helper_image_repo: "busybox"
local_path_provisioner_helper_image_tag: "latest"
local_path_provisioner_resources: {}

View File

@@ -35,10 +35,6 @@ spec:
valueFrom:
fieldRef:
fieldPath: metadata.namespace
{% if local_path_provisioner_resources %}
resources:
{{ local_path_provisioner_resources | to_nice_yaml | indent(10) | trim }}
{% endif %}
volumes:
- name: config-volume
configMap:

View File

@@ -0,0 +1,28 @@
---
ingress_nginx_namespace: "ingress-nginx"
ingress_nginx_host_network: false
ingress_nginx_service_type: LoadBalancer
ingress_nginx_service_nodeport_http: ""
ingress_nginx_service_nodeport_https: ""
ingress_nginx_service_annotations: {}
ingress_publish_status_address: ""
ingress_nginx_publish_service: "{{ ingress_nginx_namespace }}/ingress-nginx"
ingress_nginx_nodeselector:
kubernetes.io/os: "linux"
ingress_nginx_tolerations: []
ingress_nginx_insecure_port: 80
ingress_nginx_secure_port: 443
ingress_nginx_metrics_port: 10254
ingress_nginx_configmap: {}
ingress_nginx_configmap_tcp_services: {}
ingress_nginx_configmap_udp_services: {}
ingress_nginx_extra_args: []
ingress_nginx_termination_grace_period_seconds: 300
ingress_nginx_class: nginx
ingress_nginx_without_class: true
ingress_nginx_default: false
ingress_nginx_webhook_enabled: false
ingress_nginx_webhook_job_ttl: 1800
ingress_nginx_opentelemetry_enabled: false
ingress_nginx_probe_initial_delay_seconds: 10

View File

@@ -0,0 +1,69 @@
---
- name: NGINX Ingress Controller | Create addon dir
file:
path: "{{ kube_config_dir }}/addons/ingress_nginx"
state: directory
owner: root
group: root
mode: "0755"
when:
- inventory_hostname == groups['kube_control_plane'][0]
- name: NGINX Ingress Controller | Templates list
set_fact:
ingress_nginx_templates:
- { name: 00-namespace, file: 00-namespace.yml, type: ns }
- { name: cm-ingress-nginx, file: cm-ingress-nginx.yml, type: cm }
- { name: cm-tcp-services, file: cm-tcp-services.yml, type: cm }
- { name: cm-udp-services, file: cm-udp-services.yml, type: cm }
- { name: sa-ingress-nginx, file: sa-ingress-nginx.yml, type: sa }
- { name: clusterrole-ingress-nginx, file: clusterrole-ingress-nginx.yml, type: clusterrole }
- { name: clusterrolebinding-ingress-nginx, file: clusterrolebinding-ingress-nginx.yml, type: clusterrolebinding }
- { name: role-ingress-nginx, file: role-ingress-nginx.yml, type: role }
- { name: rolebinding-ingress-nginx, file: rolebinding-ingress-nginx.yml, type: rolebinding }
- { name: ingressclass-nginx, file: ingressclass-nginx.yml, type: ingressclass }
- { name: ds-ingress-nginx-controller, file: ds-ingress-nginx-controller.yml, type: ds }
ingress_nginx_template_for_service:
- { name: svc-ingress-nginx, file: svc-ingress-nginx.yml, type: svc }
ingress_nginx_templates_for_webhook:
- { name: admission-webhook-configuration, file: admission-webhook-configuration.yml, type: sa }
- { name: sa-admission-webhook, file: sa-admission-webhook.yml, type: sa }
- { name: clusterrole-admission-webhook, file: clusterrole-admission-webhook.yml, type: clusterrole }
- { name: clusterrolebinding-admission-webhook, file: clusterrolebinding-admission-webhook.yml, type: clusterrolebinding }
- { name: role-admission-webhook, file: role-admission-webhook.yml, type: role }
- { name: rolebinding-admission-webhook, file: rolebinding-admission-webhook.yml, type: rolebinding }
- { name: admission-webhook-job, file: admission-webhook-job.yml, type: job }
- { name: svc-ingress-nginx-controller-admission, file: svc-ingress-nginx-controller-admission.yml, type: svc }
- name: NGINX Ingress Controller | Append extra templates to NGINX Ingress Template list for service
set_fact:
ingress_nginx_templates: "{{ ingress_nginx_templates + ingress_nginx_template_for_service }}"
when: not ingress_nginx_host_network
- name: NGINX Ingress Controller | Append extra templates to NGINX Ingress Templates list for webhook
set_fact:
ingress_nginx_templates: "{{ ingress_nginx_templates + ingress_nginx_templates_for_webhook }}"
when: ingress_nginx_webhook_enabled
- name: NGINX Ingress Controller | Create manifests
template:
src: "{{ item.file }}.j2"
dest: "{{ kube_config_dir }}/addons/ingress_nginx/{{ item.file }}"
mode: "0644"
with_items: "{{ ingress_nginx_templates }}"
register: ingress_nginx_manifests
when:
- inventory_hostname == groups['kube_control_plane'][0]
- name: NGINX Ingress Controller | Apply manifests
kube:
name: "{{ item.item.name }}"
namespace: "{{ ingress_nginx_namespace }}"
kubectl: "{{ bin_dir }}/kubectl"
resource: "{{ item.item.type }}"
filename: "{{ kube_config_dir }}/addons/ingress_nginx/{{ item.item.file }}"
state: "latest"
with_items: "{{ ingress_nginx_manifests.results }}"
when:
- inventory_hostname == groups['kube_control_plane'][0]

View File

@@ -0,0 +1,7 @@
---
apiVersion: v1
kind: Namespace
metadata:
name: {{ ingress_nginx_namespace }}
labels:
name: {{ ingress_nginx_namespace }}

View File

@@ -0,0 +1,30 @@
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
name: ingress-nginx-admission
webhooks:
- admissionReviewVersions:
- v1
clientConfig:
service:
name: ingress-nginx-controller-admission
namespace: {{ ingress_nginx_namespace }}
path: /networking/v1/ingresses
port: 443
failurePolicy: Fail
matchPolicy: Equivalent
name: validate.nginx.ingress.kubernetes.io
rules:
- apiGroups:
- networking.k8s.io
apiVersions:
- v1
operations:
- CREATE
- UPDATE
resources:
- ingresses
sideEffects: None

Some files were not shown because too many files have changed in this diff Show More