Agones – Advanced

Docs: Scheduling and Autoscaling

Thu, 03 Jan 2019 05:45:05 +0000

Cluster Autoscaler

Kubernetes has a cluster node autoscaler that works with a wide variety of cloud providers.

The default scheduling strategy (Packed) is designed to work with the Kubernetes autoscaler out of the box.

The autoscaler will automatically add Nodes to the cluster when GameServers don’t have room to be scheduled on the clusters, and then scale down when there are empty Nodes with no GameServers running on them.

This means that scaling Fleets up and down can be used to control the size of the cluster, as the cluster autoscaler will adjust the size of the cluster to match the resource needs of one or more Fleets running on it.

To enable and configure autoscaling on your cloud provider, check their connector implementation, or their cloud specific documentation.

Google Kubernetes Engine

Azure Kubernetes Service

Cluster Autoscaler on Azure Kubernetes Service (AKS) - Preview

Fleet Autoscaling

Fleet autoscaling is the only type of autoscaling that exists in Agones. It is currently available as a buffer autoscaling strategy or as a webhook driven strategy, such that you can provide your own autoscaling logic.

Have a look at the Create a Fleet Autoscaler quickstart, the Create a Webhook Fleet Autoscaler quickstart, and the Fleet Autoscaler Specification for details.

Autoscaling Concepts

To facilitate autoscaling, we need to combine several concepts and functionality, as described below.

Allocation Scheduling

Allocation scheduling refers to the order in which GameServers, and specifically their backing Pods are chosen from across the Kubernetes cluster within a given Fleet when allocation occurs.

Pod Scheduling

Each GameServer is backed by a Kubernetes Pod. Pod scheduling refers to the strategy that is in place that determines which node in the Kubernetes cluster the Pod is assigned to, when it is created.

Fleet Scale Down Strategy

Fleet Scale Down strategy refers to the order in which the GameServers that belong to a Fleet are deleted, when Fleets are shrunk in size.

Fleet Scheduling

There are two scheduling strategies for Fleets - each designed for different types of Kubernetes Environments.

Packed

apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: simple-udp
spec:
  replicas: 100
  scheduling: Packed
  template:
    spec:
      ports:
      - containerPort: 7654
      template:
        spec:
          containers:
          - name: simple-udp
            image: gcr.io/agones-images/udp-server:0.21

This is the default Fleet scheduling strategy. It is designed for dynamic Kubernetes environments, wherein you wish to scale up and down as load increases or decreases, such as in a Cloud environment where you are paying for the infrastructure you use.

It attempts to pack as much as possible into the smallest set of nodes, to make scaling infrastructure down as easy as possible.

This affects the Cluster autoscaler, Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.

Cluster Autoscaler

To ensure that the Cluster Autoscaler doesn’t attempt to evict and move GameServer Pods onto new Nodes during gameplay, Agones adds the annotation "cluster-autoscaler.kubernetes.io/safe-to-evict": "false" to the backing Pod.

Allocation Scheduling Strategy

Under the “Packed” strategy, allocation will prioritise allocating GameServers to nodes that are running on Nodes that already have allocated GameServers running on them.

Pod Scheduling Strategy

Under the “Packed” strategy, Pods will be scheduled using the PodAffinity with a preferredDuringSchedulingIgnoredDuringExecution affinity with hostname topology. This attempts to group together GameServer Pods within as few nodes in the cluster as it can.

Note

The default Kubernetes scheduler doesn’t do a perfect job of packing, but it’s a good enough job for what we need - at least at this stage.

Fleet Scale Down Strategy

With the “Packed” strategy, Fleets will remove Ready GameServers from Nodes with the least number of Ready and Allocated GameServers on them. Attempting to empty Nodes so that they can be safely removed.

Distributed

apiVersion: "agones.dev/v1"
kind: Fleet
metadata:
  name: simple-udp
spec:
  replicas: 100
  scheduling: Distributed
  template:
    spec:
      ports:
      - containerPort: 7654
      template:
        spec:
          containers:
          - name: simple-udp
            image: gcr.io/agones-images/udp-server:0.21

This Fleet scheduling strategy is designed for static Kubernetes environments, such as when you are running Kubernetes on bare metal, and the cluster size rarely changes, if at all.

This attempts to distribute the load across the entire cluster as much as possible, to take advantage of the static size of the cluster.

This affects Allocation Scheduling, Pod Scheduling and Fleet Scale Down Scheduling.

Cluster Autoscaler

Since this strategy is not aimed at clusters that autoscale, this strategy does nothing for the cluster autoscaler.

Allocation Scheduling Strategy

Under the “Distributed” strategy, allocation will prioritise allocating GameServers to nodes that have the least number of allocated GameServers on them.

Pod Scheduling Strategy

Under the “Distributed” strategy, Pod scheduling is provided by the default Kubernetes scheduler, which will attempt to distribute the GameServer Pods across as many nodes as possible.

Fleet Scale Down Strategy

With the “Distributed” strategy, Fleets will remove Ready GameServers from Nodes with at random, to ensure a distributed load is maintained.

Docs: Limiting CPU & Memory

Thu, 03 Jan 2019 05:45:15 +0000

As a short description:

CPU Requests are limits that are applied when there is CPU congestion, and as such can burst above their set limits.
CPU Limits are hard limits on how much CPU time the particular container gets access to.

This is useful for game servers, not just as a mechanism to distribute compute resources evenly, but also as a way to advice the Kubernetes scheduler how many game server processes it is able to fit into a given node in the cluster.

It’s worth reading the Managing Compute Resources for Containers Kubernetes documentation for more details on “requests” and “limits” to both CPU and Memory, and how to configure them.

GameServers

Since the GameServer specification provides a full PodSpecTemplate, we can take advantage of both resource limits and requests in our GameServer configurations.

For example, to set a CPU limit on our GameServer configuration of 250m/0.25 of a CPU, we could do so as followed:

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  name: "simple-udp"
spec:
  ports:
  - name: default
    containerPort: 7654
  template:
    spec:
      containers:
      - name: simple-udp
        image: gcr.io/agones-images/udp-server:0.21
        resources:
          limit:
            cpu: "250m" #this is our limit here

If you do not set a limit or request, the default is set by Kubernetes at a 100m CPU request.

SDK GameServer sidecar

You may also want to tweak the CPU request or limits on the SDK GameServer sidecar process that spins up alongside each game server container.

You can do this through the Helm configuration when installing Agones.

By default, this is set to having a CPU request value of 30m, with no hard CPU limit. This ensures that the sidecar always has enough CPU to function, but it is configurable in case a lower, or higher value is required on your clusters, or if you desire hard limit.

Docs: Allocator Service

Tue, 19 May 2020 05:45:05 +0000

To allocate a game server, Agones in addition to GameServerAllocations , provides a gRPC service with mTLS authentication, called agones-allocator, which is on stable version , starting on agones v1.6.

The gRPC service is accessible through a Kubernetes service that is externalized using a load balancer. For the gRPC request to succeed, a client certificate must be provided that is in the authorization list of the allocator service.

The remainder of this article describes how to manually make a successful allocation request using the gRPC API.

Find the external IP

The service is hosted under the same namespace as the Agones controller. To find the external IP of your allocator service, replace agones-system namespace with the namespace to which Agones is deployed and execute the following command:

kubectl get service agones-allocator -n agones-system

The output of the command should look like:

NAME                        TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)            AGE
agones-allocator            LoadBalancer   10.55.251.73    34.82.195.204   443:30250/TCP      7d22h

Server TLS certificate

Replace the default server TLS certificate with a certificate with CN and subjectAltName. There are multiple approaches to generate a certificate. Agones recommends using cert-manager.io solution for cluster level certificate management.

In order to use cert-manager solution, first, install cert-manager on the cluster. Then, configure an Issuer/ClusterIssuer resource and last configure a Certificate resource to manage allocator-tls Secret.

Here is an example of using a self-signed ClusterIssuer for configuring allocator-tls Secret:

#!/bin/bash
# Create a self-signed ClusterIssuer
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: selfsigned
spec:
  selfSigned: {}
EOF

EXTERNAL_IP=`kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`

# Create a Certificate with IP for the allocator-tls secret
cat <<EOF | kubectl apply -f -
apiVersion: cert-manager.io/v1alpha2
kind: Certificate
metadata:
  name: allocator-selfsigned-cert
  namespace: agones-system
spec:
  commonName: ${EXTERNAL_IP}
  ipAddresses:
    - ${EXTERNAL_IP}
  secretName: allocator-tls
  issuerRef:
    name: selfsigned
    kind: ClusterIssuer
EOF

# Optional: Store the secret ca.crt in a file to be used by the client for the server authentication
TLS_CA_FILE=ca.crt
TLS_CA_VALUE=`kubectl get secret allocator-tls -n agones-system -ojsonpath='{.data.ca\.crt}'`
echo ${TLS_CA_VALUE} | base64 -d > ${TLS_CA_FILE}

# In case of MacOS
# echo ${TLS_CA_VALUE} | base64 -D > ${TLS_CA_FILE}

# Add ca.crt to the allocator-tls-ca Secret
kubectl get secret allocator-tls-ca -o json -n agones-system | jq '.data["tls-ca.crt"]="'${TLS_CA_VALUE}'"' | kubectl apply -f -

Client Certificate

Because agones-allocator uses an mTLS authentication mechanism, client must provide a certificate that is accepted by the server. Here is an example of generating a client certificate. For the agones-allocator service to accept the newly generate client certificate, the generated client certificate CA or public portion of the certificate must be added to a kubernetes secret called allocator-client-ca.

#!/bin/bash

KEY_FILE=client.key
CERT_FILE=client.crt

openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout ${KEY_FILE} -out ${CERT_FILE}

CERT_FILE_VALUE=`cat ${CERT_FILE} | base64 -w 0`

# In case of MacOS
# CERT_FILE_VALUE=`cat ${CERT_FILE} | base64`

# white-list client certificate
kubectl get secret allocator-client-ca -o json -n agones-system | jq '.data["client_trial.crt"]="'${CERT_FILE_VALUE}'"' | kubectl apply -f -

The last command creates a new entry in the secret data map called client_trial.crt for allocator-client-ca and stores it. You can also achieve this by kubectl edit secret allocator-client-ca -n agones-system, and then add the entry.

Restart pods

Restart pods to get the new TLS certificate loaded to the agones-allocator service.

kubectl get pods -n agones-system -o=name | grep agones-allocator | xargs kubectl delete -n agones-system

Send allocation request

Now the service is ready to accept requests from the client with the generated certificates. Create a fleet and send a gRPC request to agones-allocator. To start, take a look at the allocation gRPC client examples in golang and C# languages. In the following, the golang gRPC client example is used to allocate a Game Server in the default namespace.

#!/bin/bash

NAMESPACE=default # replace with any namespace
EXTERNAL_IP=`kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`
KEY_FILE=client.key
CERT_FILE=client.crt
TLS_CA_FILE=ca.crt

go run examples/allocator-client/main.go --ip ${EXTERNAL_IP} \
    --port 443 \
    --namespace ${NAMESPACE} \
    --key ${KEY_FILE} \
    --cert ${CERT_FILE} \
    --cacert ${TLS_CA_FILE}

If your matchmaker is external to the cluster on which your game servers are hosted, the agones-allocator provides the gRPC API to allocate game services using mTLS authentication, which can scale independently to the Agones controller.

Docs: Multi-cluster Allocation

Fri, 25 Oct 2019 05:45:05 +0000

Beta

This feature is in a pre-release state and might change.

There may be different types of clusters, such as on-premise, and Google Kubernetes Engine (GKE), used by a game to help with the cost-saving and availability. For this purpose, Agones provides a mechanism to define priorities on the clusters. Priorities are defined on GameServerAllocationPolicy agones CRD. A matchmaker can enable the multi-cluster rules on a request and target agones-allocator endpoint in any of the clusters and get resources allocated on the cluster with the highest priority. If the cluster with the highest priority is overloaded, the allocation request is redirected to the cluster with the next highest priority.

The remainder of this article describes how to enable multi-cluster allocation.

Define Cluster Priority

GameServerAllocationPolicy is the CRD defined by Agones for setting multi-cluster allocation rules. In addition to cluster priority, it describes the connection information for the target cluster, including the game server namespace, agones-allocator endpoint and client K8s secrets name for redirecting the allocation request. Here is an example of setting the priority for a cluster and it’s connection rules. One such resource should be defined per cluster. For clusters with the same priority, the cluster is chosen with a probability relative to its weight.

In the following example the policy is defined for cluster B in cluster A.

cat <<EOF | kubectl apply -f -
apiVersion: multicluster.agones.dev/v1
kind: GameServerAllocationPolicy
metadata:
  name: allocator-cluster-B
  namespace: cluster-A-ns
spec:
  connectionInfo:
    allocationEndpoints:
    - 34.82.195.204
    clusterName: "clusterB"
    namespace: cluster-B-ns
    secretName: allocator-client-to-cluster-B
    serverCa: c2VydmVyQ0E=
  priority: 1
  weight: 100
EOF

To define the local cluster priority, similarly, an allocation rule should be defined, while leaving allocationEndpoints unset. If the local cluster priority is not defined, the allocation from the local cluster happens only if allocation from other clusters with the existing allocation rules is unsuccessful.

serverCa is the server TLS CA public certificate, set only if the remote server certificate is not signed by a public CA (e.g. self-signed). If this field is not specified, the certificate can also be specified in the ca.crt field of the client secret (i.e. the secret referred to in the secretName field).

Establish trust

To accept allocation requests from other clusters, agones-allocator for cluster B should be configured to accept the client’s certificate from cluster A and the cluster A’s client should be configured to accept the server TLS certificate, if it is not signed by a public Certificate Authority (CA).

Follow the steps to configure the agones allocator gRPC service. The client certificate pair in the mentioned document is stored as a K8s secret. Here are the secrets to set:

1.Client certificate to talk to other clusters:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: allocator-client-to-cluster-B
  namespace: cluster-A-ns
type: Opaque
data:
  tls.crt: <REDACTED>
  tls.key: <REDACTED>
  ca.crt: <REDACTED>
EOF

The certificates are base 64 string of the certificate file e.g. cat ${CERT_FILE} | base64 -w 0

Agones recommends using cert-manager.io solution for generating client certificates.

2.Add client CA to the list of authorized client certificates by agones-allocator in the targeted cluster.

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: allocator-client-ca
  namespace: agones-system
type: Opaque
data:
  client1.crt: <REDACTED>
  client2.crt: <REDACTED>
  …
  clientN.crt: <REDACTED>
EOF

Allocate multi-cluster

To enable multi-cluster allocation, set multiClusterSetting.enabled to true in allocation.proto and send allocation requests. For more information visit agones-allocator. In the following, using allocator-client sample , a multi-cluster allocation request is sent to the agones-allocator service.

#!/bin/bash
EXTERNAL_IP=`kubectl get services agones-allocator -n agones-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}'`

NAMESPACE=default # replace with any namespace

go run examples/allocator-client/main.go --ip ${EXTERNAL_IP} \
    --namespace ${NAMESPACE} \
    --key ${KEY_FILE} \
    --cert ${CERT_FILE} \
    --cacert ${TLS_CA_FILE} \
    --multicluster true

Docs: GameServer Pod Service Accounts

Thu, 14 Mar 2019 04:30:37 +0000

Default Settings

By default, Agones sets up service accounts and sets them appropriately for the Pods that are created for GameServers.

Since Agones provides GameServer Pods with a sidecar container that needs access to Agones Custom Resource Definitions, Pods are configured with a service account with extra RBAC permissions to ensure that it can read and modify the resources it needs.

Since service accounts apply to all containers in a Pod, Agones will automatically overwrite the mounted key for the service account in the container that is running the dedicated game server in the backing Pod. This is done since game server containers are exposed publicly, and generally don’t require the extra permissions to access aspects of the Kubernetes API.

Bringing your own Service Account

If needed, you can provide your own service account on the Pod specification in the GameServer configuration.

For example:

apiVersion: "agones.dev/v1"
kind: GameServer
metadata:
  generateName: "simple-udp-"
spec:
  ports:
  - name: default
    containerPort: 7654
  template:
    spec:
      serviceAccountName: my-special-service-account # a custom service account
      containers:
      - name: simple-udp
        image: gcr.io/agones-images/udp-server:0.21

If a service account is configured, the mounted key is not overwritten, as it assumed that you want to have full control of the service account and underlying RBAC permissions.