Hybrid Cloud Patterns

Deploying the Multicluster DevSecOps Pattern


  1. An OpenShift cluster (Go to the OpenShift console). Cluster must have a dynamic StorageClass to provision PersistentVolumes. See also sizing your cluster.
  2. A second OpenShift cluster for production (optional but desirable)
  3. A third OpenShift cluster for secure CI pipelines (optional but desirable)
  4. A GitHub account (and a token for it with repositories permissions, to read from and write to your forks)
  5. The helm binary, see here

If you do not have running Red Hat OpenShift clusters you can start one on a public or private cloud by using Red Hat’s cloud service.

Credentials Required in Pattern

In addition to the openshift cluster, you will need to prepare a number of secrets, or credentials, which will be used in the pattern in various ways. To do this, copy the values-secret.yaml template to your home directory as values-secret.yaml and replace the explanatory text as follows:

  • Your git repository username and password. The password must be base64 encoded.
    # Go to: https://github.com/settings/tokens
    # Then: echo -n 'your string value' | base64
    username: USERNAME
    password: 'encoded password in single quotes'
  • You application secret. TBD This may change when the application is changed.
    # Secret used for demonstrating vault storage, external secrets, and ACM distribution secure
    secret: PLAINTEXT

Preparing to deploy

  1. Install the installation tooling dependencies. You will need:

{% include prerequisite-tools.md %}

  1. Fork the Multicluster DevSecOps repository on GitHub. It is necessary to fork because your fork will be updated as part of the GitOps and DevSecOps processes.

  2. Clone the forked copy of the multicluster-devsecops repository. Use branch v1.0.

    git clone git@github.com:{your-username}/multicluster-devsecops.git
    cd multicluster-devsecops
    git checkout v1.0
  3. You could create your own branch where your specific values will be pushed to:

    git checkout -b my-branch
  4. A values-secret.yaml file is used to automate setup of secrets needed for:

    • A Git repository (E.g. Github, GitLab etc.)
    • Quay registry deployment secrets.
    • Any application secrets that are needed.

    DO NOT COMMIT THIS FILE. You do not want to push personal credentials to GitHub. Instead copy the template file values-secret.yaml.template to your home directory. Change the values in that file to ones that fit your environment.

    cp values-secret.yaml.template ~/values-secret.yaml
    vi ~/values-secret.yaml
  5. Customize the deployment for your cluster. Change the appropriate values in values-global.yaml

    vi values-global.yaml
    git add values-global.yaml
    git commit values-global.yaml
    git push origin my-branch

Getting Started Video

Make sure to set up the values-secret.yaml and values-global.yaml correctly (see above). For a demonstration of the deployment, click on the image below to launch the video.


How to deploy

  1. You can deploy the pattern using the validated pattern operator. If you do use the operator then skip to Validating the Environment below.

  2. Preview the changes

    make show
  3. Login to your cluster using oc login or exporting the KUBECONFIG

    oc login


    export KUBECONFIG=~/my-ocp-env/hub
  4. Apply the changes to your cluster

    make install

Validating the Environment

  1. Check the operators have been installed

    OpenShift Console UI -> Installed Operators
  2. Navigate to the OpenShift GitOps instances using the links on the top right hand side of the screen.

    The most important ArgoCD instance to examine at this point is hub-gitops-server. This is where all the applications for the hub (datacenter), including the test environment, can be tracked.

  3. Apply the secrets from the values-secret.yaml to the secrets management Vault. This can be done through Vault’s UI - manually without the file. The required secrets and scopes are:

    • secret/git git username & password (GitHub token)
    • secret/quay The admin username and password and email.
    • secret/imageregistry Quay.io or DockerHub username & password

    Or you can set up the secrets using the command-line by running the following (Ansible) playbook.


    Using the Vault UI check that the secrets have been setup.

    For more information on secrets management see here. For information on Hashicorp’s Vault see here

  4. Check all applications are synchronized in OpenShift GitOps

    Multicluster DevSecOps GitOps overview

Check the ACM policy deployment

  1. After ACM is installed a message regarding a “Web console update is available” may be displayed. Click on the “Refresh web console” link.


  2. Navigate to the ACM hub console. On the upper-left side you’ll see a pull down labeled “local-cluster”. Select “All Clusters” from this pull down. This will navigate to the ACM console and to its “Clusters” section


  3. The Governance dashboard shows high level information on Policy set violations and Policy violations.


  4. Navigate to the Governance page and select the Policy sets Governance tab. There are two policy sets deployed, one for the hub and one for managed clusters.


  5. Explore the Policies tab and select some policies to examine. The image below shows ACM policy status for a three cluster deployment.


Checking the ACS deployment

  1. Select the stackrox Project (namespace). Navigate to the OCP Networking->Routes page. Click on the central route location URL. It might take a few minutes for this link to be active. When it does it will launch a new tab with the ACS Central login page.

    ACS Central route

  2. Return to the OCP console tab and navigate to the Workload->Secrets page. Find the central-htpasswd secret and select it.

    ACS Central secret

  3. On the central-htpasswd page, scroll to the Data section and select the copy icon on the right in the password field.

    ACS Central password

  4. Return to the ACS Central tab and paste the password into thee password field. Make sure that the Username is admin.

    ACS Central password

  5. This will bring you to the ACS Central dashboard page. At first it may not show any clusters showing but as the ACS secured deployment on the hub syncs with ACS central on the hub then information will start to show.

    ACS Central dashboard

Return to this dashboard later after deploying the development and production clusters so you can see their information in this dashboard. All clusters in this pattern are ACS secured and therefore ought to show up in this dashboard when those clusters join the hub and are fully deployed.

Check the Quay deployment

  1. Select the quay-enterprise project (namespace). Navigate to the OCP Networking->Routes page. Click on the quay-registry-quay route location URL (standard Quay naming apparently). It might take a few minutes for this link to be active. When it does it will launch a new tab with the Quay login page.

    Quay route

  2. An initial quayadmin account has already been created for you as part of the deployment. The password is quayadmin123. If you want to change initial admin user name and password you can do so by editing the charts/hub/quay/values.yaml or by adding those entries to the values-global.yaml file. Log in using the username and password.

  3. After logging in, the private Quay registry dashboard will be displayed.

    Quay dashboard

Completing the Quay Bridge with a bearer token

Managed clusters use a Quay Bridge in order to provide integration between the cluster and Quay Enterprise running on the hub/central cluster. The Quay Bridge looks like a local OpenShift registry but acts as a proxy to the Quay Enterprise registry. Currently there is a manual step to completing the Quay Bridge setup for managed clusters.

  1. Log in to Red Hat Quay through the web UI.

  2. Select the organization for which the external application will be configured.

  3. On the navigation pane, select Applications.

  4. Select Create New Application and enter a name for the new application, for example, openshift.

  5. On the OAuth Applications page, select your application, for example, devel-automation.

  6. On the navigation pane, select Generate Token.

  7. Select the following fields and press Generate Access Token at the bottom of the page:

  • Administer Organization

  • Administer Repositories

  • Create Repositories

  • View all visible repositories

  • Read/Write to any accessible repositories

  • Administer User

  • Read User Information

    GitOps Devel app

  1. Review the assigned permissions.

  2. Select Authorize Application and then confirm confirm the authorization by selecting Authorize Application at the bottom of the page.

  3. Save/copy the generated access token.

  4. At a command line prompt that has KUBECONFIG set to the central/hub cluster’s auth/kubeconfig file, run the following command with the token that was saved/copied above.

$ oc create secret -n openshift-operators generic quay-integration --from-literal=token=<access_token>

There is a ACM policy that will make sure that this is copied out to the managed clusters. If there are any problems with the managed cluster’s Quay Bridge quay-integration token, you can run the same command on the managed cluster.

Creating an ACS/Quay integration

Advanced Cluster Security needs to be integrated with Quay Enterprise registry. Currently there is no way to automate this as it requires the above manual step to generate the OAuth token.

  1. On the ACS console, under ==Platform Configuration== on the left hand side, select ==Integrations==.

  2. Under Image Integrations select ==Red Hat Quay.io==

  3. In the Integrations > Quay.io page select ==New Integration== and fill out the form: Give it a name like hub-quay and select Registry as the type. Provide the URL for Quay Enterprise and the OAuth token generated in above. Press ==Save==. Here is an example.

ACS Quay Integration

Next Steps

Help & Feedback{: .btn .fs-5 .mb-4 .mb-md-0 .mr-2 } Report Bugs{: .btn .btn-red .fs-5 .mb-4 .mb-md-0 .mr-2 }

Once the hub has been setup correctly and confirmed to be working, you can:

  1. Add a dedicated development cluster to deploy the CI pipelines using ACM

  2. Add a dedicated production cluster to deploy production using ACM

  3. Once the hub, production and devel clusters have been deployed you will want to check out and test the Multi-Cluster DevSecOps demo code. You can find that here TBD

    a. Making configuration changes with GitOps TBD a. Making application changes using DevOps TBD


Probably wont work

  1. Turn off auto-sync

    helm upgrade manuela . --values ~/values-secret.yaml --set global.options.syncPolicy=Manual

  2. Remove the ArgoCD applications (except for manuela-datacenter)

    a. Browse to ArgoCD a. Go to Applications a. Click delete a. Type the application name to confirm a. Chose “Foreground” as the propagation policy a. Repeat

  3. Wait until the deletions succeed

    tbd should be the only remaining application

  4. Complete the uninstall

    helm delete tbd

  5. Check all namespaces and operators have been removed