Lab 4 - Monitoring using SRE Golden Signals

Monitoring using SRE Golden Signals with IBM Cloud Pak for Multicloud Management

• Lab Overview
• Prerequisite
• Business Context
• Add Managed Clusters
• Deploy cloud native monitoring
• Deploy Bookinfo Application
• Deploy synthetics monitoring Point-of-Presence
• Explore SRE Golden Signals
• Summary

Lab Overview

• Understand Cloud Pak for Multicloud Management Monitoring module
• Learn how to add cloud native monitoring to the managed cluster
• learn how to gather monitoring metrics from the managed cluster
• Learn how to use SRE Golden Signals to monitor application running on the managed cluster

Prerequisite

• You need to provision your own copy of the CP4MCM 2.x environment, start it and verify for correct startup (check here).

Business Context

• Hub cluster includes management console, federated monitoring, and all the controllers.
• Managed cluster includes klusterlet components that communicate status back to the Hub cluster.

• Add a managed cluster (if not done already)
• Deploy cloud native monitoring to the managed cluster
• Visualize monitored resources in a managed cluster
• Install the Synthetic Monitoring Point-of-Presence
• Deploy an monitored application to managed cluster
• Define the synthetic monitoring of the application and define a Service Level Objective for it
• Explore application service topology and SRE Golden Signals provided by data collectors embeded in the application runtimes

Add Managed Clusters

1. To start the lab, you should be in your Cloud Pak for Multicloud Management Web Console. If you are not, check here how to open your console page.

   

2. Now, let’s explore the Cluster view. Click the hamburger Menu (1) and select Automated Infrastructure -> Clusters (2).

   

3. You can add a cluster by Importing an existing cluster or provisioning a new cluster using a Service Library. We use the first option. Select Import an Existing cluster (1) and click Import (2).

   

4. Enter microk8s for cluster name (1) and microk8s for namespace (2). Click Generate command to continue (3).

   

5. A curl command is generated that you will use to add the new cluster. Click Copy command button (1) and click View cluster (2) to see the new managed cluster details page.

   

6. Go back to the desktop and open the terminal window to MicroK8s cluster clicking the MicroK8s Terminal link.

   

7. MicroK8s terminal has a yellow background. To verify the cluster status run the following command in the MicroK8s window

   kubectl get nodes
   Copied!

   

   Great, you are accessing the managed cluster. Now you are ready to execute the generated command.

8. Paste the generated command that you previously copied in the clipboard. When you run the command, several Kubernetes objects are created in the multicluster-endpoint namespace.

   

   If you see the error as before - just run the command again.

9. You can view the progress by entering the command:

   kubectl get pods -n multicluster-endpoint
   Copied!

   Make sure all the pods are in the running state.

   

10. Back to the browser window, click View cluster and make sure that the cluster status is Ready now (if necessary, refresh the details page).

    

11. On the page navigation breadcrumb, click on Clusters link

    

12. Now you can see your managed cluster in a clusters list. You need to add labels to identify your new cluster. Labels are then used by the PlacementRules and PlacementPolicies to select clusters where different applications and policies should be deployed.

    In the Cluster Management lab you could learn how to add labels using a web console. Below you will use alternative methods of modifying labels using a CLI.

    First, you need to install a multicluster plugin to the cloudctl command. Open the green terminal to the Management Hub and run the following commands

    curl -kLo cloudctl-mc-plugin https://cp-console.apps.demo.ibmdte.net/rcm/plugins/mc-linux-amd64
    cloudctl login -a https://cp-console.apps.demo.ibmdte.net -u bob -p Passw0rd -n default
    cloudctl plugin install -f cloudctl-mc-plugin
    Copied!

    Now, you can manage your remote clusters with the cli. To list the labels for microk8s cluster, in a green Management Hub terminal run:

    cloudctl mc describe cluster microk8s -n microk8s -c hub0
    Copied!

    To add a new label environment=QA run the following command in a green Management Hub terminal

    cloudctl mc label cluster microk8s environment=QA -n microk8s
    Copied!

    The remote Kubernetes cluster is now registered as a managed cluster. In the next step, you will deploy the cloud native monitoring to that mamnaged cluster.

Deploy cloud native monitoring

1. First, you need to label target cluster with ibm.com/cloud-native-monitoring=enabled. You can do this in web console, or run the following command in a green terminal titled Management Hub

   cloudctl mc label cluster microk8s ibm.com/cloud-native-monitoring=enabled -n microk8s
   Copied!

   

2. Second, you need to add the target cluster namespace as a managed resource to your team. Again, you can do this in web console (Menu -> Administer -> Identity and access -> Teams tab, select the team ‘operations’, edit the Resources), or run the following command in a green terminal titled Management Hub

   cloudctl iam resource-add operations -r crn:v1:icp:private:k8:mycluster:n/microk8s:::
   Copied!

   This command adds a namespace microk8s (this namespace was automatically created during cluster import and contains the cluster CRD) as a managed resource to the team operations. This operations triggers the deployment process of cloud-native-monitoring operator to the target cluster.

   You can check that the operator was deployed running the following commands in a yellow terminal connected to MicroK8s cluster

   kubectl get pods -n cp4mcm-cloud-native-monitoring
   Copied!

   

3. Finally, you need to open the Monitoring module console as a user belonging to the operations team - user bob in our case. Select “hamburger” menu in the top-left corner and then Monitor health (2) and Infrastructure monitoring.

   

   If you are redirected to login page, login again as user bob with password Passw0rd. Eventually, you should see the Resources tab shown below:

   

   The page probably won’t show any resources yet, because the process to install Kubernetes data collector was just started. You can check if it was successfully deployed running the previous command again (in yellow terminal).

   

   When it is deployed, after 1-2 minutes you can refresh the browser, and you should see the view showing now all the resources from the managed cluster.

   

   Click the Kubernetes cluster.

   

   Then click microk8s.

   

   The next view, let you explore the metrics and events releated to the selected resource. On top, you can see the Event timeline (1) for wihich you can change the time window (from 30 minutes to 1 week). Deployment topology widget (2) shows how the specific resource is located within the cluster.

   

   If you hover over the numbers on the Timeline (1) you can see the details of the events. You can also click the ‘How do I read this’ link (2) on the Deployment topology to learn hot to use that widget.

   

   Feel free to explore other resources and their metrics. For example you can click the worker node icon on the Deployment topology widget.

   

Deploy Bookinfo Application

1. To start, clone the git repository containing the yaml files needed to deploy the application. Using the green terminal window, run the following commands on the Management Hub

   git clone https://github.com/dymaczew/charts
   cd charts
   Copied!

2. Apply the application resources running the following commands:

   /home/ibmuser/oclogin.sh
   oc apply -f bookinfo-multiclister-2020.2.1
   oc apply -f kubernetes-1.18-ingress-deployable.yaml
   Copied!

   If you happen to run Kubernetes 1.19 on the your managed cluster (outside the Skytap environment) use kubernetes-1.19-ingress-deployable.yaml. Ingress API was changed in 1.19 and extended with new mandatory fields

   The output looks like the following:

   

   Don’t worry about the error in the last line - it just means that ImagePolicy admission controller is not used in the environment.

3. Now, let’s edit the placement rule for the application to get it deployed on the managed cluster (we tagged in with label “environment=QA”). Navigate to application view, selecting the “hamburger” menu in the top-left corner, then Observe environments and Overview

   

   On the overview page, scroll down to the Applications section and select bookinfo-app link. We recommend this way, becasue for some reason (probably a bug?) the application is not listed directly on the Hybrid Applications list

   

4. When the Application view opens, click on the bookinfo-rule icon

   

5. Edit the placement rule in the editor on the right, changing the value of the label from Dev to QA (1) and then apply you changes with the button on top of the editor(2)

   

6. After a few seconds, you should notice that the application is now being placed on the microk8s cluster. The deployment takes usually 1-2 minutes needed to pull images and start containers.

   

7. You can verify if the application was successfuly deployed and is accessible by opening a new browser tab and entering the URL: bookinfo.10.0.0.2.nip.io. You should reach the page that looks like below:

   

Exploring synthetics monitoring

1. Open the Monitoring module interface (Select “hamburger” menu, then Monitor Health, then Infrastructure monitoring). Then select the Synthetic results tab

   

2. On the page Synthetic results page, you should see the automatically configured monitor for ingress deployed as a part of Bookinfo application. Click on the monitor name to explore details

   

3. You can adjust the scope of the timeline, reducing the windows to last 30 minutes (1). You can also select to see a Response time graph or a response time Breakdown (2). This helps to diagnose issues related to name resolution on establishing the SSL session.

   

4. When you scroll down you can see the results of actual tests. Select any dot on the graph to see detailed breakdown of the response.

   

   You can deploy more agents in different locations to have your applications tested for availability and response time from a customer perspective. Deploying own agents allows also to configure more sophisticated test schedules. Let’s see how it works.

5. Go to the green terminal titled Management hub and run the following commands to unpack the synthetics agent binaries that were downloaded for you.

   cd
   tar xvf app_mgmt_syntheticpop_xlinux.tar.gz
   cd app_mgmt_syntheticpop_xlinux
   Copied!

6. To configure the synthetic monitoring agent you need a config pack that will instruct agent where to look for test definitions and where to send the gathered data. Go back to you browser, click the Synthetic results breadcrumb in the top-left corner of the screen. Then select the Administration tab.

   

7. On the Administration view select the Integrations tile, then New integration button.

   

8. Click Configure under the Monitoring Data Collectors tile

   

9. Do not provide any name, just click the Download file button and save the file to your workstation

   

10. Go back to the green terminal titled Management Hub and run the following commands to preconfigure and install the Synthetic Monitoring agent

    ./config-pop.sh -f /home/ibmuser/Downloads/ibm-cloud-apm-dc-configpack.tar
    Copied!

    Answer the installation wizard with the following values:

    You will configure a new local point of presence (PoP).
    Enter a name for your PoP. Your PoP will be identified by this name: pop_user1 -- A point of presence name
    The PoP name is set to pop_user1
    Enter the name of the country in which your PoP is located: USA -- Use any country
    The country name is set to USA
    Enter the name of the city in which your PoP is located: Las Vegas -- Use any city
    Copied!

    Finally, run the following command to start the agent:

    ./start-pop.sh
    Copied!

11. Now, when you have additional Point-of-Presence (syntectic monitoring agent) installed, go back to your browser and navigate to the Administration page, and then select the Synthetics tile.

    

    Click Create button

    

    Give your test a name and description.

    

    Scroll down and select the test type (Webpage). You can notice there are other types of tests available - you can for example replay web session recorded with Selenium, or create API tests using either SOAP or REST APIs.

    

    In the next step you need to provide the URL of the bookinfo main application page. Scroll down and provide the following values:

    URL: http://bookinfo.10.0.0.2.nip.io:9080/productpage?u=normal

    IMPORTANT Use the above URL, don’t worry that the screenshots below shows differnt one!

    Threshold value for Warning: 1

    Threshold value for Critical: 2

    

    Click Verify test

    

    On the next page, change the test frequency to 1 minute and make sure that your previously installed PoP agent is selected

    

    Click Finish at the bottom of the page

    

    Now, you have synthetics agent that will generate the traffic against the Bookinfo application. You can move to the final part of the tutorial, exploring tools available for Site Reliability Engineers.

Explore SRE Golden Signals

1. In the browser with the Cloud Pak user interface, click the Resources tab

   

2. Select “Kubernetes Services”

   You will see a list of kubernetes services that are running in your environment

   

3. Click the link for the “productpage” resource

   You will navigate to the page for the productpage microservice. Let’s explore this page as seen below

   

4. Deployment topology

   In the upper left corner, you see the “Deployment topology”. You’ve seen this before in the context of the kubernetes cluster. Now, you’re viewing it in the context of the productpage microservice. What the topology is showing you is that this microservice is deployed to one pod on one node in the cluster. If you scaled out the deployment to 2 pods, then you would see 2 pods in the Deployment topology.

5. Golden Signals

   Next, look at the golden signals on the right side of the page. The 4 graphs labeled Latency, Errors, Traffic, and Saturation are the Golden Signals. These are the most important metrics for Site Reliability Engineering (SRE) as they show the metrics imoprtant from the end-user perspective, that have been normalized for different application/middleware domains. Let’s explore Latency a little more.

6. Latency

   Flyover the Latency Graph. You’ll see a graph showing the latency shown in different percentiles (50th, 90th, and 95th). By using percentiles, you get a much better idea how the applicaiton is performing.

   

   Next, select the dropdown list in the Filter. The default behavior is to show 50th, 90th, and 95th percentile for all URLs. But, sometimes you want to filter the data.

   

   Select 1 or more of the URLs for the productpage microservice

   View the latency data for the URLs that you selected.

   Within filters, select the icon on the far right for the “/productpage?u=normal” URL.

   

   After you click the icon, you will see some very useful information as seen below. At the top of the page, you see a scatter plot chart that allows you to see a distribution of the requests. This is a very useful way to visualize the transactions because it allows you to see patterns and outliers.

   Below that, expand one of the requests and you will see a breakdown of where the request spent its time.

   

   Close the Trace Breakdown window by clicking the “X” in the upper right corner.

   

   Now, click the 3 vertical dots in the upper right corner of the Latency graph and select “Latency Options”

   

   Notice that you can customize the latency options. Either change the latency percentiles or add/delete lines from the graph. Try it out.

   

7. Next, examine the Service dependencies

   

   The service dependency shows a 1-hop topology of the microservices. For the productpage service, it shows that there are clients connecting to the service and there is a dependency on “details” and “reviews”.

   Click on “reviews” icon. You will navigate to that microservice and see the 1-hop topology for the “review” service. Examine the golden signals for the “reviews” service.

8. Full Service Topology

   Most of the time, the 1-hop topology is good enough to diagnose the root cause of a problem. But, sometimes you need to see additional information. Click the “expand to the full screen” icon in the upper right corner of the service dependencies to expand the view.

   

   The view you see comes from an embedded capability called Agile Service Manager (ASM). ASM allows you to expand to more than hop in the topology. It also allows you to visualize changes that are occurring in the application. Since change introduces most of the problems in IT, this is a powerful capability.

   

   Let’s start by switching to a 2-hop topology. Select the dropdown in the top-middle of the screen and change the value to “2”. Then click “Render”

   

   We won’t examine it here, but ASM allows you to hide/show some additional objects in the topology. In this topology, you see the microservice topology. If you want, you can add the pods into the topology. To add/hide elements on the page, click the Filter icon to the left of the “Render” button.

   

   ASM has powerful capabilities to show you what’s changing. This includes topology changes, state changes, and property changes. We won’t be exploring that capability since there haven’t been any changes to the application. Feel free to explore additional ASM capabilities. When you are done exploring, you can click on one of the icons for the microservices and you will navigate back to the Golden Signal view.

9. Drill Down into the Runtime

   Sometimes you need additional details that can only be gathered from the data collector that is running within the runtime. If the app server (python, Node.js, JVM, golang) is instrumented with a lightweight data collector, you can click on the container and drilldown into the runtime metrics.

   Click on the “container” in the Deployment topology.

   

   You are now viewing the detailed container metrics for this microservice. To navigate to the detailed metrics reported by the data collector, scroll down and click the appropriate name in Related resources window.

   

   The runtime page shows selection of most important metrics for a selected runtime type

   

   To expore any other metric, scroll down the page and expand the Custom metrics section, picking the metric you want and additional filtering and display options.

   

   This concludes the exercise. You now understand how to naviagate Golden Signals view.

   Additional resources:

   • Golden Signals video on YouTube
   • Blog: Golden Signals explained

Summary

• Understand Cloud Pak for Multicloud Management Monitoring module
• Learn how to add cloud native monitoring to the managed cluster
• learn how to gather monitoring metrics from the managed cluster
• Learn how to use SRE Golden Signals to monitor application running on the managed cluster

• Cloud Pak for Multicloud Management home page
• Cloud Pak for Multicloud Management Demos