Using Grafana for app container metrics, distributed tracing, and observability

This tutorial takes you creating a simple Steeltoe app with actuators, logging, and distributed tracing. With that app running you then export the data to an instance of Prometheus and visualize things in a Grafana dashboard.

Note

For more detailed examples, please refer to the Management solution in the Steeltoe Samples Repository.

First, clone to accompanying repo that contains all the needed assets

1. git clone https://github.com/steeltoeoss-incubator/observability.git
   

2. cd observability/grafana
   

3. Have a look at what things are provided: PS C:\tmp\observability\grafana> ls

   Name                Description
   ----                ----
   dashboard.json      The Grafana dashboard definition
   dashboard.yml       Grafana dashboard provider
   datasource.yml      Grafana datasource definition for prometheus and influxdb
   docker-compose.yml  Docker file to start all containers
   prometheus.yml      Prometheus scrape configs
   telegraf.conf       Telegraf inputs and output configuration
   

Next, create a .NET Core WebAPI with the correct Steeltoe dependencies

1. Create a new ASP.NET Core WebAPI app with the Steeltoe Initializr

2. Name the project "GrafanaObservability"

3. Add the "Actuators" dependency

4. Add the "Dynamic Logging" dependency

5. Add the "Docker" dependency

6. Click Generate to download a zip containing the new project

7. Extract the zipped project and open in your IDE of choice

8. Add the other needed actuators in startup.cs

   using Steeltoe.Management.Endpoint.Metrics;
   using Steeltoe.Management.Tracing;
   using Steeltoe.Management.Exporter.Tracing;
   
   public class Startup {
     public void ConfigureServices(IServiceCollection services) {
       services.AddPrometheusActuator(Configuration);
       services.AddMetricsActuator(Configuration);
       services.AddDistributedTracing(Configuration);
       services.AddZipkinExporter(Configuration);
     }
   
     public void Configure(IApplicationBuilder app, IWebHostEnvironment env) {
       app.UsePrometheusActuator();
       app.UseMetricsActuator();
       app.UseTracingExporter();
     }
   }
   

9. Set the actuator path and exposure addresses in appsettings.json

   {
     "Logging": {
       "LogLevel": {
         "Default": "Debug",
         "System": "Information",
         "Microsoft": "Information"
       }
     },
     "spring": {
       "application": {
         "name": "GrafanaObservability"
       }
     },
     "management": {
       "endpoints": {
         "actuator": {
           "exposure": {
             "include": ["*"]
           }
         }
       },
       "metrics": {
         "exporter": {
           "cloudfoundry": {
             "validateCertificates": false
           }
         }
       },
       "tracing": {
         "alwaysSample": true,
         "useShortTraceIds ": true,
         "exporter": {
           "zipkin": {
             "endpoint": "http://zipkin:9411/api/v2/spans",
             "validateCertificates": false
           }
         }
       }
     }
   }
   

10. Adjust docker-compose.yml to include the path to the .NET project by replacing <ABSOLUTE_PATH_TO_PROJECT> with the absolute path to the folder holding the .csproj file.

Next, deploy everything with docker compose

1. Build the image using the provided docker-compose file.

   PS C:\tmp\observability\grafana> docker-compose up -d
   

   Note

   If you get a permissionerror message about "db.lock" close your IDE. Then run docker-compose command again.

2. Confirm everything started successfully by running docker-compose ps and checking the State. Output should look similar to this:

   Name           Command                        State     Ports
   -----------------------------------------------------------------------------------------------------------
   grafana        /run.sh                          Up      0.0.0.0:3000->3000/tcp
   influxdb       /entrypoint.sh influxd           Up      0.0.0.0:8086->8086/tcp
   prometheus     /bin/prometheus --config.f ...   Up      0.0.0.0:9090->9090/tcp
   steeltoe-app   dotnet Grafana_Observabili ...   Up      0.0.0.0:80->80/tcp
   telegraf       /entrypoint.sh --config=/e ...   Up      8092/udp, 8094/tcp, 8125/udp, 0.0.0.0:9273->9273/tcp
   zipkin         /busybox/sh run.sh               Up      0.0.0.0:9411->9411/tcp
   

Finally, navigate to Grafana to see the default dashboard showing the app's metrics

Grafana is available at http://localhost:3000

Note

The default username is admin and the password is admin.