Monitor Docker Containers with Prometheus and Grafana

In this article we are going to cover How to Monitor Docker Containers with Prometheus and Grafana |monitoring docker containers with prometheus and grafana

Prerequisites

To follow this tutorial, you will need:

  • AWS Ubuntu 22.04 LTS Instance.
  • User with sudo access (see the Initial Server Setup with Ubuntu 22.04 tutorial for details).
  • Install Docker on Ubuntu Server with required Permission

How to Monitor Docker Containers with Prometheus and Grafana

Step #1:Create EC2, Install, start and enable Docker service through AWS EC2 User data

Launch an EC2 instance. Follow the usual steps for launching an EC2 instance through the AWS Management Console.

Navigate to the “Advanced Details” section. This section is typically located towards the bottom of the launch wizard

Paste your user data script or cloud-init directives in the “User data” field, as shown below

paste the below code

#!/bin/bash

# Update the system packages
sudo apt-get update
sudo apt-get upgrade -y

# Install Docker dependencies
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common

# Add Docker's official GPG key in Ubuntu
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

# Add Docker repository in Ubuntu 
echo "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

# Update package information and install Docker on Ubuntu
sudo apt-get update
sudo apt-get install -y docker-ce docker-ce-cli containerd.io

# Start Docker service
sudo systemctl start docker

# Enable Docker to start on system boot
sudo systemctl enable docker

#Adjust Docker Permissions:
sudo chmod 666 /var/run/docker.sock

once EC2 Instance is ready check docker service

sudo systemctl status docker

Output:

 docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2024-03-01 14:13:23 UTC; 4min 21s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 7699 (dockerd)
      Tasks: 8
     Memory: 27.1M
        CPU: 330ms
     CGroup: /system.slice/docker.service
             └─7699 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Step #2:Create Nginx Docker Containers

Create index.html for Nginx docker Image

sudo nano index.html

paste the below code

<!doctype html>
<html>
 <body style="backgroud-color:rgb(49, 214, 220);"><center>
    <head>
     <title>Docker Project</title>
    </head>
    <body>
     <p>Welcome to my Docker Project!<p>
        <p>Today's Date and Time is: <span id='date-time'></span><p>
        <script>
             var dateAndTime = new Date();
             document.getElementById('date-time').innerHTML=dateAndTime.toLocaleString();
        </script>
        </body>
</html>

Save and exit the index.html file

Create Dockerfile file Nginx

sudo nano Dockerfile
FROM nginx:1.10.1-alpine
COPY index.html /usr/share/nginx/html
EXPOSE 8080
CMD ["nginx", "-g", "daemon off;"]

Save and exit the Dockerfile file

Build the Docker image using the following command:

docker build -t nginx .

Run the Docker container using the following command

 docker run -d -p 8080:80 nginx

check the Nginx Docker container running status

docker ps

Output:

check nginx docker container running status
http://65.2.187.33:8080/

Output:

run nginx docker container on browser

Step #3:Run Prometheus using Docker Container

Create prometheus.yaml file

sudo nano prometheus.yaml

paste the below configs


global:
  scrape_interval: 15s # By default, scrape targets every 15 seconds.

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
    monitor: 'Docker-monitor'

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # Override global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s

    static_configs:
      - targets: ['SERVER_IP:9090']

save this file and exit.

Run Prometheus Docker Container

sudo docker run -d --name prometheus -v $(pwd):/etc/config -p 9090:9090 prom/prometheus --config.file=/etc/config/prometheus.yaml

Check prometheus docker container running status

docker ps

Output:

check prometheus docker container running status

if prometheus docker not running, then run below command to start prometheus docker container

docker run -p 9090:9090 -d prom/prometheus 

Access Prometheus Page through Browser

access prometheus on browser

Step #3:Integrate Docker Container with Grafana

Locate the configuration file of the Docker daemon, daemon.json. The file is usually found in the /etc/docker/ directory.

If the daemon.json file is not present in the specified location, create a new file named daemon.json.

Open the daemon.json file using a text editor.

sudo nano /etc/docker/daemon.json

Add the following code snippet to the daemon.json file:

{
        "metrics-addr" : "0.0.0.0:9323",
        "experimental" : true
}

save and exit the file.

For apply this change you have to restart docker service 

sudo systemctl restart docker

Start prometheus and nginx docker containers again

docker run -p 9090:9090 -d prom/prometheus 
docker run -d -p 8080:80 nginx

Now wait some time and then check logs on

http://<public-ip>:9323/metrics

You should get docker logs on webpage like following.

prometheus docker metrics

Step #4:Add New Docker Job Targets in Prometheus

list all docker containers

docker ps

Output:

docker ps

Edit the prometheus.yml located in prom/prometheus container

docker exec -it 35f48c303e80 /bin/sh

Go to Prometheus config file location in Prometheus container

cd /etc/prometheus
vi prometheus.yml

Add Docker job in prometheus.yaml file

- job_name: 'docker'
  scrape_interval: 5s
  static_configs:
    - targets: ['65.2.187.33:9323']

add docker targets in prometheus configs

Exit after saving

exit

After above step you have to restart Prometheus docker container

docker restart <container id>

The Target of docker will added in prometheus targets section.

docker container targets in prometheus

Step #5:Install and Run Grafana using Docker

  • To run the latest stable version of Grafana, run the following command
docker run -d -p 3000:3000 --name=grafana grafana/grafana

Output:

run grafana using docker container

Check Grafana Container running status

docker ps

Output:

check grafana container running status

login to Grafana using Grafana default username and password: admin

grafana default username and password

Step #6:Add Prometheus Data Source in Grafana

Access the Configuration menu: Click on the gear icon (cogwheel) in the Grafana sidebar to open the Configuration menu.

Open the Data Sources section: Navigate to the Data Sources section by clicking on it within the Configuration menu.

Add a new data source: Click on the Add data source button.

Select Prometheus data source type: From the list of available data source types, choose Prometheus.

Configure the data source settings:

URL: Enter the URL of your Prometheus server. The default URL is typically http://localhost:9090/.

Access: You can choose the access method for your Prometheus server. The options may include “Direct” or “Proxy” depending on your setup. Refer to the Grafana documentation for detailed information on access methods.

Save and test the data source: Click the Save & Test button to save the newly configured Prometheus data source and test the connection.

Once the test is successful, you can use the Prometheus data source to create dashboards and visualize metrics in Grafana.

add prometheus data sources in grafana

save and test.

save prometheus data sources in grafana

Step #7:Monitor Docker Containers with Grafana

Below are example Prometheus Docker Metrics

  • engine_daemmon_engine_cpus_cpus – Shows number of Engine CPU metrics
  • engine_daemon_container_states_containers{state=”stopped”} – Shows stopped containers
  • engine_daemon_container_states_containers{state=”paused”} – Shows paused containers
  • engine_daemon_container_states_containers{state=”running”} – Shows running containers

Click on Dashboards and then click on Create Dashboard -> Add Visualization -> select Prometheus Data Source

In the Metrics browser selection select engine_daemmon_engine_cpus_cpus then select job and job name is docker. Then click Run queries will fetch the number of Engine CPU.

create docker metrics dashoard in grafana

To create Grafana Dashboard for number of containers running, follow above steps and add below prometheis metrics as shown below

engine_daemon_container_states_containers{state=”running”} – Shows running containers

check number of Containers in Grafana

Conclusion:

We have covered How to Monitor Docker Containers with Prometheus and Grafana |monitoring docker containers with prometheus and grafana.

Related Articles:

How to Install Prometheus on Ubuntu 22.04 LTS

Secure Grafana with Nginx, Reverse Proxy and Certbot

Reference:

Grafana Installation official page

About DevOps Hint

DevOps Hint Founded in 2020 Community Site where you can find about How to Guides, Articles and Troubleshooting Tips for DevOps, SRE, Sysadmins and Developers.

Leave a Comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Share via
Copy link