15 April 2016

Datadog Integration with Postgres/MongoDB/Apache/Elasticsearch

Datadog has integration with almost any popular services. I have integrated our Datadog to specifically monitor our needed services.

Here is how I did it.

Step 1: Datadog Agent

Install Datadog agent for your specific OS in your server. You can download the specific agent in the Datadog UI under Integrations -> Agent.

Step 2: Configure Agent

You need to configure the YAML file for each specific service you would like to be monitored.

Postgres:

1. Create a read-only Postgres role with access to pg_stat_database

In our case, we have a 3 servers clustered Postgres with PG-Pool. So you need to create the Postgres role and PG-Pool user in each 3 servers.

A.  Create PG role
Connect to your pg instance in each server and run following queries: 
$ psql -h server -U postgres -W
postgres# create user datadog with password 'datadog';
postgres# grant SELECT ON pg_stat_database to datadog;
B. Create encrypted password with MD5 for PG-Pool
Login to each server and run pg_md5:
$ sudo pg_md5 -p -m -u datadog
Supply the same password as you've provided during creating the role.

This command will create encrypted MD5 hash for the user and updates the following pg-pool config:
$ cat /etc/pgpool-II/pool_passwd
postgres:md534fd2d6b187dh4e693heefabd348hdg528
pg_pgpool:md534fsfbc26e4bdda8c0ed3dc468hdb659d
nrpe:md5dcs3458a49d39a90859f6d389b5399hdg638
datadog:md5f34ce5fd8ab74af35d881a9fa60jd73
Test if you are able to connect with your created role:
$ psql -h server -U datadog postgres -c \
"select * from pg_stat_database LIMIT(1);"
&& echo -e "\e[0;32mPostgres connection - OK\e[0m" || \ 
|| echo -e "\e[0;31mCannot connect to Postgres\e[0m"
2. Configure the Datadog agent

postgres.yaml
Create or open datadog postgres.yaml config file:
$ sudo vi /etc/dd-agent/conf.d/postgres.yaml
Fill it as per your environment values

init_config:

instances:
  - host: <IP>
    port: 5432
    username: datadog
    password: datadog
    ssl: true

In my case I had to define the config file in a way that datadog connects to my postgres via SSL with ssl: true as connections been defined to be always SSL encrypted to my postgres. When I didn't define the config file in this way for the first time, I received this error when Datadog tried to connect to postgres:
Error: md5 authentication for user datadog failed.
Based on my pool_hba.conf file, all connections to all databases from all sources need to be md5 encrypted:
$ sudo cat /etc/pgpool-II/pool_hba.conf
# TYPE  DATABASE    USER        CIDR-ADDRESS          METHOD
# "local" is for Unix domain socket connections only
local   all         postgres                          trust
local   all         all                               md5
# IPv4 local connections:
host    all         postgres    127.0.0.1/32          trust
host    all         all         127.0.0.1/32          md5
host    all         all         0.0.0.0/0             md5
3. Restart the Datadog agent and check the status:

$ sudo /etc/init.d/datadog-agent restart
$ sudo /etc/init.d/datadog-agent info

MongoDB:

Same as Postgres, we need a user with read only role on admin database.

1. Create Mongo user
Connect to mongo server on admin database and create the datadog user:

$ mongo <server>/admin -ssl --sslAllowInvalidCertificates -u root -p <passwd>

> db.createUser({
  "user":"datadog",
  "pwd": "datadog",
  "roles" : [
    {role: 'read', db: 'admin' },
    {role: 'clusterMonitor', db: 'admin'},
    {role: 'read', db: 'local' }
  ]
})

2. Configure mongo.yaml

sudo vi /etc/dd-agent/conf.d/mongo.yaml
Fill it as per your environment values

init_config:

instances:
  - server: mongodb://datadog:datadog@localhost:27017/admin

    ssl: true
    ssl_cert_reqs: 0
    additional_metrics:
      - durability
      - locks
      - top
In my case, mongo is using SSL but certificate has expired. So I had to define ssl: true and ssl_cert_reqs: 0 so it ingonres the SSL cert. Fail to define that, I received `Mongodb: Connection Closed` error in datadog.

3. Restart the Datadog agent and check the status:
$ sudo /etc/init.d/datadog-agent restart
$ sudo /etc/init.d/datadog-agent info

Apache:

In case of Apache, it is quiet straight forward except you need to make sure you have mod_status loaded in your apache service. Another requirement is that ExtendedStatus need to be set to on, which is set on by default in apache version 2.3.6 onward. 

1. Configure apache.yaml

sudo vi /etc/dd-agent/conf.d/apache.yaml
Fill it as per your environment values

init_config:

instances:
  - apache_status_url: http://<IP>:6666/server-status

    disable_ssl_validation: true
2. Restart the Datadog agent and check the status:

$ sudo /etc/init.d/datadog-agent restart
$ sudo /etc/init.d/datadog-agent info

Elasticsearch

There is no trick here for Elasticsearch integrations.

Configure and restart the agent:
$ sudo vim /etc/dd-agent/conf.d/elastic.yaml
Fill it as per your environment values
init_config: null
instances:
-   tags:
    - role:production
    url: http://localhost:9200

$ sudo /etc/init.d/datadog-agent restart
$ sudo /etc/init.d/datadog-agent info