Graphite is an open source project to collect and graph data that changes over time. It contains three components: a network server to receive the data, a special data storage format, and a web interface to show graphs. Graphite is developed in Python programming language and is designed to be scalable and to handle high volume data.

Carbon is the server daemon developed using Twisted event driven framework to handle a large amount of traffic for a lot of clients. It receives data packets and stores them in whisper files. The data packets contain a name (called a metric), a numeric value and a timestamp. Carbon receives data over TCP (default port is 2003) in a simple text line format containing the metric name, the value and the timestamp separated by space. Using dots in metric names causes Carbon to create a hierarchy. Carbon sends nothing in response and the connection could be used to send more data.

Carbon has an aggregator and a relay service as well, but the main daemon that receives the data and stores whisper files is carbon-cache. If you need to distribute the data upon arrival (to multiple carbon-caches or integrate with other services), you could configure and use carbon-relay daemon. For better performance Carbon also accepts pickled (Python serialized) data. This format is used between carbon-relay and carbon-cache (default port is 2004).

Whisper files are time series database files (like RRDtool) written specially to store numeric data over time. Each whisper file has a header section for meta data and a set of archive sections. Each archive section stores a sequence of pairs of timestamp and a numeric value.

Graphite web interface reads whisper files and renders graph images. Graphite web interface is written using Python Django web framework and uses Cairo library to generate graph images, so Python Cairo bindings are required to be installed.

It is possible to install Carbon and Graphite separately. By default Carbon and Graphite are installed on the same path on /opt/graphite. This path is called GRAPHITE_ROOT, and most other paths are configured to be relative to this path by default. This path (like other important paths) could be specified using an environment variable. For more information about the project and different installation instructions please refer to graphite documentations. We install Carbon and Graphite using pip, Python package management tool.

sudo pip install django django-tagging
sudo pip install whisper carbon graphite-web

It's recommended to run the service as a separate user:

sudo useradd -d /opt/graphite -s /bin/false graphite

Carbon should be configured using a few config files in $GRAPHITE_CONF_DIR path, which is by default conf directory in $GRAPHITE_ROOT (/opt/graphite/conf). After the installation examples for all config files are available to be used as a base for desired configurations. These config files are required for Carbon to start:

  • carbon.conf: configures behavior of the Carbon services, like setting the process user, port numbers to listen, enable and configure relay, etc.
  • storage-schemas.conf: specification of whisper files, how often to aggregate data points. by default it would keep 1 minute data point for 1 day.

If you need to keep more fine grained data points for a group of metrics, you could define a different storage schema for them. For example we would like to have more data points for our database metrics (whose name start with db.), by keeping 5 second data points for 1 day, aggregated 10 seconds for a week, and 30 second aggregates for a month. Modify storage-schemas.conf file like this:

[carbon]
pattern = ^carbon\.
retentions = 60:90d

[db]
pattern = ^db\.
retentions = 5s:1d, 10s:7d, 30s:30d, 60s:90d

[default_1min_for_1day]
pattern = .*
retentions = 60s:1d

Carbon stores the whisper files in the $GRAPHITE_STORAGE_DIR path, which is by default storage directory under $GRAPHITE_ROOT (/opt/graphite/storage). Both Carbon and Graphite store log files which are by default stored in log directory under this storage path. So It is require that graphite user has write permission to the storage path.

Start the carbon-cache service:

sudo chown -R graphite:graphite /opt/graphite/storage
sudo /opt/graphite/bin/carbon-cache start

When whisper library is installed, it comes with a set of simple command line tools to display data in whisper files. Use whisper-info.py, whisper-dump.py and whisper-fetch.py to view the data and whisper-create.py and whisper-update.py to store data.

But we have Graphite for data visualization. Graphite is configured by /opt/graphite/webapp/graphite/local_settings.py. After installation a sample config file is available, so save a copy of the file with proper name and edit the file. This file is a Python source file that defines variables to configure Graphite. Set the timezone used for the app, set the location for GRAPHITE_ROOT, or custom path to where whisper files are stored. Default values are all configured so they match the default installation of Carbon. The one required part is the database access information in the file. For simplicity we are going to use a sqlite DB file so uncomment these lines (if these lines are not in you config file, append these lines):

DATABASES = {
    'default': {
        'NAME': '/opt/graphite/storage/graphite.db',
        'ENGINE': 'django.db.backends.sqlite3',
        'USER': '',
        'PASSWORD': '',
        'HOST': '',
        'PORT': ''
    }
}

We should create the sqlite file first by using the Django manage.py script:

sudo python /opt/graphite/webapp/graphite/manage.py syncdb

Now the sqlite file is created and ready to be used. Set proper permissions for it and check your Graphite installation by running its development web server:

sudo chown graphite:graphite /opt/graphite/storage/graphite.db
sudo -u graphite -- /opt/graphite/bin/run-graphite-devel-server.py --port=8080 /opt/graphite

Note that this web server is not suitable for production and is only used for development/testing purposes. Graphite (like other Django based web apps) could be served using any WSGI server and a proper WSGI server should be used for production. we are going to use gunicorn in this guide.

Graphite comes with a sample WSGI entry point file in /opt/graphite/conf/graphite.wsgi.example. Copy this file to /opt/graphite/conf/graphite_wsgi.py and then start gunicorn:

sudo pip install gunicorn
cd /opt/graphite/conf
sudo gunicorn -b '0.0.0.0:8080' -u graphite -g graphite -w 2 graphite_wsgi:application

If after installing Graphite, it failed to generate graphs, make sure pycairo is installed on your system. Installing pycairo from a prebuilt binary package (like from the OS package management) is easier since installing it using pip requires building the library from source, So you would need a C compiler, Python headers, Cairo and Cairo development files.

sudo apt-get install python-cairo

And now graphite can render graph images.