Logging gunicorn messages through journald to syslog-ng

by Sebastien Mirolo on Wed, 22 Dec 2021

We are interested here to log messages from a gunicorn application to journald, eventually forwarding the messages to syslog-ng.

gunicorn.conf

The first step is write the gunicorn messages to stdout and stderr.

Terminal

$ cat /etc/gunicorn/webapp.conf
...
accesslog="-"
errorlog="-"
...

Systemd service

There are often problems with Type=forking when it comes to journald logging. We thus do not daemonize gunicorn and use a Type=simple.

Since we will forward the log to syslog-ng, we want to make sure the entries will appear with the identifier we chose so as to filter on it. Hence we define SyslogIdentifier.

Terminal

$ cat /usr/lib/systemd/system/webapp.service
...
[Service]
Type=simple
User=webapp
PIDFile=/var/run/webapp/webapp.pid
EnvironmentFile=-/etc/sysconfig/webapp
WorkingDirectory=/app/reps/webapp
ExecStart=/app/bin/gunicorn -c /etc/gunicorn/webapp.conf webapp.wsgi
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s QUIT $MAINPID
SyslogIdentifier=app.webapp
PrivateTmp=true
...

journald.conf

Forwarding journald messages to syslog is straigtforward.

Terminal

$ diff -u prev /etc/systemd/journald.conf
...
-#ForwardToSyslog=no
+ForwardToSyslog=yes
...

syslog-ng.conf

We add a filter to write gunicorn forwarded message to a file, and a rule to exclude them from being written in the default logfile.

Terminal

$ cat /etc/syslog-ng/conf.d/webapp.conf
...
filter f_webapp { program("app.webapp"); };

destination d_webapp { file("/var/log/gunicorn/app.webapp.log"); };

log { source(s_sys); filter(f_webapp); destination(d_webapp); };

Terminal

$ diff -u prev /etc/syslog-ng/syslog-ng.conf
...
  filter f_default    { level(info..emerg) and
+                       not (program("app.*")
                        or facility(mail)
                        or facility(authpriv)
                        or facility(cron)); };
...

logrotate.conf

Finally since log files are rotated, we add a rule for the log files defined earlier (/var/log/gunicorn/webapp.log) in /etc/logrotate.d/syslog.

Terminal

$ diff -u prev /etc/logrotate.d/syslog
  /var/log/cron
  /var/log/maillog
  /var/log/messages
  /var/log/secure
  /var/log/spooler
+ /var/log/gunicorn/app.*.log
  {
...

Voila, we just need to restart all the services involved.

Terminal

systemctl daemon-reload
systemctl restart syslog-ng.service
systemctl restart webapp.service

to debug, we look at the journal and the final log file.

Terminal

sudo journalctl --since "5 min ago" -l _SYSTEMD_UNIT=webapp.service
tail -f /var/log/gunicorn/app.webapp.log

Logging gunicorn messages through journald to syslog-ng

gunicorn.conf

Systemd service

journald.conf

syslog-ng.conf

logrotate.conf

More to read