Offline docs (switch to live docs)

About MAAS logging

Errors or typos? Topics missing? Hard to read? Let us know.

This page describes basic MAAS logging operations. For more details, refer to these reference pages:

Logging updates in MAAS 3.5

In 3.5, the MAAS snap uses the pebble service manager instead of supervisord. This means that the systemd component was renamed to snap.maas.pebble.service. It contains Pebble logs, as well as intercepted stdout of the services running under Pebble. The region and rack logs are cached there, as well (i.e., regiond.log and rackd.log are no more (supervisord was redirecting the stdout of the respective services).

Here is a per-service breakdown how logging works in MAAS 3.5:

Pebble

Pebble logs to stdout, redirecting the services to stdout if run with --verbose (currently in use). It logs additional debug information when run with envvar PEBBLE_DEBUG=1 (currently in use).

Log entry format

2023-07-24T11:12:25.495Z [pebble] GET /v1/services?names=bind9 57.716µs
      2023-07-24T11:12:26.392Z [SERVICE_NAME] SERVICE STDOUT

Commands to access the log

To access only the pebble logs:

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[pebble\]"

To access pebble and its services logs:

journalctl -u snap.maas.pebble.service

Regiond

Regiond logs to stdout, redirected to journalctl by Pebble.

Own log format

2023-07-25 07:23:28 maasserver.rpc.regionservice: [info] Message

Pebble-proxied log format

2023-07-25T07:23:28.730Z [regiond] 2023-07-25 07:23:28 maasserver.rpc.regionservice: [info] Message

Commands to access the log

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[regiond\]"

Rackd

Rackd logs to stdout, redirected to journalctl by Pebble.

Own log format

2023-06-23 13:54:46 provisioningserver.rpc.clusterservice: [info] Message

Pebble-proxied log format

2023-06-23T13:54:46.391Z [rackd] 2023-06-23 13:54:46 provisioningserver.rpc.clusterservice: [info] Message

Commands to access the log

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[rackd\]"

API Server

The API server logs to stdout, redirected to journalctl by Pebble.

Own log format

INFO:     Started server process [24428]

Pebble-proxied log format

2023-07-24T10:25:37.602Z [apiserver] INFO:     Started server process [24428]

Commands to access the log

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[apiserver\]"

HTTP (nginx)

HTTP logs to its own files (/var/snap/maas/common/log/http/(access|error).log), while stdout (redirected by Pebble) contains only critical errors that cannot be redirected to the error log.

Pebble-proxied log format

2023-06-23T13:54:46.391Z [http] nginx: [alert] could not open error log file

Commands to access the log

Own logs

less /var/snap/maas/common/log/http/access.log (or error.log)

`stdout` redirected by Pebble:

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[http\]"

Proxy (squid)

The squid proxy logs to its own files (/var/snap/maas/common/log/proxy/(access|cache|store).log), while stdout (redirected by Pebble) contains init messages and errors.

Pebble proxied log format:

2023-06-23T13:54:41.114Z [proxy] 2023/06/23 13:54:41| Starting Squid Cache version 5.2

Commands to access the log

Own logs

less /var/snap/maas/common/log/proxy/access.log (or other log)

`stdout` redirected by Pebble:

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[proxy\]"

NTP (chrony)

At first glance, chrony seems to have a separate directory for log files (/var/snap/maas/common/log/chrony), but no log files are typically present there. Instead, used stdout (redirected by Pebble), which contains init message and errors.

Pebble proxied log format:

2023-06-26T12:48:01.272Z [ntp] 2023-06-26T12:48:01Z chronyd version 4.2 starting

Own log format

2023-06-26T12:48:01Z chronyd version 4.2 starting

Commands to access the log

`stdout` redirected by Pebble:

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[ntp\]"

DNS (bind9)

DNS logs everything to stdout, ignoring any configuration parameters defined in the snap. As usual, stdout is redirected by Pebble.

Pebble proxied log format:

2023-06-23T13:54:43.268Z [bind9] 23-Jun-2023 13:54:43.264 BIND 9 is maintained by Internet Systems Consortium

Own log format

23-Jun-2023 13:54:43.264 BIND 9 is maintained by Internet Systems Consortium

Commands to access the log

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[bind9\]"

Syslog (rsyslog)

Syslog logs machines enlistment and boot syslogs to /var/snap/maas/common/log/rsyslog/%HOSTNAME%/, and logs various MAAS-tagged records to /var/snap/maas/common/log/maas.log. Otherwise, stdout (redirected by Pebble) contains internal messages/errors.

Pebble proxied log format:

2023-07-24T05:38:56.522Z [syslog] Message

Own log format

<some message>

Commands to access the log

Own logs

less /var/snap/maas/common/log/rsyslog/PATH/TO/LOG

`stdout` redirected by Pebble:

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[syslog\]"

DHCP (dhcpd, dhcpd6)

DHCP logs everything through syslogd, while stdout (redirected by Pebble) contains init messages. There is no particularly easy way to differentiate between dhcpd and dhcpd6, although you can grep for a “PID file: /var/snap/maas/common/maas/dhcp/dhcpd6?.pid” message, and then use the PID to filter logs in journal, like this:

journalctl -et dhcpd -u snap.maas.pebble.service SYSLOG_PID=25799

Pebble proxied log format:

2023-06-27T11:24:47.674Z [dhcpd] Internet Systems Consortium DHCP Server 4.4.1

Own log format

<plain old DHCP message>

Commands to access the log

Own logs

journalctl -t dhcpd -u snap.maas.pebble.service

`stdout` redirected by Pebble:

journalctl -u snap.maas.pebble.service --case-sensitive -g "^[0-9TZ:.-]{24} \[dhcpd\]"

Remote syslog (3.4 UI)

To enable remote logging, choose Settings > Network > Syslog, and enter your syslog server’s IP/URL under Remote syslog server to forward machine logs.

Remote syslog (3.3– UI)

To enable remote logging, choose Settings > Network services > Syslog; add your syslog server’s IP/UL; and click Save.

Remote syslog (CLI)

maas $PROFILE maas set-config name="remote_syslog" value="$SYSLOG_FQDN"
      # Example for setting syslog server to 192.168.100.11:
      maas $PROFILE maas set-config name="remote_syslog" value=192.168.100.11
      # To reset to default (sending syslog to MAAS region controllers):
      maas $PROFILE maas set-config name="remote_syslog" value="

Note: Only machine syslog information is forwarded, not MAAS controllers syslog files.

Direct log access

Logs can be found at the following paths depending on your installation type (snap or package):

Snap installation:
- /var/snap/maas/common/log/maas.log
- /var/snap/maas/common/log/regiond.log
- /var/snap/maas/common/log/rackd.log
- /var/snap/maas/common/log/rsyslog/MACHINE_NAME/RELEVANT_DATE/messages
Package installation:
- /var/log/maas/maas.log
- /var/log/maas/regiond.log
- /var/log/maas/rackd.log
- /var/log/maas/rsyslog/MACHINE_NAME/RELEVANT_DATE/messages

Logs can be extensive and challenging to search. The MAAS web UI does not categorise events by type.

Events query (CLI)

The most efficient way to review events is using the events query CLI sub-command. This command allows filtering and summarising events. Use jq and various filters to simplify output interpretation.

Basic queries

maas $PROFILE events query

This command returns a lengthy JSON output, which can be simplified using jq and various filters, making it more readable.

Using jq with events

A jq command example for readable output:

maas $PROFILE events query limit=20 | jq -r '(["USERNAME","NODE","HOSTNAME","LEVEL","DATE","TYPE","EVENT"] | (., map(length*"-"))), (.events[] | [.username,.node,.hostname,.level,.created,.type,.description]) | @tsv' | column -t -s$'\t'

Filter parameters

The events query command supports multiple filters:

hostname: Events for a specific node hostname.
mac_address: Events for nodes with specified MAC addresses.
id: Events for nodes with specific system IDs.
zone: Events for nodes in a particular zone.
level: Filter by event level (AUDIT, CRITICAL, DEBUG, ERROR, INFO, WARNING).
limit: Maximum number of events to return.
before/after: Start returning events before or after a specific event ID.

Example usage of these filters can narrow down event listings significantly.

Auditing finesse

Audit events, tagged with AUDIT, record MAAS configuration changes and machine state transitions. They’re essential for tracking user actions and system updates, especially in multi-user environments.

Use audit events alongside jq and command-line text tools to analyse actions like machine deletions, configuration changes, and user activities. This can provide insights into system changes and help identify areas for attention or improvement.