SELinux Troubleshooting¶
When installing Nautobot for the first time on a Redhat-based Linux Distribution, SELinux may prevent the Nautobot stack from working properly. An example is SELinux preventing the HTTP daemon (NGINX, et al.) from communicating to the Django application stack on the backend.
Determine if SELinux is the Culprit¶
An example of a broken application can be seen in the Nginx error logs below:
$ sudo tail -f /var/log/nginx/error.log
2021/02/26 15:16:55 [crit] 67245#0: *494 connect() to 127.0.0.1:8080 failed (13: Permission denied) while connecting to upstream, client: 47.221.167.40, server: nautobot.example.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "nautobot.example.com"
2021/02/26 15:16:56 [crit] 67245#0: *494 connect() to 127.0.0.1:8080 failed (13: Permission denied) while connecting to upstream, client: 47.221.167.40, server: nautobot.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "nautobot.example.com", referrer: "https://nautobot.example.com/"
2021/02/26 15:16:58 [crit] 67245#0: *544 connect() to 127.0.0.1:8080 failed (13: Permission denied) while connecting to upstream, client: 47.221.167.40, server: nautobot.example.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "nautobot.example.com"
2021/02/26 15:16:58 [crit] 67245#0: *544 connect() to 127.0.0.1:8080 failed (13: Permission denied) while connecting to upstream, client: 47.221.167.40, server: nautobot.example.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "nautobot.example.com", referrer: "https://nautobot.example.com/"
A quick way to verify that SELinux is preventing the application from working is to first, verify that SELinux is indeed enabled and operating in enforcing mode, and second, temporarily put SELinux in permissive mode. With SELinux in permissive mode, the application stack can be tested again. If the application starts working as expected, then SELinux is most likely the culprit.
To put SELinux in permissive mode, execute the setenforce command with the 0 flag.
# setenforce 0
# sestatus | egrep 'SELinux status|Current mode'
SELinux status: enabled
Current mode: permissive
With SELinux in permissive mode, test the application stack and ensure everything is working properly. If the application is working, put SELinux back into enforcing mode. This is done by executing the setenforce command with the 1 flag.
# setenforce 1
# sestatus | egrep 'SELinux status|Current mode'
SELinux status: enabled
Current mode: enforcing
Troubleshoot SELinux¶
Troubleshooting SELinux in most instances is straightforward. Using the sealert command to parse /var/log/audit/audit.log is the fastest way to pin-point SELinux specific issues. In many cases, sealert will also provide guidance as to how to resolve the issue.
# sealert -a /var/log/audit/audit.log
100% done
found 1 alerts in /var/log/audit/audit.log
--------------------------------------------------------------------------------
SELinux is preventing /usr/sbin/nginx from name_connect access on the tcp_socket port 8080.
***** Plugin connect_ports (85.9 confidence) suggests *********************
If you want to allow /usr/sbin/nginx to connect to network port 8080
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 8080
where PORT_TYPE is one of the following: dns_port_t, dnssec_port_t, kerberos_port_t, ocsp_port_t.
***** Plugin catchall_boolean (7.33 confidence) suggests ******************
If you want to allow httpd to can network connect
Then you must tell SELinux about this by enabling the 'httpd_can_network_connect' boolean.
Do
setsebool -P httpd_can_network_connect 1
***** Plugin catchall_boolean (7.33 confidence) suggests ******************
If you want to allow nis to enabled
Then you must tell SELinux about this by enabling the 'nis_enabled' boolean.
Do
setsebool -P nis_enabled 1
***** Plugin catchall (1.35 confidence) suggests **************************
If you believe that nginx should be allowed name_connect access on the port 8080 tcp_socket by default.
Then you should report this as a bug.
You can generate a local policy module to allow this access.
Do
allow this access for now by executing:
# ausearch -c 'nginx' --raw | audit2allow -M my-nginx
# semodule -X 300 -i my-nginx.pp
Additional Information:
Source Context system_u:system_r:httpd_t:s0
Target Context system_u:object_r:unreserved_port_t:s0
Target Objects port 8080 [ tcp_socket ]
Source nginx
Source Path /usr/sbin/nginx
Port 8080
Host <Unknown>
Source RPM Packages nginx-1.14.1-9.module_el8.0.0+184+e34fea82.x86_64
Target RPM Packages
SELinux Policy RPM selinux-policy-targeted-3.14.3-54.el8_3.2.noarch
Local Policy RPM selinux-policy-targeted-3.14.3-54.el8_3.2.noarch
Selinux Enabled True
Policy Type targeted
Enforcing Mode Enforcing
Host Name nautobot.example.com
Platform Linux nautobot.example.com
4.18.0-240.1.1.el8_3.x86_64 #1 SMP Thu Nov 19
17:20:08 UTC 2020 x86_64 x86_64
Alert Count 5
First Seen 2021-02-26 15:16:55 UTC
Last Seen 2021-02-26 15:23:12 UTC
Local ID b83bb817-85f6-4f5c-b6e0-eee3acc85504
Raw Audit Messages
type=AVC msg=audit(1614352992.209:585): avc: denied { name_connect } for pid=67245 comm="nginx" dest=8080 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:unreserved_port_t:s0 tclass=tcp_socket permissive=0
type=SYSCALL msg=audit(1614352992.209:585): arch=x86_64 syscall=connect success=no exit=EACCES a0=12 a1=55d061477358 a2=10 a3=7ffc0c62296c items=0 ppid=67243 pid=67245 auid=4294967295 uid=988 gid=985 euid=988 suid=988 fsuid=988 egid=985 sgid=985 fsgid=985 tty=(none) ses=4294967295 comm=nginx exe=/usr/sbin/nginx subj=system_u:system_r:httpd_t:s0 key=(null)ARCH=x86_64 SYSCALL=connect AUID=unset UID=nginx GID=nginx EUID=nginx SUID=nginx FSUID=nginx EGID=nginx SGID=nginx FSGID=nginx
Hash: nginx,httpd_t,unreserved_port_t,tcp_socket,name_connect
In the first few lines of the audit, sealert details what SELinux is blocking and provides some options to remedy the issue.
Since Nginx is communicating with the Nautobot application via HTTP, the second option is the best option.
SELinux is preventing /usr/sbin/nginx from name_connect access on the tcp_socket port 8080.
***** Plugin connect_ports (85.9 confidence) suggests *********************
If you want to allow /usr/sbin/nginx to connect to network port 8080
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 8080
where PORT_TYPE is one of the following: dns_port_t, dnssec_port_t, kerberos_port_t, ocsp_port_t.
***** Plugin catchall_boolean (7.33 confidence) suggests ******************
If you want to allow httpd to can network connect
Then you must tell SELinux about this by enabling the 'httpd_can_network_connect' boolean.
Do
setsebool -P httpd_can_network_connect 1
Executing setsebool -P httpd_can_network_connect 1 should remedy the issue. Verify this by executing the setsebool command, verify that SELinux is enabled and in enforcing mode via the sestatus command, and test the application stack for functionality.
The first curl command demonstrates the failure. Nginx responds with a HTTP response code of 502, indicating that it is unable to communicate with the Nautobot application. After executing the setsebool command, curl is used again to verify that Nginx is able to communicate with the Nautobot application. This is verified with the HTTP response code of 200.
# curl -ik https://nautobot.example.com
HTTP/1.1 502 Bad Gateway
Server: nginx/1.14.1
Date: Fri, 26 Feb 2021 15:41:22 GMT
Content-Type: text/html
Content-Length: 173
Connection: keep-alive
# sestatus
SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: enforcing
Mode from config file: enforcing
Policy MLS status: enabled
Policy deny_unknown status: allowed
Memory protection checking: actual (secure)
Max kernel policy version: 32
# setsebool -P httpd_can_network_connect 1
# curl -ik https://nautobot.example.com
HTTP/1.1 200 OK
Server: nginx/1.14.1
Date: Fri, 26 Feb 2021 15:41:49 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 18698
Connection: keep-alive
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
X-Frame-Options: DENY
Vary: Cookie, Origin