Solution

Triage

1. Check SELinux status:

   getenforce
   sestatus
   

2. Check audit log for denials:

   ausearch -m AVC -ts recent
   grep "inventory" /var/log/audit/audit.log | grep denied
   

3. Check file contexts:

   ls -lZ /opt/inventory-api/bin/server
   ls -ldZ /opt/inventory-api/
   

4. Translate the denial:

   ausearch -m AVC -ts recent | audit2why
   

Root Cause

The binary /opt/inventory-api/bin/server was deployed by copying files to /opt/inventory-api/. Files in /opt inherit the SELinux context usr_t by default, not bin_t. When the systemd service tries to execute the binary, SELinux denies the execution because the init_t domain (systemd) is not allowed to execute files labeled usr_t.

Additionally, the application tries to: 1. Bind to port 9090, which is not in the default http_port_t SELinux type. 2. Read configuration from /opt/inventory-api/config/, which has the usr_t context.

All three denials must be resolved for the service to start.

Fix

1. Set the correct file context for the binary:

   semanage fcontext -a -t bin_t '/opt/inventory-api/bin(/.*)?'
   restorecon -Rv /opt/inventory-api/bin/
   

2. Set the correct context for config and data directories:

   semanage fcontext -a -t etc_t '/opt/inventory-api/config(/.*)?'
   restorecon -Rv /opt/inventory-api/config/
   

3. Allow the service to bind to port 9090:

   semanage port -a -t http_port_t -p tcp 9090
   

4. If additional denials exist, generate a custom policy module:

   # Collect all denials in permissive mode first:
   semanage permissive -a init_t
   systemctl start inventory-api
   # Let it run for a few minutes to collect all denials
   ausearch -m AVC -ts recent | audit2allow -M inventory-api
   semodule -i inventory-api.pp
   # Remove permissive exception:
   semanage permissive -d init_t
   

5. Start the service:

   systemctl start inventory-api
   systemctl status inventory-api
   

Rollback / Safety

• File context changes are persistent and survive reboots.
• If the custom policy module causes issues, remove it: semodule -r inventory-api.
• To test without committing, use permissive mode for the specific domain rather than the entire system.
• Keep SELinux in enforcing mode. Never disable it permanently in production.

Common Traps

• Disabling SELinux as the fix. This is the most common and worst response. SELinux is a critical security control.
• Using chcon instead of semanage fcontext + restorecon. chcon changes are temporary and lost when restorecon is run or during filesystem relabel.
• Running audit2allow blindly. audit2allow may generate overly permissive policies. Always review the generated policy before installing.
• Not collecting all denials. Fix one denial, restart, hit another denial, repeat. Use permissive mode for the domain to collect ALL denials at once.
• Forgetting restorecon after semanage fcontext. semanage fcontext updates the policy database but does not change file labels on disk. restorecon applies the policy to existing files.