TrueNAS SCALE Angelfish –> Bluefin Docker/SMB Migration Issues

TrueNAS Scale Bluefin has now completed beta testing and a release candidate is available to the public ready for ‘home use’.
I’ve personably been using the Angelfish train (22.02.4) since it’s RC2 which was released in December of last year.

Bluefin promises to add many new capabilities to your TrueNAS machine including:

  • Scale-Up Performance Improvements
  • Cluster Feature Improvements
  • Active-Active reliability for storage
  • Linux-based Security Enhancements
  • Enterprise Support

 

Earlier this week I upgraded both my TrueNAS servers from 22.02.4 to 22.12-RC1, the upgrade process itself completed without issues, however I was left with a handful of docker containers stuck in the ‘Deploying phase’ without any explanation…

TrueNAS_SCALE_Bluefin-300x221

The Issue

 

Once the update was completed and the TrueNAS server restarted, all docker containers re-deploy and attempt to run. In my case Sonarr and Plex got stuck on the ‘Deploying’ phase.
By looking at the logs for the docker container, I could see there was a fundamental issue with the configuration of the app.

{“kind”:”Status”,”apiVersion”:”v1″,”metadata”:{},”status”:”Failure”,”message”:”container \”CONTAINER_NAME” in pod \”POD_NAME” is waiting to start: CreateContainerError”,”reason”:”BadRequest”,”code”:400}

The Fix

 

The only thing in common with the two docker containers that refused to deploy was their host path mapping. Both Plex and Sonarr had been setup with my media dataset mounted to their container.
Removing this host path allowed the containers to deploy and run.

My initial thoughts were that the permissions on the ACL for this dataset must have changed during the upgrade and now prevent the app user from accessing it.

However, setting the ACLs to ‘Everyone’ did not solve the issue, something else was going on.

SMB

 

The vast majority of my docker containers have host paths mounted to them, but what was different about this media dataset that would cause these issues?
After scouring the TrueNAS forums I was able to find this post by user sgt_jamez, this user also experienced the same issues after moving to the nighty’s back in June of 2022.

Eventually someone in the replies realised that this issue only occurred if the host path mounted was also used as a SMB share.

As it happens, the host path I was having issues with was also a SMB share on my system.

 

Disabling the SMB share and then re-deploying the docker apps resulted in them fully deploying and running this time. Obviously, this is just a workaround and not a proper solution. (Once the app was running, re-enabling the SMB share is possible without issues).

Screenshot 2022-11-18 155901

iXsystems response

 

iXsystems are the company behind the open source TrueNAS software, fortunately the Senior Vice President Of Engineering, Kris Moore was ready to answer questions regarding the issue.

In this post, Kris explains that this is actually a feature put in place to “prevent foot-shooting when you have SMB shares + ACLS that containers may not play nicely with”.

 

In short, SMB share paths cannot be used as host paths within docker container on the Bluefin train. Unfortunately, there is no check for this when making docker apps (as of 22.12-RC1).

Solutions

There are two solutions to the problem:

  1. Change the SMB share directory.

    This is what I ended up doing, I moved my Plex Media Files down a directory and set the SMB share to the root folder of the dataset. This means that the SMB share is different from the docker host path mount.

  2. Bypass the safety check

    iXsystems have added in a flag to remove the safety check, Kris made in clear in his response that this is to be used at your own risk as. (The command can be seen below)
    (This setting is persistent)

app kubernetes update validate_host_path=false

Previous
Next