docs: recommend RKNN_THREADS=3 for multi-core Rockchip NPUs

Document shared-weight multi-context pools: keep WORKERS=1, set threads
to 3, and raise Immich job concurrency so each NPU core can stay busy.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
todorangrg 2026-07-31 15:46:23 +02:00
parent deeaea00c8
commit 7c9bc94383
2 changed files with 6 additions and 4 deletions

View file

@ -83,7 +83,9 @@ You do not need to redo any machine learning jobs after enabling hardware accele
- RKNPU driver V0.9.8 or later must be available in the host server
- You may confirm this by running `cat /sys/kernel/debug/rknpu/version` to check the version
- Optional: Configure your `.env` file, see [environment variables](/install/environment-variables) for RKNN specific settings
- In particular, setting `MACHINE_LEARNING_RKNN_THREADS` to 2 or 3 can _dramatically_ improve performance for RK3576 and RK3588 compared to the default of 1, at the expense of multiplying the amount of RAM each model uses by that amount.
- For RK3576 and RK3588, set `MACHINE_LEARNING_RKNN_THREADS=3` so each of the three NPU cores can service a concurrent request. Contexts share model weights, so RAM barely grows versus the single-core default while sustained throughput can approach **3×**.
- Keep `MACHINE_LEARNING_WORKERS=1` unless you have a specific reason not to: each worker process still loads its own copy of the weights.
- Raise Immich job concurrency (Smart Search / Face Detection, and related ML jobs) to at least `3` so those NPU cores stay busy under sustained load.
## Setup
@ -96,7 +98,7 @@ You do not need to redo any machine learning jobs after enabling hardware accele
You can confirm the device is being recognized and used by checking its utilization. There are many tools to display this, such as `nvtop` for NVIDIA or Intel, `intel_gpu_top` for Intel, and `radeontop` for AMD.
You can also check the logs of the `immich-machine-learning` container. When a Smart Search or Face Detection job begins, or when you search with text in Immich, you should either see a log for `Available ORT providers` containing the relevant provider (e.g. `CUDAExecutionProvider` in the case of CUDA), or a `Loaded ANN model` log entry without errors in the case of ARM NN.
You can also check the logs of the `immich-machine-learning` container. When a Smart Search or Face Detection job begins, or when you search with text in Immich, you should either see a log for `Available ORT providers` containing the relevant provider (e.g. `CUDAExecutionProvider` in the case of CUDA), a `Loaded ANN model` log entry without errors in the case of ARM NN, or a `Loading RKNN model ... with N worker(s)` log entry for RKNN.
#### Single Compose File
@ -173,4 +175,4 @@ Note that you should increase job concurrencies to increase overall utilization
- If `MACHINE_LEARNING_RKNN_THREADS` is at the default of 1, RKNPU will have substantially lower throughput for ML jobs than ARM NN in most cases, but similar latency (such as when searching)
- If `MACHINE_LEARNING_RKNN_THREADS` is set to 3, it will be somewhat faster than ARM NN at FP32, but somewhat slower than ARM NN if `MACHINE_LEARNING_ANN_FP16_TURBO` is enabled
- When other tasks also use the GPU (like transcoding), RKNPU has a significant advantage over ARM NN as it uses the otherwise idle NPU instead of competing for GPU usage
- Lower RAM usage if `MACHINE_LEARNING_RKNN_THREADS` is at the default of 1, but significantly higher if greater than 1 (which is necessary for it to fully utilize the NPU and hence be comparable in speed to ARM NN)
- Lower RAM usage than earlier RKNN builds: multi-core pools share weights via context duplication, so setting `MACHINE_LEARNING_RKNN_THREADS=3` no longer multiplies model memory by the thread count. Prefer that setting on RK3576/RK3588, keep `MACHINE_LEARNING_WORKERS=1`, and raise Immich ML job concurrency to match so each NPU core can service a concurrent request (~3× sustained throughput).

View file

@ -178,7 +178,7 @@ Redis (Sentinel) URL example JSON before encoding:
| `MACHINE_LEARNING_MAX_BATCH_SIZE__FACIAL_RECOGNITION` | Set the maximum number of faces that will be processed at once by the facial recognition model | None (`1` if using OpenVINO) | machine learning |
| `MACHINE_LEARNING_MAX_BATCH_SIZE__OCR` | Set the maximum number of boxes that will be processed at once by the OCR model | `6` | machine learning |
| `MACHINE_LEARNING_RKNN` | Enable RKNN hardware acceleration if supported | `True` | machine learning |
| `MACHINE_LEARNING_RKNN_THREADS` | How many threads of RKNN runtime should be spun up while inferencing. | `1` | machine learning |
| `MACHINE_LEARNING_RKNN_THREADS` | Number of RKNN worker contexts (NPU cores) to use per model. On RK3576/RK3588, `3` is recommended for ~3× sustained throughput; weights are shared across contexts so RAM barely grows vs `1`. Keep `MACHINE_LEARNING_WORKERS` at `1` and raise Immich ML job concurrency to match. | `1` | machine learning |
| `MACHINE_LEARNING_MODEL_ARENA` | Pre-allocates CPU memory to avoid memory fragmentation | true | machine learning |
| `MACHINE_LEARNING_OPENVINO_PRECISION` | If set to FP16, uses half-precision floating-point operations for faster inference with reduced accuracy (one of [`FP16`, `FP32`], applies only to OpenVINO) | `FP32` | machine learning |