mirror of
https://github.com/immich-app/immich
synced 2026-08-15 13:03:57 +00:00
docs: recommend RKNN_THREADS=3 for multi-core Rockchip NPUs
Document shared-weight multi-context pools: keep WORKERS=1, set threads to 3, and raise Immich job concurrency so each NPU core can stay busy. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
parent
deeaea00c8
commit
7c9bc94383
2 changed files with 6 additions and 4 deletions
|
|
@ -83,7 +83,9 @@ You do not need to redo any machine learning jobs after enabling hardware accele
|
|||
- RKNPU driver V0.9.8 or later must be available in the host server
|
||||
- You may confirm this by running `cat /sys/kernel/debug/rknpu/version` to check the version
|
||||
- Optional: Configure your `.env` file, see [environment variables](/install/environment-variables) for RKNN specific settings
|
||||
- In particular, setting `MACHINE_LEARNING_RKNN_THREADS` to 2 or 3 can _dramatically_ improve performance for RK3576 and RK3588 compared to the default of 1, at the expense of multiplying the amount of RAM each model uses by that amount.
|
||||
- For RK3576 and RK3588, set `MACHINE_LEARNING_RKNN_THREADS=3` so each of the three NPU cores can service a concurrent request. Contexts share model weights, so RAM barely grows versus the single-core default while sustained throughput can approach **3×**.
|
||||
- Keep `MACHINE_LEARNING_WORKERS=1` unless you have a specific reason not to: each worker process still loads its own copy of the weights.
|
||||
- Raise Immich job concurrency (Smart Search / Face Detection, and related ML jobs) to at least `3` so those NPU cores stay busy under sustained load.
|
||||
|
||||
## Setup
|
||||
|
||||
|
|
@ -96,7 +98,7 @@ You do not need to redo any machine learning jobs after enabling hardware accele
|
|||
|
||||
You can confirm the device is being recognized and used by checking its utilization. There are many tools to display this, such as `nvtop` for NVIDIA or Intel, `intel_gpu_top` for Intel, and `radeontop` for AMD.
|
||||
|
||||
You can also check the logs of the `immich-machine-learning` container. When a Smart Search or Face Detection job begins, or when you search with text in Immich, you should either see a log for `Available ORT providers` containing the relevant provider (e.g. `CUDAExecutionProvider` in the case of CUDA), or a `Loaded ANN model` log entry without errors in the case of ARM NN.
|
||||
You can also check the logs of the `immich-machine-learning` container. When a Smart Search or Face Detection job begins, or when you search with text in Immich, you should either see a log for `Available ORT providers` containing the relevant provider (e.g. `CUDAExecutionProvider` in the case of CUDA), a `Loaded ANN model` log entry without errors in the case of ARM NN, or a `Loading RKNN model ... with N worker(s)` log entry for RKNN.
|
||||
|
||||
#### Single Compose File
|
||||
|
||||
|
|
@ -173,4 +175,4 @@ Note that you should increase job concurrencies to increase overall utilization
|
|||
- If `MACHINE_LEARNING_RKNN_THREADS` is at the default of 1, RKNPU will have substantially lower throughput for ML jobs than ARM NN in most cases, but similar latency (such as when searching)
|
||||
- If `MACHINE_LEARNING_RKNN_THREADS` is set to 3, it will be somewhat faster than ARM NN at FP32, but somewhat slower than ARM NN if `MACHINE_LEARNING_ANN_FP16_TURBO` is enabled
|
||||
- When other tasks also use the GPU (like transcoding), RKNPU has a significant advantage over ARM NN as it uses the otherwise idle NPU instead of competing for GPU usage
|
||||
- Lower RAM usage if `MACHINE_LEARNING_RKNN_THREADS` is at the default of 1, but significantly higher if greater than 1 (which is necessary for it to fully utilize the NPU and hence be comparable in speed to ARM NN)
|
||||
- Lower RAM usage than earlier RKNN builds: multi-core pools share weights via context duplication, so setting `MACHINE_LEARNING_RKNN_THREADS=3` no longer multiplies model memory by the thread count. Prefer that setting on RK3576/RK3588, keep `MACHINE_LEARNING_WORKERS=1`, and raise Immich ML job concurrency to match so each NPU core can service a concurrent request (~3× sustained throughput).
|
||||
|
|
|
|||
|
|
@ -178,7 +178,7 @@ Redis (Sentinel) URL example JSON before encoding:
|
|||
| `MACHINE_LEARNING_MAX_BATCH_SIZE__FACIAL_RECOGNITION` | Set the maximum number of faces that will be processed at once by the facial recognition model | None (`1` if using OpenVINO) | machine learning |
|
||||
| `MACHINE_LEARNING_MAX_BATCH_SIZE__OCR` | Set the maximum number of boxes that will be processed at once by the OCR model | `6` | machine learning |
|
||||
| `MACHINE_LEARNING_RKNN` | Enable RKNN hardware acceleration if supported | `True` | machine learning |
|
||||
| `MACHINE_LEARNING_RKNN_THREADS` | How many threads of RKNN runtime should be spun up while inferencing. | `1` | machine learning |
|
||||
| `MACHINE_LEARNING_RKNN_THREADS` | Number of RKNN worker contexts (NPU cores) to use per model. On RK3576/RK3588, `3` is recommended for ~3× sustained throughput; weights are shared across contexts so RAM barely grows vs `1`. Keep `MACHINE_LEARNING_WORKERS` at `1` and raise Immich ML job concurrency to match. | `1` | machine learning |
|
||||
| `MACHINE_LEARNING_MODEL_ARENA` | Pre-allocates CPU memory to avoid memory fragmentation | true | machine learning |
|
||||
| `MACHINE_LEARNING_OPENVINO_PRECISION` | If set to FP16, uses half-precision floating-point operations for faster inference with reduced accuracy (one of [`FP16`, `FP32`], applies only to OpenVINO) | `FP32` | machine learning |
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue