mirror of
https://github.com/immich-app/immich.git
synced 2026-02-12 11:58:15 +03:00
feat: Use postgres as a queue
We've been keen to try this for a while as it means we can remove redis as a dependency, which makes Immich easier to setup and run. This replaces bullmq with a bespoke postgres queue. Jobs in the queue are processed either immediately via triggers and notifications, or eventually if a notification is missed.
This commit is contained in:
Thomas Way
@@ -278,7 +278,7 @@ You can use [Smart Search](/docs/features/searching.md) for this to some extent.
|
||||
|
||||
### I'm getting a lot of "faces" that aren't faces, what can I do?
|
||||
|
||||
You can increase the MIN DETECTION SCORE to 0.8 to help prevent bad thumbnails. Setting the score too high (above 0.9) might filter out too many real faces depending on the library used. If you just want to hide specific faces, you can adjust the 'MIN FACES DETECTED' setting in the administration panel
|
||||
You can increase the MIN DETECTION SCORE to 0.8 to help prevent bad thumbnails. Setting the score too high (above 0.9) might filter out too many real faces depending on the library used. If you just want to hide specific faces, you can adjust the 'MIN FACES DETECTED' setting in the administration panel
|
||||
to increase the bar for what the algorithm considers a "core face" for that person, reducing the chance of bad thumbnails being chosen.
|
||||
|
||||
### The immich_model-cache volume takes up a lot of space, what could be the problem?
|
||||
@@ -367,12 +367,6 @@ You need to [enable WebSockets](/docs/administration/reverse-proxy/) on your rev
|
||||
|
||||
Immich components are typically deployed using docker. To see logs for deployed docker containers, you can use the [Docker CLI](https://docs.docker.com/engine/reference/commandline/cli/), specifically the `docker logs` command. For examples, see [Docker Help](/docs/guides/docker-help.md).
|
||||
|
||||
### How can I reduce the log verbosity of Redis?
|
||||
|
||||
To decrease Redis logs, you can add the following line to the `redis:` section of the `docker-compose.yml`:
|
||||
|
||||
` command: redis-server --loglevel warning`
|
||||
|
||||
### How can I run Immich as a non-root user?
|
||||
|
||||
You can change the user in the container by setting the `user` argument in `docker-compose.yml` for each service.
|
||||
@@ -380,7 +374,6 @@ You may need to add mount points or docker volumes for the following internal co
|
||||
|
||||
- `immich-machine-learning:/.config`
|
||||
- `immich-machine-learning:/.cache`
|
||||
- `redis:/data`
|
||||
|
||||
The non-root user/group needs read/write access to the volume mounts, including `UPLOAD_LOCATION` and `/cache` for machine-learning.
|
||||
|
||||
@@ -425,7 +418,7 @@ After removing the containers and volumes, there are a few directories that need
|
||||
- `UPLOAD_LOCATION` contains all the media uploaded to Immich.
|
||||
|
||||
:::note Portainer
|
||||
If you use portainer, bring down the stack in portainer. Go into the volumes section
|
||||
If you use portainer, bring down the stack in portainer. Go into the volumes section
|
||||
and remove all the volumes related to immich then restart the stack.
|
||||
:::
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@ Immich uses a traditional client-server design, with a dedicated database for da
|
||||
|
||||
<img alt="Immich Architecture" src={AppArchitecture} className="p-4 dark:bg-immich-dark-primary my-4" />
|
||||
|
||||
The diagram shows clients communicating with the server's API via REST. The server communicates with downstream systems (i.e. Redis, Postgres, Machine Learning, file system) through repository interfaces. Not shown in the diagram, is that the server is split into two separate containers `immich-server` and `immich-microservices`. The microservices container does not handle API requests or schedule cron jobs, but primarily handles incoming job requests from Redis.
|
||||
The diagram shows clients communicating with the server's API via REST. The server communicates with downstream systems (i.e. Postgres, Machine Learning, file system) through repository interfaces. Not shown in the diagram, is that the server is split into two separate containers `immich-server` and `immich-microservices`. The microservices container does not handle API requests or schedule cron jobs, but primarily handles incoming job requests from Postgres.
|
||||
|
||||
## Clients
|
||||
|
||||
@@ -53,7 +53,6 @@ The Immich backend is divided into several services, which are run as individual
|
||||
1. `immich-server` - Handle and respond to REST API requests, execute background jobs (thumbnail generation, metadata extraction, transcoding, etc.)
|
||||
1. `immich-machine-learning` - Execute machine learning models
|
||||
1. `postgres` - Persistent data storage
|
||||
1. `redis`- Queue management for background jobs
|
||||
|
||||
### Immich Server
|
||||
|
||||
@@ -111,7 +110,3 @@ Immich persists data in Postgres, which includes information about access and au
|
||||
:::info
|
||||
See [Database Migrations](./database-migrations.md) for more information about how to modify the database to create an index, modify a table, add a new column, etc.
|
||||
:::
|
||||
|
||||
### Redis
|
||||
|
||||
Immich uses [Redis](https://redis.com/) via [BullMQ](https://docs.bullmq.io/) to manage job queues. Some jobs trigger subsequent jobs. For example, Smart Search and Facial Recognition relies on thumbnail generation and automatically run after one is generated.
|
||||
|
||||
@@ -23,7 +23,6 @@ This environment includes the services below. Additional details are available i
|
||||
- Server - [`/server`](https://github.com/immich-app/immich/tree/main/server)
|
||||
- Web app - [`/web`](https://github.com/immich-app/immich/tree/main/web)
|
||||
- Machine learning - [`/machine-learning`](https://github.com/immich-app/immich/tree/main/machine-learning)
|
||||
- Redis
|
||||
- PostgreSQL development database with exposed port `5432` so you can use any database client to access it
|
||||
|
||||
All the services are packaged to run as with single Docker Compose command.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Scaling Immich
|
||||
|
||||
Immich is built with modern deployment practices in mind, and the backend is designed to be able to run multiple instances in parallel. When doing this, the only requirement you need to be aware of is that every instance needs to be connected to the shared infrastructure. That means they should all have access to the same Postgres and Redis instances, and have the same files mounted into the containers.
|
||||
Immich is built with modern deployment practices in mind, and the backend is designed to be able to run multiple instances in parallel. When doing this, the only requirement you need to be aware of is that every instance needs to be connected to the shared infrastructure. That means they should all have access to the same Postgres instance, and have the same files mounted into the containers.
|
||||
|
||||
Scaling can be useful for many reasons. Maybe you have a gaming PC that you want to use for transcoding and thumbnail generation, or perhaps you run a Kubernetes cluster across a handful of powerful servers that you want to make use of.
|
||||
|
||||
@@ -16,4 +16,4 @@ By default, each running `immich-server` container comes with multiple internal
|
||||
|
||||
## Scaling down
|
||||
|
||||
In the same way you can scale up to multiple containers, you can also choose to scale down. All state is stored in Postgres, Redis, and the filesystem so there is no risk in stopping a running immich-server container, for example if you want to use your GPU to play some games. As long as there is an API worker running you will still be able to browse Immich, and jobs will wait to be processed until there is a worker available for them.
|
||||
In the same way you can scale up to multiple containers, you can also choose to scale down. All state is stored in Postgres and the filesystem so there is no risk in stopping a running immich-server container, for example if you want to use your GPU to play some games. As long as there is an API worker running you will still be able to browse Immich, and jobs will wait to be processed until there is a worker available for them.
|
||||
|
||||
@@ -98,54 +98,6 @@ When `DB_URL` is defined, the `DB_HOSTNAME`, `DB_PORT`, `DB_USERNAME`, `DB_PASSW
|
||||
|
||||
:::
|
||||
|
||||
## Redis
|
||||
|
||||
| Variable | Description | Default | Containers |
|
||||
| :--------------- | :------------- | :-----: | :--------- |
|
||||
| `REDIS_URL` | Redis URL | | server |
|
||||
| `REDIS_SOCKET` | Redis socket | | server |
|
||||
| `REDIS_HOSTNAME` | Redis host | `redis` | server |
|
||||
| `REDIS_PORT` | Redis port | `6379` | server |
|
||||
| `REDIS_USERNAME` | Redis username | | server |
|
||||
| `REDIS_PASSWORD` | Redis password | | server |
|
||||
| `REDIS_DBINDEX` | Redis DB index | `0` | server |
|
||||
|
||||
:::info
|
||||
All `REDIS_` variables must be provided to all Immich workers, including `api` and `microservices`.
|
||||
|
||||
`REDIS_URL` must start with `ioredis://` and then include a `base64` encoded JSON string for the configuration.
|
||||
More information can be found in the upstream [ioredis] documentation.
|
||||
|
||||
When `REDIS_URL` or `REDIS_SOCKET` are defined, the `REDIS_HOSTNAME`, `REDIS_PORT`, `REDIS_USERNAME`, `REDIS_PASSWORD`, and `REDIS_DBINDEX` variables are ignored.
|
||||
:::
|
||||
|
||||
Redis (Sentinel) URL example JSON before encoding:
|
||||
|
||||
<details>
|
||||
<summary>JSON</summary>
|
||||
|
||||
```json
|
||||
{
|
||||
"sentinels": [
|
||||
{
|
||||
"host": "redis-sentinel-node-0",
|
||||
"port": 26379
|
||||
},
|
||||
{
|
||||
"host": "redis-sentinel-node-1",
|
||||
"port": 26379
|
||||
},
|
||||
{
|
||||
"host": "redis-sentinel-node-2",
|
||||
"port": 26379
|
||||
}
|
||||
],
|
||||
"name": "redis-sentinel"
|
||||
}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Machine Learning
|
||||
|
||||
| Variable | Description | Default | Containers |
|
||||
@@ -212,16 +164,10 @@ the `_FILE` variable should be set to the path of a file containing the variable
|
||||
| `DB_USERNAME` | `DB_USERNAME_FILE`<sup>\*1</sup> |
|
||||
| `DB_PASSWORD` | `DB_PASSWORD_FILE`<sup>\*1</sup> |
|
||||
| `DB_URL` | `DB_URL_FILE`<sup>\*1</sup> |
|
||||
| `REDIS_PASSWORD` | `REDIS_PASSWORD_FILE`<sup>\*2</sup> |
|
||||
|
||||
\*1: See the [official documentation][docker-secrets-docs] for
|
||||
details on how to use Docker Secrets in the Postgres image.
|
||||
|
||||
\*2: See [this comment][docker-secrets-example] for an example of how
|
||||
to use a Docker secret for the password in the Redis container.
|
||||
|
||||
[tz-list]: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
|
||||
[docker-secrets-example]: https://github.com/docker-library/redis/issues/46#issuecomment-335326234
|
||||
[docker-secrets-docs]: https://github.com/docker-library/docs/tree/master/postgres#docker-secrets
|
||||
[docker-secrets]: https://docs.docker.com/engine/swarm/secrets/
|
||||
[ioredis]: https://ioredis.readthedocs.io/en/latest/README/#connect-to-redis
|
||||
|
||||
@@ -107,8 +107,6 @@ Accept the default option or select the **Machine Learning Image Type** for your
|
||||
|
||||
Immich's default is `postgres` but you should consider setting the **Database Password** to a custom value using only the characters `A-Za-z0-9`.
|
||||
|
||||
The **Redis Password** should be set to a custom value using only the characters `A-Za-z0-9`.
|
||||
|
||||
Accept the **Log Level** default of **Log**.
|
||||
|
||||
Leave **Hugging Face Endpoint** blank. (This is for downloading ML models from a different source.)
|
||||
@@ -242,7 +240,7 @@ className="border rounded-xl"
|
||||
:::info
|
||||
Some Environment Variables are not available for the TrueNAS SCALE app. This is mainly because they can be configured through GUI options in the [Edit Immich screen](#edit-app-settings).
|
||||
|
||||
Some examples are: `IMMICH_VERSION`, `UPLOAD_LOCATION`, `DB_DATA_LOCATION`, `TZ`, `IMMICH_LOG_LEVEL`, `DB_PASSWORD`, `REDIS_PASSWORD`.
|
||||
Some examples are: `IMMICH_VERSION`, `UPLOAD_LOCATION`, `DB_DATA_LOCATION`, `TZ`, `IMMICH_LOG_LEVEL`, `DB_PASSWORD`.
|
||||
:::
|
||||
|
||||
## Updating the App
|
||||
|
||||
@@ -17,9 +17,9 @@ Immich can easily be installed and updated on Unraid via:
|
||||
|
||||
:::
|
||||
|
||||
In order to install Immich from the Unraid CA, you will need an existing Redis and PostgreSQL 14 container, If you do not already have Redis or PostgreSQL you can install them from the Unraid CA, just make sure you choose PostgreSQL **14**.
|
||||
In order to install Immich from the Unraid CA, you will need an existing PostgreSQL 14 container, If you do not already have PostgreSQL you can install it from the Unraid CA, just make sure you choose PostgreSQL **14**.
|
||||
|
||||
Once you have Redis and PostgreSQL running, search for Immich on the Unraid CA, choose either of the templates listed and fill out the example variables.
|
||||
Once you have PostgreSQL running, search for Immich on the Unraid CA, choose either of the templates listed and fill out the example variables.
|
||||
|
||||
For more information about setting up the community image see [here](https://github.com/imagegenius/docker-immich#application-setup)
|
||||
|
||||
@@ -45,63 +45,63 @@ width="70%"
|
||||
alt="Select Plugins > Compose.Manager > Add New Stack > Label it Immich"
|
||||
/>
|
||||
|
||||
3. Select the cogwheel ⚙️ next to Immich and click "**Edit Stack**"
|
||||
4. Click "**Compose File**" and then paste the entire contents of the [Immich Docker Compose](https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml) file into the Unraid editor. Remove any text that may be in the text area by default. Note that Unraid v6.12.10 uses version 24.0.9 of the Docker Engine, which does not support healthcheck `start_interval` as defined in the `database` service of the Docker compose file (version 25 or higher is needed). This parameter defines an initial waiting period before starting health checks, to give the container time to start up. Commenting out the `start_interval` and `start_period` parameters will allow the containers to start up normally. The only downside to this is that the database container will not receive an initial health check until `interval` time has passed.
|
||||
3. Select the cogwheel ⚙️ next to Immich and click "**Edit Stack**"
|
||||
4. Click "**Compose File**" and then paste the entire contents of the [Immich Docker Compose](https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml) file into the Unraid editor. Remove any text that may be in the text area by default. Note that Unraid v6.12.10 uses version 24.0.9 of the Docker Engine, which does not support healthcheck `start_interval` as defined in the `database` service of the Docker compose file (version 25 or higher is needed). This parameter defines an initial waiting period before starting health checks, to give the container time to start up. Commenting out the `start_interval` and `start_period` parameters will allow the containers to start up normally. The only downside to this is that the database container will not receive an initial health check until `interval` time has passed.
|
||||
|
||||
<details >
|
||||
<summary>Using an existing Postgres container? Click me! Otherwise proceed to step 5.</summary>
|
||||
<ul>
|
||||
<li>Comment out the database service</li>
|
||||
<img
|
||||
src={require('./img/unraid02.webp').default}
|
||||
width="50%"
|
||||
alt="Comment out database service in the compose file"
|
||||
/>
|
||||
<li>Comment out the database dependency for <b>each service</b> <i>(example in screenshot below only shows 2 of the services - ensure you do this for all services)</i></li>
|
||||
<img
|
||||
src={require('./img/unraid03.webp').default}
|
||||
width="50%"
|
||||
alt="Comment out every reference to the database service in the compose file"
|
||||
/>
|
||||
<li>Comment out the volumes</li>
|
||||
<img
|
||||
src={require('./img/unraid04.webp').default}
|
||||
width="20%"
|
||||
alt="Comment out database volume"
|
||||
/>
|
||||
</ul>
|
||||
</details>
|
||||
<details >
|
||||
<summary>Using an existing Postgres container? Click me! Otherwise proceed to step 5.</summary>
|
||||
<ul>
|
||||
<li>Comment out the database service</li>
|
||||
<img
|
||||
src={require('./img/unraid02.webp').default}
|
||||
width="50%"
|
||||
alt="Comment out database service in the compose file"
|
||||
/>
|
||||
<li>Comment out the database dependency for <b>each service</b> <i>(example in screenshot below only shows 2 of the services - ensure you do this for all services)</i></li>
|
||||
<img
|
||||
src={require('./img/unraid03.webp').default}
|
||||
width="50%"
|
||||
alt="Comment out every reference to the database service in the compose file"
|
||||
/>
|
||||
<li>Comment out the volumes</li>
|
||||
<img
|
||||
src={require('./img/unraid04.webp').default}
|
||||
width="20%"
|
||||
alt="Comment out database volume"
|
||||
/>
|
||||
</ul>
|
||||
</details>
|
||||
|
||||
5. Click "**Save Changes**", you will be prompted to edit stack UI labels, just leave this blank and click "**Ok**"
|
||||
6. Select the cog ⚙️ next to Immich, click "**Edit Stack**", then click "**Env File**"
|
||||
7. Paste the entire contents of the [Immich example.env](https://github.com/immich-app/immich/releases/latest/download/example.env) file into the Unraid editor, then **before saving** edit the following:
|
||||
5. Click "**Save Changes**", you will be prompted to edit stack UI labels, just leave this blank and click "**Ok**"
|
||||
6. Select the cog ⚙️ next to Immich, click "**Edit Stack**", then click "**Env File**"
|
||||
7. Paste the entire contents of the [Immich example.env](https://github.com/immich-app/immich/releases/latest/download/example.env) file into the Unraid editor, then **before saving** edit the following:
|
||||
|
||||
- `UPLOAD_LOCATION`: Create a folder in your Images Unraid share and place the **absolute** location here > For example my _"images"_ share has a folder within it called _"immich"_. If I browse to this directory in the terminal and type `pwd` the output is `/mnt/user/images/immich`. This is the exact value I need to enter as my `UPLOAD_LOCATION`
|
||||
- `DB_DATA_LOCATION`: Change this to use an Unraid share (preferably a cache pool, e.g. `/mnt/user/appdata/postgresql/data`). This uses the `appdata` share. Do also create the `postgresql` folder, by running `mkdir /mnt/user/{share_location}/postgresql/data`. If left at default it will try to use Unraid's `/boot/config/plugins/compose.manager/projects/[stack_name]/postgres` folder which it doesn't have permissions to, resulting in this container continuously restarting.
|
||||
- `UPLOAD_LOCATION`: Create a folder in your Images Unraid share and place the **absolute** location here > For example my _"images"_ share has a folder within it called _"immich"_. If I browse to this directory in the terminal and type `pwd` the output is `/mnt/user/images/immich`. This is the exact value I need to enter as my `UPLOAD_LOCATION`
|
||||
- `DB_DATA_LOCATION`: Change this to use an Unraid share (preferably a cache pool, e.g. `/mnt/user/appdata/postgresql/data`). This uses the `appdata` share. Do also create the `postgresql` folder, by running `mkdir /mnt/user/{share_location}/postgresql/data`. If left at default it will try to use Unraid's `/boot/config/plugins/compose.manager/projects/[stack_name]/postgres` folder which it doesn't have permissions to, resulting in this container continuously restarting.
|
||||
|
||||
<img
|
||||
src={require('./img/unraid05.webp').default}
|
||||
width="70%"
|
||||
alt="Absolute location of where you want immich images stored"
|
||||
/>
|
||||
<img
|
||||
src={require('./img/unraid05.webp').default}
|
||||
width="70%"
|
||||
alt="Absolute location of where you want immich images stored"
|
||||
/>
|
||||
|
||||
<details >
|
||||
<summary>Using an existing Postgres container? Click me! Otherwise proceed to step 8.</summary>
|
||||
<p>Update the following database variables as relevant to your Postgres container:</p>
|
||||
<ul>
|
||||
<li><code>DB_HOSTNAME</code></li>
|
||||
<li><code>DB_USERNAME</code></li>
|
||||
<li><code>DB_PASSWORD</code></li>
|
||||
<li><code>DB_DATABASE_NAME</code></li>
|
||||
<li><code>DB_PORT</code></li>
|
||||
</ul>
|
||||
</details>
|
||||
<details >
|
||||
<summary>Using an existing Postgres container? Click me! Otherwise proceed to step 8.</summary>
|
||||
<p>Update the following database variables as relevant to your Postgres container:</p>
|
||||
<ul>
|
||||
<li><code>DB_HOSTNAME</code></li>
|
||||
<li><code>DB_USERNAME</code></li>
|
||||
<li><code>DB_PASSWORD</code></li>
|
||||
<li><code>DB_DATABASE_NAME</code></li>
|
||||
<li><code>DB_PORT</code></li>
|
||||
</ul>
|
||||
</details>
|
||||
|
||||
8. Click "**Save Changes**" followed by "**Compose Up**" and Unraid will begin to create the Immich containers in a popup window. Once complete you will see a message on the popup window stating _"Connection Closed"_. Click "**Done**" and go to the Unraid "**Docker**" page
|
||||
8. Click "**Save Changes**" followed by "**Compose Up**" and Unraid will begin to create the Immich containers in a popup window. Once complete you will see a message on the popup window stating _"Connection Closed"_. Click "**Done**" and go to the Unraid "**Docker**" page
|
||||
|
||||
> Note: This can take several minutes depending on your Internet speed and Unraid hardware
|
||||
> Note: This can take several minutes depending on your Internet speed and Unraid hardware
|
||||
|
||||
9. Once on the Docker page you will see several Immich containers, one of them will be labelled `immich_server` and will have a port mapping. Visit the `IP:PORT` displayed in your web browser and you should see the Immich admin setup page.
|
||||
9. Once on the Docker page you will see several Immich containers, one of them will be labelled `immich_server` and will have a port mapping. Visit the `IP:PORT` displayed in your web browser and you should see the Immich admin setup page.
|
||||
|
||||
<img
|
||||
src={require('./img/unraid06.webp').default}
|
||||
@@ -122,7 +122,7 @@ alt="Go to Docker Tab and visit the address listed next to immich-web"
|
||||
width="90%"
|
||||
alt="Go to Docker Tab and visit the address listed next to immich-web"
|
||||
/>
|
||||
|
||||
|
||||
</details>
|
||||
|
||||
:::tip
|
||||
|
||||
Reference in New Issue
Block a user