immich/server/apps/microservices/src/processors/asset-uploaded.processor.ts

74 lines
2.6 KiB
TypeScript
Raw Normal View History

WIP refactor container and queuing system (#206) * refactor microservices to machine-learning * Update tGithub issue template with correct task syntax * Added microservices container * Communicate between service based on queue system * added dependency * Fixed problem with having to import BullQueue into the individual service * Added todo * refactor server into monorepo with microservices * refactor database and entity to library * added simple migration * Move migrations and database config to library * Migration works in library * Cosmetic change in logging message * added user dto * Fixed issue with testing not able to find the shared library * Clean up library mapping path * Added webp generator to microservices * Update Github Action build latest * Fixed issue NPM cannot install due to conflict witl Bull Queue * format project with prettier * Modified docker-compose file * Add GH Action for Staging build: * Fixed GH action job name * Modified GH Action to only build & push latest when pushing to main * Added Test 2e2 Github Action * Added Test 2e2 Github Action * Implemented microservice to extract exif * Added cronjob to scan and generate webp thumbnail at midnight * Refactor to ireduce hit time to database when running microservices * Added error handling to asset services that handle read file from disk * Added video transcoding queue to process one video at a time * Fixed loading spinner on web while loading covering the info panel * Add mechanism to show new release announcement to web and mobile app (#209) * Added changelog page * Fixed issues based on PR comments * Fixed issue with video transcoding run on the server * Change entry point content for backward combatibility when starting up server * Added announcement box * Added error handling to failed silently when the app version checking is not able to make the request to GITHUB * Added new version announcement overlay * Update message * Added messages * Added logic to check and show announcement * Add method to handle saving new version * Added button to dimiss the acknowledge message * Up version for deployment to the app store
2022-06-11 16:12:06 -05:00
import { InjectQueue, OnQueueActive, OnQueueCompleted, OnQueueWaiting, Process, Processor } from '@nestjs/bull';
import { Job, Queue } from 'bull';
import { AssetEntity, AssetType } from '@app/database/entities/asset.entity';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { randomUUID } from 'crypto';
@Processor('asset-uploaded-queue')
export class AssetUploadedProcessor {
constructor(
@InjectQueue('thumbnail-generator-queue')
private thumbnailGeneratorQueue: Queue,
@InjectQueue('metadata-extraction-queue')
private metadataExtractionQueue: Queue,
@InjectQueue('video-conversion-queue')
private videoConversionQueue: Queue,
@InjectRepository(AssetEntity)
private assetRepository: Repository<AssetEntity>,
) {}
/**
* Post processing uploaded asset to perform the following function if missing
* 1. Generate JPEG Thumbnail
* 2. Generate Webp Thumbnail <-> if JPEG thumbnail exist
* 3. EXIF extractor
* 4. Reverse Geocoding
*
* @param job asset-uploaded
*/
@Process('asset-uploaded')
async processUploadedVideo(job: Job) {
const {
asset,
fileName,
fileSize,
hasThumbnail,
}: { asset: AssetEntity; fileName: string; fileSize: number; hasThumbnail: boolean } = job.data;
if (hasThumbnail) {
// The jobs below depends on the existence of jpeg thumbnail
await this.thumbnailGeneratorQueue.add('generate-webp-thumbnail', { asset }, { jobId: randomUUID() });
await this.metadataExtractionQueue.add('tag-image', { asset }, { jobId: randomUUID() });
await this.metadataExtractionQueue.add('detect-object', { asset }, { jobId: randomUUID() });
} else {
// Generate Thumbnail -> Then generate webp, tag image and detect object
await this.thumbnailGeneratorQueue.add('generate-jpeg-thumbnail', { asset }, { jobId: randomUUID() });
WIP refactor container and queuing system (#206) * refactor microservices to machine-learning * Update tGithub issue template with correct task syntax * Added microservices container * Communicate between service based on queue system * added dependency * Fixed problem with having to import BullQueue into the individual service * Added todo * refactor server into monorepo with microservices * refactor database and entity to library * added simple migration * Move migrations and database config to library * Migration works in library * Cosmetic change in logging message * added user dto * Fixed issue with testing not able to find the shared library * Clean up library mapping path * Added webp generator to microservices * Update Github Action build latest * Fixed issue NPM cannot install due to conflict witl Bull Queue * format project with prettier * Modified docker-compose file * Add GH Action for Staging build: * Fixed GH action job name * Modified GH Action to only build & push latest when pushing to main * Added Test 2e2 Github Action * Added Test 2e2 Github Action * Implemented microservice to extract exif * Added cronjob to scan and generate webp thumbnail at midnight * Refactor to ireduce hit time to database when running microservices * Added error handling to asset services that handle read file from disk * Added video transcoding queue to process one video at a time * Fixed loading spinner on web while loading covering the info panel * Add mechanism to show new release announcement to web and mobile app (#209) * Added changelog page * Fixed issues based on PR comments * Fixed issue with video transcoding run on the server * Change entry point content for backward combatibility when starting up server * Added announcement box * Added error handling to failed silently when the app version checking is not able to make the request to GITHUB * Added new version announcement overlay * Update message * Added messages * Added logic to check and show announcement * Add method to handle saving new version * Added button to dimiss the acknowledge message * Up version for deployment to the app store
2022-06-11 16:12:06 -05:00
}
// Video Conversion
if (asset.type == AssetType.VIDEO) {
await this.videoConversionQueue.add('mp4-conversion', { asset }, { jobId: randomUUID() });
} else {
// Extract Metadata/Exif for Images - Currently the library cannot extract EXIF for video yet
await this.metadataExtractionQueue.add(
'exif-extraction',
{
asset,
fileName,
fileSize,
},
{ jobId: randomUUID() },
);
}
// Extract video duration if uploaded from the web
if (asset.type == AssetType.VIDEO && asset.duration == '0:00:00.000000') {
await this.metadataExtractionQueue.add('extract-video-length', { asset }, { jobId: randomUUID() });
}
WIP refactor container and queuing system (#206) * refactor microservices to machine-learning * Update tGithub issue template with correct task syntax * Added microservices container * Communicate between service based on queue system * added dependency * Fixed problem with having to import BullQueue into the individual service * Added todo * refactor server into monorepo with microservices * refactor database and entity to library * added simple migration * Move migrations and database config to library * Migration works in library * Cosmetic change in logging message * added user dto * Fixed issue with testing not able to find the shared library * Clean up library mapping path * Added webp generator to microservices * Update Github Action build latest * Fixed issue NPM cannot install due to conflict witl Bull Queue * format project with prettier * Modified docker-compose file * Add GH Action for Staging build: * Fixed GH action job name * Modified GH Action to only build & push latest when pushing to main * Added Test 2e2 Github Action * Added Test 2e2 Github Action * Implemented microservice to extract exif * Added cronjob to scan and generate webp thumbnail at midnight * Refactor to ireduce hit time to database when running microservices * Added error handling to asset services that handle read file from disk * Added video transcoding queue to process one video at a time * Fixed loading spinner on web while loading covering the info panel * Add mechanism to show new release announcement to web and mobile app (#209) * Added changelog page * Fixed issues based on PR comments * Fixed issue with video transcoding run on the server * Change entry point content for backward combatibility when starting up server * Added announcement box * Added error handling to failed silently when the app version checking is not able to make the request to GITHUB * Added new version announcement overlay * Update message * Added messages * Added logic to check and show announcement * Add method to handle saving new version * Added button to dimiss the acknowledge message * Up version for deployment to the app store
2022-06-11 16:12:06 -05:00
}
}