Different video beginning with black fade in marked as duplicates

[nicolaswurtz]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Hi!
All my videos that begin with some black frames (old cameras with built-in fade in "effect") are considered identical by de-duplication tool.
Thank you for considering my issue ;-)

The OS that Immich Server is running on

Debian 6.12.41-1

Version of Immich Server

2.7.3

Version of Immich Mobile App

2.7.3

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

#
# WARNING: To install Immich, follow our guide: https://immich.app/docs/install/docker-compose
#
# Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.

name: immich

networks:
  traefik_default:
    external: true

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false
    networks:
      - traefik_default

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
    networks:
      - traefik_default

  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:facc1d2c3462975c34e10fccb167bfa92b0e0dbd992fc282c29a61c3243afb11
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
    networks:
      - traefik_default

  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.4.3-pgvectors0.2.0@sha256:32324a2f41df5de9efe1af166b7008c3f55646f8d0e00d9550c16c9822366b4a
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    shm_size: 128mb
    restart: always
    networks:
      - traefik_default

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
IMMICH_TELEMETRY_INCLUDE=all

# The location where your uploaded files are stored
UPLOAD_LOCATION=./library
# The location where your database files are stored
DB_DATA_LOCATION=./postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=XXXXXXX

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=XXXXXXX
DB_DATABASE_NAME=XXXXXXX

Reproduction steps

1. Add different videos files that begin with some black frames
2. Wait deduplicate tool cronjob
3. Check deduplicate tool, they should be marked as duplicates

Relevant log output

Additional information

No response

[bo0tzz]

This is to some degree "working as expected". The deduplication utility operates only on the thumbnails, videos don't get any special treatment there. When generating thumbnails for videos that start with black, the code makes some attempt to find a more meaningful frame, but has to give up at some point.

[nicolaswurtz]

Thx for your response, I understand. I'll try to deal with it ;)