Merge branch 'duplicate-detection'
This commit is contained in:
@@ -5,6 +5,15 @@ CHITAI_TOKEN_SECRET=secret
|
||||
CHITAI_DEFAULT_LIBRARY_NAME=Books
|
||||
CHITAI_DEFAULT_LIBRARY_PATH="libraries/books"
|
||||
|
||||
# Duplicate detection when importing files (optional).
|
||||
# Scope: "library" compares against the library being uploaded to, "global" against
|
||||
# every library, "off" disables the check.
|
||||
CHITAI_DUPLICATE_SCOPE=library
|
||||
|
||||
# Where the consume watcher parks files it refused as duplicates. Keep it outside
|
||||
# CHITAI_CONSUME_PATH, or the watcher picks them straight back up.
|
||||
CHITAI_DUPLICATE_PATH="duplicates"
|
||||
|
||||
# You probably should not change these
|
||||
CHITAI_API_URL="http://backend:8000"
|
||||
CHITAI_API_DEBUG=false
|
||||
|
||||
@@ -88,6 +88,14 @@ The backend owns files on disk, not just rows:
|
||||
- **Layout** — `services/filesystem_library.py` (`BookPathGenerator`) renders a Jinja2 template
|
||||
against book metadata to decide where a book lives under the library's `root_path`
|
||||
(default: `author/series/position - title/`).
|
||||
- **One directory per book, never shared.** The generated path is a pure function of the metadata,
|
||||
so two books with the same author and title produce the same one — two editions, or an
|
||||
`allow_duplicates` copy. `BookService._reserve_book_path` moves the later one to `title (2)`
|
||||
before anything is written, and `update_book` reserves the same way so a rename cannot move a
|
||||
book in on top of another. This matters because `book.path` is what deletes, moves and file
|
||||
lookups act on: books sharing a directory means one overwrites the other's files, and deleting
|
||||
either takes both. `_unused_path` does the same job for filenames within a directory.
|
||||
A book that already has a `path` keeps it — `add_files` must follow the book, not the template.
|
||||
- **Metadata extraction** — `services/metadata_extractor.py` reads EPUB (ebooklib) and PDF
|
||||
(pypdfium2) files; extracted values fill only *empty* fields on the incoming payload.
|
||||
- **Covers** — converted to WebP with a UUID filename under `settings.book_cover_path`, served by a
|
||||
@@ -100,6 +108,140 @@ The backend owns files on disk, not just rows:
|
||||
if it differs, moves the directory contents and prunes empty parents. Keep that in mind before
|
||||
changing metadata handling.
|
||||
|
||||
## Duplicate detection
|
||||
|
||||
Every ingest path screens incoming files against what is already stored, keyed on
|
||||
**`(hash, size)`** — never the hash alone, because it samples 12 KiB (see below) and
|
||||
EPUBs from one toolchain often share their first window. `FileMetadata.hash` carries a
|
||||
plain, deliberately **non-unique** index: a collision must not be able to fail an import,
|
||||
and older databases may already hold duplicates.
|
||||
|
||||
Scope comes from `CHITAI_DUPLICATE_SCOPE` (`library`, the default | `global` | `off`).
|
||||
|
||||
The policy differs by how deliberate the import is:
|
||||
|
||||
| Path | Behaviour |
|
||||
| --- | --- |
|
||||
| `create_many_from_files` (browser bulk) | Skip per file, skip a whole group whose files are all known, report everything skipped in `ImportResult.duplicates`. Re-dropping a folder to pick up what is new is the case this serves. |
|
||||
| `create_book` (single, with metadata) | All-or-nothing: raises `DuplicateFilesError`, which `controllers/book.py` renders as a **409** carrying the refused files in `extra`. |
|
||||
| `add_files` | A file the book already carries is a no-op; one stored under another book raises `DuplicateFilesError`. |
|
||||
| `create_many_from_existing_files` (consume watcher) | Skips, and **moves the file to `CHITAI_DUPLICATE_PATH/<library slug>/`** — nothing is deleted, and it cannot stay put because `watchfiles` only reports additions. That path must stay outside `consume_path` or the watcher re-imports it and tries to read the directory name as a library slug. |
|
||||
|
||||
`allow_duplicates=true` overrides all of it, on every endpoint. Keep that working — the
|
||||
hash is not proof of identity, so a wrong verdict has to be recoverable, and a scripted
|
||||
import needs a way through. The **web UI deliberately does not offer it**: storing the
|
||||
same bytes twice splits reading progress and shelf membership across two records that
|
||||
can never converge, which is nothing anyone wants on purpose.
|
||||
|
||||
Three things to preserve when touching this code:
|
||||
|
||||
- **A match only counts while the file is on disk.** `find_duplicate_files` stats each
|
||||
candidate, and `add_files` writes a missing file back into the row that already
|
||||
describes it (`_restore_file`) instead of adding a second row beside it. The hash
|
||||
lives in the database and the file does not, so without this a file deleted behind
|
||||
the app's back would go on refusing its own replacement.
|
||||
|
||||
- **Screening runs before anything is written.** `fingerprint_upload` reads the spooled
|
||||
upload and rewinds it; the resulting fingerprints are handed to `_save_book_files`,
|
||||
which skips its own `StreamingHasher` when it already has the answer. Passing them
|
||||
through is what keeps the file from being read twice.
|
||||
- **`_screen_for_duplicates` extends the `known` dict as it goes**, so the same bytes
|
||||
submitted twice in one request are caught. Those duplicates report `book_id: None` —
|
||||
there is no row to point at yet.
|
||||
|
||||
`POST /books/duplicate-files` answers the same question from fingerprints alone, for
|
||||
clients that want to ask before uploading anything.
|
||||
|
||||
### Book-level detection — a different question
|
||||
|
||||
The file check answers "are these the same bytes?". `find_duplicate_books` answers "is
|
||||
this the same book?", which a re-scan, a re-zipped EPUB or another edition cannot be
|
||||
asked with a hash. Two signals, either sufficient: a **shared identifier**, or a
|
||||
**matching normalized title with at least one shared author**.
|
||||
|
||||
It **never blocks**. A metadata match is a guess — a work shares title and author with
|
||||
its own translation, its own second edition and its own audiobook — so the book is
|
||||
created and the candidates are reported alongside it in
|
||||
`ImportResult.possible_duplicates`. File-level dedupe keeps its refuse/skip behaviour;
|
||||
that one is near-certain and this one is not. Do not "improve" this into a refusal.
|
||||
|
||||
Two rules narrow it, both applied in Python over the small candidate set:
|
||||
|
||||
- **A shared author is required for a title match.** Without it every book the
|
||||
extractors gave up on and titled `Unknown` is a duplicate of every other one. A book
|
||||
with no authors can therefore only match on an identifier.
|
||||
- **The same series at a different `series_position` disqualifies a match.** A trilogy
|
||||
shares an author and often most of its title; the position is the library saying
|
||||
outright that these are two books.
|
||||
|
||||
A book is compared under **several title keys, not one** (`_title_keys`). Ebook files
|
||||
are overwhelmingly named `Title - Author.epub`, and wherever nothing inside the file
|
||||
overrode that name the author ended up in the title column — so one copy is stored as
|
||||
`Building Microservices` and another as `Building Microservices Sam Newman`. Both
|
||||
directions are generated, the author stripped off and the author added on, which is why
|
||||
the query is `normalized_title.in_(keys)` rather than `==`. This is still exact matching
|
||||
on an indexed column: no similarity score, nothing to tune. It does not weaken the
|
||||
shared-author requirement, which is a separate condition.
|
||||
|
||||
`find_duplicate_book_groups` is the library-wide pass behind
|
||||
`GET /books/duplicate-books`, since the import-time check says nothing about a
|
||||
collection someone already has. It buckets books by every key they carry and merges the
|
||||
buckets with union-find, so A~B by ISBN and B~C by title land in one group. Pairs in
|
||||
`duplicate_dismissals` are never merged — a reader disagreeing with one pairing must not
|
||||
silently break a group that stands on other evidence.
|
||||
|
||||
### Author names have one stored form
|
||||
|
||||
`Author.name` is always the canonical form, produced by `format_author_name`. Extractors
|
||||
hand over whatever the file said — `Newman, Sam;` from a `DC:creator` list, `Sam Newman`
|
||||
from a PDF, `Sam Newman.epub` from a filename — and storing those verbatim is how one
|
||||
person becomes four rows in the sidebar, four entries in the author filter, and four
|
||||
books that never look like each other.
|
||||
|
||||
This is **display** canonicalization, distinct from `normalize_author`, which throws
|
||||
away case, accents and spacing to build a comparison key nobody sees. Tidying only
|
||||
removes what an extractor added: a trailing separator, a file extension, and the
|
||||
`Surname, Given` ordering. It never touches case or accents — `Michał Płachta` and
|
||||
`Steve McConnell` are the author's own spelling, not something to correct.
|
||||
|
||||
Three places have to agree, and `Author` keeps them together: the `@validates("name")`
|
||||
hook, `unique_hash`, and `unique_filter`. `as_unique_async` looks a row up with the
|
||||
filter and then constructs with the validator, so if the lookup used the raw name and
|
||||
the insert used the tidy one, every variant spelling would miss the existing row and
|
||||
then collide with it on the unique index.
|
||||
|
||||
A form the rule does not recognise is **left exactly as it was found** — `Dave Thomas,
|
||||
Andy Hunt` is two people in one string, and flipping it would invent a third. Leaving a
|
||||
mess visible beats rewriting it wrongly.
|
||||
|
||||
### The normalized columns are written by validators
|
||||
|
||||
`Book.normalized_title`, `Author.normalized_name` and `Identifier.normalized_value` are
|
||||
derived from `services/matching.py` and kept current by **SQLAlchemy `@validates` hooks
|
||||
on the models**, not by any service. Assigning them directly is always wrong.
|
||||
|
||||
This is deliberate and it is invisible at the call sites: `BookService` writes titles
|
||||
through at least three paths (`to_model_on_create`, `to_model_on_update`, and the
|
||||
`setattr` loop in `_populate_with_unique_relationships`), and a validator is the only
|
||||
thing a fourth cannot bypass. The columns carry plain, **non-unique** btree indexes —
|
||||
two spellings collapsing onto one value is the entire point.
|
||||
|
||||
`services/matching.py` is imported *inside* those validators rather than at module
|
||||
scope: reaching it initialises the `chitai.services` package, which imports the
|
||||
services, which import the models. Keep the local import.
|
||||
|
||||
The validators only fire on write, so a migration that adds one of these columns must
|
||||
backfill existing rows through the same helpers — see the `data_upgrades()` hook in
|
||||
`2026-08-15_add_book_matching_keys_and_duplicate__4358e7d4743a.py`.
|
||||
|
||||
**Changing anything in `services/matching.py` needs a revision that recomputes them.**
|
||||
The keys are derived and already written, so a normalization change silently invalidates
|
||||
every stored row: a book written under the old rules just stops matching one written
|
||||
under the new rules, with nothing to show that anything is wrong. Copy
|
||||
`2026-08-15_recompute_book_matching_keys_ed41acf21270.py`, which exists because
|
||||
`normalize_title` learned to strip compact edition markers (`2E`, `5e`). It is
|
||||
idempotent and safe to re-run.
|
||||
|
||||
## KOReader hashing
|
||||
|
||||
`services/utils.py` reimplements KOReader's partial-MD5 document identifier: 1 KiB samples at
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
"""add file hash index
|
||||
|
||||
Revision ID: e9c2c7e875ae
|
||||
Revises: 6d72d1bbc0ee
|
||||
Create Date: 2026-08-13 14:52:09.341906
|
||||
|
||||
"""
|
||||
|
||||
import warnings
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from advanced_alchemy.types import EncryptedString, EncryptedText, GUID, ORA_JSONB, DateTimeUTC, StoredObject, PasswordHash, FernetBackend
|
||||
from advanced_alchemy.types.encrypted_string import PGCryptoBackend
|
||||
from advanced_alchemy.types.password_hash.argon2 import Argon2Hasher
|
||||
from advanced_alchemy.types.password_hash.passlib import PasslibHasher
|
||||
from advanced_alchemy.types.password_hash.pwdlib import PwdlibHasher
|
||||
from sqlalchemy import Text # noqa: F401
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from collections.abc import Sequence
|
||||
|
||||
__all__ = ["downgrade", "upgrade", "schema_upgrades", "schema_downgrades", "data_upgrades", "data_downgrades"]
|
||||
|
||||
sa.GUID = GUID
|
||||
sa.DateTimeUTC = DateTimeUTC
|
||||
sa.ORA_JSONB = ORA_JSONB
|
||||
sa.EncryptedString = EncryptedString
|
||||
sa.EncryptedText = EncryptedText
|
||||
sa.StoredObject = StoredObject
|
||||
sa.PasswordHash = PasswordHash
|
||||
sa.Argon2Hasher = Argon2Hasher
|
||||
sa.PasslibHasher = PasslibHasher
|
||||
sa.PwdlibHasher = PwdlibHasher
|
||||
sa.FernetBackend = FernetBackend
|
||||
sa.PGCryptoBackend = PGCryptoBackend
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'e9c2c7e875ae'
|
||||
down_revision = '6d72d1bbc0ee'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with warnings.catch_warnings():
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
with op.get_context().autocommit_block():
|
||||
schema_upgrades()
|
||||
data_upgrades()
|
||||
|
||||
def downgrade() -> None:
|
||||
with warnings.catch_warnings():
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
with op.get_context().autocommit_block():
|
||||
data_downgrades()
|
||||
schema_downgrades()
|
||||
|
||||
def schema_upgrades() -> None:
|
||||
"""schema upgrade migrations go here."""
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
with op.batch_alter_table('file_metadata', schema=None) as batch_op:
|
||||
batch_op.create_index('ix_file_metadata_hash', ['hash'], unique=False)
|
||||
|
||||
# ### end Alembic commands ###
|
||||
|
||||
def schema_downgrades() -> None:
|
||||
"""schema downgrade migrations go here."""
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
with op.batch_alter_table('file_metadata', schema=None) as batch_op:
|
||||
batch_op.drop_index('ix_file_metadata_hash')
|
||||
|
||||
# ### end Alembic commands ###
|
||||
|
||||
def data_upgrades() -> None:
|
||||
"""Add any optional data upgrade migrations here!"""
|
||||
|
||||
def data_downgrades() -> None:
|
||||
"""Add any optional data downgrade migrations here!"""
|
||||
+163
@@ -0,0 +1,163 @@
|
||||
"""add book matching keys and duplicate dismissals
|
||||
|
||||
Revision ID: 4358e7d4743a
|
||||
Revises: e9c2c7e875ae
|
||||
Create Date: 2026-08-15 15:09:02.708914
|
||||
|
||||
"""
|
||||
|
||||
import warnings
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from advanced_alchemy.types import EncryptedString, EncryptedText, GUID, ORA_JSONB, DateTimeUTC, StoredObject, PasswordHash, FernetBackend
|
||||
from advanced_alchemy.types.encrypted_string import PGCryptoBackend
|
||||
from advanced_alchemy.types.password_hash.argon2 import Argon2Hasher
|
||||
from advanced_alchemy.types.password_hash.passlib import PasslibHasher
|
||||
from advanced_alchemy.types.password_hash.pwdlib import PwdlibHasher
|
||||
from sqlalchemy import Text # noqa: F401
|
||||
|
||||
__all__ = ["downgrade", "upgrade", "schema_upgrades", "schema_downgrades", "data_upgrades", "data_downgrades"]
|
||||
|
||||
sa.GUID = GUID
|
||||
sa.DateTimeUTC = DateTimeUTC
|
||||
sa.ORA_JSONB = ORA_JSONB
|
||||
sa.EncryptedString = EncryptedString
|
||||
sa.EncryptedText = EncryptedText
|
||||
sa.StoredObject = StoredObject
|
||||
sa.PasswordHash = PasswordHash
|
||||
sa.Argon2Hasher = Argon2Hasher
|
||||
sa.PasslibHasher = PasslibHasher
|
||||
sa.PwdlibHasher = PwdlibHasher
|
||||
sa.FernetBackend = FernetBackend
|
||||
sa.PGCryptoBackend = PGCryptoBackend
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '4358e7d4743a'
|
||||
down_revision = 'e9c2c7e875ae'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with warnings.catch_warnings():
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
with op.get_context().autocommit_block():
|
||||
schema_upgrades()
|
||||
data_upgrades()
|
||||
|
||||
def downgrade() -> None:
|
||||
with warnings.catch_warnings():
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
with op.get_context().autocommit_block():
|
||||
data_downgrades()
|
||||
schema_downgrades()
|
||||
|
||||
def schema_upgrades() -> None:
|
||||
"""schema upgrade migrations go here."""
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
op.create_table('duplicate_dismissals',
|
||||
sa.Column('id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
|
||||
sa.Column('book_a_id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
|
||||
sa.Column('book_b_id', sa.BigInteger().with_variant(sa.Integer(), 'sqlite'), nullable=False),
|
||||
sa.ForeignKeyConstraint(['book_a_id'], ['books.id'], name=op.f('fk_duplicate_dismissals_book_a_id_books'), ondelete='cascade'),
|
||||
sa.ForeignKeyConstraint(['book_b_id'], ['books.id'], name=op.f('fk_duplicate_dismissals_book_b_id_books'), ondelete='cascade'),
|
||||
sa.PrimaryKeyConstraint('id', name=op.f('pk_duplicate_dismissals')),
|
||||
sa.UniqueConstraint('book_a_id', 'book_b_id', name=op.f('uq_duplicate_dismissals_book_a_id'))
|
||||
)
|
||||
with op.batch_alter_table('duplicate_dismissals', schema=None) as batch_op:
|
||||
batch_op.create_index(batch_op.f('ix_duplicate_dismissals_book_a_id'), ['book_a_id'], unique=False)
|
||||
batch_op.create_index(batch_op.f('ix_duplicate_dismissals_book_b_id'), ['book_b_id'], unique=False)
|
||||
|
||||
# `server_default` so the column can be added to a table that already has rows;
|
||||
# `data_upgrades` fills in the real keys immediately afterwards.
|
||||
with op.batch_alter_table('authors', schema=None) as batch_op:
|
||||
batch_op.add_column(sa.Column('normalized_name', sa.String(), nullable=False, server_default=''))
|
||||
batch_op.create_index(batch_op.f('ix_authors_normalized_name'), ['normalized_name'], unique=False)
|
||||
|
||||
with op.batch_alter_table('books', schema=None) as batch_op:
|
||||
batch_op.add_column(sa.Column('normalized_title', sa.String(), nullable=False, server_default=''))
|
||||
batch_op.create_index(batch_op.f('ix_books_normalized_title'), ['normalized_title'], unique=False)
|
||||
|
||||
with op.batch_alter_table('identifiers', schema=None) as batch_op:
|
||||
batch_op.add_column(sa.Column('normalized_value', sa.String(), nullable=True))
|
||||
batch_op.create_index(batch_op.f('ix_identifiers_normalized_value'), ['normalized_value'], unique=False)
|
||||
|
||||
# ### end Alembic commands ###
|
||||
|
||||
def schema_downgrades() -> None:
|
||||
"""schema downgrade migrations go here."""
|
||||
# ### commands auto generated by Alembic - please adjust! ###
|
||||
with op.batch_alter_table('identifiers', schema=None) as batch_op:
|
||||
batch_op.drop_index(batch_op.f('ix_identifiers_normalized_value'))
|
||||
batch_op.drop_column('normalized_value')
|
||||
|
||||
with op.batch_alter_table('books', schema=None) as batch_op:
|
||||
batch_op.drop_index(batch_op.f('ix_books_normalized_title'))
|
||||
batch_op.drop_column('normalized_title')
|
||||
|
||||
with op.batch_alter_table('authors', schema=None) as batch_op:
|
||||
batch_op.drop_index(batch_op.f('ix_authors_normalized_name'))
|
||||
batch_op.drop_column('normalized_name')
|
||||
|
||||
with op.batch_alter_table('duplicate_dismissals', schema=None) as batch_op:
|
||||
batch_op.drop_index(batch_op.f('ix_duplicate_dismissals_book_b_id'))
|
||||
batch_op.drop_index(batch_op.f('ix_duplicate_dismissals_book_a_id'))
|
||||
|
||||
op.drop_table('duplicate_dismissals')
|
||||
# ### end Alembic commands ###
|
||||
|
||||
def data_upgrades() -> None:
|
||||
"""
|
||||
Fill the matching keys in for rows that already exist.
|
||||
|
||||
The validators on the models only fire when something is written, so without this
|
||||
every book imported before today is invisible to duplicate detection. Run through
|
||||
the same helpers the validators use, so a backfilled row and a freshly written one
|
||||
are guaranteed to agree.
|
||||
"""
|
||||
from chitai.services.matching import (
|
||||
normalize_author,
|
||||
normalize_identifier,
|
||||
normalize_title,
|
||||
)
|
||||
|
||||
connection = op.get_bind()
|
||||
|
||||
books = connection.execute(sa.text("SELECT id, title FROM books")).fetchall()
|
||||
_apply(
|
||||
connection,
|
||||
"UPDATE books SET normalized_title = :key WHERE id = :id",
|
||||
[{"id": id, "key": normalize_title(title)} for id, title in books],
|
||||
)
|
||||
|
||||
authors = connection.execute(sa.text("SELECT id, name FROM authors")).fetchall()
|
||||
_apply(
|
||||
connection,
|
||||
"UPDATE authors SET normalized_name = :key WHERE id = :id",
|
||||
[{"id": id, "key": normalize_author(name)} for id, name in authors],
|
||||
)
|
||||
|
||||
identifiers = connection.execute(
|
||||
sa.text("SELECT id, name, value FROM identifiers")
|
||||
).fetchall()
|
||||
_apply(
|
||||
connection,
|
||||
"UPDATE identifiers SET normalized_value = :key WHERE id = :id",
|
||||
[
|
||||
{"id": id, "key": normalize_identifier(name, value)}
|
||||
for id, name, value in identifiers
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def _apply(connection, statement: str, parameters: list[dict]) -> None:
|
||||
"""Run one update per row, in batches, skipping the work when there are none."""
|
||||
batch_size = 1000
|
||||
|
||||
for start in range(0, len(parameters), batch_size):
|
||||
connection.execute(sa.text(statement), parameters[start : start + batch_size])
|
||||
|
||||
|
||||
def data_downgrades() -> None:
|
||||
"""Add any optional data downgrade migrations here!"""
|
||||
@@ -0,0 +1,130 @@
|
||||
"""canonicalize author names
|
||||
|
||||
Revision ID: 49a9e85a0ffc
|
||||
Revises: ed41acf21270
|
||||
Create Date: 2026-08-15 15:59:47.331545
|
||||
|
||||
"""
|
||||
|
||||
import warnings
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from advanced_alchemy.types import EncryptedString, EncryptedText, GUID, ORA_JSONB, DateTimeUTC, StoredObject, PasswordHash, FernetBackend
|
||||
from advanced_alchemy.types.encrypted_string import PGCryptoBackend
|
||||
from advanced_alchemy.types.password_hash.argon2 import Argon2Hasher
|
||||
from advanced_alchemy.types.password_hash.passlib import PasslibHasher
|
||||
from advanced_alchemy.types.password_hash.pwdlib import PwdlibHasher
|
||||
from sqlalchemy import Text # noqa: F401
|
||||
|
||||
__all__ = ["downgrade", "upgrade", "schema_upgrades", "schema_downgrades", "data_upgrades", "data_downgrades"]
|
||||
|
||||
sa.GUID = GUID
|
||||
sa.DateTimeUTC = DateTimeUTC
|
||||
sa.ORA_JSONB = ORA_JSONB
|
||||
sa.EncryptedString = EncryptedString
|
||||
sa.EncryptedText = EncryptedText
|
||||
sa.StoredObject = StoredObject
|
||||
sa.PasswordHash = PasswordHash
|
||||
sa.Argon2Hasher = Argon2Hasher
|
||||
sa.PasslibHasher = PasslibHasher
|
||||
sa.PwdlibHasher = PwdlibHasher
|
||||
sa.FernetBackend = FernetBackend
|
||||
sa.PGCryptoBackend = PGCryptoBackend
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = '49a9e85a0ffc'
|
||||
down_revision = 'ed41acf21270'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with warnings.catch_warnings():
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
with op.get_context().autocommit_block():
|
||||
schema_upgrades()
|
||||
data_upgrades()
|
||||
|
||||
def downgrade() -> None:
|
||||
with warnings.catch_warnings():
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
with op.get_context().autocommit_block():
|
||||
data_downgrades()
|
||||
schema_downgrades()
|
||||
|
||||
def schema_upgrades() -> None:
|
||||
"""schema upgrade migrations go here."""
|
||||
pass
|
||||
|
||||
def schema_downgrades() -> None:
|
||||
"""schema downgrade migrations go here."""
|
||||
pass
|
||||
|
||||
def data_upgrades() -> None:
|
||||
"""
|
||||
Rewrite every author into the canonical form, merging the rows that collide.
|
||||
|
||||
`Author.name` is only now guaranteed tidy — until this revision extractors wrote
|
||||
whatever the file said, so one person could hold several rows: "Sam Newman" beside
|
||||
"Newman, Sam;" beside "Sam Newman.epub" (the last from a filename whose extension
|
||||
was never stripped). Each showed up as its own author in the sidebar and its own
|
||||
filter, and no amount of fixing the extractors repairs a row already written.
|
||||
|
||||
Rows that canonicalize onto one name are merged into the lowest id, which keeps
|
||||
whichever row the library has been referring to longest. `book.path` is stored, not
|
||||
derived, so renaming an author moves nothing on disk.
|
||||
"""
|
||||
from chitai.services.matching import format_author_name, normalize_author
|
||||
|
||||
connection = op.get_bind()
|
||||
authors = connection.execute(sa.text("SELECT id, name FROM authors")).fetchall()
|
||||
|
||||
groups: dict[str, list[int]] = {}
|
||||
for id, name in sorted(authors):
|
||||
# A name with nothing left of it after tidying is left exactly as it was:
|
||||
# merging those together would invent one author out of several unrelated
|
||||
# broken rows, which is worse than leaving the mess visible.
|
||||
if canonical := format_author_name(name):
|
||||
groups.setdefault(canonical, []).append(id)
|
||||
|
||||
for canonical, ids in groups.items():
|
||||
winner, losers = ids[0], ids[1:]
|
||||
|
||||
for loser in losers:
|
||||
# A book credited to both rows would otherwise breach the
|
||||
# (book_id, author_id) unique constraint the moment the link is repointed.
|
||||
connection.execute(
|
||||
sa.text(
|
||||
"DELETE FROM book_author_links WHERE author_id = :loser AND book_id IN"
|
||||
" (SELECT book_id FROM book_author_links WHERE author_id = :winner)"
|
||||
),
|
||||
{"loser": loser, "winner": winner},
|
||||
)
|
||||
connection.execute(
|
||||
sa.text(
|
||||
"UPDATE book_author_links SET author_id = :winner"
|
||||
" WHERE author_id = :loser"
|
||||
),
|
||||
{"loser": loser, "winner": winner},
|
||||
)
|
||||
connection.execute(
|
||||
sa.text("DELETE FROM authors WHERE id = :loser"), {"loser": loser}
|
||||
)
|
||||
|
||||
# Only after the losers are gone, or this collides with the unique index.
|
||||
connection.execute(
|
||||
sa.text(
|
||||
"UPDATE authors SET name = :name, normalized_name = :key WHERE id = :id"
|
||||
),
|
||||
{"id": winner, "name": canonical, "key": normalize_author(canonical)},
|
||||
)
|
||||
|
||||
|
||||
def data_downgrades() -> None:
|
||||
"""
|
||||
Nothing to undo.
|
||||
|
||||
The rows a merge removed are gone, and the spellings it replaced were never
|
||||
recorded anywhere else — there is nothing to restore them from.
|
||||
"""
|
||||
@@ -0,0 +1,122 @@
|
||||
"""recompute book matching keys
|
||||
|
||||
Revision ID: ed41acf21270
|
||||
Revises: 4358e7d4743a
|
||||
Create Date: 2026-08-15 15:44:28.341020
|
||||
|
||||
"""
|
||||
|
||||
import warnings
|
||||
|
||||
import sqlalchemy as sa
|
||||
from alembic import op
|
||||
from advanced_alchemy.types import EncryptedString, EncryptedText, GUID, ORA_JSONB, DateTimeUTC, StoredObject, PasswordHash, FernetBackend
|
||||
from advanced_alchemy.types.encrypted_string import PGCryptoBackend
|
||||
from advanced_alchemy.types.password_hash.argon2 import Argon2Hasher
|
||||
from advanced_alchemy.types.password_hash.passlib import PasslibHasher
|
||||
from advanced_alchemy.types.password_hash.pwdlib import PwdlibHasher
|
||||
from sqlalchemy import Text # noqa: F401
|
||||
|
||||
__all__ = ["downgrade", "upgrade", "schema_upgrades", "schema_downgrades", "data_upgrades", "data_downgrades"]
|
||||
|
||||
sa.GUID = GUID
|
||||
sa.DateTimeUTC = DateTimeUTC
|
||||
sa.ORA_JSONB = ORA_JSONB
|
||||
sa.EncryptedString = EncryptedString
|
||||
sa.EncryptedText = EncryptedText
|
||||
sa.StoredObject = StoredObject
|
||||
sa.PasswordHash = PasswordHash
|
||||
sa.Argon2Hasher = Argon2Hasher
|
||||
sa.PasslibHasher = PasslibHasher
|
||||
sa.PwdlibHasher = PwdlibHasher
|
||||
sa.FernetBackend = FernetBackend
|
||||
sa.PGCryptoBackend = PGCryptoBackend
|
||||
|
||||
# revision identifiers, used by Alembic.
|
||||
revision = 'ed41acf21270'
|
||||
down_revision = '4358e7d4743a'
|
||||
branch_labels = None
|
||||
depends_on = None
|
||||
|
||||
|
||||
def upgrade() -> None:
|
||||
with warnings.catch_warnings():
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
with op.get_context().autocommit_block():
|
||||
schema_upgrades()
|
||||
data_upgrades()
|
||||
|
||||
def downgrade() -> None:
|
||||
with warnings.catch_warnings():
|
||||
warnings.filterwarnings("ignore", category=UserWarning)
|
||||
with op.get_context().autocommit_block():
|
||||
data_downgrades()
|
||||
schema_downgrades()
|
||||
|
||||
def schema_upgrades() -> None:
|
||||
"""schema upgrade migrations go here."""
|
||||
pass
|
||||
|
||||
def schema_downgrades() -> None:
|
||||
"""schema downgrade migrations go here."""
|
||||
pass
|
||||
|
||||
def data_upgrades() -> None:
|
||||
"""
|
||||
Recompute every matching key against the current normalization.
|
||||
|
||||
The keys are derived, so changing a helper in `services/matching.py` silently
|
||||
invalidates every row already written — a book stored under the old rules simply
|
||||
stops matching one stored under the new ones, with nothing to show that anything
|
||||
is wrong. `normalize_title` learned to strip the compact edition markers a cover
|
||||
actually carries ("2E", "5e"), which moved `Building Microservices, 2E` onto the
|
||||
same key as `Building Microservices`.
|
||||
|
||||
Any later change to those helpers wants a revision that looks exactly like this
|
||||
one. It is idempotent and safe to re-run.
|
||||
"""
|
||||
from chitai.services.matching import (
|
||||
normalize_author,
|
||||
normalize_identifier,
|
||||
normalize_title,
|
||||
)
|
||||
|
||||
connection = op.get_bind()
|
||||
|
||||
books = connection.execute(sa.text("SELECT id, title FROM books")).fetchall()
|
||||
_apply(
|
||||
connection,
|
||||
"UPDATE books SET normalized_title = :key WHERE id = :id",
|
||||
[{"id": id, "key": normalize_title(title)} for id, title in books],
|
||||
)
|
||||
|
||||
authors = connection.execute(sa.text("SELECT id, name FROM authors")).fetchall()
|
||||
_apply(
|
||||
connection,
|
||||
"UPDATE authors SET normalized_name = :key WHERE id = :id",
|
||||
[{"id": id, "key": normalize_author(name)} for id, name in authors],
|
||||
)
|
||||
|
||||
identifiers = connection.execute(
|
||||
sa.text("SELECT id, name, value FROM identifiers")
|
||||
).fetchall()
|
||||
_apply(
|
||||
connection,
|
||||
"UPDATE identifiers SET normalized_value = :key WHERE id = :id",
|
||||
[
|
||||
{"id": id, "key": normalize_identifier(name, value)}
|
||||
for id, name, value in identifiers
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
def _apply(connection, statement: str, parameters: list[dict]) -> None:
|
||||
"""Run one update per row, in batches, skipping the work when there are none."""
|
||||
batch_size = 1000
|
||||
|
||||
for start in range(0, len(parameters), batch_size):
|
||||
connection.execute(sa.text(statement), parameters[start : start + batch_size])
|
||||
|
||||
|
||||
def data_downgrades() -> None:
|
||||
"""Add any optional data downgrade migrations here!"""
|
||||
@@ -1,9 +1,25 @@
|
||||
from enum import StrEnum
|
||||
|
||||
from pydantic import Field, PostgresDsn, computed_field
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
from advanced_alchemy.extensions.litestar import (
|
||||
SQLAlchemyAsyncConfig,
|
||||
)
|
||||
|
||||
|
||||
class DuplicateScope(StrEnum):
|
||||
"""How widely an incoming file is compared against what is already stored."""
|
||||
|
||||
LIBRARY = "library"
|
||||
"""Only files in the library being uploaded to count as duplicates."""
|
||||
|
||||
GLOBAL = "global"
|
||||
"""A file already held by any library counts as a duplicate."""
|
||||
|
||||
OFF = "off"
|
||||
"""No duplicate detection at all."""
|
||||
|
||||
|
||||
class Settings(BaseSettings):
|
||||
version: str = Field("0.0.1")
|
||||
project_name: str = Field("chitai")
|
||||
@@ -33,6 +49,14 @@ class Settings(BaseSettings):
|
||||
# Path to consume directory
|
||||
consume_path: str = Field("./consume")
|
||||
|
||||
# Duplicate detection
|
||||
duplicate_scope: DuplicateScope = Field(DuplicateScope.LIBRARY)
|
||||
|
||||
# Where the consume watcher parks files it refused as duplicates. Must sit
|
||||
# outside `consume_path`, or the watcher picks them straight back up and
|
||||
# tries to resolve the directory name as a library slug.
|
||||
duplicate_path: str = Field("./duplicates")
|
||||
|
||||
@computed_field
|
||||
@property
|
||||
def postgres_uri(self) -> PostgresDsn:
|
||||
|
||||
@@ -12,7 +12,12 @@ from litestar.params import Dependency, Body
|
||||
from litestar.enums import RequestEncodingType
|
||||
from litestar.response import File, Stream
|
||||
from litestar.exceptions import HTTPException
|
||||
from litestar.status_codes import HTTP_400_BAD_REQUEST
|
||||
from litestar.status_codes import (
|
||||
HTTP_200_OK,
|
||||
HTTP_204_NO_CONTENT,
|
||||
HTTP_400_BAD_REQUEST,
|
||||
HTTP_409_CONFLICT,
|
||||
)
|
||||
from litestar.datastructures import UploadFile
|
||||
from advanced_alchemy.service.pagination import OffsetPagination
|
||||
from advanced_alchemy.filters import CollectionFilter
|
||||
@@ -23,6 +28,24 @@ from chitai.services import dependencies as deps
|
||||
from chitai import schemas as s
|
||||
from chitai.database import models as m
|
||||
from chitai.services import BookService, BookProgressService
|
||||
from chitai.services.book import DuplicateFilesError
|
||||
|
||||
|
||||
def _duplicate_conflict(exc: DuplicateFilesError) -> HTTPException:
|
||||
"""
|
||||
Turn refused files into a 409 the caller can act on.
|
||||
|
||||
The files ride along in `extra` so the client can name them and offer to send them
|
||||
again with `allow_duplicates`, rather than being told only that something clashed.
|
||||
"""
|
||||
return HTTPException(
|
||||
status_code=HTTP_409_CONFLICT,
|
||||
detail="These files are already in the library",
|
||||
extra=[
|
||||
s.DuplicateFileRead.model_validate(duplicate).model_dump(mode="json")
|
||||
for duplicate in exc.duplicates
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
class BookController(Controller):
|
||||
@@ -63,6 +86,7 @@ class BookController(Controller):
|
||||
books_service: BookService,
|
||||
library: m.Library,
|
||||
data: Annotated[s.BookCreate, Body(media_type=RequestEncodingType.MULTI_PART)],
|
||||
allow_duplicates: bool = False,
|
||||
) -> s.BookRead:
|
||||
"""
|
||||
Create a new book with metadata and files.
|
||||
@@ -73,6 +97,10 @@ class BookController(Controller):
|
||||
Path Parameters:
|
||||
library_id: The ID of the library the book belongs to.
|
||||
|
||||
Query Parameters:
|
||||
allow_duplicates: If True, store the files even if the library already
|
||||
holds them.
|
||||
|
||||
Request Body:
|
||||
data: Book creation data including metadata and files.
|
||||
|
||||
@@ -83,9 +111,17 @@ class BookController(Controller):
|
||||
Returns:
|
||||
The created book as a BookRead schema.
|
||||
|
||||
Raises:
|
||||
HTTPException: 409 if any of the files is already in the library.
|
||||
"""
|
||||
|
||||
result = await books_service.create_book(data, library)
|
||||
try:
|
||||
result = await books_service.create_book(
|
||||
data, library, screen_duplicates=not allow_duplicates
|
||||
)
|
||||
except DuplicateFilesError as exc:
|
||||
raise _duplicate_conflict(exc)
|
||||
|
||||
book = await books_service.get(result.id)
|
||||
return books_service.to_schema(book, schema_type=s.BookRead)
|
||||
|
||||
@@ -97,13 +133,20 @@ class BookController(Controller):
|
||||
data: Annotated[
|
||||
s.BooksCreateFromFiles, Body(media_type=RequestEncodingType.MULTI_PART)
|
||||
],
|
||||
) -> OffsetPagination[s.BookRead]:
|
||||
allow_duplicates: bool = False,
|
||||
) -> s.BooksUploadResult:
|
||||
"""
|
||||
Create multiple books from uploaded files.
|
||||
|
||||
Groups files by directory and creates separate books for each group.
|
||||
Metadata is automatically extracted from the files.
|
||||
|
||||
Files the library already holds are skipped rather than refused, and reported
|
||||
back so the caller can say which ones did not make it in and why.
|
||||
|
||||
Query Parameters:
|
||||
allow_duplicates: If True, store every file, even one already held.
|
||||
|
||||
Request Body:
|
||||
data: Container with list of uploaded files.
|
||||
|
||||
@@ -112,19 +155,200 @@ class BookController(Controller):
|
||||
library: The library the books belong to.
|
||||
|
||||
Returns:
|
||||
Paginated list of created books.
|
||||
The books created, and the files skipped as duplicates.
|
||||
"""
|
||||
try:
|
||||
results = await books_service.create_many_from_files(data, library)
|
||||
result = await books_service.create_many_from_files(
|
||||
data, library, allow_duplicates=allow_duplicates
|
||||
)
|
||||
except ValueError:
|
||||
raise HTTPException(
|
||||
status_code=HTTP_400_BAD_REQUEST, detail="Must upload at least one file"
|
||||
)
|
||||
|
||||
books = await books_service.list(
|
||||
CollectionFilter("id", [result.id for result in results])
|
||||
books = (
|
||||
await books_service.list(
|
||||
CollectionFilter("id", [book.id for book in result.books])
|
||||
)
|
||||
if result.books
|
||||
else []
|
||||
)
|
||||
return books_service.to_schema(books, schema_type=s.BookRead)
|
||||
|
||||
return s.BooksUploadResult(
|
||||
created=[
|
||||
books_service.to_schema(book, schema_type=s.BookRead) for book in books
|
||||
],
|
||||
skipped=[
|
||||
s.DuplicateFileRead.model_validate(duplicate)
|
||||
for duplicate in result.duplicates
|
||||
],
|
||||
possible_duplicates=[
|
||||
s.PossibleDuplicateRead.model_validate(possible)
|
||||
for possible in result.possible_duplicates
|
||||
],
|
||||
)
|
||||
|
||||
@get(path="duplicate-books")
|
||||
async def list_duplicate_books(
|
||||
self, books_service: BookService, library: m.Library
|
||||
) -> list[s.DuplicateBookGroupRead]:
|
||||
"""
|
||||
Report books already in the library that look like copies of one another.
|
||||
|
||||
The import-time check only ever sees what is arriving, so this is what covers
|
||||
a collection someone already has. Matching is on metadata and therefore a
|
||||
guess: a group is a question for the reader, not a verdict.
|
||||
|
||||
Query Parameters:
|
||||
library_id: The library to review.
|
||||
|
||||
Injected Dependencies:
|
||||
books_service: The book service for database operations.
|
||||
library: The library to review.
|
||||
|
||||
Returns:
|
||||
One entry per group of two or more books. Empty when there is nothing to
|
||||
review, or when duplicate detection is switched off.
|
||||
"""
|
||||
groups = await books_service.find_duplicate_book_groups(library)
|
||||
|
||||
return [
|
||||
s.DuplicateBookGroupRead(
|
||||
books=[s.DuplicateBookRead.model_validate(book) for book in group]
|
||||
)
|
||||
for group in groups
|
||||
]
|
||||
|
||||
@post(path="merge")
|
||||
async def merge_books(
|
||||
self, books_service: BookService, library: m.Library, data: s.BookMerge
|
||||
) -> s.BookRead:
|
||||
"""
|
||||
Fold several books into one and delete the records folded in.
|
||||
|
||||
The survivor keeps its id, so links and bookmarks still resolve. Files, reading
|
||||
progress, shelves, tags and unheld identifiers move onto it; metadata is only
|
||||
changed by what `metadata` names, because choosing between two titles is the
|
||||
reader's judgement rather than this endpoint's.
|
||||
|
||||
Nothing is removed from disk — a wrong merge should cost metadata that can be
|
||||
retyped, not a book.
|
||||
|
||||
Query Parameters:
|
||||
library_id: The library the books belong to.
|
||||
|
||||
Request Body:
|
||||
data: The survivor, the books to fold in, and the resolved metadata.
|
||||
|
||||
Injected Dependencies:
|
||||
books_service: The book service for database operations.
|
||||
library: The library the books belong to.
|
||||
|
||||
Returns:
|
||||
The surviving book.
|
||||
|
||||
Raises:
|
||||
HTTPException: 400 if fewer than two distinct books were named, one is
|
||||
unknown, or they do not all belong to one library.
|
||||
"""
|
||||
try:
|
||||
book = await books_service.merge_books(
|
||||
data.survivor_id,
|
||||
data.merged_ids,
|
||||
library,
|
||||
metadata=data.metadata.model_dump(exclude_unset=True)
|
||||
if data.metadata
|
||||
else None,
|
||||
)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=HTTP_400_BAD_REQUEST, detail=str(exc))
|
||||
|
||||
return books_service.to_schema(book, schema_type=s.BookRead)
|
||||
|
||||
@post(path="duplicate-books/dismissals", status_code=HTTP_204_NO_CONTENT)
|
||||
async def dismiss_duplicate_books(
|
||||
self, books_service: BookService, data: s.DuplicateDismissal
|
||||
) -> None:
|
||||
"""
|
||||
Record that two books are not the same book.
|
||||
|
||||
Without this the review screen proposes the same wrong pair forever, which is
|
||||
how a reader learns to stop looking at it.
|
||||
|
||||
Request Body:
|
||||
data: The two book IDs. Order does not matter.
|
||||
|
||||
Injected Dependencies:
|
||||
books_service: The book service for database operations.
|
||||
|
||||
Raises:
|
||||
HTTPException: 400 if the two IDs are the same or either book is unknown.
|
||||
"""
|
||||
try:
|
||||
await books_service.dismiss_duplicates(data.book_a_id, data.book_b_id)
|
||||
except ValueError as exc:
|
||||
raise HTTPException(status_code=HTTP_400_BAD_REQUEST, detail=str(exc))
|
||||
|
||||
@delete(path="duplicate-books/dismissals")
|
||||
async def restore_duplicate_books(
|
||||
self, books_service: BookService, book_a_id: int, book_b_id: int
|
||||
) -> None:
|
||||
"""
|
||||
Undo a dismissal, so the pair is proposed again.
|
||||
|
||||
Query Parameters:
|
||||
book_a_id: One of the two books.
|
||||
book_b_id: The other. Order does not matter.
|
||||
|
||||
Injected Dependencies:
|
||||
books_service: The book service for database operations.
|
||||
"""
|
||||
await books_service.restore_duplicates(book_a_id, book_b_id)
|
||||
|
||||
# A question, not a change: 200 rather than the 201 a POST would default to.
|
||||
@post(path="duplicate-files", status_code=HTTP_200_OK)
|
||||
async def check_duplicate_files(
|
||||
self,
|
||||
books_service: BookService,
|
||||
library: m.Library,
|
||||
data: list[s.FileFingerprint],
|
||||
) -> list[s.DuplicateFileRead]:
|
||||
"""
|
||||
Report which of the given files the library already holds.
|
||||
|
||||
Lets a client ask before it uploads anything, which is the difference between
|
||||
re-sending a folder of books and re-sending twelve kilobytes of hashes.
|
||||
|
||||
Query Parameters:
|
||||
library_id: The library to check against.
|
||||
|
||||
Request Body:
|
||||
data: Hash and size for each file, optionally with the name to echo back.
|
||||
|
||||
Injected Dependencies:
|
||||
books_service: The book service for database operations.
|
||||
library: The library to check against.
|
||||
|
||||
Returns:
|
||||
One entry per submitted file that is already stored. Files that are not
|
||||
are absent.
|
||||
"""
|
||||
matches = await books_service.find_duplicate_files(
|
||||
((item.hash, item.size) for item in data), library
|
||||
)
|
||||
|
||||
return [
|
||||
s.DuplicateFileRead(
|
||||
filename=item.filename or match.filename,
|
||||
hash=match.hash,
|
||||
size=match.size,
|
||||
library_id=match.library_id,
|
||||
book_id=match.book_id,
|
||||
book_title=match.book_title,
|
||||
)
|
||||
for item in data
|
||||
if (match := matches.get((item.hash, item.size))) is not None
|
||||
]
|
||||
|
||||
@get(path="/{book_id:int}")
|
||||
async def get_book_by_id(
|
||||
@@ -303,13 +527,20 @@ class BookController(Controller):
|
||||
],
|
||||
library: m.Library,
|
||||
books_service: BookService,
|
||||
allow_duplicates: bool = False,
|
||||
) -> s.BookRead:
|
||||
"""
|
||||
Add files to an existing book.
|
||||
|
||||
A file the book already carries is ignored, so re-sending one is harmless.
|
||||
|
||||
Path Parameters:
|
||||
book_id: The ID of the book to modify
|
||||
|
||||
Query Parameters:
|
||||
allow_duplicates: If True, store the files even if the library already
|
||||
holds them.
|
||||
|
||||
Request Body:
|
||||
files: The files to add to the book
|
||||
|
||||
@@ -320,9 +551,17 @@ class BookController(Controller):
|
||||
Returns:
|
||||
The modified book
|
||||
|
||||
Raises:
|
||||
HTTPException: 409 if a file is already stored under a different book.
|
||||
"""
|
||||
|
||||
await books_service.add_files(book_id, data, library)
|
||||
try:
|
||||
await books_service.add_files(
|
||||
book_id, data, library, allow_duplicates=allow_duplicates
|
||||
)
|
||||
except DuplicateFilesError as exc:
|
||||
raise _duplicate_conflict(exc)
|
||||
|
||||
book = await books_service.get(book_id)
|
||||
return books_service.to_schema(book, schema_type=s.BookRead)
|
||||
|
||||
|
||||
@@ -3,6 +3,7 @@ from .book import Book, Identifier, FileMetadata
|
||||
from .book_list import BookList, BookListLink
|
||||
from .book_progress import BookProgress
|
||||
from .book_series import BookSeries
|
||||
from .duplicate_dismissal import DuplicateDismissal
|
||||
from .kosync_device import KosyncDevice
|
||||
from .kosync_progress import KosyncProgress
|
||||
from .library import Library
|
||||
|
||||
@@ -5,6 +5,7 @@ from sqlalchemy import ColumnElement, ForeignKey, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped
|
||||
from sqlalchemy.orm import mapped_column
|
||||
from sqlalchemy.orm import relationship
|
||||
from sqlalchemy.orm import validates
|
||||
|
||||
from advanced_alchemy.base import BigIntAuditBase, BigIntBase
|
||||
from advanced_alchemy.mixins import UniqueMixin
|
||||
@@ -16,18 +17,55 @@ if TYPE_CHECKING:
|
||||
class Author(BigIntAuditBase, UniqueMixin):
|
||||
__tablename__ = "authors"
|
||||
|
||||
# Always the canonical form — see `_canonicalize`. Extractors hand over whatever
|
||||
# the file happened to say: "Newman, Sam;" from a `DC:creator` list, or
|
||||
# "Sam Newman.epub" from a filename. Storing those verbatim is how one person ends
|
||||
# up as several rows in the sidebar.
|
||||
name: Mapped[str] = mapped_column(unique=True, index=True)
|
||||
|
||||
# Kept current by `_canonicalize` too — never assign it directly. Not unique: two
|
||||
# spellings that survive canonicalization, "Steve Mcconnell" and "Steve McConnell",
|
||||
# are still one person to a reader, which is what this column exists to express.
|
||||
normalized_name: Mapped[str] = mapped_column(default="", index=True)
|
||||
|
||||
description: Mapped[Optional[str]]
|
||||
|
||||
@validates("name")
|
||||
def _canonicalize(self, _key: str, name: str) -> str:
|
||||
"""
|
||||
Store the tidied name, and derive the matching key from it.
|
||||
|
||||
A validator so no write can get around it, and `unique_hash` / `unique_filter`
|
||||
below tidy the same way so `as_unique_async` looks the row up under the name it
|
||||
would actually be stored as. All three have to agree: if the lookup used the
|
||||
raw name and the insert used the tidy one, every variant spelling would miss
|
||||
the existing row and then collide with it on the unique index.
|
||||
"""
|
||||
# Imported here rather than at module scope: `chitai.services.matching` cannot
|
||||
# be reached without initialising the `chitai.services` package, which imports
|
||||
# the services, which import this module.
|
||||
from chitai.services.matching import format_author_name, normalize_author
|
||||
|
||||
name = format_author_name(name)
|
||||
self.normalized_name = normalize_author(name)
|
||||
return name
|
||||
|
||||
@classmethod
|
||||
def _tidy(cls, name: str) -> str:
|
||||
"""The name as `_canonicalize` would store it."""
|
||||
from chitai.services.matching import format_author_name
|
||||
|
||||
return format_author_name(name)
|
||||
|
||||
@classmethod
|
||||
def unique_hash(cls, name: str) -> Hashable:
|
||||
"""Generate a unique hash for deduplication."""
|
||||
return name
|
||||
return cls._tidy(name)
|
||||
|
||||
@classmethod
|
||||
def unique_filter(cls, name: str) -> ColumnElement[bool]:
|
||||
"""SQL filter for finding existing records."""
|
||||
return cls.name == name
|
||||
return cls.name == cls._tidy(name)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"Author({self.name!r})"
|
||||
|
||||
@@ -2,13 +2,10 @@ from datetime import date
|
||||
from typing import TYPE_CHECKING, Any, Optional
|
||||
|
||||
from sqlalchemy import Index, ForeignKey, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship
|
||||
from sqlalchemy.orm import mapped_column
|
||||
from sqlalchemy.orm import relationship
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
|
||||
from sqlalchemy.ext.orderinglist import ordering_list
|
||||
from sqlalchemy.ext.associationproxy import association_proxy
|
||||
from sqlalchemy.ext.associationproxy import AssociationProxy
|
||||
from sqlalchemy.orm.collections import attribute_keyed_dict
|
||||
|
||||
from advanced_alchemy.base import BigIntAuditBase, BigIntBase
|
||||
|
||||
@@ -44,6 +41,13 @@ class Book(BigIntAuditBase):
|
||||
library: Mapped["Library"] = relationship(back_populates="books")
|
||||
|
||||
title: Mapped[str]
|
||||
|
||||
# Kept current by `_normalize_title` below — never assign it directly.
|
||||
#
|
||||
# Deliberately not unique: two spellings collapsing onto one value is the whole
|
||||
# point of the column, and a second edition is allowed to exist.
|
||||
normalized_title: Mapped[str] = mapped_column(default="", index=True)
|
||||
|
||||
subtitle: Mapped[Optional[str]]
|
||||
description: Mapped[Optional[str]]
|
||||
published_date: Mapped[Optional[date]]
|
||||
@@ -111,6 +115,24 @@ class Book(BigIntAuditBase):
|
||||
def progress(self) -> Optional["BookProgress"]:
|
||||
return self.progress_records[0] if self.progress_records else None
|
||||
|
||||
@validates("title")
|
||||
def _normalize_title(self, _key: str, title: str) -> str:
|
||||
"""
|
||||
Derive `normalized_title` from whatever writes the title.
|
||||
|
||||
A validator rather than a service call because `BookService` sets titles from
|
||||
at least three places — `to_model_on_create`, `to_model_on_update` and the
|
||||
`setattr` loop in `_populate_with_unique_relationships` — and a fourth would
|
||||
otherwise leave the key silently stale.
|
||||
"""
|
||||
# Imported here rather than at module scope: `chitai.services.matching` cannot
|
||||
# be reached without initialising the `chitai.services` package, which imports
|
||||
# the services, which import this module.
|
||||
from chitai.services.matching import normalize_title
|
||||
|
||||
self.normalized_title = normalize_title(title)
|
||||
return title
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"Book({self.title=!r})"
|
||||
|
||||
@@ -132,6 +154,24 @@ class Identifier(BigIntBase):
|
||||
book_id: Mapped[int] = mapped_column(ForeignKey("books.id", ondelete="cascade"))
|
||||
value: Mapped[str]
|
||||
|
||||
# Kept current by `_normalize` below — never assign it directly. Null for an
|
||||
# identifier that cannot carry a match: a per-build UUID, or an ISBN that fails
|
||||
# its own checksum.
|
||||
normalized_value: Mapped[Optional[str]] = mapped_column(index=True)
|
||||
|
||||
@validates("name", "value")
|
||||
def _normalize(self, key: str, value: str) -> str:
|
||||
"""Recompute `normalized_value` whenever either half of the pair changes."""
|
||||
from chitai.services.matching import normalize_identifier
|
||||
|
||||
name = value if key == "name" else self.name
|
||||
raw = value if key == "value" else self.value
|
||||
|
||||
self.normalized_value = (
|
||||
normalize_identifier(name, raw) if name and raw else None
|
||||
)
|
||||
return value
|
||||
|
||||
def __repr__(self):
|
||||
return f"Identifier({self.name!r} : {self.value!r})"
|
||||
|
||||
@@ -139,6 +179,15 @@ class Identifier(BigIntBase):
|
||||
class FileMetadata(BigIntBase):
|
||||
__tablename__ = "file_metadata"
|
||||
|
||||
__table_args__ = (
|
||||
# Deliberately not unique. The hash is KOReader's partial MD5, which samples
|
||||
# 12 KiB of the file, so two genuinely different files can collide — and an
|
||||
# existing database may already hold duplicates, which a unique index would
|
||||
# refuse to build over. Duplicate detection pairs it with `size` and treats a
|
||||
# match as advisory, so this only has to make the lookup cheap.
|
||||
Index("ix_file_metadata_hash", "hash"),
|
||||
)
|
||||
|
||||
book_id: Mapped[int] = mapped_column(ForeignKey("books.id", ondelete="cascade"))
|
||||
book: Mapped[Book] = relationship(back_populates="files")
|
||||
hash: Mapped[str]
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
from sqlalchemy import ForeignKey, UniqueConstraint
|
||||
from sqlalchemy.orm import Mapped, mapped_column
|
||||
|
||||
from advanced_alchemy.base import BigIntBase
|
||||
|
||||
|
||||
class DuplicateDismissal(BigIntBase):
|
||||
"""
|
||||
Two books a reader has said are not the same book.
|
||||
|
||||
Title and author matching is probabilistic, so it will keep proposing a second
|
||||
edition, a translation and a sequel that shares its predecessor's name. A review
|
||||
screen with no way to disagree with it nags forever, which is how people learn to
|
||||
ignore a screen.
|
||||
|
||||
The pair is stored ordered — `book_a_id` is always the lower id — so "A and B" and
|
||||
"B and A" are one row and the unique constraint can do its job. Use `pair()` rather
|
||||
than assigning the columns directly.
|
||||
"""
|
||||
|
||||
__tablename__ = "duplicate_dismissals"
|
||||
__table_args__ = (UniqueConstraint("book_a_id", "book_b_id"),)
|
||||
|
||||
book_a_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("books.id", ondelete="cascade"), index=True
|
||||
)
|
||||
book_b_id: Mapped[int] = mapped_column(
|
||||
ForeignKey("books.id", ondelete="cascade"), index=True
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def pair(first: int, second: int) -> tuple[int, int]:
|
||||
"""The two book ids in the order this table stores them."""
|
||||
return (first, second) if first <= second else (second, first)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"DuplicateDismissal({self.book_a_id!r}, {self.book_b_id!r})"
|
||||
@@ -4,7 +4,15 @@ from .book import (
|
||||
BookProgressCreate,
|
||||
BookProgressRead,
|
||||
BooksCreateFromFiles,
|
||||
BooksUploadResult,
|
||||
BookMerge,
|
||||
BookMetadataUpdate,
|
||||
DuplicateBookGroupRead,
|
||||
DuplicateBookRead,
|
||||
DuplicateDismissal,
|
||||
DuplicateFileRead,
|
||||
FileFingerprint,
|
||||
PossibleDuplicateRead,
|
||||
FileMetadataRead,
|
||||
BookSeriesRead,
|
||||
)
|
||||
|
||||
@@ -129,6 +129,83 @@ class BooksCreateFromFiles(BaseModel):
|
||||
model_config = ConfigDict(arbitrary_types_allowed=True)
|
||||
|
||||
|
||||
class FileFingerprint(BaseModel):
|
||||
"""What a client can say about a file it has not uploaded yet."""
|
||||
|
||||
hash: str
|
||||
size: int
|
||||
filename: str = ""
|
||||
|
||||
|
||||
class DuplicateFileRead(BaseModel):
|
||||
"""A file that was not stored because the library already holds its bytes."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
filename: str
|
||||
hash: str
|
||||
size: int
|
||||
library_id: int
|
||||
|
||||
# Null when the match was another file in the same upload, which has no row yet.
|
||||
book_id: int | None = None
|
||||
book_title: str | None = None
|
||||
|
||||
|
||||
class DuplicateBookRead(BaseModel):
|
||||
"""
|
||||
A stored book that may be the same book as another one.
|
||||
|
||||
Unlike `DuplicateFileRead` this is a guess: the evidence is metadata two editions
|
||||
of one work legitimately share. Nothing was refused on the strength of it.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
book_id: int
|
||||
title: str
|
||||
authors: list[str]
|
||||
library_id: int
|
||||
cover_image: Path | None = None
|
||||
|
||||
# Why it matched: "identifier" and/or "title-author".
|
||||
matched_on: list[str]
|
||||
|
||||
|
||||
class PossibleDuplicateRead(BaseModel):
|
||||
"""A book that was imported, together with what it might be a second copy of."""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
book_id: int
|
||||
title: str
|
||||
candidates: list[DuplicateBookRead]
|
||||
|
||||
|
||||
class DuplicateBookGroupRead(BaseModel):
|
||||
"""Books the library holds that all look like copies of one book."""
|
||||
|
||||
books: list[DuplicateBookRead]
|
||||
|
||||
|
||||
class DuplicateDismissal(BaseModel):
|
||||
"""Two books a reader is saying are not the same book."""
|
||||
|
||||
book_a_id: int
|
||||
book_b_id: int
|
||||
|
||||
|
||||
class BooksUploadResult(BaseModel):
|
||||
"""The outcome of a multi-file upload: what was created, and what was skipped."""
|
||||
|
||||
created: list["BookRead"]
|
||||
skipped: list[DuplicateFileRead]
|
||||
|
||||
# Created, not skipped — these are books that went in and look like something the
|
||||
# library already had. The reader decides what to do about it.
|
||||
possible_duplicates: list[PossibleDuplicateRead] = Field(default_factory=list)
|
||||
|
||||
|
||||
class BookMetadataUpdate(BaseModel):
|
||||
title: str | None = None
|
||||
subtitle: str | None = None
|
||||
@@ -170,6 +247,20 @@ class BookMetadataUpdate(BaseModel):
|
||||
return v
|
||||
|
||||
|
||||
class BookMerge(BaseModel):
|
||||
"""
|
||||
Fold several books into one.
|
||||
|
||||
`metadata` is the reader's resolution of the fields the records disagreed on.
|
||||
Anything it does not name keeps the survivor's value — merging metadata is a
|
||||
judgement, so nothing is guessed on the caller's behalf.
|
||||
"""
|
||||
|
||||
survivor_id: int
|
||||
merged_ids: list[int]
|
||||
metadata: Optional["BookMetadataUpdate"] = None
|
||||
|
||||
|
||||
class BookProgressCreate(BaseModel):
|
||||
percentage: float
|
||||
epub_cfi: str | None = None
|
||||
|
||||
+1296
-44
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,7 @@
|
||||
import asyncio
|
||||
from pathlib import Path
|
||||
from collections import defaultdict
|
||||
from chitai.config import settings
|
||||
from chitai.database.models.library import Library
|
||||
from chitai.services import BookService, LibraryService
|
||||
from chitai.services.metadata_extractor import Extractor
|
||||
@@ -111,13 +112,32 @@ class ConsumeDirectoryWatcher:
|
||||
"""Process a batch of files."""
|
||||
try:
|
||||
|
||||
books = await self.book_service.create_many_from_existing_files(
|
||||
result = await self.book_service.create_many_from_existing_files(
|
||||
list(file_paths),
|
||||
self.watch_path / Path(library_slug),
|
||||
library=await self._get_library(library_slug),
|
||||
)
|
||||
|
||||
print(f"Created {len(books)} books!")
|
||||
print(f"Created {len(result.books)} books!")
|
||||
|
||||
if result.duplicates:
|
||||
print(
|
||||
f"Moved {len(result.duplicates)} already-stored file(s) "
|
||||
f"to {settings.duplicate_path}"
|
||||
)
|
||||
|
||||
# Imported all the same — a metadata match is a guess, and there is nobody
|
||||
# here to ask. The library's duplicates screen is where these get decided.
|
||||
for possible in result.possible_duplicates:
|
||||
names = ", ".join(
|
||||
f"{candidate.title} (#{candidate.book_id})"
|
||||
for candidate in possible.candidates
|
||||
)
|
||||
print(
|
||||
f"Imported {possible.title!r} (#{possible.book_id}), which may "
|
||||
f"already be in the library as: {names}"
|
||||
)
|
||||
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error processing batch: {e}")
|
||||
|
||||
@@ -127,6 +127,31 @@ def create_book_filter_dependencies(
|
||||
# Get base filters first
|
||||
filters = create_filter_dependencies(config, dep_defaults)
|
||||
|
||||
# OVERRIDE: id filter typed by the configured id type, not always `str`
|
||||
#
|
||||
# advanced_alchemy's `provide_id_filter` annotates `ids` as `list[str]` and
|
||||
# ignores `config["id_filter"]` entirely, so `?ids=12` reaches the database as
|
||||
# the string "12" and Postgres refuses `bigint = character varying`. Nothing
|
||||
# called `?ids=` until the duplicates screen needed to fetch a handful of books
|
||||
# by id, which is why it went unnoticed.
|
||||
if id_type := config.get("id_filter"):
|
||||
id_field = config.get("id_field", "id")
|
||||
|
||||
def provide_typed_id_filter(
|
||||
ids=Parameter(query="ids", default=None, required=False),
|
||||
) -> CollectionFilter:
|
||||
return CollectionFilter(field_name=id_field, values=ids)
|
||||
|
||||
# Attached as a type object rather than written as an annotation: this module
|
||||
# has `from __future__ import annotations`, so a written one is stored as the
|
||||
# string "Optional[list[id_type]]" and resolved against module globals, where
|
||||
# a local named `id_type` does not exist.
|
||||
provide_typed_id_filter.__annotations__["ids"] = Optional[list[id_type]]
|
||||
|
||||
filters[dep_defaults.ID_FILTER_DEPENDENCY_KEY] = Provide(
|
||||
provide_typed_id_filter, sync_to_thread=False
|
||||
)
|
||||
|
||||
# OVERRIDE: Custom search filter with trigram search
|
||||
if config.get("search"):
|
||||
search_fields = config.get("search")
|
||||
|
||||
@@ -0,0 +1,207 @@
|
||||
# src/chitai/services/matching.py
|
||||
|
||||
"""
|
||||
Normalisation for book-level duplicate detection.
|
||||
|
||||
Two copies of one book rarely agree on how it is written down. One says
|
||||
`The Metamorphosis`, the other `Metamorphosis`; one credits `Kafka, Franz`, the other
|
||||
`Franz Kafka`; one carries the ISBN-10 and the other the ISBN-13 of the same edition.
|
||||
These functions reduce each of those to a single key, so the comparison is an equality
|
||||
test the database can index rather than a similarity score nobody can explain.
|
||||
|
||||
Everything here is pure: the keys are computed once and stored on the row (see
|
||||
`Book.normalized_title`, `Author.normalized_name`, `Identifier.normalized_value`), so
|
||||
no Postgres extension is needed at query time.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import unicodedata
|
||||
|
||||
from chitai.services.utils import is_valid_isbn, isbn10_to_isbn13
|
||||
|
||||
|
||||
# Asides a title carries that say nothing about which book it is:
|
||||
# "Frankenstein (Illustrated)", "Dune [Deluxe]".
|
||||
_BRACKETED = re.compile(r"[(\[{][^)\]}]*[)\]}]")
|
||||
|
||||
# Edition and format qualifiers, matched only as a *trailing* run of words. Anchoring
|
||||
# to the end is what keeps "The Illustrated Man" a book and "Moby Dick Illustrated" a
|
||||
# format note — a qualifier trails the title, it is never the thing the title is about.
|
||||
_EDITION_NOISE = re.compile(
|
||||
r"\s+(?:"
|
||||
# "2nd edition", but also the compact forms publishers actually print on a
|
||||
# cover: "2E", "3 Ed", "5e". The number alone is never enough — "Catch 22" is
|
||||
# a title and must survive.
|
||||
r"\d+(?:st|nd|rd|th)?\s*(?:edition|edn|ed|e)"
|
||||
r"|(?:first|second|third|fourth|fifth|sixth|new|revised|expanded|updated|"
|
||||
r"annotated|illustrated|unabridged|abridged|complete|definitive|deluxe|"
|
||||
r"anniversary|collectors|international|kindle|paperback|hardcover|hardback|"
|
||||
r"ebook|audiobook)"
|
||||
r"(?:\s+(?:and|&)\s+\w+)*"
|
||||
r"(?:\s+ed(?:ition|n)?)?"
|
||||
r")$"
|
||||
)
|
||||
|
||||
_LEADING_ARTICLE = re.compile(r"^(?:the|a|an)\s+")
|
||||
|
||||
# Anything that is not a letter, a digit or a space, once accents are gone.
|
||||
_PUNCTUATION = re.compile(r"[^0-9a-z ]+")
|
||||
|
||||
_WHITESPACE = re.compile(r"\s+")
|
||||
|
||||
# Junk an extractor leaves on the end of a name: the `;` from a `DC:creator` list, the
|
||||
# `.epub` from a filename the author's name was read out of.
|
||||
_TRAILING_SEPARATORS = " ;,&/"
|
||||
_FILE_EXTENSION = re.compile(
|
||||
r"\.(?:epub|pdf|mobi|azw3?|djvu|fb2|txt|cbz|cbr)$", re.IGNORECASE
|
||||
)
|
||||
|
||||
# `J. R. R.` survives punctuation stripping as three one-letter words; `J.R.R.` as one.
|
||||
# Joining any run of them makes both `jrr`.
|
||||
_INITIAL_RUN = re.compile(r"\b(?:[a-z] )+[a-z]\b")
|
||||
|
||||
# ISBNs are the same number under several names; everything else keeps its own.
|
||||
_ISBN_NAMES = {"isbn", "isbn-10", "isbn10", "isbn-13", "isbn13"}
|
||||
|
||||
# Generated fresh for every build of a file, so two copies of one book never share one.
|
||||
# Matching on them would only re-find files the hash check already catches.
|
||||
_PER_BUILD_NAMES = {"uuid", "urn:uuid"}
|
||||
|
||||
# Below this an identifier is not specific enough to be evidence: a Calibre `id` of
|
||||
# "42" would otherwise pair two unrelated books.
|
||||
_MIN_IDENTIFIER_LENGTH = 4
|
||||
|
||||
|
||||
def _fold(text: str) -> str:
|
||||
"""Casefolded, accent-free, punctuation-free, single-spaced."""
|
||||
decomposed = unicodedata.normalize("NFKD", text.casefold())
|
||||
unaccented = "".join(c for c in decomposed if not unicodedata.combining(c))
|
||||
|
||||
return _WHITESPACE.sub(" ", _PUNCTUATION.sub(" ", unaccented)).strip()
|
||||
|
||||
|
||||
def normalize_title(title: str | None) -> str:
|
||||
"""
|
||||
Reduce a title to the key two copies of one book should share.
|
||||
|
||||
`Book.subtitle` is already split off by `Extractor.format_book_title`, so only what
|
||||
is left in the title column is considered here.
|
||||
|
||||
Args:
|
||||
title: The title as it was stored.
|
||||
|
||||
Returns:
|
||||
The comparison key, or an empty string if nothing survives normalisation —
|
||||
which is the signal not to match on the title at all.
|
||||
"""
|
||||
if not title:
|
||||
return ""
|
||||
|
||||
folded = _fold(_BRACKETED.sub(" ", title).replace("&", " and "))
|
||||
|
||||
# Repeated because qualifiers stack: "Dune Deluxe Edition Illustrated".
|
||||
while (trimmed := _EDITION_NOISE.sub("", folded)) != folded:
|
||||
folded = trimmed
|
||||
|
||||
# An article says nothing, but a title that is only an article is not improved by
|
||||
# having none, and neither is one that noise removal emptied out.
|
||||
return _LEADING_ARTICLE.sub("", folded, count=1) or folded
|
||||
|
||||
|
||||
def format_author_name(name: str | None) -> str:
|
||||
"""
|
||||
Tidy an author's name into the one form the library writes them in.
|
||||
|
||||
Distinct from `normalize_author`, which throws away case, accents and spacing to
|
||||
build a comparison key. This one is what a reader sees, so it keeps everything
|
||||
that belongs to the name and only removes what an extractor added: a trailing
|
||||
separator left over from a creator list, a file extension carried in from a
|
||||
filename, and the `Surname, Given` ordering that EPUBs file names under.
|
||||
|
||||
Args:
|
||||
name: The name as the file or filename gave it.
|
||||
|
||||
Returns:
|
||||
The name to store, or an empty string if there is nothing left of it.
|
||||
"""
|
||||
if not name:
|
||||
return ""
|
||||
|
||||
tidied = _FILE_EXTENSION.sub("", name.strip().strip(_TRAILING_SEPARATORS).strip())
|
||||
|
||||
if tidied.count(",") == 1:
|
||||
surname, given = (part.strip() for part in tidied.split(","))
|
||||
|
||||
# Only when the part before the comma is a single word. "Dave Thomas, Andy
|
||||
# Hunt" is two people in one string, and flipping it would invent a third
|
||||
# person who does not exist. Leaving an unrecognised form alone is the safe
|
||||
# failure; rewriting it wrongly is not.
|
||||
if surname and given and " " not in surname:
|
||||
tidied = f"{given} {surname}"
|
||||
|
||||
return _WHITESPACE.sub(" ", tidied).strip()
|
||||
|
||||
|
||||
def normalize_author(name: str | None) -> str:
|
||||
"""
|
||||
Reduce an author's name to the key their other books should share.
|
||||
|
||||
Deliberately not reduced to surname plus initial: that collides unrelated people,
|
||||
and a wrong match here is a book pointed at a stranger's shelf.
|
||||
|
||||
Args:
|
||||
name: The name as it was stored, in either `Franz Kafka` or `Kafka, Franz` form.
|
||||
|
||||
Returns:
|
||||
The comparison key, or an empty string if nothing survives normalisation.
|
||||
"""
|
||||
if not name:
|
||||
return ""
|
||||
|
||||
# `Kafka, Franz` is one name written backwards. More than one comma is a list, or a
|
||||
# suffix, and guessing at either does more harm than leaving it alone.
|
||||
if name.count(",") == 1:
|
||||
surname, forename = name.split(",")
|
||||
name = f"{forename.strip()} {surname.strip()}"
|
||||
|
||||
folded = _fold(name)
|
||||
|
||||
return _INITIAL_RUN.sub(lambda run: run.group().replace(" ", ""), folded)
|
||||
|
||||
|
||||
def normalize_identifier(name: str, value: str) -> str | None:
|
||||
"""
|
||||
Reduce one identifier to a `scheme:value` key, if it can carry a match at all.
|
||||
|
||||
Args:
|
||||
name: What kind of identifier it is, as stored.
|
||||
value: The identifier itself.
|
||||
|
||||
Returns:
|
||||
The key, or None when the identifier is no use for matching: a per-build UUID,
|
||||
something too short to be evidence, or an ISBN that fails its own checksum.
|
||||
"""
|
||||
name = (name or "").strip().casefold()
|
||||
value = (value or "").strip()
|
||||
|
||||
if not name or not value or name in _PER_BUILD_NAMES:
|
||||
return None
|
||||
|
||||
if name in _ISBN_NAMES:
|
||||
digits = re.sub(r"[^0-9Xx]", "", value).upper()
|
||||
|
||||
if not is_valid_isbn(digits):
|
||||
return None
|
||||
|
||||
# One scheme for both forms: a publisher prints whichever it likes, and the
|
||||
# ISBN-10 and ISBN-13 of an edition are the same number written twice.
|
||||
isbn = digits if len(digits) == 13 else isbn10_to_isbn13(digits)
|
||||
return f"isbn:{isbn}" if isbn else None
|
||||
|
||||
folded = _fold(value) or value.casefold()
|
||||
if len(folded) < _MIN_IDENTIFIER_LENGTH:
|
||||
return None
|
||||
|
||||
return f"{name}:{folded}"
|
||||
@@ -3,7 +3,6 @@
|
||||
# TODO: Code is a mess. Clean it up and add docstrings
|
||||
|
||||
# Standard library
|
||||
from abc import ABC, abstractmethod
|
||||
import datetime
|
||||
from pathlib import Path
|
||||
from io import BytesIO
|
||||
@@ -31,6 +30,147 @@ from chitai.services.utils import (
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# Identifier schemes an EPUB can declare, mapped onto the names the rest of the app
|
||||
# uses. A scheme arrives either as an `opf:scheme` attribute or as a prefix on the
|
||||
# value itself (`urn:isbn:…`, `calibre:…`), and the two say the same thing.
|
||||
_IDENTIFIER_SCHEMES = {
|
||||
"isbn": "isbn",
|
||||
"isbn10": "isbn-10",
|
||||
"isbn-10": "isbn-10",
|
||||
"isbn13": "isbn-13",
|
||||
"isbn-13": "isbn-13",
|
||||
"uuid": "uuid",
|
||||
"calibre": "calibre",
|
||||
"doi": "doi",
|
||||
"asin": "asin",
|
||||
"amazon": "asin",
|
||||
"mobi-asin": "asin",
|
||||
"google": "google",
|
||||
"goodreads": "goodreads",
|
||||
}
|
||||
|
||||
# `scheme:rest`, with an optional `urn:` in front of it.
|
||||
_SCHEME_PREFIX = re.compile(r"^(?:urn:)?([A-Za-z][A-Za-z0-9.-]*):(.+)$")
|
||||
|
||||
_UUID = re.compile(r"^[0-9a-f]{8}(?:-[0-9a-f]{4}){3}-[0-9a-f]{12}$", re.IGNORECASE)
|
||||
|
||||
|
||||
def parse_identifier(value: str, scheme: str | None = None) -> tuple[str, str] | None:
|
||||
"""
|
||||
Work out what one raw identifier is, and what it is worth storing as.
|
||||
|
||||
EPUBs write the same ISBN as `9780486282114`, `978-0-486-28211-4` and
|
||||
`urn:isbn:978-0-486-28211-4`, and carry plenty of identifiers that are not ISBNs
|
||||
at all. Validating the string verbatim keeps only the first form and throws the
|
||||
rest away, so normalise first and name whatever survives.
|
||||
|
||||
Args:
|
||||
value: The identifier as the file wrote it.
|
||||
scheme: What the file said it is, if it said anything — an `opf:scheme`
|
||||
attribute. A prefix on the value takes precedence over this.
|
||||
|
||||
Returns:
|
||||
The `(name, value)` to store, or `None` when there is nothing usable: an
|
||||
empty value, or one declared to be an ISBN that fails its own checksum.
|
||||
"""
|
||||
value = (value or "").strip()
|
||||
if not value:
|
||||
return None
|
||||
|
||||
name = _IDENTIFIER_SCHEMES.get((scheme or "").strip().casefold())
|
||||
|
||||
# An unrecognised prefix is part of the value rather than a scheme —
|
||||
# "http://example.com/book" is not an identifier called "http".
|
||||
if (match := _SCHEME_PREFIX.match(value)) and (
|
||||
prefixed := _IDENTIFIER_SCHEMES.get(match.group(1).casefold())
|
||||
):
|
||||
name = prefixed
|
||||
value = match.group(2).strip()
|
||||
|
||||
if name is None or name.startswith("isbn"):
|
||||
digits = re.sub(r"[^0-9Xx]", "", value).upper()
|
||||
if is_valid_isbn(digits):
|
||||
return f"isbn-{len(digits)}", digits
|
||||
|
||||
# Something that announced itself as an ISBN and is not one carries no
|
||||
# information: storing it would link the reader to a page that does not exist.
|
||||
if name is not None:
|
||||
return None
|
||||
|
||||
return name or ("uuid" if _UUID.match(value) else "id"), value
|
||||
|
||||
|
||||
# Numbered editions, in the forms covers and catalogue records actually use:
|
||||
# "3rd Edition", "2E", "4e", "8_e", "/6e", "(2nd edition)", "Third International Edition".
|
||||
_ORDINAL_WORDS = {
|
||||
"first": 1, "second": 2, "third": 3, "fourth": 4, "fifth": 5, "sixth": 6,
|
||||
"seventh": 7, "eighth": 8, "ninth": 9, "tenth": 10, "eleventh": 11, "twelfth": 12,
|
||||
}
|
||||
|
||||
# Words that sit between the number and "Edition" and belong to the same statement.
|
||||
_EDITION_QUALIFIER = (
|
||||
r"(?:international|global|revised|updated|expanded|anniversary|deluxe|student|"
|
||||
r"instructors?|annotated|illustrated|reprint)"
|
||||
)
|
||||
|
||||
_EDITION = re.compile(
|
||||
rf"""
|
||||
[\s,;:/\-–—(\[]+ # the separator the statement hangs off
|
||||
(?:
|
||||
(?P<num>\d{{1,2}})\s*(?:st|nd|rd|th)?[\s_]*
|
||||
(?:{_EDITION_QUALIFIER}\s+)*(?:edition\b|edn\b|ed\b\.?|e\b)
|
||||
| (?P<word>{"|".join(_ORDINAL_WORDS)})\s+
|
||||
(?:{_EDITION_QUALIFIER}\s+)*(?:edition\b|edn\b|ed\b\.?)
|
||||
)
|
||||
[\s)\]]* # and its closing bracket, if it had one
|
||||
""",
|
||||
re.IGNORECASE | re.VERBOSE,
|
||||
)
|
||||
|
||||
|
||||
def split_edition(title: str | None) -> tuple[str | None, int | None]:
|
||||
"""
|
||||
Separate a numbered edition statement from the title it is written into.
|
||||
|
||||
"Fluent Python, 2nd Edition" is one book with a field for the edition, not a
|
||||
title. Left in place it also splits the library: the second edition never looks
|
||||
like the first, and neither matches the copy whose file simply did not mention it.
|
||||
|
||||
The number is what makes this safe. Nothing is stripped without one, so
|
||||
"Catch 22" and "Blade Runner 2049" keep their numbers and "Global Edition" —
|
||||
which is a variant, not a numbered edition, and has nowhere to go in an
|
||||
integer column — is left in the title where it can still be read.
|
||||
|
||||
Args:
|
||||
title: The title as the file or filename gave it.
|
||||
|
||||
Returns:
|
||||
The title without the edition statement, and the edition number. The title
|
||||
unchanged and None when there is no numbered edition in it, or when removing
|
||||
it would leave nothing behind.
|
||||
"""
|
||||
if not title:
|
||||
return title, None
|
||||
|
||||
if (match := _EDITION.search(title)) is None:
|
||||
return title, None
|
||||
|
||||
edition = (
|
||||
int(match["num"]) if match["num"] else _ORDINAL_WORDS[match["word"].casefold()]
|
||||
)
|
||||
|
||||
stripped = _EDITION.sub(" ", title)
|
||||
stripped = re.sub(r"\s{2,}", " ", stripped)
|
||||
stripped = re.sub(r"\s+([,;:.!?])", r"\1", stripped) # "Works : What" → "Works: What"
|
||||
stripped = stripped.strip(" ,;:-–—/")
|
||||
|
||||
# A title that is only an edition statement is not improved by having none.
|
||||
if not stripped:
|
||||
return title, None
|
||||
|
||||
return stripped, edition
|
||||
|
||||
|
||||
class FileExtractor(Protocol):
|
||||
@classmethod
|
||||
async def extract_metadata(
|
||||
@@ -54,15 +194,36 @@ class Extractor:
|
||||
# EPUB tends to give better metadata results over pdf
|
||||
sorted_files = sorted(files, key=lambda f: Extractor._get_file_priority(f))
|
||||
|
||||
# Identifiers accumulate across formats instead of replacing each other. Every
|
||||
# other field is a single value where the later, better-trusted format simply
|
||||
# wins, but identifiers are a *collection*: a book holding an EPUB and a PDF
|
||||
# genuinely carries what both of them declare, and merging the dict wholesale
|
||||
# threw away everything the earlier format found. An EPUB that declares an
|
||||
# ASIN, a Google volume id and a Calibre id kept none of them once a PDF
|
||||
# contributed a single ISBN.
|
||||
identifiers: dict[str, str] = {}
|
||||
|
||||
for file in sorted_files:
|
||||
match get_file_extension(file):
|
||||
case "epub":
|
||||
metadata = metadata | await EpubExtractor.extract_metadata(file)
|
||||
extracted = await EpubExtractor.extract_metadata(file)
|
||||
case "pdf":
|
||||
metadata = metadata | await PdfExtractor.extract_metadata(file)
|
||||
extracted = await PdfExtractor.extract_metadata(file)
|
||||
case _:
|
||||
break
|
||||
|
||||
# First writer wins per name, and the files are already ordered by how
|
||||
# far their metadata can be trusted. A `dc:identifier` the publisher
|
||||
# declared outranks an ISBN scraped out of a PDF's copyright page, which
|
||||
# routinely prints the ISBNs of other formats and older editions too.
|
||||
for name, value in (extracted.pop("identifiers", None) or {}).items():
|
||||
identifiers.setdefault(name, value)
|
||||
|
||||
metadata = metadata | extracted
|
||||
|
||||
if identifiers:
|
||||
metadata["identifiers"] = identifiers
|
||||
|
||||
# Get metadata from file names
|
||||
for file in files:
|
||||
metadata = FilenameExtractor.extract_metadata(file) | metadata
|
||||
@@ -75,7 +236,14 @@ class Extractor:
|
||||
|
||||
# format the title
|
||||
if metadata.get('title', None):
|
||||
title, subtitle = Extractor.format_book_title(metadata["title"])
|
||||
# Before the subtitle split, so the edition cannot be mistaken for one:
|
||||
# "How Linux Works, 3rd Edition: What Every Superuser Should Know" has to
|
||||
# lose the edition first for the colon count to mean anything.
|
||||
title, edition = split_edition(metadata["title"])
|
||||
if edition is not None:
|
||||
metadata.setdefault("edition", edition)
|
||||
|
||||
title, subtitle = Extractor.format_book_title(title)
|
||||
metadata["title"] = title
|
||||
metadata["subtitle"] = subtitle
|
||||
|
||||
@@ -236,7 +404,7 @@ class PdfExtractor(FileExtractor):
|
||||
try:
|
||||
return datetime.datetime.strptime(date_portion, "%Y%m%d").date()
|
||||
|
||||
except Exception as e:
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
@@ -363,15 +531,23 @@ class EpubExtractor(FileExtractor):
|
||||
|
||||
@classmethod
|
||||
def _extract_identifiers(cls, epub: epub.EpubBook) -> dict[str, str]:
|
||||
"""
|
||||
Every `DC:identifier` the file carries, keyed by what kind of thing it is.
|
||||
|
||||
Non-ISBN identifiers are kept: `Identifier` is a free-form name/value pair, so
|
||||
a Calibre id or an ASIN costs nothing to store and is one more thing two copies
|
||||
of a book can be recognised by.
|
||||
"""
|
||||
identifiers = {}
|
||||
|
||||
for id in epub.get_metadata("DC", "identifier"):
|
||||
if is_valid_isbn(id[0]):
|
||||
if len(id[0]) == 13:
|
||||
identifiers.update({"isbn-13": id[0]})
|
||||
for value, attributes in epub.get_metadata("DC", "identifier"):
|
||||
scheme = None
|
||||
if isinstance(attributes, dict):
|
||||
scheme = attributes.get("opf:scheme") or attributes.get("scheme")
|
||||
|
||||
elif len(id[0]) == 10:
|
||||
identifiers.update({"isbn-10": id[0]})
|
||||
if (parsed := parse_identifier(value, scheme)) is not None:
|
||||
name, parsed_value = parsed
|
||||
identifiers[name] = parsed_value
|
||||
|
||||
return identifiers
|
||||
|
||||
@@ -380,7 +556,7 @@ class EpubExtractor(FileExtractor):
|
||||
try:
|
||||
return epub.get_metadata("DC", "description")[0][0]
|
||||
|
||||
except:
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
@@ -389,15 +565,15 @@ class EpubExtractor(FileExtractor):
|
||||
date_str = epub.get_metadata("DC", "date")[0][0].split("T")[0]
|
||||
return datetime.date.fromisoformat(date_str)
|
||||
|
||||
except:
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
def _extract_publisher(cls, epub: epub.EpubBook) -> str | None:
|
||||
try:
|
||||
epub.get_metadata("DC", "publisher")[0][0]
|
||||
return epub.get_metadata("DC", "publisher")[0][0]
|
||||
|
||||
except:
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
@classmethod
|
||||
@@ -421,7 +597,7 @@ class EpubExtractor(FileExtractor):
|
||||
cover_item = epub.get_item_with_id(cover_id)
|
||||
if cover_item:
|
||||
return PIL.Image.open(BytesIO(cover_item.content))
|
||||
except Exception as e:
|
||||
except Exception:
|
||||
pass # Fallback to next strategy
|
||||
|
||||
# Strategy 2: Search image filenames for "cover" keyword
|
||||
@@ -489,7 +665,11 @@ class FilenameExtractor(FileExtractor):
|
||||
elif isinstance(input, Path):
|
||||
filename = get_filename(input, ext=False)
|
||||
elif isinstance(input, UploadFile):
|
||||
filename = Path(input.filename).name
|
||||
# `.stem`, not `.name`: the extension is not part of the metadata, and
|
||||
# this is the browser upload path, so keeping it is how a library fills
|
||||
# up with authors called "Sam Newman.epub". The other two branches have
|
||||
# always stripped it.
|
||||
filename = Path(input.filename).stem
|
||||
else:
|
||||
raise ValueError("Input type not supported")
|
||||
|
||||
|
||||
@@ -3,13 +3,11 @@
|
||||
# Standard library
|
||||
from __future__ import annotations
|
||||
|
||||
import errno
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
from typing import TYPE_CHECKING, BinaryIO
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from hashlib import _Hash
|
||||
from typing import BinaryIO
|
||||
|
||||
# Third-party libraries
|
||||
import PIL
|
||||
@@ -32,6 +30,9 @@ KO_STEP = 1024
|
||||
KO_SAMPLE_SIZE = 1024
|
||||
KO_INDICES = range(-1, 11) # -1 to 10 inclusive
|
||||
|
||||
# How much is read at a time while hashing.
|
||||
HASH_CHUNK_SIZE = 262144 # 256 KiB
|
||||
|
||||
|
||||
def _lshift32(val: int, shift: int) -> int:
|
||||
"""
|
||||
@@ -100,10 +101,9 @@ async def calculate_koreader_hash(file_path: Path) -> str:
|
||||
offsets = _get_koreader_offsets()
|
||||
|
||||
file_pos = 0
|
||||
chunk_size = 262144 # 256 KiB
|
||||
|
||||
async with aiofiles.open(file_path, "rb") as f:
|
||||
while chunk := await f.read(chunk_size):
|
||||
while chunk := await f.read(HASH_CHUNK_SIZE):
|
||||
_partial_md5_from_chunk(chunk, hasher, offsets, file_pos)
|
||||
file_pos += len(chunk)
|
||||
|
||||
@@ -132,6 +132,49 @@ class StreamingHasher:
|
||||
"""Return the final hash."""
|
||||
return self.hasher.hexdigest()
|
||||
|
||||
@property
|
||||
def size(self) -> int:
|
||||
"""Total number of bytes fed in so far."""
|
||||
return self.position
|
||||
|
||||
|
||||
async def fingerprint_upload(file: UploadFile) -> tuple[str, int]:
|
||||
"""
|
||||
Calculate the hash and byte size of an uploaded file without storing it.
|
||||
|
||||
Duplicate detection has to answer before anything is written to the library, so
|
||||
the file is read here and rewound for whoever writes it afterwards.
|
||||
|
||||
Args:
|
||||
file: The uploaded file to read.
|
||||
|
||||
Returns:
|
||||
The file's `(hash, size)` pair.
|
||||
"""
|
||||
hasher = StreamingHasher()
|
||||
|
||||
await file.seek(0)
|
||||
while chunk := await file.read(HASH_CHUNK_SIZE):
|
||||
hasher.update(chunk)
|
||||
await file.seek(0)
|
||||
|
||||
return hasher.hexdigest(), hasher.size
|
||||
|
||||
|
||||
async def fingerprint_file(file_path: Path) -> tuple[str, int]:
|
||||
"""
|
||||
Calculate the hash and byte size of a file already on disk.
|
||||
|
||||
Args:
|
||||
file_path: Path to the file to read.
|
||||
|
||||
Returns:
|
||||
The file's `(hash, size)` pair.
|
||||
"""
|
||||
stats = await aios.stat(file_path)
|
||||
return await calculate_koreader_hash(file_path), stats.st_size
|
||||
|
||||
|
||||
##################################
|
||||
# Filesystem related utilities #
|
||||
##################################
|
||||
@@ -184,7 +227,16 @@ async def move_file(src_path: Path, dest_path: Path, create_dirs=True) -> None:
|
||||
if dest_dir: # Only create if there's a directory path
|
||||
await aios.makedirs(dest_dir, exist_ok=True)
|
||||
|
||||
await aios.rename(src_path, dest_path)
|
||||
try:
|
||||
await aios.rename(src_path, dest_path)
|
||||
except OSError as exc:
|
||||
if exc.errno != errno.EXDEV:
|
||||
raise
|
||||
|
||||
# Source and destination are on different filesystems, which rename cannot
|
||||
# cross. Libraries, the consume directory and the duplicates directory are
|
||||
# all configured separately, so they can easily be separate mounts.
|
||||
shutil.move(str(src_path), str(dest_path))
|
||||
|
||||
async def move_dir_contents(source_dir: Path | str, target_dir: Path | str) -> None:
|
||||
"""
|
||||
@@ -432,7 +484,7 @@ def is_valid_isbn(isbn: str) -> bool:
|
||||
return is_valid_isbn13(isbn)
|
||||
else:
|
||||
return False
|
||||
except:
|
||||
except Exception:
|
||||
return False
|
||||
|
||||
|
||||
@@ -457,6 +509,29 @@ def is_valid_isbn10(isbn: str) -> bool:
|
||||
return str(check_digit) == isbn[-1] or (check_digit == 10 and isbn[-1] in "Xx")
|
||||
|
||||
|
||||
def isbn10_to_isbn13(isbn: str) -> str | None:
|
||||
"""
|
||||
Convert an ISBN-10 to the ISBN-13 naming the same edition.
|
||||
|
||||
The two are the same number written twice: prefix `978`, drop the ISBN-10 check
|
||||
digit, recompute the check digit under the ISBN-13 rule. Matching only works if
|
||||
both forms collapse onto one, since a publisher may print either.
|
||||
|
||||
Args:
|
||||
isbn: A 10-character ISBN, digits and an optional trailing `X` only.
|
||||
|
||||
Returns:
|
||||
The equivalent ISBN-13, or None if the input is not a valid ISBN-10.
|
||||
"""
|
||||
if not is_valid_isbn(isbn) or len(isbn) != 10:
|
||||
return None
|
||||
|
||||
digits = f"978{isbn[:9]}"
|
||||
total = sum(int(digit) * (1 if i % 2 == 0 else 3) for i, digit in enumerate(digits))
|
||||
|
||||
return f"{digits}{(10 - total % 10) % 10}"
|
||||
|
||||
|
||||
def is_valid_isbn13(isbn: str) -> bool:
|
||||
"""
|
||||
Validate an ISBN-13 number using its check digit.
|
||||
|
||||
@@ -40,7 +40,13 @@ pytest_plugins = [
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _patch_settings(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setattr(settings, "book_cover_path", f"{tmp_path}/covers")
|
||||
# PIL will not create the directory it is asked to save into, so anything that
|
||||
# imports a file carrying a cover needs it to exist first.
|
||||
covers = tmp_path / "covers"
|
||||
covers.mkdir()
|
||||
|
||||
monkeypatch.setattr(settings, "book_cover_path", str(covers))
|
||||
monkeypatch.setattr(settings, "duplicate_path", str(tmp_path / "duplicates"))
|
||||
|
||||
|
||||
@pytest.fixture(name="engine")
|
||||
|
||||
@@ -37,7 +37,8 @@ from pathlib import Path
|
||||
(
|
||||
Path("tests/data_files/Calculus Made Easy - Silvanus Thompson.pdf"),
|
||||
2,
|
||||
"The Project Gutenberg eBook #33283: Calculus Made Easy, 2nd Edition",
|
||||
# The ", 2nd Edition" is split off into `edition`, not kept in the title.
|
||||
"The Project Gutenberg eBook #33283: Calculus Made Easy",
|
||||
["Silvanus Phillips Thompson"],
|
||||
),
|
||||
],
|
||||
@@ -208,6 +209,21 @@ async def test_get_book_file(
|
||||
assert downloaded_content == file_content
|
||||
|
||||
|
||||
async def test_list_books_by_id(populated_authenticated_client: AsyncClient) -> None:
|
||||
"""
|
||||
`?ids=` has to reach the database as integers.
|
||||
|
||||
advanced_alchemy's stock id filter annotates the parameter as `list[str]` whatever
|
||||
the configured id type, so the ids arrived as strings and Postgres refused to
|
||||
compare a bigint primary key against them. Nothing called it until a screen needed
|
||||
to fetch a handful of books by id.
|
||||
"""
|
||||
response = await populated_authenticated_client.get("/books?ids=1&ids=2&pageSize=10")
|
||||
|
||||
assert response.status_code == 200
|
||||
assert sorted(book["id"] for book in response.json()["items"]) == [1, 2]
|
||||
|
||||
|
||||
async def test_get_book_by_id(populated_authenticated_client: AsyncClient) -> None:
|
||||
"""Test retrieving a specific book by ID."""
|
||||
|
||||
@@ -366,7 +382,8 @@ async def test_create_multiple_books_from_directory(
|
||||
|
||||
assert response.status_code == 201
|
||||
data = response.json()
|
||||
assert len(data.get("items") or data.get("data")) >= 1
|
||||
assert len(data["created"]) == 2
|
||||
assert data["skipped"] == []
|
||||
|
||||
|
||||
async def test_create_books_from_parent_directory_keeps_embedded_title(
|
||||
@@ -396,7 +413,7 @@ async def test_create_books_from_parent_directory_keeps_embedded_title(
|
||||
|
||||
assert response.status_code == 201
|
||||
|
||||
books = response.json()["items"]
|
||||
books = response.json()["created"]
|
||||
assert len(books) == 1
|
||||
assert books[0]["title"] == "Metamorphosis"
|
||||
|
||||
@@ -419,11 +436,336 @@ async def test_create_books_groups_formats_within_one_folder(
|
||||
|
||||
assert response.status_code == 201
|
||||
|
||||
books = response.json()["items"]
|
||||
books = response.json()["created"]
|
||||
assert len(books) == 1
|
||||
assert len(books[0]["files"]) == 2
|
||||
|
||||
|
||||
class TestDuplicateHandling:
|
||||
"""A file the library already holds must not be stored a second time."""
|
||||
|
||||
epub_path = Path("tests/data_files/Metamorphosis - Franz Kafka.epub")
|
||||
|
||||
def upload(self, name: str | None = None) -> list[tuple[str, tuple]]:
|
||||
return [
|
||||
(
|
||||
"files",
|
||||
(
|
||||
name or self.epub_path.name,
|
||||
self.epub_path.read_bytes(),
|
||||
"application/epub+zip",
|
||||
),
|
||||
)
|
||||
]
|
||||
|
||||
async def test_bulk_upload_reports_skipped_files(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
"""Re-dropping a folder must import what is new and name what was not."""
|
||||
first = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload()
|
||||
)
|
||||
assert first.status_code == 201
|
||||
created = first.json()["created"][0]
|
||||
|
||||
second = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload()
|
||||
)
|
||||
|
||||
assert second.status_code == 201
|
||||
result = second.json()
|
||||
assert result["created"] == []
|
||||
assert len(result["skipped"]) == 1
|
||||
|
||||
skipped = result["skipped"][0]
|
||||
assert skipped["filename"] == self.epub_path.name
|
||||
assert skipped["book_id"] == created["id"]
|
||||
assert skipped["book_title"] == created["title"]
|
||||
|
||||
async def test_bulk_upload_can_be_forced(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload()
|
||||
)
|
||||
|
||||
response = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1&allow_duplicates=true", files=self.upload()
|
||||
)
|
||||
|
||||
assert response.status_code == 201
|
||||
assert len(response.json()["created"]) == 1
|
||||
assert response.json()["skipped"] == []
|
||||
|
||||
async def test_single_book_create_conflicts(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
"""Naming files deliberately earns a refusal rather than a silent drop."""
|
||||
await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload()
|
||||
)
|
||||
|
||||
response = await authenticated_client.post(
|
||||
"/books?library_id=1", files=self.upload(), data={"library_id": 1}
|
||||
)
|
||||
|
||||
assert response.status_code == 409
|
||||
assert response.json()["extra"][0]["filename"] == self.epub_path.name
|
||||
|
||||
forced = await authenticated_client.post(
|
||||
"/books?library_id=1&allow_duplicates=true",
|
||||
files=self.upload(),
|
||||
data={"library_id": 1},
|
||||
)
|
||||
assert forced.status_code == 201
|
||||
|
||||
async def test_adding_another_books_file_conflicts(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
created = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload()
|
||||
)
|
||||
book_id = created.json()["created"][0]["id"]
|
||||
|
||||
other = await authenticated_client.post(
|
||||
"/books?library_id=1",
|
||||
files=[
|
||||
(
|
||||
"files",
|
||||
(
|
||||
"war.epub",
|
||||
Path("tests/data_files/The Art of War - Sun Tzu.epub").read_bytes(),
|
||||
"application/epub+zip",
|
||||
),
|
||||
)
|
||||
],
|
||||
data={"library_id": 1},
|
||||
)
|
||||
other_id = other.json()["id"]
|
||||
|
||||
response = await authenticated_client.post(
|
||||
f"/books/{book_id}/files",
|
||||
files=[
|
||||
(
|
||||
"files",
|
||||
(
|
||||
"war.epub",
|
||||
Path("tests/data_files/The Art of War - Sun Tzu.epub").read_bytes(),
|
||||
"application/epub+zip",
|
||||
),
|
||||
)
|
||||
],
|
||||
)
|
||||
|
||||
assert response.status_code == 409
|
||||
assert response.json()["extra"][0]["book_id"] == other_id
|
||||
|
||||
async def test_resending_a_books_own_file_changes_nothing(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
created = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload()
|
||||
)
|
||||
book_id = created.json()["created"][0]["id"]
|
||||
|
||||
response = await authenticated_client.post(
|
||||
f"/books/{book_id}/files", files=self.upload()
|
||||
)
|
||||
|
||||
assert response.status_code == 201
|
||||
assert len(response.json()["files"]) == 1
|
||||
|
||||
async def test_duplicates_can_be_checked_before_uploading(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
"""The pre-flight check answers from hashes alone, with no file sent."""
|
||||
created = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload()
|
||||
)
|
||||
book = created.json()["created"][0]
|
||||
stored = book["files"][0]
|
||||
|
||||
response = await authenticated_client.post(
|
||||
"/books/duplicate-files?library_id=1",
|
||||
json=[
|
||||
{
|
||||
"hash": stored["hash"],
|
||||
"size": stored["size"],
|
||||
"filename": "local-copy.epub",
|
||||
},
|
||||
{"hash": stored["hash"], "size": stored["size"] + 1},
|
||||
{"hash": "0" * 32, "size": 1234},
|
||||
],
|
||||
)
|
||||
|
||||
assert response.status_code == 200
|
||||
|
||||
matches = response.json()
|
||||
assert len(matches) == 1
|
||||
assert matches[0]["filename"] == "local-copy.epub"
|
||||
assert matches[0]["book_id"] == book["id"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
class TestDuplicateBooks:
|
||||
"""A second copy of a book is imported and reported, never refused."""
|
||||
|
||||
epub_path = Path("tests/data_files/Metamorphosis - Franz Kafka.epub")
|
||||
|
||||
def upload(self, name: str, pad: bool = False) -> list[tuple[str, tuple]]:
|
||||
"""
|
||||
The fixture, optionally padded so it is a different file and the same book.
|
||||
|
||||
Padding the archive changes its size and its sampled hash without disturbing
|
||||
the metadata, which is exactly the case the file-level check cannot see.
|
||||
"""
|
||||
data = self.epub_path.read_bytes() + (b"\0" * 64 if pad else b"")
|
||||
|
||||
return [("files", (name, data, "application/epub+zip"))]
|
||||
|
||||
async def test_a_second_edition_is_created_and_reported(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
first = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
|
||||
)
|
||||
original = first.json()["created"][0]
|
||||
|
||||
second = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("second.epub", pad=True)
|
||||
)
|
||||
|
||||
assert second.status_code == 201
|
||||
body = second.json()
|
||||
|
||||
# Created, not skipped: a metadata match is a guess, and refusing a legitimate
|
||||
# second edition costs more than a note does.
|
||||
assert len(body["created"]) == 1
|
||||
assert body["skipped"] == []
|
||||
|
||||
assert len(body["possible_duplicates"]) == 1
|
||||
possible = body["possible_duplicates"][0]
|
||||
assert possible["book_id"] == body["created"][0]["id"]
|
||||
|
||||
candidate = possible["candidates"][0]
|
||||
assert candidate["book_id"] == original["id"]
|
||||
assert candidate["title"] == original["title"]
|
||||
assert candidate["authors"] == ["Franz Kafka"]
|
||||
assert "title-author" in candidate["matched_on"]
|
||||
|
||||
async def test_the_review_screen_groups_them(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
first = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
|
||||
)
|
||||
second = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("second.epub", pad=True)
|
||||
)
|
||||
|
||||
book_ids = sorted(
|
||||
[first.json()["created"][0]["id"], second.json()["created"][0]["id"]]
|
||||
)
|
||||
|
||||
response = await authenticated_client.get("/books/duplicate-books?library_id=1")
|
||||
|
||||
assert response.status_code == 200
|
||||
groups = response.json()
|
||||
assert len(groups) == 1
|
||||
assert [book["book_id"] for book in groups[0]["books"]] == book_ids
|
||||
|
||||
async def test_a_dismissed_group_stays_dismissed(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
first = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
|
||||
)
|
||||
second = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("second.epub", pad=True)
|
||||
)
|
||||
|
||||
pair = {
|
||||
"book_a_id": first.json()["created"][0]["id"],
|
||||
"book_b_id": second.json()["created"][0]["id"],
|
||||
}
|
||||
|
||||
dismissed = await authenticated_client.post(
|
||||
"/books/duplicate-books/dismissals", json=pair
|
||||
)
|
||||
assert dismissed.status_code == 204
|
||||
|
||||
response = await authenticated_client.get("/books/duplicate-books?library_id=1")
|
||||
assert response.json() == []
|
||||
|
||||
restored = await authenticated_client.delete(
|
||||
"/books/duplicate-books/dismissals"
|
||||
f"?book_a_id={pair['book_b_id']}&book_b_id={pair['book_a_id']}"
|
||||
)
|
||||
assert restored.status_code == 204
|
||||
|
||||
response = await authenticated_client.get("/books/duplicate-books?library_id=1")
|
||||
assert len(response.json()) == 1
|
||||
|
||||
async def test_two_books_merge_into_one(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
"""The survivor keeps its id and gains the other's file; the other is gone."""
|
||||
first = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
|
||||
)
|
||||
second = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("second.epub", pad=True)
|
||||
)
|
||||
keep = first.json()["created"][0]
|
||||
fold = second.json()["created"][0]
|
||||
|
||||
response = await authenticated_client.post(
|
||||
"/books/merge?library_id=1",
|
||||
json={
|
||||
"survivor_id": keep["id"],
|
||||
"merged_ids": [fold["id"]],
|
||||
"metadata": {"title": "Metamorphosis", "edition": 2},
|
||||
},
|
||||
)
|
||||
|
||||
assert response.status_code == 201
|
||||
merged = response.json()
|
||||
|
||||
assert merged["id"] == keep["id"]
|
||||
assert merged["edition"] == 2
|
||||
assert len(merged["files"]) == 2
|
||||
|
||||
# The folded record is gone, and the group it formed with it.
|
||||
assert (await authenticated_client.get(f"/books/{fold['id']}")).status_code == 404
|
||||
groups = await authenticated_client.get("/books/duplicate-books?library_id=1")
|
||||
assert groups.json() == []
|
||||
|
||||
async def test_merging_an_unknown_book_is_refused(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
created = await authenticated_client.post(
|
||||
"/books/fromFiles?library_id=1", files=self.upload("first.epub")
|
||||
)
|
||||
keep = created.json()["created"][0]["id"]
|
||||
|
||||
response = await authenticated_client.post(
|
||||
"/books/merge?library_id=1",
|
||||
json={"survivor_id": keep, "merged_ids": [9999]},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
async def test_dismissing_an_unknown_book_is_refused(
|
||||
self, authenticated_client: AsyncClient
|
||||
) -> None:
|
||||
response = await authenticated_client.post(
|
||||
"/books/duplicate-books/dismissals",
|
||||
json={"book_a_id": 1, "book_b_id": 9999},
|
||||
)
|
||||
|
||||
assert response.status_code == 400
|
||||
|
||||
|
||||
# NOTE: the multi-book ZIP download is covered at the service level, in
|
||||
# tests/unit/test_services/test_book_service.py. Driving `/books/download` through
|
||||
# AsyncTestClient hangs in fixture teardown: it is the only `Stream` endpoint in the
|
||||
|
||||
@@ -0,0 +1,233 @@
|
||||
"""Tests for the normalization behind book-level duplicate detection."""
|
||||
|
||||
import pytest
|
||||
|
||||
from chitai.services.matching import (
|
||||
format_author_name,
|
||||
normalize_author,
|
||||
normalize_identifier,
|
||||
normalize_title,
|
||||
)
|
||||
from chitai.services.metadata_extractor import parse_identifier
|
||||
from chitai.services.utils import isbn10_to_isbn13
|
||||
|
||||
|
||||
class TestNormalizeTitle:
|
||||
"""Two copies of one book rarely agree on how the title is written."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("title", "expected"),
|
||||
[
|
||||
("The Metamorphosis", "metamorphosis"),
|
||||
("Metamorphosis", "metamorphosis"),
|
||||
("METAMORPHOSIS", "metamorphosis"),
|
||||
("A Tale of Two Cities", "tale of two cities"),
|
||||
("An Enquiry", "enquiry"),
|
||||
# Accents, punctuation and ampersands are spelling, not identity.
|
||||
("Les Misérables", "les miserables"),
|
||||
("Moby Dick; Or, The Whale", "moby dick or the whale"),
|
||||
("Sense & Sensibility", "sense and sensibility"),
|
||||
# Bracketed asides and trailing edition noise say nothing about the book.
|
||||
("Frankenstein (Illustrated)", "frankenstein"),
|
||||
("Frankenstein [Kindle Edition]", "frankenstein"),
|
||||
("Frankenstein, 2nd Edition", "frankenstein"),
|
||||
# The compact forms a cover actually carries.
|
||||
("Building Microservices, 2E", "building microservices"),
|
||||
("Building Microservices 2e", "building microservices"),
|
||||
("Frankenstein 3 Ed", "frankenstein"),
|
||||
("Dungeons & Dragons 5e", "dungeons and dragons"),
|
||||
("Frankenstein Revised Edition", "frankenstein"),
|
||||
("Dune Deluxe Edition Illustrated", "dune"),
|
||||
("", ""),
|
||||
(None, ""),
|
||||
],
|
||||
)
|
||||
def test_titles_that_should_agree(self, title: str | None, expected: str) -> None:
|
||||
assert normalize_title(title) == expected
|
||||
|
||||
def test_a_qualifier_that_is_the_title_survives(self) -> None:
|
||||
"""A trailing qualifier is noise; the same word at the front is the book."""
|
||||
assert normalize_title("The Illustrated Man") == "illustrated man"
|
||||
|
||||
def test_normalization_never_empties_a_title(self) -> None:
|
||||
"""An article-only title is not improved by having no article left."""
|
||||
assert normalize_title("The") == "the"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"title", ["Catch 22", "Fahrenheit 451", "Blade Runner 2049", "1984", "Apollo 13"]
|
||||
)
|
||||
def test_a_number_is_not_an_edition(self, title: str) -> None:
|
||||
"""Edition stripping keys on the `e`; a bare number is part of the title."""
|
||||
assert normalize_title(title) == title.casefold()
|
||||
|
||||
|
||||
class TestNormalizeAuthor:
|
||||
"""One person, written down several ways."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("name", "expected"),
|
||||
[
|
||||
("Franz Kafka", "franz kafka"),
|
||||
("Kafka, Franz", "franz kafka"),
|
||||
("KAFKA, FRANZ", "franz kafka"),
|
||||
("Émile Zola", "emile zola"),
|
||||
("Doyle, Arthur Conan", "arthur conan doyle"),
|
||||
# Runs of initials are joined, so spacing them out changes nothing.
|
||||
("J.R.R. Tolkien", "jrr tolkien"),
|
||||
("J. R. R. Tolkien", "jrr tolkien"),
|
||||
("JRR Tolkien", "jrr tolkien"),
|
||||
("", ""),
|
||||
(None, ""),
|
||||
],
|
||||
)
|
||||
def test_names_that_should_agree(self, name: str | None, expected: str) -> None:
|
||||
assert normalize_author(name) == expected
|
||||
|
||||
def test_two_people_are_not_reduced_together(self) -> None:
|
||||
"""Surname plus initial would collide unrelated writers; it is not used."""
|
||||
assert normalize_author("Charles Dickens") != normalize_author("Colin Dexter")
|
||||
|
||||
def test_a_list_is_left_alone(self) -> None:
|
||||
"""More than one comma is a list or a suffix, and guessing does more harm."""
|
||||
assert normalize_author("Smith, John, Jr.") == "smith john jr"
|
||||
|
||||
|
||||
class TestFormatAuthorName:
|
||||
"""What gets stored and shown, as opposed to what gets compared."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("written", "expected"),
|
||||
[
|
||||
# A leftover separator from a `DC:creator` list.
|
||||
("Newman, Sam;", "Sam Newman"),
|
||||
("Sam Newman ", "Sam Newman"),
|
||||
(" Dan Vanderkam ", "Dan Vanderkam"),
|
||||
# `Surname, Given` is how EPUBs file a name, not how anyone reads it.
|
||||
("Kleppmann, Martin", "Martin Kleppmann"),
|
||||
("Huxley, Aldous", "Aldous Huxley"),
|
||||
("Liu, Cixin", "Cixin Liu"),
|
||||
# An extension carried in from the filename the name was read out of.
|
||||
("Sam Newman.epub", "Sam Newman"),
|
||||
("Franz Kafka.mobi", "Franz Kafka"),
|
||||
("Brian W. Kernighan.epub", "Brian W. Kernighan"),
|
||||
("", ""),
|
||||
(None, ""),
|
||||
],
|
||||
)
|
||||
def test_names_are_tidied(self, written: str | None, expected: str) -> None:
|
||||
assert format_author_name(written) == expected
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"written",
|
||||
[
|
||||
# Two people in one string. Flipping it would invent a third.
|
||||
"Dave Thomas, Andy Hunt",
|
||||
"Mark Richards, Neal Ford",
|
||||
# A compound surname is not recognised, and is left alone rather than
|
||||
# rearranged wrongly.
|
||||
"García Márquez, Gabriel",
|
||||
],
|
||||
)
|
||||
def test_an_unrecognised_form_is_left_alone(self, written: str) -> None:
|
||||
assert format_author_name(written) == written
|
||||
|
||||
def test_case_and_accents_belong_to_the_author(self) -> None:
|
||||
"""Tidying removes what an extractor added; it does not correct spelling."""
|
||||
assert format_author_name("Michał Płachta.epub") == "Michał Płachta"
|
||||
assert format_author_name("Steve McConnell") == "Steve McConnell"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"written", ["Newman, Sam;", "Sam Newman.epub", "Kleppmann, Martin"]
|
||||
)
|
||||
def test_tidying_is_idempotent(self, written: str) -> None:
|
||||
"""`unique_filter` tidies a name that may already be tidy; it must not drift."""
|
||||
once = format_author_name(written)
|
||||
assert format_author_name(once) == once
|
||||
|
||||
|
||||
class TestNormalizeIdentifier:
|
||||
"""Identifiers only help if the same edition produces the same key."""
|
||||
|
||||
def test_isbn_10_and_isbn_13_are_one_key(self) -> None:
|
||||
assert normalize_identifier("isbn-10", "0486282112") == "isbn:9780486282114"
|
||||
assert normalize_identifier("isbn-13", "9780486282114") == "isbn:9780486282114"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"written", ["978-0-486-28211-4", "978 0 486 28211 4", "9780486282114"]
|
||||
)
|
||||
def test_formatting_is_not_part_of_an_isbn(self, written: str) -> None:
|
||||
assert normalize_identifier("isbn", written) == "isbn:9780486282114"
|
||||
|
||||
def test_an_isbn_that_fails_its_checksum_is_no_evidence(self) -> None:
|
||||
assert normalize_identifier("isbn-13", "9780486282115") is None
|
||||
|
||||
def test_uuids_are_refused(self) -> None:
|
||||
"""Generated per build, so they only re-find what the hash check catches."""
|
||||
assert normalize_identifier("uuid", "3f2b1c4e-1111-2222-3333-444455556666") is None
|
||||
assert normalize_identifier("urn:uuid", "3f2b1c4e-1111-2222-3333-444455556666") is None
|
||||
|
||||
def test_other_schemes_keep_their_own_key(self) -> None:
|
||||
assert normalize_identifier("asin", "B000FC0PDA") == "asin:b000fc0pda"
|
||||
assert normalize_identifier("ASIN", "b000fc0pda") == "asin:b000fc0pda"
|
||||
|
||||
def test_something_too_short_is_not_evidence(self) -> None:
|
||||
"""A Calibre id of "42" would otherwise pair two unrelated books."""
|
||||
assert normalize_identifier("calibre", "42") is None
|
||||
|
||||
@pytest.mark.parametrize(("name", "value"), [("", "1234567"), ("asin", "")])
|
||||
def test_half_an_identifier_is_no_identifier(self, name: str, value: str) -> None:
|
||||
assert normalize_identifier(name, value) is None
|
||||
|
||||
|
||||
class TestIsbnConversion:
|
||||
def test_isbn_10_converts_to_its_isbn_13(self) -> None:
|
||||
assert isbn10_to_isbn13("0486282112") == "9780486282114"
|
||||
|
||||
def test_a_trailing_x_is_a_digit(self) -> None:
|
||||
assert isbn10_to_isbn13("043942089X") == "9780439420891"
|
||||
|
||||
@pytest.mark.parametrize("isbn", ["0486282113", "9780486282114", "nonsense"])
|
||||
def test_anything_that_is_not_an_isbn_10_converts_to_nothing(self, isbn: str) -> None:
|
||||
assert isbn10_to_isbn13(isbn) is None
|
||||
|
||||
|
||||
class TestParseIdentifier:
|
||||
"""What an EPUB writes, and what is worth storing for it."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"written",
|
||||
[
|
||||
"9780486282114",
|
||||
"978-0-486-28211-4",
|
||||
"urn:isbn:9780486282114",
|
||||
"urn:isbn:978-0-486-28211-4",
|
||||
"ISBN:978-0-486-28211-4",
|
||||
],
|
||||
)
|
||||
def test_isbns_survive_however_they_are_written(self, written: str) -> None:
|
||||
assert parse_identifier(written) == ("isbn-13", "9780486282114")
|
||||
|
||||
def test_the_scheme_attribute_is_read_too(self) -> None:
|
||||
assert parse_identifier("0-486-28211-2", "ISBN") == ("isbn-10", "0486282112")
|
||||
|
||||
def test_non_isbn_identifiers_are_kept(self) -> None:
|
||||
assert parse_identifier("urn:uuid:3f2b1c4e-1111-2222-3333-444455556666") == (
|
||||
"uuid",
|
||||
"3f2b1c4e-1111-2222-3333-444455556666",
|
||||
)
|
||||
assert parse_identifier("calibre:1234") == ("calibre", "1234")
|
||||
assert parse_identifier("B000FC0PDA", "mobi-asin") == ("asin", "B000FC0PDA")
|
||||
|
||||
def test_an_unrecognised_prefix_is_part_of_the_value(self) -> None:
|
||||
""""http://example.com/book" is not an identifier called "http"."""
|
||||
assert parse_identifier("http://www.gutenberg.org/5200") == (
|
||||
"id",
|
||||
"http://www.gutenberg.org/5200",
|
||||
)
|
||||
|
||||
def test_a_declared_isbn_that_is_not_one_is_dropped(self) -> None:
|
||||
assert parse_identifier("urn:isbn:not-an-isbn") is None
|
||||
|
||||
@pytest.mark.parametrize("written", ["", " ", None])
|
||||
def test_nothing_yields_nothing(self, written: str | None) -> None:
|
||||
assert parse_identifier(written) is None
|
||||
@@ -1,7 +1,13 @@
|
||||
import pytest
|
||||
from ebooklib import epub
|
||||
from pathlib import Path
|
||||
from datetime import date
|
||||
from chitai.services.metadata_extractor import EpubExtractor
|
||||
from chitai.services.metadata_extractor import (
|
||||
EpubExtractor,
|
||||
Extractor,
|
||||
PdfExtractor,
|
||||
split_edition,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio()
|
||||
@@ -15,3 +21,172 @@ class TestEpubExtractor:
|
||||
assert metadata["authors"] == ["Herman Melville"]
|
||||
assert metadata["published_date"] == date(year=2001, month=7, day=1)
|
||||
|
||||
|
||||
|
||||
EPUB = Path("tests/data_files/Metamorphosis - Franz Kafka.epub")
|
||||
PDF = Path("tests/data_files/Calculus Made Easy - Silvanus Thompson.pdf")
|
||||
|
||||
|
||||
@pytest.mark.asyncio()
|
||||
class TestIdentifierMerging:
|
||||
"""A book's formats each contribute identifiers; none of them replaces the rest."""
|
||||
|
||||
async def test_every_format_contributes(self, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
"""
|
||||
Identifiers are a collection, not a single value.
|
||||
|
||||
Merging the whole dict meant the last format to report won outright: an EPUB
|
||||
declaring an ASIN, a Google volume id and a Calibre id kept none of them once
|
||||
a PDF contributed one ISBN.
|
||||
"""
|
||||
|
||||
async def epub(_file):
|
||||
return {
|
||||
"title": "How Linux Works",
|
||||
"identifiers": {"isbn-13": "9781718500419", "asin": "1718500408"},
|
||||
}
|
||||
|
||||
async def pdf(_file):
|
||||
return {"identifiers": {"isbn-13": "9781718500402", "isbn-10": "1593270356"}}
|
||||
|
||||
monkeypatch.setattr(EpubExtractor, "extract_metadata", epub)
|
||||
monkeypatch.setattr(PdfExtractor, "extract_metadata", pdf)
|
||||
|
||||
metadata = await Extractor.extract_metadata(
|
||||
[Path("How Linux Works.epub"), Path("How Linux Works.pdf")]
|
||||
)
|
||||
|
||||
assert metadata["identifiers"] == {
|
||||
# Declared by the publisher's toolchain, so it outranks the PDF's, which
|
||||
# was scraped off a copyright page that also prints the print edition's.
|
||||
"isbn-13": "9781718500419",
|
||||
"asin": "1718500408",
|
||||
"isbn-10": "1593270356",
|
||||
}
|
||||
|
||||
async def test_a_second_format_that_finds_nothing_erases_nothing(self) -> None:
|
||||
"""The PDF fixture carries no ISBN, so it must leave the EPUB's alone."""
|
||||
metadata = await Extractor.extract_metadata([EPUB, PDF])
|
||||
|
||||
assert metadata["identifiers"] == {"id": "http://www.gutenberg.org/5200"}
|
||||
|
||||
async def test_one_format_on_its_own_is_unaffected(self) -> None:
|
||||
metadata = await Extractor.extract_metadata([EPUB])
|
||||
|
||||
assert metadata["identifiers"] == {"id": "http://www.gutenberg.org/5200"}
|
||||
|
||||
async def test_no_identifiers_anywhere_leaves_the_field_absent(self) -> None:
|
||||
"""An empty dict would count as extracted metadata and overwrite nothing."""
|
||||
metadata = await Extractor.extract_metadata([PDF])
|
||||
|
||||
assert "identifiers" not in metadata
|
||||
|
||||
|
||||
class TestSplitEdition:
|
||||
"""An edition is a field on the book, not part of what the book is called."""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("title", "stripped", "edition"),
|
||||
[
|
||||
# Every form below is one that turned up in a real library.
|
||||
("Fluent Python, 2nd Edition", "Fluent Python", 2),
|
||||
("Building Microservices, 2E", "Building Microservices", 2),
|
||||
("Digital Image Processing, 4e", "Digital Image Processing", 4),
|
||||
(
|
||||
"Network Security Essentials: Applications and Standards/6e",
|
||||
"Network Security Essentials: Applications and Standards",
|
||||
6,
|
||||
),
|
||||
(
|
||||
"Refactoring: Improving the Design of Existing Code (2nd edition)",
|
||||
"Refactoring: Improving the Design of Existing Code",
|
||||
2,
|
||||
),
|
||||
# Ordinal words, including a qualifier sitting inside the statement.
|
||||
(
|
||||
"The Art of Computer Programming: Volume 1 / Fundamental Algorithms, Third Edition",
|
||||
"The Art of Computer Programming: Volume 1 / Fundamental Algorithms",
|
||||
3,
|
||||
),
|
||||
(
|
||||
"Introduction to the Theory of Computation, Third International Edition",
|
||||
"Introduction to the Theory of Computation",
|
||||
3,
|
||||
),
|
||||
# Mid-title, before a subtitle and before a trailing author.
|
||||
(
|
||||
"How Linux Works, 3rd Edition: What Every Superuser Should Know",
|
||||
"How Linux Works: What Every Superuser Should Know",
|
||||
3,
|
||||
),
|
||||
(
|
||||
"Code Complete, 2nd Edition - Steve McConnell",
|
||||
"Code Complete - Steve McConnell",
|
||||
2,
|
||||
),
|
||||
# An underscore between the number and the "e", beside an unnumbered
|
||||
# qualifier that has nowhere to go in an integer column and so stays put.
|
||||
(
|
||||
"Cryptography and Network Security, Global Edition, 8_e - Stallings",
|
||||
"Cryptography and Network Security, Global Edition - Stallings",
|
||||
8,
|
||||
),
|
||||
],
|
||||
)
|
||||
def test_editions_are_split_out(self, title: str, stripped: str, edition: int) -> None:
|
||||
assert split_edition(title) == (stripped, edition)
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"title",
|
||||
[
|
||||
# A number alone is never an edition — these are titles.
|
||||
"Catch 22",
|
||||
"Fahrenheit 451",
|
||||
"Blade Runner 2049",
|
||||
"1984",
|
||||
"Apollo 13",
|
||||
"Slaughterhouse 5",
|
||||
"The Art of Computer Programming: Volume 1",
|
||||
# "Edition" with no number cannot be stored, so it stays where it can
|
||||
# still be read.
|
||||
"Cryptography and Network Security: Principles and Practice, Global Edition",
|
||||
"Building Microservices",
|
||||
],
|
||||
)
|
||||
def test_titles_are_left_alone(self, title: str) -> None:
|
||||
assert split_edition(title) == (title, None)
|
||||
|
||||
def test_a_title_that_is_only_an_edition_is_kept(self) -> None:
|
||||
"""Stripping must never leave a book with no title at all."""
|
||||
assert split_edition("2nd Edition") == ("2nd Edition", None)
|
||||
|
||||
@pytest.mark.parametrize("title", ["", None])
|
||||
def test_nothing_yields_nothing(self, title: str | None) -> None:
|
||||
assert split_edition(title) == (title, None)
|
||||
|
||||
|
||||
@pytest.mark.asyncio()
|
||||
class TestEditionFromFiles:
|
||||
async def test_extraction_moves_the_edition_off_the_title(self) -> None:
|
||||
"""The PDF fixture calls itself a 2nd edition in its own metadata title."""
|
||||
metadata = await Extractor.extract_metadata([PDF])
|
||||
|
||||
assert metadata["title"] == "The Project Gutenberg eBook #33283: Calculus Made Easy"
|
||||
assert metadata["edition"] == 2
|
||||
|
||||
|
||||
class TestEpubPublisher:
|
||||
"""The publisher was looked up and then dropped on the floor."""
|
||||
|
||||
def test_a_declared_publisher_is_returned(self) -> None:
|
||||
"""
|
||||
The lookup discarded its own result and fell off the end of the function, so
|
||||
every EPUB reported no publisher no matter what it said.
|
||||
"""
|
||||
book = epub.EpubBook()
|
||||
book.add_metadata("DC", "publisher", "No Starch Press")
|
||||
|
||||
assert EpubExtractor._extract_publisher(book) == "No Starch Press"
|
||||
|
||||
def test_no_publisher_is_none(self) -> None:
|
||||
assert EpubExtractor._extract_publisher(epub.EpubBook()) is None
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,146 @@
|
||||
# Implementation brief: move Duplicates into library settings
|
||||
|
||||
Written for an agent picking this up cold. Read the repo-root `AGENTS.md` and
|
||||
`frontend/AGENTS.md` first — this brief assumes both.
|
||||
|
||||
**This is a frontend-only change.** The backend already scopes everything by library
|
||||
(`GET /books/duplicate-books?library_id=`), so no endpoint, schema or migration is
|
||||
involved.
|
||||
|
||||
## Where this starts from
|
||||
|
||||
The duplicates review screen exists and works. It currently lives at
|
||||
|
||||
```
|
||||
frontend/src/routes/(root)/(library)/library/[libraryId]/duplicates/
|
||||
+page.server.ts loads the groups, plus the full Book records merge needs
|
||||
+page.svelte group cards, "Not duplicates", "Merge…"
|
||||
```
|
||||
|
||||
and is reached from a **Duplicates entry in the main sidebar**
|
||||
(`frontend/src/lib/components/layout/nav-main.svelte`), which is what this change
|
||||
removes.
|
||||
|
||||
Settings today is a flat, entirely global four-item nav
|
||||
(`frontend/src/routes/(root)/settings/+layout.svelte`): Account, Appearance, Libraries,
|
||||
Devices. `settings/libraries/+page.svelte` is a single table of every library whose rows
|
||||
link *out* to the library itself. **There is nowhere that means "settings for this
|
||||
library"** — that is the gap this change fills.
|
||||
|
||||
## What to build — option B
|
||||
|
||||
Libraries expands in the settings nav. Every library is a sub-item; selecting one swaps
|
||||
the pane; Duplicates is a section inside that pane. All libraries and all their sections
|
||||
end up one click apart.
|
||||
|
||||
```
|
||||
/settings/libraries the existing table (leave it as the index)
|
||||
/settings/libraries/[libraryId] redirects to the first section
|
||||
/settings/libraries/[libraryId]/duplicates the review screen, moved
|
||||
```
|
||||
|
||||
Suggested files:
|
||||
|
||||
| Path | What |
|
||||
| --- | --- |
|
||||
| `settings/libraries/[libraryId]/+layout.svelte` | Library name, and the section tabs |
|
||||
| `settings/libraries/[libraryId]/+page.ts` | `redirect(303, …/duplicates)` |
|
||||
| `settings/libraries/[libraryId]/duplicates/+page.server.ts` | Moved verbatim |
|
||||
| `settings/libraries/[libraryId]/duplicates/+page.svelte` | Moved verbatim |
|
||||
|
||||
Duplicates is the **only** real section today. Build the tab strip so General and Danger
|
||||
zone have somewhere obvious to land, but do not invent them now — an empty tab is worse
|
||||
than no tab.
|
||||
|
||||
## The nav
|
||||
|
||||
In `settings/+layout.svelte`, `items` is a flat `as const` array matched on
|
||||
`page.route.id`. Libraries needs to render its children beneath it:
|
||||
|
||||
```svelte
|
||||
{#each libraryState.libraries as library (library.id)}
|
||||
<a href={resolve('/(root)/settings/libraries/[libraryId]/duplicates', {
|
||||
libraryId: String(library.id) })}> … </a>
|
||||
{/each}
|
||||
```
|
||||
|
||||
`getLibraryState()` **is** available under `/settings` — it is set in
|
||||
`(root)/+layout.svelte`, above the settings group, and `settings/libraries/+page.svelte`
|
||||
already uses it. No new load function is needed to list the libraries.
|
||||
|
||||
**Active state is matched on route id, not pathname.** There is a comment in
|
||||
`settings/+layout.svelte` explaining why: `resolve()` returns an absolute path on the
|
||||
client and a relative one during SSR, so a pathname comparison is false on the server and
|
||||
true after hydration, and the highlight flashes in. A nested library item is active when
|
||||
the route id matches **and** `page.params.libraryId === String(library.id)` — both, or
|
||||
every library lights up at once.
|
||||
|
||||
## Things that will bite
|
||||
|
||||
1. **Remove the sidebar entry in the same change.** `nav-main.svelte` gained a
|
||||
`Duplicates` item and a `CopyCheck` import when the screen was built. Delete both, and
|
||||
delete the old route directory. Doing the removal and the move together is the point —
|
||||
split across two commits the screen is unreachable in between.
|
||||
|
||||
2. **Delete the old route, do not leave it.** Two live copies of a screen that both write
|
||||
is how they drift.
|
||||
|
||||
3. **`setBookSelectionState` is not available under `/settings`.** It is set in
|
||||
`(root)/(library)/+layout.svelte`, which the settings group is not inside. This is
|
||||
fine — the duplicates page uses `BookImage` directly, not `book-thumbnail.svelte`, and
|
||||
`MergeBooks` takes its `libraryId` as a prop. **Verify this stays true** if you touch
|
||||
either component; a `getBookSelectionState()` under settings returns `undefined` and
|
||||
fails at the first access, not at import.
|
||||
|
||||
4. **Keep `depends('app:duplicate-books')`.** Both the dismiss action and `MergeBooks`
|
||||
call `invalidate('app:duplicate-books')` to make a resolved group leave the screen.
|
||||
Drop it and the page silently stops refreshing. `MergeBooks` also invalidates
|
||||
`app:books`, which is a no-op under settings and should stay that way.
|
||||
|
||||
5. **The settings shell is height-constrained.** `settings/+layout.svelte` is
|
||||
`h-[calc(100vh-var(--header-height)-2rem)]` with `overflow-auto` on the content pane.
|
||||
The review screen is a long list of cards — it must scroll *inside* that pane. Its
|
||||
current `mx-auto max-w-5xl` wrapper will want revisiting.
|
||||
|
||||
6. **Three levels of nav is option B's known cost.** Nav → library → section, and the
|
||||
pane is narrower than the full-width route the screen was designed against. The group
|
||||
cards are `w-36` covers in a wrapping flex row, so they reflow, but check a group of
|
||||
four at a narrow window before calling it done.
|
||||
|
||||
7. **`resolve()` must be a direct call in markup** for `svelte/no-navigation-without-resolve`.
|
||||
Where `nav-main.svelte` computes a url through a variable it carries an
|
||||
`eslint-disable-next-line`; prefer the direct call over inheriting that.
|
||||
|
||||
8. **The loader depends on the `?ids=` fix.** `+page.server.ts` fetches full `Book`
|
||||
records with `listBooks({ ids, pageSize })` because the merge workbench needs
|
||||
identifiers, description and publisher, which `DuplicateBookRead` does not carry.
|
||||
advanced_alchemy's stock id filter types that parameter as `list[str]` regardless of
|
||||
config, which made Postgres refuse `bigint = character varying`; the override lives in
|
||||
`backend/src/chitai/services/dependencies.py` (`create_book_filter_dependencies`).
|
||||
If `GET /books?ids=1&ids=2` 500s, that override is missing — do not work around it in
|
||||
the loader.
|
||||
|
||||
## Out of scope
|
||||
|
||||
The **General** and **Danger zone** sections (rename, path template, read-only, consume
|
||||
directory, delete), and any change to the merge workbench itself. The toolbar entry point
|
||||
for merge — select 2+ books in the library view — is unrelated and stays where it is.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
pnpm check # baseline: 30 errors, 1 warning, 8 files — none of them yours
|
||||
pnpm lint # not clean either; check the files you touched, not the tree
|
||||
pnpm build
|
||||
```
|
||||
|
||||
By hand, with a library that has a duplicate group:
|
||||
|
||||
- Settings → Libraries lists every library beneath it; clicking one opens its pane.
|
||||
- Duplicates shows the same groups the old route did, and scrolls inside the settings pane.
|
||||
- **Not duplicates** removes the group and it stays gone after a reload.
|
||||
- **Merge…** opens the workbench, merges, and the group leaves the screen.
|
||||
- The main sidebar no longer has a Duplicates entry, and
|
||||
`/library/<id>/duplicates` no longer resolves.
|
||||
- A library with no duplicates shows the empty state, not a blank pane.
|
||||
+6
-4
@@ -157,10 +157,12 @@ but take a baseline first, because neither is clean (see below).
|
||||
|
||||
Observed in the current tree — don't mistake these for intentional patterns to copy:
|
||||
|
||||
- `src/lib/schema/openapi/schema.d.ts` is **stale** — it predates the `BookProgress` rework and has
|
||||
no `BookProgressRead`, so `book.progress` types as `{}`. It is the source of most of the errors
|
||||
`pnpm check` reports; regenerating it should clear them. **Baseline as of 2026-08-11: 37 errors,
|
||||
1 warning, 11 files.** Get your own baseline before assuming an error is yours.
|
||||
- `pnpm check` is not clean. **Baseline as of 2026-08-13: 30 errors, 1 warning, 8 files**, most of
|
||||
them in `src/routes/api/[...path]/+server.ts` (see below). Get your own baseline before assuming
|
||||
an error is yours. `src/lib/schema/openapi/schema.d.ts` was regenerated on that date and is
|
||||
current; regenerate it again after any backend API change, with
|
||||
`pnpm exec openapi-typescript http://localhost:8000/schema/openapi.json -o src/lib/schema/openapi/schema.d.ts`
|
||||
against a backend running **your** branch — a stale server silently writes a stale file.
|
||||
- `pnpm lint` does not pass either — 131 pre-existing ESLint errors, 35 of them
|
||||
`svelte/no-navigation-without-resolve` on plain `href`s, plus ~59 files Prettier would rewrite
|
||||
(mostly vendored shadcn components). Check the files you touched, not the whole tree.
|
||||
|
||||
@@ -8,12 +8,31 @@ import {
|
||||
deleteBooksSchema,
|
||||
editBookMetadataSchema,
|
||||
updateBookProgressSchema,
|
||||
duplicateDismissalSchema,
|
||||
bookMergeSchema,
|
||||
type Book,
|
||||
type BooksUploadResult,
|
||||
type DuplicateBookGroup,
|
||||
bookFilesUpload
|
||||
} from '$lib/schema/index';
|
||||
import { stringCoerce, type PaginatedResponse } from '$lib/schema/common';
|
||||
import { createQueryParams } from '$lib/utils';
|
||||
|
||||
/**
|
||||
* The backend's own message for a failed response, rather than its JSON envelope.
|
||||
*
|
||||
* A refused duplicate answers 409 with a `detail` worth reading and the offending
|
||||
* files in `extra`; passing the body through whole puts JSON in front of the reader.
|
||||
*/
|
||||
function detailOf(body: string): string {
|
||||
try {
|
||||
const parsed = JSON.parse(body);
|
||||
return typeof parsed?.detail === 'string' ? parsed.detail : body;
|
||||
} catch {
|
||||
return body;
|
||||
}
|
||||
}
|
||||
|
||||
export const getBook = query(stringCoerce, async (id): Promise<Book> => {
|
||||
const { locals } = getRequestEvent();
|
||||
|
||||
@@ -68,26 +87,29 @@ export const updateBookCover = form(bookCoverUpload, async ({ book_id, file }) =
|
||||
return await response.json();
|
||||
});
|
||||
|
||||
export const uploadBooks = form(booksUpload, async ({ library_id, files }) => {
|
||||
const { locals } = getRequestEvent();
|
||||
export const uploadBooks = form(
|
||||
booksUpload,
|
||||
async ({ library_id, files }): Promise<BooksUploadResult> => {
|
||||
const { locals } = getRequestEvent();
|
||||
|
||||
const formData = new FormData();
|
||||
files.forEach((file) => {
|
||||
formData.append('files', file);
|
||||
});
|
||||
const formData = new FormData();
|
||||
files.forEach((file) => {
|
||||
formData.append('files', file);
|
||||
});
|
||||
|
||||
const response = await locals.api.postMultipart(
|
||||
`/books/fromFiles?library_id=${library_id}`,
|
||||
formData
|
||||
);
|
||||
const response = await locals.api.postMultipart(
|
||||
`/books/fromFiles?library_id=${library_id}`,
|
||||
formData
|
||||
);
|
||||
|
||||
if (!response.ok) {
|
||||
const message = await response.text();
|
||||
error(response.status, message);
|
||||
if (!response.ok) {
|
||||
const message = await response.text();
|
||||
error(response.status, message);
|
||||
}
|
||||
|
||||
return await response.json();
|
||||
}
|
||||
|
||||
return await response.json();
|
||||
});
|
||||
);
|
||||
|
||||
export const uploadBookFiles = form(bookFilesUpload, async ({ book_id, files }) => {
|
||||
const { locals } = getRequestEvent();
|
||||
@@ -100,8 +122,9 @@ export const uploadBookFiles = form(bookFilesUpload, async ({ book_id, files })
|
||||
const response = await locals.api.postMultipart(`/books/${book_id}/files`, formData);
|
||||
|
||||
if (!response.ok) {
|
||||
const message = await response.text();
|
||||
error(response.status, message);
|
||||
// 409 here means the file is already stored under a different book, which is
|
||||
// something the reader can act on — so the message has to survive the trip.
|
||||
error(response.status, detailOf(await response.text()));
|
||||
}
|
||||
|
||||
return await response.json();
|
||||
@@ -133,6 +156,65 @@ export const deleteBookFiles = command(deleteBookFilesSchema, async ({ book_id,
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* Books already in the library that look like copies of one another.
|
||||
*
|
||||
* Metadata only, so every group is a question rather than a verdict — which is why
|
||||
* the screen it feeds offers a way to disagree.
|
||||
*/
|
||||
export const listDuplicateBooks = query(
|
||||
stringCoerce,
|
||||
async (libraryId): Promise<DuplicateBookGroup[]> => {
|
||||
const { locals } = getRequestEvent();
|
||||
|
||||
const response = await locals.api.get(`/books/duplicate-books?library_id=${libraryId}`);
|
||||
|
||||
if (!response.ok) error(response.status, detailOf(await response.text()));
|
||||
|
||||
return await response.json();
|
||||
}
|
||||
);
|
||||
|
||||
/**
|
||||
* Fold several books into one and delete the records folded in.
|
||||
*
|
||||
* Irreversible, so the caller is expected to have shown what is about to happen.
|
||||
*/
|
||||
export const mergeBooks = command(
|
||||
bookMergeSchema,
|
||||
async ({ library_id, ...data }): Promise<Book> => {
|
||||
const { locals } = getRequestEvent();
|
||||
|
||||
const response = await locals.api.post(`/books/merge?library_id=${library_id}`, data);
|
||||
|
||||
if (!response.ok) error(response.status, detailOf(await response.text()));
|
||||
|
||||
return await response.json();
|
||||
}
|
||||
);
|
||||
|
||||
/** Record that two books are not the same book, so the pair stops being proposed. */
|
||||
export const dismissDuplicateBooks = command(duplicateDismissalSchema, async (data) => {
|
||||
const { locals } = getRequestEvent();
|
||||
|
||||
const response = await locals.api.post('/books/duplicate-books/dismissals', data);
|
||||
|
||||
if (!response.ok) error(response.status, detailOf(await response.text()));
|
||||
});
|
||||
|
||||
/** Undo a dismissal, so the pair is proposed again. */
|
||||
export const restoreDuplicateBooks = command(duplicateDismissalSchema, async (data) => {
|
||||
const { locals } = getRequestEvent();
|
||||
|
||||
const params = createQueryParams(data);
|
||||
|
||||
const response = await locals.api.delete(
|
||||
`/books/duplicate-books/dismissals?${params.toString()}`
|
||||
);
|
||||
|
||||
if (!response.ok) error(response.status, detailOf(await response.text()));
|
||||
});
|
||||
|
||||
export const updateBookProgress = command(
|
||||
updateBookProgressSchema,
|
||||
async ({ book_ids, ...data }) => {
|
||||
|
||||
@@ -50,6 +50,22 @@
|
||||
toast.error(`${file.name} was not added`, { description: reason });
|
||||
};
|
||||
|
||||
/**
|
||||
* The API's own words, when it has any.
|
||||
*
|
||||
* Adding a file the library already holds under another book is refused with a
|
||||
* 409 naming it — far more use than "failed to add files". SvelteKit hands an
|
||||
* `error()` back as an HttpError on the client, so the message sits on `body`.
|
||||
*/
|
||||
function apiMessage(error: unknown): string | undefined {
|
||||
if (typeof error !== 'object' || error === null) return undefined;
|
||||
|
||||
const body = (error as { body?: { message?: string } }).body;
|
||||
if (typeof body?.message === 'string') return body.message;
|
||||
|
||||
return error instanceof Error ? error.message : undefined;
|
||||
}
|
||||
|
||||
function confirmDelete(file: BookFile) {
|
||||
fileToDelete = file;
|
||||
confirmOpen = true;
|
||||
@@ -148,7 +164,8 @@
|
||||
toast.success('Files added');
|
||||
} catch (error) {
|
||||
console.error('Failed to add files: ', error);
|
||||
toast.error('Failed to add files');
|
||||
toast.error(apiMessage(error) ?? 'Failed to add files');
|
||||
uploadBookFiles.fields.files.set([]);
|
||||
}
|
||||
})}
|
||||
enctype="multipart/form-data"
|
||||
|
||||
@@ -0,0 +1,149 @@
|
||||
import type { Book } from '$lib/schema';
|
||||
|
||||
/**
|
||||
* What kind of thing a metadata field holds, which is the only thing that decides
|
||||
* what you can do with it when two records disagree.
|
||||
*/
|
||||
export type FieldKind = 'text' | 'number' | 'date' | 'list' | 'keyed' | 'longtext';
|
||||
|
||||
/** An action offered on a field, beyond replacing it outright. */
|
||||
export type FieldAction = 'replace' | 'merge' | 'append';
|
||||
|
||||
export interface FieldSpec {
|
||||
/** The key sent in `BookMetadataUpdate`. */
|
||||
key: string;
|
||||
label: string;
|
||||
kind: FieldKind;
|
||||
group: string;
|
||||
/** Extra actions past `replace`, which every field has. */
|
||||
extra: FieldAction[];
|
||||
}
|
||||
|
||||
/**
|
||||
* The fields a merge can resolve, in the order and grouping the edit form uses.
|
||||
*
|
||||
* Deliberately a spec rather than markup: the merge workbench and, later, the
|
||||
* provider review screen both render from this, so a field cannot exist in one and
|
||||
* not the other. `cover` and `files` are absent because they are not choices —
|
||||
* files always come across and the cover has its own endpoint.
|
||||
*/
|
||||
export const MERGE_FIELDS: FieldSpec[] = [
|
||||
{ key: 'title', label: 'Title', kind: 'text', group: 'Identity', extra: [] },
|
||||
{ key: 'subtitle', label: 'Subtitle', kind: 'text', group: 'Identity', extra: [] },
|
||||
{ key: 'edition', label: 'Edition', kind: 'number', group: 'Identity', extra: [] },
|
||||
{ key: 'series', label: 'Series', kind: 'text', group: 'Identity', extra: [] },
|
||||
{ key: 'series_position', label: 'No.', kind: 'text', group: 'Identity', extra: [] },
|
||||
{ key: 'language', label: 'Language', kind: 'text', group: 'Identity', extra: [] },
|
||||
|
||||
// Order is meaningful for authors, so the second list is appended rather than
|
||||
// interleaved; tags are a set, so they merge.
|
||||
{
|
||||
key: 'authors',
|
||||
label: 'Authors',
|
||||
kind: 'list',
|
||||
group: 'People and subjects',
|
||||
extra: ['append']
|
||||
},
|
||||
{ key: 'tags', label: 'Tags', kind: 'list', group: 'People and subjects', extra: ['merge'] },
|
||||
|
||||
{ key: 'publisher', label: 'Publisher', kind: 'text', group: 'Publication', extra: [] },
|
||||
{ key: 'published_date', label: 'Published', kind: 'date', group: 'Publication', extra: [] },
|
||||
{ key: 'pages', label: 'Pages', kind: 'number', group: 'Publication', extra: [] },
|
||||
{
|
||||
key: 'identifiers',
|
||||
label: 'Identifiers',
|
||||
kind: 'keyed',
|
||||
group: 'Publication',
|
||||
extra: ['merge']
|
||||
},
|
||||
|
||||
{
|
||||
key: 'description',
|
||||
label: 'Description',
|
||||
kind: 'longtext',
|
||||
group: 'Description',
|
||||
extra: ['append']
|
||||
}
|
||||
];
|
||||
|
||||
/** A field's value, in the shape `BookMetadataUpdate` expects to receive it. */
|
||||
export type FieldValue = string | number | string[] | Record<string, string> | null;
|
||||
|
||||
/** Read one field off a book, flattening the relations the API returns as objects. */
|
||||
export function readField(book: Book, key: string): FieldValue {
|
||||
switch (key) {
|
||||
case 'authors':
|
||||
return book.authors.map((author) => author.name);
|
||||
case 'tags':
|
||||
return book.tags.map((tag) => tag.name);
|
||||
case 'publisher':
|
||||
return book.publisher?.name ?? null;
|
||||
case 'series':
|
||||
return book.series?.title ?? null;
|
||||
case 'identifiers':
|
||||
return book.identifiers ?? {};
|
||||
default:
|
||||
return (book as unknown as Record<string, FieldValue>)[key] ?? null;
|
||||
}
|
||||
}
|
||||
|
||||
/** Whether a field holds nothing, and so has no decision attached to it. */
|
||||
export function isEmpty(value: FieldValue): boolean {
|
||||
if (value === null || value === undefined || value === '') return true;
|
||||
if (Array.isArray(value)) return value.length === 0;
|
||||
if (typeof value === 'object') return Object.keys(value).length === 0;
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Whether two field values say the same thing, order included for lists. */
|
||||
export function isSame(left: FieldValue, right: FieldValue): boolean {
|
||||
if (isEmpty(left) && isEmpty(right)) return true;
|
||||
return JSON.stringify(left ?? null) === JSON.stringify(right ?? null);
|
||||
}
|
||||
|
||||
/**
|
||||
* Apply an action to a pair of values and return what the target becomes.
|
||||
*
|
||||
* `merge` on a keyed collection is per name and the target wins a clash, because
|
||||
* `Identifier` is unique on `(name, book_id)` — a book cannot hold both its print
|
||||
* and its ebook ISBN, so the second one has nowhere to go.
|
||||
*/
|
||||
export function applyAction(
|
||||
action: FieldAction,
|
||||
kind: FieldKind,
|
||||
target: FieldValue,
|
||||
incoming: FieldValue
|
||||
): FieldValue {
|
||||
if (action === 'replace') return incoming;
|
||||
|
||||
if (kind === 'list') {
|
||||
const current = Array.isArray(target) ? target : [];
|
||||
const extra = Array.isArray(incoming) ? incoming : [];
|
||||
// Order preserved, duplicates dropped — works for both append and merge.
|
||||
return [...new Set([...current, ...extra])];
|
||||
}
|
||||
|
||||
if (kind === 'keyed') {
|
||||
return { ...(incoming as Record<string, string>), ...(target as Record<string, string>) };
|
||||
}
|
||||
|
||||
if (kind === 'longtext') {
|
||||
const current = typeof target === 'string' ? target.trim() : '';
|
||||
const extra = typeof incoming === 'string' ? incoming.trim() : '';
|
||||
return [current, extra].filter(Boolean).join('\n\n');
|
||||
}
|
||||
|
||||
return incoming;
|
||||
}
|
||||
|
||||
/** How a value reads in the inert reference column. */
|
||||
export function displayValue(value: FieldValue): string {
|
||||
if (isEmpty(value)) return '';
|
||||
if (Array.isArray(value)) return value.join(', ');
|
||||
if (typeof value === 'object') {
|
||||
return Object.entries(value as Record<string, string>)
|
||||
.map(([name, id]) => `${name}: ${id}`)
|
||||
.join(' · ');
|
||||
}
|
||||
return String(value);
|
||||
}
|
||||
@@ -0,0 +1,394 @@
|
||||
<script lang="ts">
|
||||
import { untrack } from 'svelte';
|
||||
import { ArrowRight, GitMerge, Plus, Undo2 } from '@lucide/svelte';
|
||||
|
||||
import * as Dialog from '$lib/components/ui/dialog/index.js';
|
||||
import { Button } from '$lib/components/ui/button/index.js';
|
||||
import { Input } from '$lib/components/ui/input/index.js';
|
||||
import { Textarea } from '$lib/components/ui/textarea/index.js';
|
||||
import { Badge } from '$lib/components/ui/badge/index.js';
|
||||
import BookImage from '$lib/components/view/book-image.svelte';
|
||||
import GeneratedCover from '$lib/components/view/generated-cover.svelte';
|
||||
import type { Book } from '$lib/schema';
|
||||
|
||||
import { mergeInto } from './merge';
|
||||
|
||||
import {
|
||||
MERGE_FIELDS,
|
||||
applyAction,
|
||||
displayValue,
|
||||
isEmpty,
|
||||
isSame,
|
||||
readField,
|
||||
type FieldAction,
|
||||
type FieldSpec,
|
||||
type FieldValue
|
||||
} from './field-spec';
|
||||
|
||||
let {
|
||||
books,
|
||||
libraryId,
|
||||
open = $bindable(),
|
||||
onmerged
|
||||
}: {
|
||||
books: Book[];
|
||||
libraryId: number | string;
|
||||
open: boolean;
|
||||
/** Given the record that survived and the ones folded into it and deleted. */
|
||||
onmerged?: (survivor: Book, folded: Book[]) => void;
|
||||
} = $props();
|
||||
|
||||
// Seeded once per mount. The dialog is keyed on the group upstream, so a
|
||||
// different group gets a fresh workbench rather than the previous one's draft.
|
||||
let survivorId = $state(untrack(() => books[0]?.id));
|
||||
let candidateId = $state(untrack(() => books[1]?.id));
|
||||
let draft = $state<Record<string, FieldValue>>({});
|
||||
let busy = $state(false);
|
||||
|
||||
const survivor = $derived(books.find((book) => book.id === survivorId) ?? books[0]);
|
||||
const candidates = $derived(books.filter((book) => book.id !== survivorId));
|
||||
/** The records that will be deleted — the same set, named for what happens to them. */
|
||||
const folded = $derived(candidates);
|
||||
const candidate = $derived(candidates.find((book) => book.id === candidateId) ?? candidates[0]);
|
||||
|
||||
/** The survivor's stored value for a field, or the draft if it has been touched. */
|
||||
function current(field: FieldSpec): FieldValue {
|
||||
return field.key in draft ? draft[field.key] : readField(survivor, field.key);
|
||||
}
|
||||
|
||||
function take(field: FieldSpec, action: FieldAction) {
|
||||
draft[field.key] = applyAction(
|
||||
action,
|
||||
field.kind,
|
||||
current(field),
|
||||
readField(candidate, field.key)
|
||||
);
|
||||
}
|
||||
|
||||
function undo(field: FieldSpec) {
|
||||
delete draft[field.key];
|
||||
draft = { ...draft };
|
||||
}
|
||||
|
||||
/**
|
||||
* Fill only the fields the survivor has nothing in.
|
||||
*
|
||||
* The safe bulk action, and the one worth reaching for: it cannot overwrite a
|
||||
* value, so it needs no per-field protection to be pressed without reading.
|
||||
*/
|
||||
function fillEmpty() {
|
||||
for (const field of MERGE_FIELDS) {
|
||||
const incoming = readField(candidate, field.key);
|
||||
if (isEmpty(current(field)) && !isEmpty(incoming)) draft[field.key] = incoming;
|
||||
}
|
||||
}
|
||||
|
||||
const changed = $derived(Object.keys(draft));
|
||||
|
||||
const differing = $derived(
|
||||
candidate
|
||||
? MERGE_FIELDS.filter((field) => !isSame(current(field), readField(candidate, field.key)))
|
||||
: []
|
||||
);
|
||||
|
||||
/** Fields shown as a row: anything the two disagree on, plus anything edited. */
|
||||
const shown = $derived(
|
||||
MERGE_FIELDS.filter((field) => differing.includes(field) || field.key in draft)
|
||||
);
|
||||
|
||||
const agreed = $derived(MERGE_FIELDS.filter((field) => !shown.includes(field)));
|
||||
|
||||
const groups = $derived([...new Set(shown.map((field) => field.group))]);
|
||||
|
||||
async function submit() {
|
||||
if (busy) return;
|
||||
busy = true;
|
||||
|
||||
const merged = await mergeInto(libraryId, survivor, folded, changed.length ? draft : undefined);
|
||||
|
||||
busy = false;
|
||||
|
||||
if (merged) {
|
||||
open = false;
|
||||
onmerged?.(survivor, folded);
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
{#snippet cover(book: Book, size: string)}
|
||||
<span class="{size} shrink-0 overflow-hidden rounded-sm border bg-muted">
|
||||
{#if book.cover_image}
|
||||
<BookImage src="/api/{book.cover_image}" class="h-full w-full object-cover" />
|
||||
{:else}
|
||||
<!-- Drawn rather than left blank, so the rail tells two coverless books
|
||||
apart the same way the shelves do. -->
|
||||
<GeneratedCover {book} />
|
||||
{/if}
|
||||
</span>
|
||||
{/snippet}
|
||||
|
||||
<!-- The inert reference column: the same shape as the control opposite it, with
|
||||
nothing that invites a click. -->
|
||||
{#snippet reference(field: FieldSpec, book: Book)}
|
||||
{@const value = readField(book, field.key)}
|
||||
<div class="flex min-w-0 flex-col gap-1">
|
||||
<span class="text-[11px] text-muted-foreground">{field.label}</span>
|
||||
{#if isEmpty(value)}
|
||||
<span class="min-h-8 py-1 text-sm text-muted-foreground italic">empty</span>
|
||||
{:else if field.kind === 'list'}
|
||||
<span class="flex min-h-8 flex-wrap items-center gap-1 py-0.5">
|
||||
{#each value as string[] as item (item)}
|
||||
<Badge variant="secondary" class="font-normal">{item}</Badge>
|
||||
{/each}
|
||||
</span>
|
||||
{:else}
|
||||
<span class="min-h-8 py-1 text-sm break-words">{displayValue(value)}</span>
|
||||
{/if}
|
||||
</div>
|
||||
{/snippet}
|
||||
|
||||
<Dialog.Root bind:open>
|
||||
<Dialog.Content
|
||||
class="flex h-[85vh] max-w-[min(72rem,95vw)] flex-col gap-0 overflow-hidden p-0 sm:max-w-[min(72rem,95vw)]"
|
||||
>
|
||||
<Dialog.Header class="shrink-0 space-y-0 border-b px-5 py-3 text-left">
|
||||
<Dialog.Title class="font-serif text-base font-normal">
|
||||
Merge {books.length} books
|
||||
</Dialog.Title>
|
||||
<Dialog.Description class="text-xs">
|
||||
{candidates.length}
|
||||
{candidates.length === 1 ? 'record is' : 'records are'} deleted. Their files move onto the book
|
||||
you keep — nothing is removed from disk.
|
||||
</Dialog.Description>
|
||||
</Dialog.Header>
|
||||
|
||||
<div class="grid min-h-0 flex-1 grid-cols-1 md:grid-cols-[15rem_1fr]">
|
||||
<!-- Rail: the books being folded in, one open at a time -->
|
||||
<aside
|
||||
class="flex min-w-0 flex-col overflow-y-auto border-b bg-sidebar md:border-r md:border-b-0"
|
||||
>
|
||||
<p
|
||||
class="px-3 pt-3 pb-1 font-mono text-[10px] tracking-widest text-muted-foreground uppercase"
|
||||
>
|
||||
Taking from · {candidates.length}
|
||||
</p>
|
||||
{#each candidates as book (book.id)}
|
||||
{@const count = MERGE_FIELDS.filter(
|
||||
(field) => !isSame(readField(survivor, field.key), readField(book, field.key))
|
||||
).length}
|
||||
<button
|
||||
type="button"
|
||||
onclick={() => (candidateId = book.id)}
|
||||
class="flex items-start gap-2 border-l-2 px-3 py-2 text-left transition-colors hover:bg-muted/50 {book.id ===
|
||||
candidate?.id
|
||||
? 'border-l-primary bg-background'
|
||||
: 'border-l-transparent'}"
|
||||
>
|
||||
{@render cover(book, 'h-10 w-7')}
|
||||
<span class="min-w-0 flex-1">
|
||||
<span class="line-clamp-2 font-serif text-[13px]">{book.title}</span>
|
||||
<span class="block text-[10px] text-muted-foreground">
|
||||
#{book.id} · {book.files.length}
|
||||
{book.files.length === 1 ? 'file' : 'files'}
|
||||
</span>
|
||||
{#if count === 0}
|
||||
<Badge variant="secondary" class="mt-1 text-[10px] font-normal">
|
||||
nothing to take
|
||||
</Badge>
|
||||
{/if}
|
||||
</span>
|
||||
</button>
|
||||
{/each}
|
||||
</aside>
|
||||
|
||||
<div class="flex min-h-0 flex-col">
|
||||
<!-- Which record survives -->
|
||||
<div class="flex shrink-0 flex-wrap items-center gap-2 border-b px-5 py-2.5">
|
||||
<span class="text-[11px] text-muted-foreground">Keeping</span>
|
||||
{#each books as book (book.id)}
|
||||
<button
|
||||
type="button"
|
||||
onclick={() => (survivorId = book.id)}
|
||||
class="flex items-center gap-1.5 rounded-md border px-2 py-1 text-xs transition-colors {book.id ===
|
||||
survivorId
|
||||
? 'border-primary bg-accent text-accent-foreground'
|
||||
: 'text-muted-foreground hover:bg-muted'}"
|
||||
>
|
||||
{@render cover(book, 'h-6 w-4')}
|
||||
<span class="max-w-32 truncate">#{book.id}</span>
|
||||
</button>
|
||||
{/each}
|
||||
|
||||
<div class="ml-auto flex gap-2">
|
||||
<Button variant="outline" size="sm" onclick={fillEmpty} disabled={!candidate}>
|
||||
Fill empty fields
|
||||
</Button>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div class="min-h-0 flex-1 overflow-y-auto px-5 py-4">
|
||||
{#if !candidate}
|
||||
<p class="text-sm text-muted-foreground">Nothing left to fold in.</p>
|
||||
{:else if shown.length === 0}
|
||||
<p class="text-sm text-muted-foreground">
|
||||
These records agree on every field. Merging keeps
|
||||
<span class="font-medium text-foreground">#{survivor.id}</span> and moves the others' files
|
||||
onto it.
|
||||
</p>
|
||||
{:else}
|
||||
{#each groups as group (group)}
|
||||
<h3
|
||||
class="mt-5 border-b pb-1 font-mono text-[10px] tracking-widest text-primary uppercase first:mt-0"
|
||||
>
|
||||
{group}
|
||||
</h3>
|
||||
|
||||
{#each shown.filter((field) => field.group === group) as field (field.key)}
|
||||
{@const value = current(field)}
|
||||
{@const incoming = readField(candidate, field.key)}
|
||||
{@const edited = field.key in draft}
|
||||
<div class="grid grid-cols-[1fr_auto_1fr] items-center gap-3 border-b py-2">
|
||||
{@render reference(field, candidate)}
|
||||
|
||||
<!-- Actions, on the row rather than stacked beside it -->
|
||||
<div class="flex items-center gap-1">
|
||||
{#if edited}
|
||||
<Button
|
||||
variant="ghost"
|
||||
size="icon"
|
||||
class="size-7"
|
||||
title="Undo"
|
||||
onclick={() => undo(field)}
|
||||
>
|
||||
<Undo2 class="size-3.5" />
|
||||
<span class="sr-only">Undo {field.label}</span>
|
||||
</Button>
|
||||
{/if}
|
||||
|
||||
{#if !isSame(value, incoming)}
|
||||
{#if isEmpty(value)}
|
||||
<!-- Nothing to weigh, so it is an offer rather than a choice -->
|
||||
<Button
|
||||
size="sm"
|
||||
class="h-7 gap-1 px-2 text-[11px]"
|
||||
onclick={() => take(field, 'replace')}
|
||||
>
|
||||
<ArrowRight class="size-3.5" />
|
||||
Take
|
||||
</Button>
|
||||
{:else}
|
||||
<Button
|
||||
variant="outline"
|
||||
size="icon"
|
||||
class="size-7"
|
||||
title="Replace"
|
||||
onclick={() => take(field, 'replace')}
|
||||
>
|
||||
<ArrowRight class="size-3.5" />
|
||||
<span class="sr-only">Replace {field.label}</span>
|
||||
</Button>
|
||||
{#each field.extra as action (action)}
|
||||
<Button
|
||||
variant="outline"
|
||||
size="icon"
|
||||
class="size-7"
|
||||
title={action === 'merge' ? 'Merge' : 'Append'}
|
||||
onclick={() => take(field, action)}
|
||||
>
|
||||
{#if action === 'merge'}
|
||||
<GitMerge class="size-3.5" />
|
||||
{:else}
|
||||
<Plus class="size-3.5" />
|
||||
{/if}
|
||||
<span class="sr-only">{action} {field.label}</span>
|
||||
</Button>
|
||||
{/each}
|
||||
{/if}
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<!-- The survivor's side: the edit form, live -->
|
||||
<div class="flex min-w-0 flex-col gap-1">
|
||||
<span class="flex items-center gap-1.5 text-[11px] text-muted-foreground">
|
||||
{field.label}
|
||||
{#if edited}
|
||||
<Badge class="h-4 px-1.5 text-[9px] font-semibold">taken</Badge>
|
||||
{/if}
|
||||
</span>
|
||||
|
||||
{#if field.kind === 'longtext'}
|
||||
<Textarea
|
||||
rows={4}
|
||||
class="text-sm"
|
||||
value={(value as string) ?? ''}
|
||||
oninput={(event) => (draft[field.key] = event.currentTarget.value)}
|
||||
/>
|
||||
{:else if field.kind === 'list' || field.kind === 'keyed'}
|
||||
<!-- Edited through the take actions; typing here would need the
|
||||
tags and key/value editors, which belong to the edit form. -->
|
||||
<div
|
||||
class="flex min-h-8 flex-wrap items-center gap-1 rounded-md border bg-background px-2 py-1"
|
||||
>
|
||||
{#if isEmpty(value)}
|
||||
<span class="text-sm text-muted-foreground">—</span>
|
||||
{:else if field.kind === 'list'}
|
||||
{#each value as string[] as item (item)}
|
||||
<Badge variant="secondary" class="font-normal">{item}</Badge>
|
||||
{/each}
|
||||
{:else}
|
||||
{#each Object.entries(value as Record<string, string>) as [name, id] (name)}
|
||||
<Badge variant="secondary" class="font-mono text-[10px] font-normal">
|
||||
{name}: {id}
|
||||
</Badge>
|
||||
{/each}
|
||||
{/if}
|
||||
</div>
|
||||
{:else}
|
||||
<Input
|
||||
type={field.kind === 'number'
|
||||
? 'number'
|
||||
: field.kind === 'date'
|
||||
? 'date'
|
||||
: 'text'}
|
||||
class="h-8 text-sm"
|
||||
value={(value as string | number) ?? ''}
|
||||
oninput={(event) =>
|
||||
(draft[field.key] =
|
||||
field.kind === 'number'
|
||||
? Number(event.currentTarget.value) || null
|
||||
: event.currentTarget.value)}
|
||||
/>
|
||||
{/if}
|
||||
</div>
|
||||
</div>
|
||||
{/each}
|
||||
{/each}
|
||||
|
||||
{#if agreed.length > 0}
|
||||
<p class="pt-3 text-center text-[11px] text-muted-foreground italic">
|
||||
{agreed.map((field) => field.label).join(', ')} — identical in both
|
||||
</p>
|
||||
{/if}
|
||||
{/if}
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<Dialog.Footer class="shrink-0 items-center gap-2 border-t px-5 py-3 sm:justify-between">
|
||||
<p class="text-xs text-muted-foreground">
|
||||
{#if changed.length}
|
||||
{changed.length}
|
||||
{changed.length === 1 ? 'change' : 'changes'} pending · this cannot be undone
|
||||
{:else}
|
||||
Metadata is left as #{survivor.id} has it · this cannot be undone
|
||||
{/if}
|
||||
</p>
|
||||
<div class="flex gap-2">
|
||||
<Button variant="outline" onclick={() => (open = false)}>Cancel</Button>
|
||||
<Button onclick={submit} disabled={busy || candidates.length === 0}>
|
||||
Merge into “{survivor.title}”
|
||||
</Button>
|
||||
</div>
|
||||
</Dialog.Footer>
|
||||
</Dialog.Content>
|
||||
</Dialog.Root>
|
||||
@@ -0,0 +1,44 @@
|
||||
import { invalidate } from '$app/navigation';
|
||||
import { toast } from 'svelte-sonner';
|
||||
|
||||
import { mergeBooks } from '$lib/api/book.remote';
|
||||
import type { Book } from '$lib/schema';
|
||||
import type { FieldValue } from './field-spec';
|
||||
|
||||
/**
|
||||
* Fold `folded` into `survivor`, and tell the reader how it went.
|
||||
*
|
||||
* Shared because a merge is reachable two ways — the workbench, where the reader
|
||||
* resolved the metadata field by field, and the quick action beside it, which
|
||||
* takes the survivor's metadata as it stands. Only the `metadata` argument
|
||||
* differs, and the invalidation and the wording should not.
|
||||
*
|
||||
* @returns whether the merge went through; the caller decides what to close or
|
||||
* clear, and has no toast of its own to write either way.
|
||||
*/
|
||||
export async function mergeInto(
|
||||
libraryId: number | string,
|
||||
survivor: Book,
|
||||
folded: Book[],
|
||||
metadata?: Record<string, FieldValue>
|
||||
): Promise<boolean> {
|
||||
try {
|
||||
await mergeBooks({
|
||||
library_id: libraryId,
|
||||
survivor_id: survivor.id,
|
||||
merged_ids: folded.map((book) => book.id),
|
||||
metadata
|
||||
});
|
||||
|
||||
// Both, because a merge is started from the duplicates review and from the
|
||||
// library's selection toolbar, and each page depends on a different one.
|
||||
await Promise.all([invalidate('app:books'), invalidate('app:duplicate-books')]);
|
||||
|
||||
toast.success(`Merged into “${survivor.title}”`);
|
||||
return true;
|
||||
} catch (error) {
|
||||
console.error('Failed to merge books', error);
|
||||
toast.error(error instanceof Error ? error.message : 'Could not merge these books');
|
||||
return false;
|
||||
}
|
||||
}
|
||||
@@ -33,13 +33,15 @@
|
||||
title: 'Home',
|
||||
icon: House,
|
||||
routeId: '/(root)/(library)/library/[libraryId]',
|
||||
path: (id?: number) => resolve('/(root)/(library)/library/[libraryId]', { libraryId: String(id ?? '') })
|
||||
path: (id?: number) =>
|
||||
resolve('/(root)/(library)/library/[libraryId]', { libraryId: String(id ?? '') })
|
||||
},
|
||||
{
|
||||
title: 'Library',
|
||||
icon: LibraryBig,
|
||||
routeId: '/(root)/(library)/library/[libraryId]/view',
|
||||
path: (id?: number) => resolve('/(root)/(library)/library/[libraryId]/view', { libraryId: String(id ?? '') })
|
||||
path: (id?: number) =>
|
||||
resolve('/(root)/(library)/library/[libraryId]/view', { libraryId: String(id ?? '') })
|
||||
},
|
||||
{ title: 'Shelves', icon: Rows3, routeId: null, path: () => '#', shelves: [] }
|
||||
];
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
<script lang="ts">
|
||||
import { fly } from 'svelte/transition';
|
||||
import { prefersReducedMotion } from 'svelte/motion';
|
||||
import { ChevronDown, CircleAlert, RotateCcw, X } from '@lucide/svelte';
|
||||
import { resolve } from '$app/paths';
|
||||
import { ChevronDown, CircleAlert, Copy, RotateCcw, X } from '@lucide/svelte';
|
||||
|
||||
import { Button } from '$lib/components/ui/button/index.js';
|
||||
import { Spinner } from '$lib/components/ui/spinner/index';
|
||||
import { getUploadQueueState } from '$lib/state/upload-queue.svelte';
|
||||
import type { DuplicateBook, DuplicateFile } from '$lib/schema';
|
||||
import { formatFileSize } from '$lib/utils';
|
||||
|
||||
const queue = getUploadQueueState();
|
||||
@@ -19,10 +21,48 @@
|
||||
const heading = $derived.by(() => {
|
||||
const noun = queue.total === 1 ? 'book' : 'books';
|
||||
if (queue.active) return `Adding ${queue.settled} of ${queue.total} ${noun}`;
|
||||
if (queue.failed === 0) return `Added ${queue.done} ${noun}`;
|
||||
if (queue.done === 0) return `Couldn't add ${queue.failed} ${noun}`;
|
||||
return `Added ${queue.done}, ${queue.failed} failed`;
|
||||
if (queue.failed === 0 && queue.skipped === 0) return `Added ${queue.done} ${noun}`;
|
||||
if (queue.done === 0 && queue.skipped === 0) return `Couldn't add ${queue.failed} ${noun}`;
|
||||
|
||||
const parts = [];
|
||||
if (queue.done > 0) parts.push(`Added ${queue.done}`);
|
||||
if (queue.skipped > 0) parts.push(`${queue.skipped} already here`);
|
||||
if (queue.failed > 0) parts.push(`${queue.failed} failed`);
|
||||
return parts.join(', ');
|
||||
});
|
||||
|
||||
/**
|
||||
* What the reader needs to know about files that were not stored.
|
||||
*
|
||||
* One duplicate can name where its bytes already live; several would be a list,
|
||||
* and the row has no room for one, so they collapse to a count.
|
||||
*/
|
||||
function duplicateNote(duplicates: DuplicateFile[]) {
|
||||
if (duplicates.length > 1) return `${duplicates.length} files are already in your library`;
|
||||
|
||||
const [only] = duplicates;
|
||||
// No book to name: it matched another file in this same upload.
|
||||
return only.book_title ? `Already in ${only.book_title}` : 'Already added by this upload';
|
||||
}
|
||||
|
||||
/** The book a note can link to, when a single duplicate points at exactly one. */
|
||||
function noteTarget(duplicates: DuplicateFile[]) {
|
||||
return duplicates.length === 1 ? duplicates[0].book_id : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* What the reader needs to know about a book that went in and may already be here.
|
||||
*
|
||||
* Worded weaker than the file-level "Already in …" on purpose. That one means the
|
||||
* library holds these exact bytes; this one means the metadata agrees, which a
|
||||
* second edition, a translation and a re-scan all do. Nothing was refused.
|
||||
*/
|
||||
function possibleNote(candidates: DuplicateBook[]) {
|
||||
if (candidates.length > 1)
|
||||
return `Might already be in your library, ${candidates.length} times`;
|
||||
|
||||
return `Might already be in your library — ${candidates[0].title}`;
|
||||
}
|
||||
</script>
|
||||
|
||||
{#if queue.total > 0}
|
||||
@@ -44,6 +84,9 @@
|
||||
<Spinner class="size-4 shrink-0" />
|
||||
{:else if queue.failed > 0}
|
||||
<CircleAlert class="size-4 shrink-0 text-destructive" />
|
||||
{:else if queue.skipped > 0}
|
||||
<!-- Muted, not alarming: nothing went wrong, the books were already here. -->
|
||||
<Copy class="size-4 shrink-0 text-muted-foreground" />
|
||||
{/if}
|
||||
|
||||
<span class="min-w-0 flex-1 truncate text-sm font-medium">{heading}</span>
|
||||
@@ -78,12 +121,61 @@
|
||||
{#if !queue.collapsed}
|
||||
<ul class="flex max-h-64 flex-col divide-y overflow-y-auto">
|
||||
{#each queue.jobs as job (job.id)}
|
||||
{@const duplicates = job.duplicates ?? []}
|
||||
{@const target = noteTarget(duplicates)}
|
||||
{@const possible = job.possibleDuplicates ?? []}
|
||||
<li class="flex min-w-0 items-center gap-2 px-3 py-2">
|
||||
<span class="min-w-0 flex-1">
|
||||
<span class="block truncate text-sm" title={job.label}>{job.label}</span>
|
||||
<span class="block font-mono text-[10px] text-muted-foreground tabular-nums">
|
||||
{job.error ?? formatFileSize(job.size)}
|
||||
</span>
|
||||
|
||||
{#if job.error}
|
||||
<span class="block font-mono text-[10px] text-muted-foreground tabular-nums">
|
||||
{job.error}
|
||||
</span>
|
||||
{:else if duplicates.length > 0}
|
||||
<!--
|
||||
Reported, with no offer to override. Storing the same bytes twice
|
||||
splits reading progress and shelves across two records that can
|
||||
never converge, and there is no version of that the reader wants.
|
||||
`allow_duplicates` stays on the API for when the match is wrong.
|
||||
-->
|
||||
<span class="block min-w-0 text-[10px] text-muted-foreground">
|
||||
{#if target}
|
||||
<a
|
||||
href={resolve('/(root)/(library)/book/[bookId]', { bookId: String(target) })}
|
||||
class="truncate underline underline-offset-2"
|
||||
>
|
||||
{duplicateNote(duplicates)}
|
||||
</a>
|
||||
{:else}
|
||||
<span class="truncate">{duplicateNote(duplicates)}</span>
|
||||
{/if}
|
||||
</span>
|
||||
{:else if possible.length > 0}
|
||||
<!--
|
||||
A guess, not a fact, so it says so and stops there: the book was
|
||||
stored, and the library's duplicates screen is where a reader
|
||||
decides what to do about it.
|
||||
-->
|
||||
<span class="block min-w-0 text-[10px] text-muted-foreground">
|
||||
{#if possible.length === 1}
|
||||
<a
|
||||
href={resolve('/(root)/(library)/book/[bookId]', {
|
||||
bookId: String(possible[0].book_id)
|
||||
})}
|
||||
class="truncate underline underline-offset-2"
|
||||
>
|
||||
{possibleNote(possible)}
|
||||
</a>
|
||||
{:else}
|
||||
<span class="truncate">{possibleNote(possible)}</span>
|
||||
{/if}
|
||||
</span>
|
||||
{:else}
|
||||
<span class="block font-mono text-[10px] text-muted-foreground tabular-nums">
|
||||
{formatFileSize(job.size)}
|
||||
</span>
|
||||
{/if}
|
||||
</span>
|
||||
|
||||
<span
|
||||
@@ -96,6 +188,8 @@
|
||||
Adding
|
||||
{:else if job.status === 'done'}
|
||||
Done
|
||||
{:else if job.status === 'skipped'}
|
||||
Skipped
|
||||
{:else if job.status === 'failed'}
|
||||
Failed
|
||||
{:else}
|
||||
|
||||
@@ -1,15 +1,27 @@
|
||||
<script lang="ts">
|
||||
import { buttonVariants } from '$lib/components/ui/button/button.svelte';
|
||||
import * as Tooltip from '$lib/components/ui/tooltip/index';
|
||||
import { BookOpenCheck, Download, Trash2, SquareCheckBig, X, Album, PlusIcon } from '@lucide/svelte';
|
||||
import {
|
||||
BookOpenCheck,
|
||||
Download,
|
||||
GitMerge,
|
||||
Trash2,
|
||||
SquareCheckBig,
|
||||
SlidersHorizontal,
|
||||
X,
|
||||
Album,
|
||||
PlusIcon
|
||||
} from '@lucide/svelte';
|
||||
import * as DropdownMenu from '$lib/components/ui/dropdown-menu/index';
|
||||
import { getBookOperationsState } from '$lib/state/bookOperations.svelte';
|
||||
import { getBookSelectionState } from '$lib/state/bookSelection.svelte';
|
||||
import { getBookCollectionState } from '$lib/state/bookCollection.svelte';
|
||||
import { getLibraryState } from '$lib/state/library.svelte';
|
||||
import { getBookshelfState } from '$lib/state/bookshelf.svelte';
|
||||
import { invalidate } from '$app/navigation';
|
||||
import ShelfCreateDialog from '../forms/shelf-create-dialog.svelte';
|
||||
import MergeBooks from '../forms/merge-books/merge-books.svelte';
|
||||
import { mergeInto } from '../forms/merge-books/merge';
|
||||
import type { Book } from '$lib/schema';
|
||||
|
||||
const libraryState = getLibraryState();
|
||||
const bookshelfState = getBookshelfState();
|
||||
@@ -17,10 +29,50 @@
|
||||
const bookOps = getBookOperationsState();
|
||||
const collectionState = getBookCollectionState();
|
||||
|
||||
let selectedBooks = $derived(selectionState.getSelectedBooks())
|
||||
let selectedBooks = $derived(selectionState.getSelectedBooks());
|
||||
|
||||
let createShelfDialogOpen = $state(false)
|
||||
let createShelfDialogOpen = $state(false);
|
||||
|
||||
// The books the merge dialog opened on. Snapshotted rather than read live from
|
||||
// the selection, so clearing the selection on success cannot empty the dialog
|
||||
// underneath itself.
|
||||
let mergingBooks = $state<Book[] | null>(null);
|
||||
|
||||
// A quick merge has no dialog to disable, so the flag is what stops a slow
|
||||
// round trip being started twice.
|
||||
let quickMerging = $state(false);
|
||||
|
||||
/** Put the library back in step with a merge that went through. */
|
||||
function afterMerge(folded: Book[]) {
|
||||
// Only the folded records are gone; the survivor is still in the library.
|
||||
libraryState.activeLibrary!.total! -= folded.length;
|
||||
bookshelfState.deletedBooks(folded);
|
||||
selectionState.deselectAll();
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge without opening the workbench, keeping the book selected first.
|
||||
*
|
||||
* Selection is held in insertion order, so the first record is the one the
|
||||
* reader started from — the one they were looking at when they decided the
|
||||
* rest were copies of it.
|
||||
*/
|
||||
async function quickMerge() {
|
||||
if (quickMerging) return;
|
||||
|
||||
// Read once: the merge clears the selection, and the bookkeeping afterwards
|
||||
// still needs the records that went away.
|
||||
const [survivor, ...folded] = selectedBooks;
|
||||
if (!survivor || folded.length === 0) return;
|
||||
|
||||
quickMerging = true;
|
||||
|
||||
const merged = await mergeInto(libraryState.activeLibrary!.id, survivor, folded);
|
||||
|
||||
quickMerging = false;
|
||||
|
||||
if (merged) afterMerge(folded);
|
||||
}
|
||||
</script>
|
||||
|
||||
<!-- Mark selected as finished button -->
|
||||
@@ -95,12 +147,13 @@
|
||||
{/each}
|
||||
</DropdownMenu.Group>
|
||||
<DropdownMenu.Separator />
|
||||
<DropdownMenu.Item
|
||||
onclick={() => createShelfDialogOpen = true}
|
||||
class="text-muted-foreground ">
|
||||
<PlusIcon class="size-4" />
|
||||
New Shelf
|
||||
</DropdownMenu.Item>
|
||||
<DropdownMenu.Item
|
||||
onclick={() => (createShelfDialogOpen = true)}
|
||||
class="text-muted-foreground "
|
||||
>
|
||||
<PlusIcon class="size-4" />
|
||||
New Shelf
|
||||
</DropdownMenu.Item>
|
||||
</DropdownMenu.Content>
|
||||
</DropdownMenu.Root>
|
||||
{/snippet}
|
||||
@@ -128,6 +181,44 @@
|
||||
</Tooltip.Root>
|
||||
</Tooltip.Provider>
|
||||
|
||||
<!-- Merge selected button. Two books is the smallest thing a merge can mean, so
|
||||
it appears only once there are two. -->
|
||||
{#if selectedBooks.length > 1}
|
||||
<Tooltip.Provider>
|
||||
<Tooltip.Root ignoreNonKeyboardFocus>
|
||||
<Tooltip.Trigger>
|
||||
{#snippet child({ props })}
|
||||
<!-- A choice rather than a jump straight into the workbench: two
|
||||
copies of one book have nothing to resolve, and the dialog is a
|
||||
step in the way of saying so. -->
|
||||
<DropdownMenu.Root>
|
||||
<DropdownMenu.Trigger
|
||||
{...props}
|
||||
disabled={quickMerging}
|
||||
class={buttonVariants({ variant: 'ghost', size: 'icon' })}
|
||||
>
|
||||
<GitMerge />
|
||||
</DropdownMenu.Trigger>
|
||||
<DropdownMenu.Content>
|
||||
<DropdownMenu.Item onclick={quickMerge}>
|
||||
<GitMerge class="size-4 shrink-0" />
|
||||
Merge into first selected
|
||||
</DropdownMenu.Item>
|
||||
<DropdownMenu.Item onclick={() => (mergingBooks = selectedBooks)}>
|
||||
<SlidersHorizontal class="size-4 shrink-0" />
|
||||
Manual merge…
|
||||
</DropdownMenu.Item>
|
||||
</DropdownMenu.Content>
|
||||
</DropdownMenu.Root>
|
||||
{/snippet}
|
||||
</Tooltip.Trigger>
|
||||
<Tooltip.Content>
|
||||
<p>Merge {selectedBooks.length} books</p>
|
||||
</Tooltip.Content>
|
||||
</Tooltip.Root>
|
||||
</Tooltip.Provider>
|
||||
{/if}
|
||||
|
||||
<!-- Delete selected button -->
|
||||
<Tooltip.Provider>
|
||||
<Tooltip.Root ignoreNonKeyboardFocus>
|
||||
@@ -136,8 +227,8 @@
|
||||
bookOps.deleteDialogTitle = `Delete ${selectionState.getSelectedIds().length} books?`;
|
||||
bookOps.deleteFn = async (deleteFiles: boolean) => {
|
||||
await bookOps.deleteBooks(selectionState.getSelectedIds(), deleteFiles);
|
||||
libraryState.activeLibrary!.total! -= selectedBooks.length
|
||||
bookshelfState.deletedBooks(selectedBooks)
|
||||
libraryState.activeLibrary!.total! -= selectedBooks.length;
|
||||
bookshelfState.deletedBooks(selectedBooks);
|
||||
selectionState.deselectAll();
|
||||
};
|
||||
bookOps.deleteDialogOpen = true;
|
||||
@@ -186,14 +277,43 @@
|
||||
</Tooltip.Root>
|
||||
</Tooltip.Provider>
|
||||
|
||||
|
||||
<ShelfCreateDialog
|
||||
bind:open={createShelfDialogOpen}
|
||||
onSubmit={async (name: string) => {
|
||||
|
||||
const bookshelf = await bookshelfState.addBookshelf(name, libraryState.activeLibrary!.id, selectionState.getSelectedIds())
|
||||
selectedBooks.forEach(book => book.lists.push(bookshelf))
|
||||
selectionState.deselectAll()
|
||||
createShelfDialogOpen = false
|
||||
const bookshelf = await bookshelfState.addBookshelf(
|
||||
name,
|
||||
libraryState.activeLibrary!.id,
|
||||
selectionState.getSelectedIds()
|
||||
);
|
||||
selectedBooks.forEach((book) => book.lists.push(bookshelf));
|
||||
selectionState.deselectAll();
|
||||
createShelfDialogOpen = false;
|
||||
}}
|
||||
/>
|
||||
|
||||
<!--
|
||||
Keyed on the selection so opening merge on a different pair starts from those
|
||||
records rather than the previous dialog's draft.
|
||||
-->
|
||||
{#if mergingBooks}
|
||||
{#key mergingBooks.map((book) => book.id).join()}
|
||||
<MergeBooks
|
||||
books={mergingBooks}
|
||||
libraryId={libraryState.activeLibrary!.id}
|
||||
bind:open={
|
||||
() => mergingBooks !== null,
|
||||
(value) => {
|
||||
// Bound, not passed as a bare `true`: the dialog closes itself on an
|
||||
// outside click or Escape, and if that never reaches `mergingBooks`
|
||||
// the snapshot stays set — the menu item then re-assigns the same
|
||||
// selection, nothing changes, and the dialog never reopens.
|
||||
if (!value) mergingBooks = null;
|
||||
}
|
||||
}
|
||||
onmerged={(_survivor, folded) => {
|
||||
afterMerge(folded);
|
||||
mergingBooks = null;
|
||||
}}
|
||||
/>
|
||||
{/key}
|
||||
{/if}
|
||||
|
||||
@@ -1,7 +1,14 @@
|
||||
<script lang="ts">
|
||||
import type { Book } from '$lib/schema';
|
||||
|
||||
let { book, class: className = '' }: { book: Book; class?: string } = $props();
|
||||
// Typed by what is drawn rather than by `Book`: the duplicates review holds
|
||||
// candidates that carry a title and author names without a whole record, and a
|
||||
// `Book` satisfies this shape anyway.
|
||||
let {
|
||||
book,
|
||||
class: className = ''
|
||||
}: {
|
||||
book: { title: string; authors?: { name: string }[] | null };
|
||||
class?: string;
|
||||
} = $props();
|
||||
|
||||
/**
|
||||
* Bookcloth tones rather than a point on the hue wheel.
|
||||
|
||||
@@ -6,6 +6,49 @@ export type Book = components['schemas']['BookRead'];
|
||||
export type BookFile = components['schemas']['FileMetadataRead'];
|
||||
export type BookProgress = components['schemas']['BookProgressRead'];
|
||||
|
||||
/**
|
||||
* A file the library already held, so the upload did not store it again.
|
||||
*
|
||||
* `book_id` is null when the match was another file in the same upload — there is
|
||||
* no book to point at yet.
|
||||
*/
|
||||
export type DuplicateFile = components['schemas']['DuplicateFileRead'];
|
||||
export type BooksUploadResult = components['schemas']['BooksUploadResult'];
|
||||
|
||||
/**
|
||||
* A stored book that may be the same book as another one.
|
||||
*
|
||||
* Weaker than `DuplicateFile`, and deliberately so: that one means the library holds
|
||||
* these exact bytes, this one means the metadata agrees. A second edition and a
|
||||
* translation both look like this, so nothing is ever refused on the strength of it.
|
||||
*/
|
||||
export type DuplicateBook = components['schemas']['DuplicateBookRead'];
|
||||
|
||||
/** A book that was imported, together with what it might be a second copy of. */
|
||||
export type PossibleDuplicate = components['schemas']['PossibleDuplicateRead'];
|
||||
|
||||
/** Books the library holds that all look like copies of one book. */
|
||||
export type DuplicateBookGroup = components['schemas']['DuplicateBookGroupRead'];
|
||||
|
||||
/** The metadata a reader resolved while merging, or while reviewing a provider. */
|
||||
export type BookMetadataUpdate = components['schemas']['BookMetadataUpdate'];
|
||||
|
||||
/** Mirrors BookMerge in backend/src/chitai/schemas/book.py */
|
||||
export const bookMergeSchema = z.object({
|
||||
library_id: z.coerce.number(),
|
||||
survivor_id: z.coerce.number(),
|
||||
merged_ids: z.array(z.coerce.number()).min(1, 'Pick at least one book to fold in'),
|
||||
// Passed through untouched — the backend validates it as BookMetadataUpdate, and
|
||||
// duplicating that shape here would be two places to keep in step for no gain.
|
||||
metadata: z.record(z.string(), z.unknown()).optional()
|
||||
});
|
||||
|
||||
/** Mirrors DuplicateDismissal in backend/src/chitai/schemas/book.py */
|
||||
export const duplicateDismissalSchema = z.object({
|
||||
book_a_id: z.coerce.number(),
|
||||
book_b_id: z.coerce.number()
|
||||
});
|
||||
|
||||
export const bookQuerySchema = commonQuerySchema.extend({
|
||||
libraries: stringArrayCoerce,
|
||||
authors: stringArrayCoerce,
|
||||
@@ -97,4 +140,6 @@ export type BookQuery = z.infer<typeof bookQuerySchema>;
|
||||
export type BookEditMetadata = z.infer<typeof editBookMetadataSchema>;
|
||||
export type DeleteBook = z.infer<typeof deleteBooksSchema>;
|
||||
export type DeleteBookFiles = z.infer<typeof deleteBookFilesSchema>;
|
||||
export type DuplicateDismissal = z.infer<typeof duplicateDismissalSchema>;
|
||||
export type BookMerge = z.infer<typeof bookMergeSchema>;
|
||||
export type UpdateBookProgress = z.infer<typeof updateBookProgressSchema>;
|
||||
|
||||
+319
-11
@@ -22,6 +22,23 @@ export interface paths {
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/books/duplicate-files": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
get?: never;
|
||||
put?: never;
|
||||
/** CheckDuplicateFiles */
|
||||
post: operations["BooksDuplicateFilesCheckDuplicateFiles"];
|
||||
delete?: never;
|
||||
options?: never;
|
||||
head?: never;
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/books": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
@@ -58,6 +75,24 @@ export interface paths {
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/books/duplicate-books/dismissals": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
get?: never;
|
||||
put?: never;
|
||||
/** DismissDuplicateBooks */
|
||||
post: operations["BooksDuplicateBooksDismissalsDismissDuplicateBooks"];
|
||||
/** RestoreDuplicateBooks */
|
||||
delete: operations["BooksDuplicateBooksDismissalsRestoreDuplicateBooks"];
|
||||
options?: never;
|
||||
head?: never;
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/books/{book_id}": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
@@ -110,6 +145,40 @@ export interface paths {
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/books/duplicate-books": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
/** ListDuplicateBooks */
|
||||
get: operations["BooksDuplicateBooksListDuplicateBooks"];
|
||||
put?: never;
|
||||
post?: never;
|
||||
delete?: never;
|
||||
options?: never;
|
||||
head?: never;
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/books/merge": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
get?: never;
|
||||
put?: never;
|
||||
/** MergeBooks */
|
||||
post: operations["BooksMergeMergeBooks"];
|
||||
delete?: never;
|
||||
options?: never;
|
||||
head?: never;
|
||||
patch?: never;
|
||||
trace?: never;
|
||||
};
|
||||
"/books/progress": {
|
||||
parameters: {
|
||||
query?: never;
|
||||
@@ -637,6 +706,12 @@ export interface components {
|
||||
cover_image?: string | null;
|
||||
files?: string[];
|
||||
};
|
||||
/** BookMerge */
|
||||
BookMerge: {
|
||||
survivor_id: number;
|
||||
merged_ids: number[];
|
||||
metadata?: components["schemas"]["BookMetadataUpdate"] | null;
|
||||
};
|
||||
/** BookMetadataUpdate */
|
||||
BookMetadataUpdate: {
|
||||
title?: string | null;
|
||||
@@ -710,6 +785,46 @@ export interface components {
|
||||
BooksCreateFromFiles: {
|
||||
files?: string[];
|
||||
};
|
||||
/** BooksUploadResult */
|
||||
BooksUploadResult: {
|
||||
created: components["schemas"]["BookRead"][];
|
||||
skipped: components["schemas"]["DuplicateFileRead"][];
|
||||
possible_duplicates?: components["schemas"]["PossibleDuplicateRead"][];
|
||||
};
|
||||
/** DuplicateBookGroupRead */
|
||||
DuplicateBookGroupRead: {
|
||||
books: components["schemas"]["DuplicateBookRead"][];
|
||||
};
|
||||
/** DuplicateBookRead */
|
||||
DuplicateBookRead: {
|
||||
book_id: number;
|
||||
title: string;
|
||||
authors: string[];
|
||||
library_id: number;
|
||||
cover_image?: string | null;
|
||||
matched_on: string[];
|
||||
};
|
||||
/** DuplicateDismissal */
|
||||
DuplicateDismissal: {
|
||||
book_a_id: number;
|
||||
book_b_id: number;
|
||||
};
|
||||
/** DuplicateFileRead */
|
||||
DuplicateFileRead: {
|
||||
filename: string;
|
||||
hash: string;
|
||||
size: number;
|
||||
library_id: number;
|
||||
book_id?: number | null;
|
||||
book_title?: string | null;
|
||||
};
|
||||
/** FileFingerprint */
|
||||
FileFingerprint: {
|
||||
hash: string;
|
||||
size: number;
|
||||
/** @default */
|
||||
filename: string;
|
||||
};
|
||||
/** FileMetadataRead */
|
||||
FileMetadataRead: {
|
||||
id: number;
|
||||
@@ -777,6 +892,12 @@ export interface components {
|
||||
refresh_token?: string | null;
|
||||
expires_in?: number | null;
|
||||
};
|
||||
/** PossibleDuplicateRead */
|
||||
PossibleDuplicateRead: {
|
||||
book_id: number;
|
||||
title: string;
|
||||
candidates: components["schemas"]["DuplicateBookRead"][];
|
||||
};
|
||||
/** PublisherRead */
|
||||
PublisherRead: {
|
||||
id: number;
|
||||
@@ -828,6 +949,7 @@ export interface operations {
|
||||
parameters: {
|
||||
query?: {
|
||||
library_id?: number | null;
|
||||
allow_duplicates?: boolean;
|
||||
};
|
||||
header?: never;
|
||||
path: {
|
||||
@@ -908,6 +1030,47 @@ export interface operations {
|
||||
};
|
||||
};
|
||||
};
|
||||
BooksDuplicateFilesCheckDuplicateFiles: {
|
||||
parameters: {
|
||||
query?: {
|
||||
library_id?: number | null;
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody: {
|
||||
content: {
|
||||
"application/json": components["schemas"]["FileFingerprint"][];
|
||||
};
|
||||
};
|
||||
responses: {
|
||||
/** @description Request fulfilled, document follows */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["DuplicateFileRead"][];
|
||||
};
|
||||
};
|
||||
/** @description Bad request syntax or unsupported method */
|
||||
400: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": {
|
||||
status_code: number;
|
||||
detail: string;
|
||||
extra?: null | {
|
||||
[key: string]: unknown;
|
||||
} | unknown[];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
BooksListBooks: {
|
||||
parameters: {
|
||||
query?: {
|
||||
@@ -917,7 +1080,7 @@ export interface operations {
|
||||
tags?: number[] | null;
|
||||
shelves?: number[] | null;
|
||||
progress?: string[] | null;
|
||||
ids?: string[] | null;
|
||||
ids?: number[] | null;
|
||||
searchString?: string | null;
|
||||
searchIgnoreCase?: boolean | null;
|
||||
currentPage?: number;
|
||||
@@ -969,6 +1132,7 @@ export interface operations {
|
||||
parameters: {
|
||||
query?: {
|
||||
library_id?: number | null;
|
||||
allow_duplicates?: boolean;
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
@@ -1047,6 +1211,7 @@ export interface operations {
|
||||
parameters: {
|
||||
query?: {
|
||||
library_id?: number | null;
|
||||
allow_duplicates?: boolean;
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
@@ -1060,21 +1225,86 @@ export interface operations {
|
||||
responses: {
|
||||
/** @description Document created, URL follows */
|
||||
201: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["BooksUploadResult"];
|
||||
};
|
||||
};
|
||||
/** @description Bad request syntax or unsupported method */
|
||||
400: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": {
|
||||
items?: components["schemas"]["BookRead"][];
|
||||
/** @description Maximal number of items to send. */
|
||||
limit?: number;
|
||||
/** @description Offset from the beginning of the query. */
|
||||
offset?: number;
|
||||
/** @description Total number of items. */
|
||||
total?: number;
|
||||
status_code: number;
|
||||
detail: string;
|
||||
extra?: null | {
|
||||
[key: string]: unknown;
|
||||
} | unknown[];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
BooksDuplicateBooksDismissalsDismissDuplicateBooks: {
|
||||
parameters: {
|
||||
query?: never;
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody: {
|
||||
content: {
|
||||
"application/json": components["schemas"]["DuplicateDismissal"];
|
||||
};
|
||||
};
|
||||
responses: {
|
||||
/** @description Request fulfilled, nothing follows */
|
||||
204: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content?: never;
|
||||
};
|
||||
/** @description Bad request syntax or unsupported method */
|
||||
400: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": {
|
||||
status_code: number;
|
||||
detail: string;
|
||||
extra?: null | {
|
||||
[key: string]: unknown;
|
||||
} | unknown[];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
BooksDuplicateBooksDismissalsRestoreDuplicateBooks: {
|
||||
parameters: {
|
||||
query: {
|
||||
book_a_id: number;
|
||||
book_b_id: number;
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody?: never;
|
||||
responses: {
|
||||
/** @description Request fulfilled, nothing follows */
|
||||
204: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content?: never;
|
||||
};
|
||||
/** @description Bad request syntax or unsupported method */
|
||||
400: {
|
||||
headers: {
|
||||
@@ -1254,6 +1484,84 @@ export interface operations {
|
||||
};
|
||||
};
|
||||
};
|
||||
BooksDuplicateBooksListDuplicateBooks: {
|
||||
parameters: {
|
||||
query?: {
|
||||
library_id?: number | null;
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody?: never;
|
||||
responses: {
|
||||
/** @description Request fulfilled, document follows */
|
||||
200: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["DuplicateBookGroupRead"][];
|
||||
};
|
||||
};
|
||||
/** @description Bad request syntax or unsupported method */
|
||||
400: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": {
|
||||
status_code: number;
|
||||
detail: string;
|
||||
extra?: null | {
|
||||
[key: string]: unknown;
|
||||
} | unknown[];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
BooksMergeMergeBooks: {
|
||||
parameters: {
|
||||
query?: {
|
||||
library_id?: number | null;
|
||||
};
|
||||
header?: never;
|
||||
path?: never;
|
||||
cookie?: never;
|
||||
};
|
||||
requestBody: {
|
||||
content: {
|
||||
"application/json": components["schemas"]["BookMerge"];
|
||||
};
|
||||
};
|
||||
responses: {
|
||||
/** @description Document created, URL follows */
|
||||
201: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": components["schemas"]["BookRead"];
|
||||
};
|
||||
};
|
||||
/** @description Bad request syntax or unsupported method */
|
||||
400: {
|
||||
headers: {
|
||||
[name: string]: unknown;
|
||||
};
|
||||
content: {
|
||||
"application/json": {
|
||||
status_code: number;
|
||||
detail: string;
|
||||
extra?: null | {
|
||||
[key: string]: unknown;
|
||||
} | unknown[];
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
};
|
||||
BooksProgressSetBookProgressBatch: {
|
||||
parameters: {
|
||||
query: {
|
||||
@@ -1931,7 +2239,7 @@ export interface operations {
|
||||
tags?: number[] | null;
|
||||
shelves?: number[] | null;
|
||||
progress?: string[] | null;
|
||||
ids?: string[] | null;
|
||||
ids?: number[] | null;
|
||||
searchString?: string | null;
|
||||
searchIgnoreCase?: boolean | null;
|
||||
currentPage?: number;
|
||||
@@ -2018,7 +2326,7 @@ export interface operations {
|
||||
OpdsLibraryLibraryIdCollectionTypeGetLibraryCollectionFeed: {
|
||||
parameters: {
|
||||
query?: {
|
||||
ids?: string[] | null;
|
||||
ids?: number[] | null;
|
||||
searchString?: string | null;
|
||||
searchIgnoreCase?: boolean | null;
|
||||
currentPage?: number;
|
||||
@@ -2147,7 +2455,7 @@ export interface operations {
|
||||
tags?: number[] | null;
|
||||
shelves?: number[] | null;
|
||||
progress?: string[] | null;
|
||||
ids?: string[] | null;
|
||||
ids?: number[] | null;
|
||||
searchString?: string | null;
|
||||
searchIgnoreCase?: boolean | null;
|
||||
currentPage?: number;
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
import { getContext, setContext } from 'svelte';
|
||||
import { invalidate } from '$app/navigation';
|
||||
|
||||
import type { Book, PaginatedResponse } from '$lib/schema';
|
||||
import type { Book, BooksUploadResult, DuplicateBook, DuplicateFile } from '$lib/schema';
|
||||
|
||||
export type UploadStatus = 'queued' | 'uploading' | 'done' | 'failed';
|
||||
export type UploadStatus = 'queued' | 'uploading' | 'done' | 'skipped' | 'failed';
|
||||
|
||||
export interface UploadJob {
|
||||
id: string;
|
||||
@@ -15,10 +15,23 @@ export interface UploadJob {
|
||||
status: UploadStatus;
|
||||
error?: string;
|
||||
book?: Book;
|
||||
/**
|
||||
* Files the library already held, which were not stored again. A job with
|
||||
* nothing left over lands as `skipped`; one that had something new is `done`
|
||||
* and still carries these, since the reader asked for those files too.
|
||||
*/
|
||||
duplicates?: DuplicateFile[];
|
||||
/**
|
||||
* Books already in the library that the one this job created might be a second
|
||||
* copy of. Much weaker than `duplicates`: the bytes are new and only the metadata
|
||||
* agrees, which a second edition and a translation both do. The book was stored.
|
||||
*/
|
||||
possibleDuplicates?: DuplicateBook[];
|
||||
}
|
||||
|
||||
export interface UploadSummary {
|
||||
created: number;
|
||||
skipped: number;
|
||||
failed: number;
|
||||
firstBook?: Book;
|
||||
}
|
||||
@@ -70,12 +83,30 @@ export class UploadQueueState {
|
||||
|
||||
readonly total = $derived(this.jobs.length);
|
||||
readonly done = $derived(this.jobs.filter((job) => job.status === 'done').length);
|
||||
readonly skipped = $derived(this.jobs.filter((job) => job.status === 'skipped').length);
|
||||
readonly failed = $derived(this.jobs.filter((job) => job.status === 'failed').length);
|
||||
readonly active = $derived(
|
||||
this.jobs.some((job) => job.status === 'queued' || job.status === 'uploading')
|
||||
);
|
||||
readonly current = $derived(this.jobs.find((job) => job.status === 'uploading'));
|
||||
readonly settled = $derived(this.done + this.failed);
|
||||
readonly settled = $derived(this.done + this.skipped + this.failed);
|
||||
|
||||
/**
|
||||
* Whether the run left something the reader still has to see.
|
||||
*
|
||||
* A skipped book is not a failure, but it is the only place that says the file
|
||||
* was already here — and the only place to override it from. A book that may
|
||||
* already be in the library counts too: it is a note the reader has to actually
|
||||
* read, and a tray that clears itself after six seconds is one they never will.
|
||||
*/
|
||||
readonly needsAttention = $derived(
|
||||
this.jobs.some(
|
||||
(job) =>
|
||||
job.status === 'failed' ||
|
||||
(job.duplicates?.length ?? 0) > 0 ||
|
||||
(job.possibleDuplicates?.length ?? 0) > 0
|
||||
)
|
||||
);
|
||||
|
||||
#running = false;
|
||||
#dismissTimer: ReturnType<typeof setTimeout> | undefined;
|
||||
@@ -129,20 +160,20 @@ export class UploadQueueState {
|
||||
|
||||
release() {
|
||||
this.#held = false;
|
||||
if (!this.active && this.failed === 0 && this.total > 0) this.#scheduleDismiss();
|
||||
if (!this.active && !this.needsAttention && this.total > 0) this.#scheduleDismiss();
|
||||
}
|
||||
|
||||
/**
|
||||
* Clears itself only when everything worked. A run with failures stays until
|
||||
* dismissed — it is the only record of what did not make it in, and the only
|
||||
* place to retry from.
|
||||
* Clears itself only when everything went in cleanly. A run with failures or
|
||||
* skipped files stays until dismissed — it is the only record of what did not
|
||||
* make it in, and the only place to retry or override from.
|
||||
*/
|
||||
#scheduleDismiss() {
|
||||
clearTimeout(this.#dismissTimer);
|
||||
if (this.#held) return;
|
||||
|
||||
this.#dismissTimer = setTimeout(() => {
|
||||
if (!this.active && this.failed === 0) this.jobs = [];
|
||||
if (!this.active && !this.needsAttention) this.jobs = [];
|
||||
}, DISMISS_AFTER_MS);
|
||||
}
|
||||
|
||||
@@ -169,26 +200,35 @@ export class UploadQueueState {
|
||||
const index = this.jobs.findIndex((job) => job.status === 'queued');
|
||||
if (index === -1) break;
|
||||
|
||||
const job = this.jobs[index];
|
||||
this.#patch(index, { status: 'uploading' });
|
||||
|
||||
try {
|
||||
const body = new FormData();
|
||||
for (const file of this.jobs[index].files) body.append('files', file);
|
||||
for (const file of job.files) body.append('files', file);
|
||||
|
||||
const response = await fetch(
|
||||
`/api/books/fromFiles?library_id=${encodeURIComponent(String(this.jobs[index].libraryId))}`,
|
||||
`/api/books/fromFiles?library_id=${encodeURIComponent(String(job.libraryId))}`,
|
||||
{ method: 'POST', body }
|
||||
);
|
||||
|
||||
if (!response.ok) throw new Error(`The server returned ${response.status}`);
|
||||
|
||||
const result: PaginatedResponse<Book> = await response.json();
|
||||
const book = result.items[0];
|
||||
const result: BooksUploadResult = await response.json();
|
||||
const book = result.created[0];
|
||||
|
||||
created += result.total ?? result.items.length;
|
||||
created += result.created.length;
|
||||
firstBook ??= book;
|
||||
|
||||
this.#patch(index, { status: 'done', book });
|
||||
// Nothing created means every file in this folder was already here.
|
||||
// That is not a failure, and it is not something to hide either.
|
||||
this.#patch(index, {
|
||||
status: result.created.length === 0 ? 'skipped' : 'done',
|
||||
book,
|
||||
duplicates: result.skipped,
|
||||
// One job is one book, so it has at most one set of candidates.
|
||||
possibleDuplicates: result.possible_duplicates?.[0]?.candidates
|
||||
});
|
||||
} catch (error) {
|
||||
// One bad book must not take the rest of the queue with it.
|
||||
console.error(`Failed to upload ${this.jobs[index].label}`, error);
|
||||
@@ -205,9 +245,9 @@ export class UploadQueueState {
|
||||
|
||||
if (created > 0) await invalidate('app:books');
|
||||
|
||||
if (this.failed === 0) this.#scheduleDismiss();
|
||||
if (!this.needsAttention) this.#scheduleDismiss();
|
||||
|
||||
onFinished?.({ created, failed: this.failed, firstBook });
|
||||
onFinished?.({ created, skipped: this.skipped, failed: this.failed, firstBook });
|
||||
}
|
||||
|
||||
#patch(index: number, changes: Partial<UploadJob>) {
|
||||
|
||||
@@ -1,9 +1,14 @@
|
||||
<script lang="ts">
|
||||
import { page } from '$app/state';
|
||||
import { resolve } from '$app/paths';
|
||||
import { getLibraryState } from '$lib/state/library.svelte';
|
||||
|
||||
let { children } = $props();
|
||||
|
||||
// Set in (root)/+layout.svelte, above this group, so listing the libraries in
|
||||
// the nav needs no load function.
|
||||
const libraryState = getLibraryState();
|
||||
|
||||
// Route ids rather than paths: resolve() is called in the markup so it stays a
|
||||
// direct call the lint rule can see, and the active check compares route ids.
|
||||
// A pathname comparison would miss during SSR, where resolve() returns a
|
||||
@@ -14,12 +19,28 @@
|
||||
{ title: 'Libraries', routeId: '/(root)/settings/libraries' },
|
||||
{ title: 'Devices', routeId: '/(root)/settings/devices' }
|
||||
] as const;
|
||||
|
||||
/**
|
||||
* Whether a nested library item is the one being looked at.
|
||||
*
|
||||
* Both halves are needed: the route id alone is shared by every library, so
|
||||
* matching on it would light all of them up at once.
|
||||
*/
|
||||
function isActiveLibrary(id: number) {
|
||||
return (
|
||||
page.route.id?.startsWith('/(root)/settings/libraries/[libraryId]') === true &&
|
||||
page.params.libraryId === String(id)
|
||||
);
|
||||
}
|
||||
</script>
|
||||
|
||||
<div class="flex h-[calc(100vh-var(--header-height)-2rem)] flex-col">
|
||||
<h1 class="font-serif text-xl font-medium tracking-tight">Settings</h1>
|
||||
<div class="mt-6 flex flex-1 gap-8 overflow-hidden">
|
||||
<nav class="flex w-48 shrink-0 flex-col gap-1">
|
||||
<!-- Scrolls on its own now that it holds a list that grows with the number
|
||||
of libraries; the row above it is overflow-hidden, so without this a
|
||||
long list is clipped rather than reachable. -->
|
||||
<nav class="flex w-48 shrink-0 flex-col gap-1 overflow-y-auto">
|
||||
{#each items as item}
|
||||
<a
|
||||
href={resolve(item.routeId)}
|
||||
@@ -30,6 +51,26 @@
|
||||
>
|
||||
{item.title}
|
||||
</a>
|
||||
|
||||
<!-- Libraries is the one item with children: a library's own settings
|
||||
hang off it, so every library and every section it has stay one
|
||||
click from here. -->
|
||||
{#if item.routeId === '/(root)/settings/libraries'}
|
||||
{#each libraryState.libraries as library (library.id)}
|
||||
<a
|
||||
href={resolve('/(root)/settings/libraries/[libraryId]/duplicates', {
|
||||
libraryId: String(library.id)
|
||||
})}
|
||||
class="ml-3 truncate rounded-md border-l px-3 py-1.5 text-sm transition-colors hover:bg-muted {isActiveLibrary(
|
||||
library.id
|
||||
)
|
||||
? 'bg-muted font-medium'
|
||||
: 'text-muted-foreground'}"
|
||||
>
|
||||
{library.name}
|
||||
</a>
|
||||
{/each}
|
||||
{/if}
|
||||
{/each}
|
||||
</nav>
|
||||
<div class="flex-1 overflow-auto">{@render children()}</div>
|
||||
|
||||
@@ -0,0 +1,48 @@
|
||||
<script lang="ts">
|
||||
import { page } from '$app/state';
|
||||
import { resolve } from '$app/paths';
|
||||
import { getLibraryState } from '$lib/state/library.svelte';
|
||||
|
||||
let { children } = $props();
|
||||
|
||||
// Set in (root)/+layout.svelte, above the settings group, so the libraries are
|
||||
// already here — this pane needs no load function of its own.
|
||||
const libraryState = getLibraryState();
|
||||
|
||||
const library = $derived(
|
||||
libraryState.libraries.find((lib) => String(lib.id) === page.params.libraryId)
|
||||
);
|
||||
|
||||
// Duplicates is the only section today. The strip exists so General and a
|
||||
// danger zone have an obvious place to land; neither is built yet, and an
|
||||
// empty tab is worse than no tab.
|
||||
//
|
||||
// Active state is matched on route id, not pathname, for the reason spelled
|
||||
// out in settings/+layout.svelte.
|
||||
const sections = [
|
||||
{ title: 'Duplicates', routeId: '/(root)/settings/libraries/[libraryId]/duplicates' }
|
||||
] as const;
|
||||
</script>
|
||||
|
||||
<div class="flex flex-col gap-4">
|
||||
<div>
|
||||
<h2 class="text-lg font-semibold">{library?.name ?? 'Library'}</h2>
|
||||
<p class="text-sm text-muted-foreground">Settings for this library</p>
|
||||
</div>
|
||||
|
||||
<nav class="flex gap-1 border-b">
|
||||
{#each sections as section (section.routeId)}
|
||||
<a
|
||||
href={resolve(section.routeId, { libraryId: page.params.libraryId! })}
|
||||
class="-mb-px border-b-2 px-3 py-2 text-sm font-medium transition-colors hover:text-foreground {page
|
||||
.route.id === section.routeId
|
||||
? 'border-foreground'
|
||||
: 'border-transparent text-muted-foreground'}"
|
||||
>
|
||||
{section.title}
|
||||
</a>
|
||||
{/each}
|
||||
</nav>
|
||||
|
||||
{@render children()}
|
||||
</div>
|
||||
@@ -0,0 +1,10 @@
|
||||
import { resolve } from '$app/paths';
|
||||
import { redirect } from '@sveltejs/kit';
|
||||
|
||||
export async function load({ params }) {
|
||||
// The library pane is its sections; land on the first one.
|
||||
redirect(
|
||||
303,
|
||||
resolve('/(root)/settings/libraries/[libraryId]/duplicates', { libraryId: params.libraryId })
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
import { listBooks, listDuplicateBooks } from '$lib/api/book.remote.js';
|
||||
|
||||
export async function load({ params, depends }) {
|
||||
// Dismissing or merging a group re-runs this, so the card leaves the screen.
|
||||
depends('app:duplicate-books');
|
||||
|
||||
const groups = await listDuplicateBooks(params.libraryId);
|
||||
|
||||
// The groups carry only enough to render a card. Merging needs the whole record —
|
||||
// identifiers, description, publisher — so fetch them in one go rather than per
|
||||
// card, and let the dialog pick out the books for its own group.
|
||||
const ids = [...new Set(groups.flatMap((group) => group.books.map((book) => book.book_id)))];
|
||||
const books = ids.length
|
||||
? await listBooks({ ids, pageSize: ids.length })
|
||||
: { items: [] };
|
||||
|
||||
return { groups, books: books.items };
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
<script lang="ts">
|
||||
import { page } from '$app/state';
|
||||
import { resolve } from '$app/paths';
|
||||
import { invalidate } from '$app/navigation';
|
||||
import { toast } from 'svelte-sonner';
|
||||
import { CopyCheck, Fingerprint, GitMerge, SlidersHorizontal, Type } from '@lucide/svelte';
|
||||
|
||||
import * as Card from '$lib/components/ui/card/index.js';
|
||||
import * as Empty from '$lib/components/ui/empty/index.js';
|
||||
import * as DropdownMenu from '$lib/components/ui/dropdown-menu/index.js';
|
||||
import { Badge } from '$lib/components/ui/badge/index.js';
|
||||
import { Button, buttonVariants } from '$lib/components/ui/button/index.js';
|
||||
import BookImage from '$lib/components/view/book-image.svelte';
|
||||
import GeneratedCover from '$lib/components/view/generated-cover.svelte';
|
||||
import { dismissDuplicateBooks } from '$lib/api/book.remote';
|
||||
import MergeBooks from '$lib/components/forms/merge-books/merge-books.svelte';
|
||||
import { mergeInto } from '$lib/components/forms/merge-books/merge';
|
||||
import type { Book, DuplicateBookGroup } from '$lib/schema';
|
||||
|
||||
let { data }: { data: { groups: DuplicateBookGroup[]; books: Book[] } } = $props();
|
||||
|
||||
// The group open in the workbench, or null when it is closed.
|
||||
let merging = $state<DuplicateBookGroup | null>(null);
|
||||
|
||||
/** The full records behind a group, in the order the group lists them. */
|
||||
function recordsFor(group: DuplicateBookGroup): Book[] {
|
||||
return group.books
|
||||
.map(({ book_id }) => data.books.find((book) => book.id === book_id))
|
||||
.filter((book): book is Book => book !== undefined);
|
||||
}
|
||||
|
||||
// Which groups have a request in flight — a dismissal or a quick merge — so a
|
||||
// slow round trip cannot be started twice.
|
||||
let dismissing = $state<number[]>([]);
|
||||
let quickMerging = $state<number[]>([]);
|
||||
|
||||
/**
|
||||
* Merge without opening the workbench, keeping the first book as it stands.
|
||||
*
|
||||
* The common case by far: the group is the same book twice, one record is as
|
||||
* good as the other, and the point is to end up with one. Resolving metadata
|
||||
* field by field is the other button.
|
||||
*/
|
||||
async function quickMerge(group: DuplicateBookGroup) {
|
||||
const key = keyOf(group);
|
||||
if (quickMerging.includes(key)) return;
|
||||
|
||||
const [survivor, ...folded] = recordsFor(group);
|
||||
if (!survivor || folded.length === 0) return;
|
||||
|
||||
quickMerging = [...quickMerging, key];
|
||||
|
||||
try {
|
||||
await mergeInto(page.params.libraryId!, survivor, folded);
|
||||
} finally {
|
||||
quickMerging = quickMerging.filter((id) => id !== key);
|
||||
}
|
||||
}
|
||||
|
||||
/** A group is named by its lowest book id, which the backend orders it by. */
|
||||
function keyOf(group: DuplicateBookGroup) {
|
||||
return group.books[0].book_id;
|
||||
}
|
||||
|
||||
function reasonLabel(reason: string) {
|
||||
if (reason === 'identifier') return 'Same identifier';
|
||||
if (reason === 'title-author') return 'Same title and author';
|
||||
return reason;
|
||||
}
|
||||
|
||||
/**
|
||||
* Dismiss every pairing in a group at once.
|
||||
*
|
||||
* A group is held together pair by pair, so saying "these are not duplicates"
|
||||
* about three books means saying it about all three pairs — dismissing only the
|
||||
* first would leave the rest of the group standing and the screen unchanged.
|
||||
*/
|
||||
async function notDuplicates(group: DuplicateBookGroup) {
|
||||
const key = keyOf(group);
|
||||
if (dismissing.includes(key)) return;
|
||||
|
||||
dismissing = [...dismissing, key];
|
||||
|
||||
const ids = group.books.map((book) => book.book_id);
|
||||
const pairs = ids.flatMap((a, index) =>
|
||||
ids.slice(index + 1).map((b) => ({ book_a_id: a, book_b_id: b }))
|
||||
);
|
||||
|
||||
try {
|
||||
await Promise.all(pairs.map((pair) => dismissDuplicateBooks(pair)));
|
||||
await invalidate('app:duplicate-books');
|
||||
toast.success('Marked as different books');
|
||||
} catch {
|
||||
toast.error('Could not mark these as different books');
|
||||
} finally {
|
||||
dismissing = dismissing.filter((id) => id !== key);
|
||||
}
|
||||
}
|
||||
</script>
|
||||
|
||||
<!-- No max width and no padding of its own: the settings pane sets the width and
|
||||
is the thing that scrolls, so a wrapper that centres inside it only makes the
|
||||
cards narrower than they need to be. -->
|
||||
<div class="flex flex-col gap-6 pb-4">
|
||||
<p class="text-sm text-muted-foreground">
|
||||
Books whose metadata matches. A second edition, a translation and a different scan of one book
|
||||
all look like this, so nothing here has been changed or removed — this is a list to read, not a
|
||||
problem to fix.
|
||||
</p>
|
||||
|
||||
{#if data.groups.length === 0}
|
||||
<Empty.Root>
|
||||
<Empty.Header>
|
||||
<Empty.Media variant="icon">
|
||||
<CopyCheck />
|
||||
</Empty.Media>
|
||||
<Empty.Title>Nothing looks duplicated</Empty.Title>
|
||||
<Empty.Description>
|
||||
No two books in this library share an identifier, or a title and an author.
|
||||
</Empty.Description>
|
||||
</Empty.Header>
|
||||
</Empty.Root>
|
||||
{:else}
|
||||
{#each data.groups as group (keyOf(group))}
|
||||
{@const busy = dismissing.includes(keyOf(group)) || quickMerging.includes(keyOf(group))}
|
||||
{@const records = recordsFor(group)}
|
||||
<Card.Root>
|
||||
<Card.Header>
|
||||
<Card.Title>{group.books.length} books look like the same book</Card.Title>
|
||||
<Card.Action class="flex gap-2">
|
||||
<Button
|
||||
variant="outline"
|
||||
size="sm"
|
||||
disabled={busy}
|
||||
onclick={() => notDuplicates(group)}
|
||||
>
|
||||
Not duplicates
|
||||
</Button>
|
||||
<!-- A choice rather than a jump straight into the workbench:
|
||||
most groups are one book twice, where there is nothing to
|
||||
resolve and the dialog is a step in the way.
|
||||
|
||||
Disabled until every record in the group came back, so
|
||||
neither action can run on a partial group. -->
|
||||
<DropdownMenu.Root>
|
||||
<DropdownMenu.Trigger
|
||||
class={buttonVariants({ size: 'sm' })}
|
||||
disabled={busy || records.length !== group.books.length}
|
||||
>
|
||||
Merge…
|
||||
</DropdownMenu.Trigger>
|
||||
<DropdownMenu.Content align="end">
|
||||
<DropdownMenu.Item onclick={() => quickMerge(group)}>
|
||||
<GitMerge class="size-4 shrink-0" />
|
||||
Merge into first selected
|
||||
</DropdownMenu.Item>
|
||||
<DropdownMenu.Item onclick={() => (merging = group)}>
|
||||
<SlidersHorizontal class="size-4 shrink-0" />
|
||||
Manual merge…
|
||||
</DropdownMenu.Item>
|
||||
</DropdownMenu.Content>
|
||||
</DropdownMenu.Root>
|
||||
</Card.Action>
|
||||
</Card.Header>
|
||||
|
||||
<Card.Content>
|
||||
<ul class="flex flex-wrap gap-4">
|
||||
{#each group.books as book (book.book_id)}
|
||||
<li class="w-36">
|
||||
<a
|
||||
href={resolve('/(root)/(library)/book/[bookId]', {
|
||||
bookId: String(book.book_id)
|
||||
})}
|
||||
class="group flex flex-col gap-2"
|
||||
>
|
||||
<span
|
||||
class="block aspect-9/12 w-full overflow-hidden rounded-sm bg-muted shadow-md transition-all group-hover:brightness-75"
|
||||
>
|
||||
{#if book.cover_image}
|
||||
<BookImage src="/api/{book.cover_image}" class="h-full w-full object-cover" />
|
||||
{:else}
|
||||
<!-- A blank swatch here is worse than anywhere else: this screen
|
||||
is a side-by-side comparison, and two of them are impossible
|
||||
to tell apart. The candidate carries author names rather than
|
||||
records, which is all the cover draws. -->
|
||||
<GeneratedCover
|
||||
book={{
|
||||
title: book.title,
|
||||
authors: book.authors.map((name) => ({ name }))
|
||||
}}
|
||||
/>
|
||||
{/if}
|
||||
</span>
|
||||
|
||||
<span class="line-clamp-2 font-serif text-sm group-hover:underline">
|
||||
{book.title}
|
||||
</span>
|
||||
</a>
|
||||
|
||||
<p class="line-clamp-2 text-xs text-muted-foreground">
|
||||
{book.authors.join(', ')}
|
||||
</p>
|
||||
|
||||
<!-- Why this book is in the group, so the reader can judge the
|
||||
evidence rather than take the grouping on trust. -->
|
||||
<p class="mt-1 flex flex-wrap gap-1">
|
||||
{#each book.matched_on as reason (reason)}
|
||||
<Badge variant="secondary" class="gap-1 text-[10px] font-normal">
|
||||
{#if reason === 'identifier'}
|
||||
<Fingerprint class="size-3" />
|
||||
{:else}
|
||||
<Type class="size-3" />
|
||||
{/if}
|
||||
{reasonLabel(reason)}
|
||||
</Badge>
|
||||
{/each}
|
||||
</p>
|
||||
</li>
|
||||
{/each}
|
||||
</ul>
|
||||
</Card.Content>
|
||||
</Card.Root>
|
||||
{/each}
|
||||
{/if}
|
||||
</div>
|
||||
|
||||
<!--
|
||||
Keyed on the group so a second merge starts from that group's records rather
|
||||
than the previous one's draft, the same way the edit dialog keys on the book.
|
||||
-->
|
||||
{#if merging}
|
||||
{#key merging.books[0].book_id}
|
||||
<MergeBooks
|
||||
books={recordsFor(merging)}
|
||||
libraryId={page.params.libraryId!}
|
||||
bind:open={
|
||||
() => merging !== null,
|
||||
(value) => {
|
||||
// Bound, not passed as a bare `true`: the dialog closes itself on an
|
||||
// outside click or Escape, and if that never reaches `merging` the
|
||||
// group stays set — the menu item then re-assigns the same group,
|
||||
// nothing changes, and the workbench never reopens.
|
||||
if (!value) merging = null;
|
||||
}
|
||||
}
|
||||
onmerged={() => (merging = null)}
|
||||
/>
|
||||
{/key}
|
||||
{/if}
|
||||
Reference in New Issue
Block a user