Every table in one vector index

Semantic search over a document store is a solved shape. You have one table, it has a text column, you embed the column. Every tutorial written since 2023 covers it.

Semantic search over a business platform is a different problem, and the difference is not size. The thing a user wants to find is spread across eight tables that have nothing textual in common: a client is a company name and a Swiss IDE number, an employee is a payroll ID and an AVS number, an invoice is an amount and a status, a document is a filename. There is no content column to embed. There is no single table to index. And the sentence the user types — the Geneva construction mandate we invoiced in March — crosses four of them.

This is the semantic layer of AltiusOne, the fiduciary and accounting platform we run for Swiss mandates. The design is one sentence with a hard constraint attached:

Every write anywhere in the application ends up in one vector index — and none of that work happens inside the request that caused it.

The two obvious designs, and why neither survives

Search each table separately. Add an embedding column to Client, another to Facture, another to Employe, and query all eight at read time. This works until you try to rank the results: eight separate similarity scores over eight differently-shaped texts are not comparable, and the merge step becomes a heuristic you will be tuning forever. It also means eight HNSW indexes, eight migrations, and eight places to remember when someone adds a field.

Denormalise by hand into a search table. Write a SearchDocument model, and a save() hook on each business model that renders it into text. This is the right shape and the wrong implementation: the rendering logic ends up spread across eight apps, each written by whoever touched that app last, and it drifts.

The third option is the same shape with the logic pulled into one place: a projection declared as data.

The projection is a dictionary

# Modèle Django → type d'entité dans le graphe
MODEL_GRAPH_CONFIG = {
    'core.Client': {
        'type_nom': 'Entreprise',
        'nom_field': 'raison_sociale',
        'attributs_fields': ['ide_number', 'forme_juridique', 'canton_rc', 'statut'],
        'source': 'systeme',
    },
    'salaires.Employe': {
        'type_nom': 'Personne',
        'nom_field': '__str__',
        'attributs_fields': ['matricule', 'avs_number', 'statut'],
        'source': 'systeme',
    },
    'facturation.Facture': {
        'type_nom': 'Facture',
        'nom_field': 'numero_facture',
        'attributs_fields': ['montant_ttc', 'statut', 'date_emission'],
        'source': 'systeme',
    },
    ...
}

Eight entries. Each says: which model, what to call the thing, and which fields carry meaning. Every mapped instance becomes one row in a single Entite table, attached back to its origin by a generic foreign key:

content_type = models.ForeignKey(
    ContentType, on_delete=models.SET_NULL,
    null=True, blank=True, related_name='+',
)
object_id = models.UUIDField(null=True, blank=True)
content_object = GenericForeignKey('content_type', 'object_id')

One table, one embedding column, one index, one ranking. The generic FK is what makes a search result clickable: a hit is not a row of text, it is the invoice, and the view can follow it.

Foreign keys project too, as a second dictionary:

RELATION_MAPPINGS = {
    'core.Mandat': {
        'client': 'Client de',
        'responsable': 'Responsable de',
    },
    'facturation.Facture': {
        'mandat': 'Facturé à',
        'client': 'Client de',
    },
    ...
}

Facture.mandat_id becomes an edge labelled Facturé à. That turns the flat index into a graph, which is what makes the mandate we invoiced in March answerable: the embedding finds the invoice, the edges reach the mandate.

Wiring is a loop, not eight post_save decorators:

def register_sync_signals():
    """Enregistre les signals post_save/post_delete pour tous les modèles mappés."""
    for model_key in MODEL_GRAPH_CONFIG:
        app_label, model_name = model_key.split('.')
        try:
            model = django_apps.get_model(app_label, model_name)
            post_save.connect(graph_post_save, sender=model,
                              dispatch_uid=f'graph_sync_save_{model_key}')
            post_delete.connect(graph_post_delete, sender=model,
                                dispatch_uid=f'graph_sync_delete_{model_key}')
        except LookupError:
            logger.warning(f"Modèle introuvable pour sync graphe: {model_key}")

The dispatch_uid matters more than it looks. AppConfig.ready() can run more than once — a management command that reloads, a test runner, a runserver autoreload — and without a dispatch UID you get two handlers, two syncs, and two Celery tasks per save. Django deduplicates by that string.

The LookupError branch is the other half. MODEL_GRAPH_CONFIG names models by string, so an app that gets renamed or removed leaves a dangling key. Logging a warning and continuing is the right call: a search projection is not worth refusing to boot over.

Resolving a field that is not a field

Three of the eight entries name __str__ as their nom_field, one names get_full_name, and one attribute path crosses a relation (role__nom). One resolver handles all of it:

def _resolve_field(instance, field_path):
    """
    Résout la valeur d'un champ, y compris les traversées FK (role__nom)
    et les callables (get_full_name, __str__).
    """
    if field_path == '__str__':
        return str(instance)

    parts = field_path.split('__')
    obj = instance
    for part in parts:
        if obj is None:
            return None
        attr = getattr(obj, part, None)
        if attr is None:
            return None
        if callable(attr) and not isinstance(attr, type):
            obj = attr()
        else:
            obj = attr
    return obj

callable(attr) and not isinstance(attr, type) is the line worth keeping. Bound methods are callable and should be called; so are model classes, which turn up whenever a path lands on a related manager or a class attribute, and calling one of those instantiates a model with no arguments — which either raises or, worse, succeeds and gives you an empty object. The isinstance(attr, type) guard is two words that prevent a category of nonsense.

The rest of the projection is deliberately dull:

def _build_attributs(instance, fields):
    attributs = {}
    for field_path in fields:
        val = _resolve_field(instance, field_path)
        if val is not None:
            # Clé = dernier segment du path (role__nom → nom)
            key = field_path.split('__')[-1]
            attributs[key] = str(val) if not isinstance(val, (str, int, float, bool)) else val
    return attributs

Decimal, date, UUID and enum members all become strings, because the target is a JSONField and because everything downstream is going to be concatenated into a sentence anyway.

The text you embed matters more than the model you embed it with

This is the part that tutorials skip, and the part that determines whether the search is any good:

def texte_pour_embedding(self):
    """Concatène les champs pertinents pour générer un embedding."""
    parts = []
    if self.type_id:
        parts.append(self.type.nom)
    parts.append(self.nom)
    if self.description:
        parts.append(self.description)
    if self.attributs:
        for key, val in self.attributs.items():
            if val:
                parts.append(f"{key}: {val}")
    if self.tags:
        parts.append(" ".join(str(t) for t in self.tags))
    return " | ".join(parts)

An invoice does not become "FAC-2026-0412". It becomes:

Facture | FAC-2026-0412 | montant_ttc: 12450.00 | statut: payee | date_emission: 2026-03-14

Three decisions are in there.

The type name goes first. Facture, Personne, Entreprise — a word that appears in the user’s query when they are thinking about a category rather than a name. Sentence-transformer models weight early tokens meaningfully, and this costs one token to make invoices from March match invoices.

Attributes are written as key: value, not bare values. 12450.00 embeds as a number with no meaning; montant_ttc: 12450.00 embeds as an amount. The key is a domain word in the same language as the query.

Empty values are dropped, not rendered as empty. statut: contributes noise in the same position where a real status would contribute signal.

The separator is | rather than a newline or a comma, for the plain reason that it does not occur in the data and so never merges two fields into one apparent phrase.

Nothing is computed in the request

The constraint that shaped everything else. The local embedding service:

class LocalEmbeddingService:
    """
    Service d'embeddings local utilisant sentence-transformers.

    Le modèle est chargé lazily au premier appel et gardé en mémoire.
    """

"""
Modèle: paraphrase-multilingual-mpnet-base-v2
- 768 dimensions (compatible pgvector existant)
- Multilingue: FR, DE, EN, IT (parfait pour la Suisse)
- ~400MB RAM, ~50ms/embedding sur CPU
- Chargé lazily au premier appel (pas d'impact au démarrage)
"""

Four hundred megabytes and fifty milliseconds. Neither is acceptable inside a request — the memory because it would be paid by every web worker, the latency because saving an invoice would visibly get slower for a feature the person saving it is not using.

Lazy loading is what keeps the memory out of the web processes. It is not a startup optimisation; it is a placement decision. The web workers never call generate_embedding, so they never touch the property, so they never pay the 400 MB. The Celery worker calls it on its first task and holds it from then on.

The signal does one thing:

@receiver(post_save, sender='graph.Entite')
def entite_post_save(sender, instance, created, **kwargs):
    """Lance la génération d'embedding après sauvegarde d'une entité."""
    update_fields = kwargs.get('update_fields')
    if update_fields and set(update_fields) <= {'embedding', 'embedding_updated_at'}:
        return

    from graph.tasks import generer_embedding_entite_task

    try:
        generer_embedding_entite_task.delay(str(instance.pk))
    except Exception as e:
        logger.warning(f"Impossible de lancer le task embedding pour {instance.pk}: {e}")

The try around .delay() is not defensive noise. .delay() talks to the broker, and the broker can be down. Without the guard, a Redis outage would turn every Facture.save() in the application into a 500 — the search index taking the whole platform down with it. A warning in the log and a stale embedding is the correct failure mode for this feature, and stating that in code is cheaper than discovering it at 3 a.m.

Two loops, closed two different ways

Signal-driven indexing has a characteristic failure: the index write triggers the indexing.

The embedding loop. mettre_a_jour_embedding finishes by saving the entity:

entite.embedding = embedding
entite.embedding_updated_at = timezone.now()
entite.save(update_fields=['embedding', 'embedding_updated_at'])

That save() fires post_save on Entite, which queues an embedding task, which saves the entity, which fires post_save. An unbounded loop of Celery tasks, each one costing fifty milliseconds of CPU on a worker, growing until someone notices the queue.

The guard is the first three lines of the receiver, and it works because update_fields is passed explicitly at the one place that writes only those two columns. The set comparison is <=, not ==, so a future write of just embedding alone is still caught.

The sync loop. sync_instance calls Entite.objects.update_or_create(...), which saves an Entite. Nothing in that path re-enters graph_post_save, because Entite is not in MODEL_GRAPH_CONFIG — but the business models themselves can re-enter, when a save() inside the sync path touches a mapped model. That one is closed with an explicit re-entrancy guard:

# Flag pour éviter la récursion (sync_instance sauvegarde Entite → signal Entite)
_sync_in_progress = set()


def graph_post_save(sender, instance, **kwargs):
    key = (sender, instance.pk)
    if key in _sync_in_progress:
        return
    _sync_in_progress.add(key)
    try:
        entite = sync_instance(instance)
        if entite:
            sync_relations(instance)
    except Exception as e:
        logger.warning(f"Erreur sync graphe pour {sender.__name__} #{instance.pk}: {e}")
    finally:
        _sync_in_progress.discard(key)

Keyed by (sender, pk) rather than a single boolean, so that syncing a Facture does not suppress the sync of the Mandat it touches. And the finally is load-bearing: an exception that left the key in the set would silently stop indexing that one record for the life of the process — the worst kind of bug, because it is invisible and it is per-row.

The broad except around the whole body is the same judgement as the broker guard. A projection failure must not roll back the business write that caused it. Saving an invoice is the user’s work; indexing it is ours.

The index

indexes = [
    models.Index(fields=['type', 'nom'], name='graph_entite_type_nom_idx'),
    models.Index(fields=['source', 'confiance'], name='graph_entite_src_conf_idx'),
    models.Index(fields=['content_type', 'object_id'], name='graph_entite_gfk_idx'),
    GistIndex(fields=['geom'], name='graph_entite_geom_gist_idx'),
    HnswIndex(
        fields=['embedding'],
        name='graph_entite_emb_hnsw_idx',
        m=16, ef_construction=64,
        opclasses=['vector_cosine_ops'],
    ),
]

Five indexes on one table, and the two that are easiest to forget are the third and the fifth.

graph_entite_gfk_idx on (content_type, object_id) is what makes the sync path cheap. Every post_save does a lookup by that pair — through update_or_create, then again in sync_relations for the source, then once per mapped foreign key for each target. That is four or five point lookups per save on a table that grows with the whole platform. Without the index they are sequential scans, and the sync path degrades from invisible to the reason saving is slow somewhere around a hundred thousand rows.

For the HNSW index, m=16, ef_construction=64 are pgvector’s defaults and the right starting point: m is the number of connections per node in the graph, ef_construction the size of the candidate list while building. Raising them buys recall and costs build time and index size, and neither is worth tuning before you have a recall complaint from a real query. vector_cosine_ops pairs with normalize_embeddings=True in the encoder — with normalised vectors, cosine distance and inner product rank identically, and choosing cosine keeps the stored numbers interpretable when you go looking at them by hand.

The same shape recurs for documents, on a second table indexed by chunk:

embedding = VectorField(dimensions=768, null=True, blank=True)
...
db_table = 'text_chunk_embeddings'
managed = False  # Table créée via migration SQL brut (0004)
unique_together = [['document', 'chunk_index']]

managed = False because the table is created by a raw SQL migration. It is worth being explicit about why that is acceptable here and not generally: the table holds a derived index, not business data. It can be dropped and rebuilt from the source rows at any time, so the usual argument for keeping Django in charge of the schema — that a migration mistake loses data — does not apply.

Rebuilding everything

Adding a field to attributs_fields changes the embedded text for every row of that model. So there is a task that walks the whole table:

@shared_task(bind=True, max_retries=1, default_retry_delay=300)
def reindexer_tous_embeddings_task(self, batch_size=100):
    entites = Entite.objects.filter(is_active=True).values_list('pk', flat=True)
    total = entites.count()
    updated = 0
    errors = 0

    for pk in entites.iterator(chunk_size=batch_size):
        try:
            if mettre_a_jour_embedding(pk):
                updated += 1
        except Exception as e:
            logger.error(f"Erreur reindex entité {pk}: {e}")
            errors += 1

    return {'status': 'ok', 'total': total, 'updated': updated, 'errors': errors}

Three deliberate choices in ten lines.

values_list('pk', flat=True) plus .iterator(). Not .all(). A full queryset of entities would materialise every row — including the 768-float embedding column already stored on each one — into the worker’s memory. Primary keys only, streamed by server-side cursor in chunks of a hundred, and each row re-fetched inside mettre_a_jour_embedding when it is actually needed. Memory stays flat regardless of table size.

max_retries=1. A reindex that fails halfway and retries from the start would redo the work it already did, and a task that costs an hour of CPU should not be silently restarted three times. Per-row errors are counted and swallowed; the task itself gets one more chance and then stops with a number in the return value.

Errors are counted, not raised. One entity whose text triggers a tokeniser edge case must not stop the other forty thousand. {'updated': 39_998, 'errors': 2} is an actionable result; a traceback with 39 998 rows unprocessed is not.

The one thing this loop deliberately does not do is fan out into forty thousand Celery tasks. Sequential in one worker is slower in wall-clock and much better behaved: the model is loaded once, the broker is not asked to hold a queue the size of the database, and a reindex cannot starve the interactive tasks that share the same workers.

What this does not solve

Three limits, stated because they are the ones a reader will hit.

The window between the write and the embedding. A record saved now is searchable by exact match immediately and by meaning a second or two later, when the worker gets to it. For a fiduciary platform that is nothing. For anything where a user searches for what they just typed, it is a visible bug, and the answer is not a faster worker — it is embedding the query and the record in the request, which returns you to the 400 MB problem.

Deletion is deactivation. delete_instance sets is_active = False on the entity and its edges rather than removing rows. The vector stays in the index and every query has to filter it out. That is the right trade for an audit-bound platform, where nothing is truly deleted anyway — but on a system with real churn, an HNSW index full of tombstones degrades, and the periodic rebuild is not optional.

The projection is only as good as its dictionary. A field nobody added to attributs_fields is a field the search cannot see, and nothing anywhere fails when that happens. The failure mode of a declarative projection is silence. The mitigation we use is that the dictionary is one file, short, and read whenever a model gains a field — which is a process, not a guarantee, and I would rather say so than pretend otherwise.

What carries over

• Project into one table. Comparable scores need a common shape. The generic FK back to the origin is what keeps the results useful.
• Declare the mapping as data. One file that a reviewer can read in a minute beats eight save() overrides that nobody reads together.
• Decide where the model is loaded, not just when. Lazy loading is a placement decision: it keeps 400 MB out of every web worker.
• Write the loop guards down, with the reason. update_fields sets and (sender, pk) re-entrancy keys look like paranoia until the day the queue grows without bound.
• Let indexing fail quietly. The broker being down must not turn a business write into a 500.
• Stream primary keys, not rows, whenever a task walks a whole table that carries a wide column.