Form fields that open a device — the orphan file problem

The form builder in PaperPoint offers twenty-nine field types. Twenty-three of them behave like form fields, however exotic they look — short text, number, date, single choice, a Swiss address, a country, GPS coordinates, a map polygon, a matrix of typed columns, a reference to a row in another form.

Six are not fields at all, and the difference is not that they are harder to render.

# Spécialisés
("audio_recording", _("Audio recording")),
("video_recording", _("Video recording")),
("signature", _("Signature")),

("single_video_camera", _("Camera - Single video")),
("single_image_camera", _("Camera - Single photo")),
("multi_image_camera", _("Camera - Multiple photos")),

An HTML form field has a contract that nobody writes down because it has never needed writing down:

The value exists at submit time, it travels in the POST that creates the record, and it belongs to that record from the first instant it exists.

Every one of those six breaks it. A forty-megabyte site video does not travel in a form POST. A person recording two minutes of audio does it before filling in the rest of the page, not after. A multi-photo camera produces its files one at a time, over several minutes, from a phone that may lose signal between the third and the fourth.

So the file has to be uploaded early — and the moment it is, it exists before the thing that owns it. That is the whole problem, and everything below is a consequence.

The orphan

The file model carries a nullable link back to the submission, and the help text is the design:

submission = models.ForeignKey(
    FormSubmission, on_delete=models.CASCADE, null=True, blank=True,
    related_name="files",
    help_text=_("Form submission this file is attached to. "
                "Null while the file is awaiting attachment to a submission."),
)
field = models.ForeignKey(
    FormField, on_delete=models.SET_NULL, null=True, blank=True,
    related_name="submitted_files",
)
uploaded_by = models.ForeignKey(
    "users.UserProfile", on_delete=models.SET_NULL, null=True, blank=True,
    related_name="uploaded_form_files",
    help_text=_("CustomUser who uploaded the file "
                "(used for orphan-attachment safety checks)."),
)

Both submission and field are nullable, which is worth pausing on: at upload time the server knows neither which submission the file will belong to nor which field it answers. The client knows, but the client has not sent the form yet, and taking its word for the field would mean trusting a value that has not been validated against the template.

The upload endpoint is therefore almost empty of business logic:

class FileUploadAPIView(APIView):
    """POST /api/v1/files/upload/ — upload a single file as an orphan FormFile.

    The file is stored without a `submission` link. When the user submits a form,
    `FormSubmissionViewSet.perform_create` walks the submission `data` JSON and
    attaches any orphan files referenced by id whose `uploaded_by` matches the
    submitter — preventing cross-user hijacking.
    """

    MAX_FILE_SIZE = 25 * 1024 * 1024  # 25 MB hard cap; per-type caps live on FormFile

    def post(self, request):
        f = request.FILES.get('file')
        ...
        instance = FormFile.objects.create(
            submission=None,
            field=None,
            file=f,
            original_filename=f.name or 'file',
            file_type=(f.content_type or 'application/octet-stream'),
            file_size=f.size,
            uploaded_by=profile,
            is_signature=is_signature,
        )

It returns an id. The client holds that id and, later, writes it into the submission JSON as the value of the field. The form POST carries a number where a file would have been.

The hijack this creates, and the fence

Here is the failure mode that the design invites, and it is not subtle. Files now exist that belong to nobody, addressed by a small integer. If attachment worked by id alone, then submitting

{"full_name": "Ada", "cv": 4127}

would attach file 4127 to my submission whether or not I uploaded it. On a platform where forms collect identity documents and site photographs, that is a read primitive over other people’s uploads: guess ids, attach, read back your own submission.

The fence is one query with one extra filter:

def attach_orphan_files(submission, profile):
    """Attach orphan FormFiles referenced by id in `submission.data` to the submission.

    Authenticated path: only files where `uploaded_by == profile` are attached.
    Anonymous path (profile is None): only files with `uploaded_by IS NULL` are attached.
    """
    ids = _extract_file_ids_from_data(submission.data)
    if not ids:
        return
    qs = FormFile.objects.filter(id__in=ids, submission__isnull=True)
    qs = qs.filter(uploaded_by__isnull=True) if profile is None else qs.filter(uploaded_by=profile)
    qs.update(submission=submission)

Three properties in six lines, and each one is doing work.

submission__isnull=True. An already-attached file cannot be re-attached. Without it, a file could be moved from one submission to another, which is worse than the read: it removes evidence from a record that has already been counted.

The uploaded_by branch. For a signed-in user, the file must be theirs. For the anonymous public path — the embed SDK, the QR-code intake — the file must have no owner at all. Note that anonymous is not the permissive case here: an anonymous submitter cannot claim an authenticated user’s orphan, because uploaded_by__isnull=True excludes every file that has one.

update(), not a loop of saves. One statement, no signals, no per-row round trip, and — because it is a single UPDATE ... WHERE, running inside the submission’s transaction — no window in which half the files are attached.

What it deliberately does not do is fail loudly. An id in the JSON that matches nothing simply attaches nothing. This is the right call for a mobile client that may retry a submission after a partial upload: the submission is recorded, the missing file is visible as a gap, and a rejected submission would have lost the twenty answers that did arrive.

Walking the JSON

The ids have to be found in a free-form dictionary, because the field types that produce files include one that produces several:

def _extract_file_ids_from_data(data):
    """Walk a submission `data` dict and collect referenced FormFile ids."""
    ids = set()
    for value in (data or {}).values():
        if isinstance(value, int):
            ids.add(value)
        elif isinstance(value, list):
            for v in value:
                if isinstance(v, int):
                    ids.add(v)
                elif isinstance(v, str) and v.isdigit():
                    ids.add(int(v))
        elif isinstance(value, str) and value.isdigit():
            ids.add(int(value))
    return ids

Two things about this deserve defending, because both look sloppy.

It accepts "4127" as well as 4127. JSON has integers, so in principle the client should send one. In practice the multi-photo widget accumulates ids in component state, a React Native TextInput somewhere in the chain has turned one into a string, and a submission arriving over an offline queue may have been serialised and reparsed twice. Being strict here does not produce a better client; it produces a photo that silently fails to attach, which the user discovers when the report comes back short.

It does not consult the form template. It could: it could look up which fields are of a file-producing type and read only those keys. It does not, because the ids are validated by ownership anyway. A number in a text field matches nothing that belongs to the submitter, so it attaches nothing. Adding a template lookup would add a query and a failure mode — a template edited between upload and submission — to re-derive a constraint the ownership filter already enforces.

The cost is real and worth naming: any integer answer that happens to equal the id of one of your own unattached files will attach it. The bound is your own files, so the worst case is that a stray upload of yours is filed against your own submission. That is a mislabelling, not a leak, and it is the trade the comment above should have said out loud.

The one that goes the other way

Signature is the exception, and the exception is instructive. A signature is a canvas, not a device: it produces a small PNG, immediately, in the same interaction as the rest of the form. So it rides along in the POST as a data URI and becomes a file server-side:

@classmethod
def create_from_signature_data(cls, signature_data, field, submission):
    if not signature_data or not signature_data.startswith("data:image"):
        return None

    try:
        format_info, imgstr = signature_data.split(";base64,")
        ext = format_info.split("/")[-1]

        filename = f"signature_{field.name}_{submission.pk}.{ext}"
        file_content = ContentFile(base64.b64decode(imgstr), name=filename)

        form_file = cls(
            submission=submission,
            field=field,
            file=file_content,
            original_filename=filename,
            file_type=f"image/{ext}",
            is_signature=True,
        )

        file_content.seek(0, io.SEEK_END)
        form_file.file_size = file_content.tell()
        file_content.seek(0)

        form_file.save()
        return form_file

    except Exception:
        logger.exception("Error creating signature file")
        return None

submission and field are both non-null on the way in. There is no orphan stage, no ownership question, and no id to guess — because the file is created by the submission rather than adopted by it. Where a device field costs an endpoint, a nullable FK and a security filter, the signature costs a split().

That is the actual lesson of the six: the cost is not the device, it is the timing. Anything small enough and fast enough to travel with the form should travel with the form.

The except block has a small history. It used to end in print(). In development that is visible; in production it goes to stdout and is lost, so a signature that failed to decode detached itself from the record with no trace anywhere — no exception, no log line, and a None return that the caller treated as “no signature given”. A contract signed on a tablet, absent from the submission, and nothing to look at. logger.exception costs one line and turns a silent data loss into a stack trace with a request behind it.

Note also that the two seek calls exist because ContentFile.size is not what you want here: the size is measured from the decoded bytes after they are in the file object, and the cursor is put back so the subsequent save() writes from the start. Getting that wrong writes a zero-byte signature, which is the kind of bug that passes every test that only checks a row exists.

What a file is allowed to be

Uploading early also means validating early, and in two places at once. At the API boundary, one flat cap:

MAX_FILE_SIZE = 25 * 1024 * 1024  # 25 MB hard cap; per-type caps live on FormFile

And on the model, caps by kind:

MAX_FILE_SIZE = 10 * 1024 * 1024   # 10 MB par défaut
MAX_IMAGE_SIZE = 5 * 1024 * 1024   # 5 MB pour les images
MAX_AUDIO_SIZE = 20 * 1024 * 1024  # 20 MB pour l'audio
MAX_VIDEO_SIZE = 50 * 1024 * 1024  # 50 MB pour la vidéo

The two sets do not agree, and the disagreement is the point: the API cap is a resource limit — it stops a request from consuming twenty-five megabytes of worker memory and disk before anything has decided the file is wanted — while the model caps are a policy — an image over five megabytes is a photo nobody resized, and a video may legitimately reach fifty. The API rejects on cost, the model rejects on meaning. Merging them into one number would mean either rejecting a legal video or accepting a costly request.

The stored name is not the submitted name:

def secure_file_upload_path(instance, filename):
    """
    Format: form_submissions/{year}/{month}/{day}/{uuid}_{sanitized_filename}
    """
    safe_filename = re.sub(r'[^\w\-\.]', '_', filename)
    safe_filename = safe_filename[:100]
    unique_id = uuid_lib.uuid4().hex[:16]

    now = datetime.now()
    return f"form_submissions/{now.year}/{now.month:02d}/{now.day:02d}/{unique_id}_{safe_filename}"

The regex is a whitelist, which is the only kind that works: .., /, and every clever encoding of them collapse to _. The UUID prefix means two people photographing IMG_0001.jpg on the same day do not collide, and the date segments keep directory listings survivable. The original name is not thrown away — it is kept in original_filename, where it is data rather than a path.

Extensions are checked against a shared list, and the public tests assert the refusal is legible rather than a 500:

def test_upload_disallowed_extension_returns_400_json(self):
    bad = SimpleUploadedFile("notes.txt", b"hello", content_type="text/plain")
    resp = self.client.post(self._upload_url(), {
        "file": bad,
        "public_token": str(self.template.public_token),
    })
    self.assertEqual(resp.status_code, 400)
    self.assertIn("extension", resp.json()["detail"].lower())

The last structural decision is the smallest and it is what keeps the six from spreading. Every field type declares which widgets it may use, and the ordinary ones have several:

"date": ["date", "text"],
"choice": ["radio", "select"],
"multiple_choice": ["checkbox", "multiselect"],
"country": ["country", "select"],

The device fields have exactly one:

# Spécialisés - un seul widget possible
"audio_recording": ["audio_recording"],
"gps_coordinates": ["gps_coordinates"],
"signature": ["signature"],

# NOUVEAUX : Camera widgets - un seul widget possible
"single_video_camera": ["single_video_camera"],
"single_image_camera": ["single_image_camera"],
"multi_image_camera": ["multi_image_camera"],

A single-element list looks like a table that wants collapsing. It is the opposite: it is a constraint written where the constraint lives. A date can fall back to a text input, and on a browser that does not support the picker it should. A camera cannot fall back to anything — there is no degraded rendering of open the camera — and the day someone adds a "single_image_camera": ["file"] fallback, they will have quietly turned a device capture into a file picker and broken every downstream assumption about provenance.

The same table also carries the default:

def save(self, *args, **kwargs):
    ...
    if not self.widget_type:
        self.widget_type = self.DEFAULT_WIDGET_MAPPING.get(self.field_type, "text")

Falling back to "text" for an unknown type is the same philosophy the rest of the form builder runs on: a misconfigured field degrades to something harmless instead of breaking the whole form.

What this costs, honestly

A device field costs, relative to a text field: one API endpoint, two nullable foreign keys, an ownership filter, a JSON walk, two tiers of size limits, and a class of bug — the orphan that is never adopted — that has no analogue in an ordinary form. Files whose submission stays null forever accumulate, and nothing in the code above collects them; that is a scheduled job this design owes and which is not written here.

What you get for it is a form that can be filled in from a phone in a basement with two bars of signal, where the video went up while the person was still typing, and where losing the connection at the moment of submit costs the submission and not the recording.

Five things worth carrying:

Ask what the field’s timing is, not what it renders. Anything that produces its value before the form is submitted has left the HTML contract, and needs an identity story of its own.
An early-uploaded file must carry its owner. uploaded_by on the row, and a filter on it at attach time — including the anonymous branch, where nobody’s is a real and enforceable owner.
Never let attachment move a file. submission__isnull=True is the difference between adopting an orphan and rewriting history.
Be permissive about the reference, strict about the right to it. Accepting "4127" costs nothing when ownership is what actually gates the attach.
Small and immediate should ride along. The signature field is the whole argument for keeping the orphan path for the cases that genuinely need it.

Six of the twenty-nine are not fields