An offline queue that loses nothing

Offline-first is usually presented as a storage problem: keep the payload, replay it later. That part takes an afternoon.

The part that takes a week is that a form is not only JSON. It carries photographs, a signature, an audio recording — and in an application designed for good connectivity those files were uploaded while the user was filling the form, so the payload contains their server ids. Take the network away and there are no ids. There is no server. There is a technician in a basement with a cracked phone, four photographs and a form that must not be lost.

This is the submission queue in the PaperPoint mobile app — field data collection where no signal is the normal case, not the edge case.

Why AsyncStorage and not SQLite

/**
 * Built on AsyncStorage (already a dependency) rather than SQLite — payloads
 * are small JSON; the only bulk is the file binaries, which stay on disk and
 * are referenced by URI.
 */

The reflex on hearing “offline queue” is SQLite: transactions, indexes, a real store. It would work and it would be more machinery than the problem has.

The measurements that decide it: a queued submission is a few kilobytes of JSON, there are rarely more than a few dozen, and access is always the whole queue — read all, filter, write all. There is no query, so there is no index to want.

And the reason the naive choice stays viable is the one on the third line: the binaries never enter the store. A photograph is a file:// URI in a JSON field; the bytes stay where the camera put them. A queue that inlined base64 images would blow past AsyncStorage’s practical limits on the second submission, and that is the design decision — not the store.

The general form: pick the boring persistence layer, and make sure the thing that would break it never goes in.

The file problem

The application uploads eagerly. You take a photograph, it uploads while you keep typing, and the numeric id it returns is written into the form values. By the time you press Send, the payload is small and the files are already on the server.

That design is right — it spreads a slow operation across the time the user is busy — and it has exactly one assumption, which is the one field work removes.

The fix is a sentinel in the value space:

export function makeOfflineUpload(base: UploadFn, collect: (f: OfflineFile) => void): UploadFn {
  return async (local, opts) => {
    const net = await NetInfo.fetch();
    if (net.isConnected) {
      try {
        return await base(local, opts);
      } catch (e) {
        if (!isNetworkError(e)) throw e; // real upload error → surface it
        // connectivity died mid-upload → defer below
      }
    }
    const placeholderId = newPlaceholderId();
    collect({
      placeholderId,
      local: {uri: local.uri, name: local.name, type: local.type},
      isSignature: opts?.isSignature,
    });
    return {id: placeholderId, original_filename: local.name, file_size: 0, file_url: null};
  };
}

Offline, the upload function returns a negative id and stashes the local file. Everything upstream — the form component, the value state, the payload builder — carries on unchanged, because what it received has the same shape as a real upload result.

This is a decorator around a strategy, and the property that makes it work is that the caller cannot tell. There is no if (offline) in the form runner. There is no offline variant of the media input. The knowledge that we are offline is confined to one function whose signature is identical to the one it replaces.

At flush time the placeholders are swapped for the real thing:

async function processOne(item: QueuedSubmission): Promise<void> {
  const map = new Map<number, number>();
  for (const f of item.files) {
    const up = item.kind === 'auth'
      ? await uploadFile(f.local, {isSignature: f.isSignature})
      : await uploadPublicFile(f.local, {...});
    map.set(f.placeholderId, up.id);
  }
  ...
  await submitForm({...item.payload, data: remap(item.payload.data, map)});
}

Why the sentinel is negative

This is the part worth stealing.

Server ids come from a positive autoincrement. Negative integers are therefore a region of the same type that the server can never produce — so a placeholder is distinguishable from a real id by inspection, with no wrapper object, no tagged union, no parallel list of which fields are file fields.

That is what lets the remapper be four lines and know nothing about the form:

function remap(data: Record<string, any>, map: Map<number, number>): Record<string, any> {
  const out: Record<string, any> = {};
  for (const k of Object.keys(data)) {
    const v = data[k];
    if (typeof v === 'number' && map.has(v)) {
      out[k] = map.get(v);
    } else if (Array.isArray(v)) {
      out[k] = v.map(x => (typeof x === 'number' && map.has(x) ? map.get(x) : x));
    } else {
      out[k] = v;
    }
  }
  return out;
}

No schema. No list of file field names. Any number anywhere in the payload that happens to be a key in the map is a placeholder, because nothing else could be. A form with thirty fields, six of them files, four of those multi-file — the function does not care and does not need updating when a field type is added.

The alternative — {__placeholder: true, id: …} — is more explicit and forces every consumer between the input and the payload to understand it. The sentinel buys transparency through the whole stack, and the price is that it only works while the two id spaces stay disjoint. Write that down next to the generator, because it is an invariant maintained by a database default that lives in another repository.

Note also that remap returns a new object rather than mutating, and is applied as {...item.payload, data: remap(...)}. The queued item is untouched. If the submit throws, the stored payload still has its placeholders and the next attempt starts from the same place — a mutating remap would leave a half-rewritten payload referencing ids for files that may not have finished uploading.

The remapping is one level deep: scalars and arrays of scalars. That covers a single-file field and a multi-file field, which is every file-bearing field there is. A value that nested file ids inside an object would pass through untouched — a bound worth knowing about before someone adds one.

Minting ids in JavaScript has a ceiling

let _ph = 0;
/** Unique negative id used as a stand-in for a not-yet-uploaded file. */
export function newPlaceholderId(): number {
  _ph = (_ph + 1) % 100000;
  return -(Date.now() * 100000 + _ph);
}

The intent is clear and correct: a timestamp so ids stay unique across app restarts — a counter that resets to zero would collide with yesterday’s queued placeholders — plus a counter so two files picked in the same millisecond differ.

And this is where JavaScript has a constraint that does not exist in Python, Java or Go, and that quietly eats the second half.

Number is a float64. Integers are exact only up to Number.MAX_SAFE_INTEGER — 9 007 199 254 740 991, about 9 × 10¹⁵. Above it, consecutive integers stop being representable and arithmetic silently rounds.

Date.now() today is about 1.78 × 10¹². Multiply by 100 000:

Date.now() * 100000  ≈  1.78 × 10¹⁷
Number.MAX_SAFE_INTEGER ≈ 9.01 × 10¹⁵

Nearly twenty times over the ceiling. At that magnitude the gap between representable doubles is 32, so adding 1, 2 or 3 produces the same number:

_ph = 1  →  178000000000000000
_ph = 2  →  178000000000000000
_ph = 3  →  178000000000000000

The counter is gone. Not truncated — absorbed. The multiplier that was meant to make room for it is precisely what pushed the sum out of the exact range, and the low digits it was supposed to occupy no longer exist.

The consequence is specific: two files stashed within the same millisecond get the same placeholder id. map.set(placeholderId, up.id) then overwrites, and both fields remap to the same uploaded file. Selecting several photographs in one gallery action is exactly how you land in the same millisecond twice.

The rule to carry away:

In JavaScript, any id built as a * m + b must satisfy a * m + b < 2⁵³. Check it with real numbers, not with the small ones in your test.

For a millisecond timestamp, that leaves a multiplier of about 5 000. A thousand is a comfortable choice — a thousand ids per millisecond, and the product stays at 1.78 × 10¹⁵, safely inside:

let _ph = 0;
export function newPlaceholderId(): number {
  _ph = (_ph + 1) % 1000;
  return -(Date.now() * 1000 + _ph);
}

BigInt is the other answer and the wrong one here: the value has to survive JSON.stringify, and BigInt throws. A persisted monotonic counter works too, and costs an await on a code path that currently has none.

The error you must never queue

export function isNetworkError(e: any): boolean {
  if (!e) return false;
  if (e.code === 'ERR_NETWORK' || e.code === 'ECONNABORTED') return true;
  if (typeof e.message === 'string' && /network|timeout|Failed to fetch/i.test(e.message)) return true;
  if (e.request && !e.response) return true; // axios: sent, no response
  if (e instanceof TypeError) return true;   // fetch failure
  return false;
}

This function decides the behaviour of the whole system, and getting it wrong in either direction is bad in a different way.

Queue too much and a validation error — a missing required field, an expired session, a deleted form — becomes a permanent queue entry. It will never succeed. It retries on every connectivity change, forever, and the user sees “1 pending” on a submission that is not pending, it is wrong, and nothing tells them which.

Queue too little and a submission is lost the moment the network flickers mid-request, which is the exact scenario the queue exists for.

So the test is not “did it fail” but “did the server ever see it?”

if (e.request && !e.response) return true; // axios: sent, no response

That line is the heart of it. Axios sets request when it sent something and response when something came back. Request without response means the message left and nothing returned — a connectivity failure, by construction, regardless of what the message says. And e instanceof TypeError is the same fact in the fetch API, where a network failure is the only thing that rejects with a TypeError while any HTTP status resolves normally.

The two string tests are the weak ones — matching on /network|timeout|Failed to fetch/i is guessing — but they are last, after the structural checks, so they only catch what the reliable tests missed.

The same distinction, at the point of submission:

if (net.isConnected && !hasFiles) {
  try {
    if (base.kind === 'auth') await submitForm(base.payload);
    else await submitPublicForm(base.payload);
    return {synced: true};
  } catch (e) {
    if (!isNetworkError(e)) throw e; // real error (validation/auth) → caller handles
    // connectivity died mid-request → fall through to queue
  }
}

throw e for a real error. Not queued, not swallowed: raised, so the form screen shows the field errors while the user is still looking at the form. A validation error deferred to a background sync is a validation error nobody will ever fix.

And note the fast path’s condition: online and no deferred files. Submitting directly when possible is not only faster, it is the only path that surfaces server-side validation synchronously. The queue is the fallback, and it is entered only when it has to be.

Draining, and the difference between two failures

let _flushing = false;

export async function flush(): Promise<{done: number; failed: number}> {
  if (_flushing) return {done: 0, failed: 0};
  const net = await NetInfo.fetch();
  if (!net.isConnected) return {done: 0, failed: 0};

  _flushing = true;
  let done = 0, failed = 0;
  try {
    const current = await getInstanceUrl();
    const items = await read();
    for (const item of items) {
      // Never replay a submission against a different workspace than the one
      // it was captured in.
      if (item.instanceUrl && current && item.instanceUrl !== current) continue;
      try {
        await processOne(item);
        await remove(item.id);
        done++;
      } catch (e: any) {
        failed++;
        await patch(item.id, {attempts: item.attempts + 1, lastError: e?.message || 'Sync failed'});
        // Stop on the first connectivity drop; transient per-item errors are
        // recorded and retried on the next flush.
        if (isNetworkError(e)) break;
      }
    }
  } finally {
    _flushing = false;
  }
  return {done, failed};
}

Four decisions, each carrying weight.

_flushing is a re-entrancy guard, not a lock. NetInfo fires liberally — Wi-Fi to cellular, a captive portal negotiating, a screen unlock — and two concurrent flushes would both read the same queue and submit everything twice. The finally matters: an unhandled throw that left _flushing true would wedge the queue for the rest of the process lifetime, and the user would have no way to know why nothing syncs.

break on a network error, continue on anything else. Once connectivity is gone, the remaining items will all fail for the same reason. Continuing would increment attempts on every one of them and record the same misleading lastError — so a single tunnel makes a twelve-item queue look like twelve broken submissions. Stopping keeps the diagnostics honest: attempts counts real attempts, not the same outage counted twelve times.

The workspace guard. A technician working across two client deployments must never have a submission captured in one replayed into the other. It is a continue, not a delete: the item stays, and it will flush when the app is pointed back at its own instance. Deleting would be losing data to protect against a mistake nobody made.

remove only after success. This is at-least-once delivery, and it is the right choice for field data: the failure mode is a duplicate, not a loss, and a duplicate is a support ticket while a loss is a wasted site visit.

It is worth being explicit about the corner it leaves. If the submit succeeds and the app is killed before remove completes — battery dies, OS reclaims the process — the item is still queued and will be sent again. The server gets two identical submissions.

The complete fix belongs on the wire: a client-generated idempotency key, allocated at enqueue time, sent with the submission, and unique-indexed server-side so the second delivery returns the first result instead of creating a row. It is one extra column and one extra header, and it is much easier to add before there is a year of production data than after.

Making the queue visible

const listeners = new Set<() => void>();
export function subscribe(l: () => void): () => void {
  listeners.add(l);
  return () => { listeners.delete(l); };
}
function notify() {
  listeners.forEach(l => {
    try { l(); } catch {
      // listener errors are not our problem
    }
  });
}

write() calls notify(), so every mutation refreshes the UI. subscribe returns its own unsubscribe function, which is the shape a React useEffect cleanup wants and the one that makes leaking a listener take deliberate effort.

The try/catch around each listener is small and load-bearing: a component that throws while re-rendering must not stop the other listeners from being told, and must certainly not propagate into write() and make a successful persist look like a failure. Notification is best-effort; persistence is not.

And the reason any of this exists is worth saying plainly. A field agent needs to know, without asking anyone, whether their morning’s work is on the server. A queue that syncs perfectly and shows nothing is indistinguishable from a queue that lost everything — and the agent will redo the work, which costs more than the bug would have.

What to take away

1. Keep binaries out of the queue. Store URIs; the boring key-value store is then sufficient.
2. When a value must stand in for a not-yet-existing id, use a region of the same type the server can never produce. Negative integers against a positive autoincrement — everything upstream stays unchanged.
3. Check your id arithmetic against 2⁵³. In JavaScript, a * m + b silently loses b once the product leaves the safe range.
4. Only queue failures where the server never saw the request. Structurally — sent, no response — not by matching error strings.
5. Raise validation errors immediately. Deferred to a background sync, nobody ever fixes them.
6. Stop the drain on a connectivity failure; continue past a per-item one. Otherwise one outage looks like twelve broken submissions.
7. Delete only after success, and add an idempotency key — at-least-once is the right trade, and the duplicate is your job to prevent on the server.
8. Show the queue. Silent success and silent loss look identical to the person doing the work.