"I don't know" and "it's empty" are not the same piece of data

An IPTV provider hands you two hundred categories and you care about twelve. So on your phone you hide the rest, get the list down to something clean, and that cleanliness shows up on the TV: that syncs fine.

Months later you change your mind and hit "show all." On the phone, all two hundred come back. On the TV, they stay hidden.

There's no error, no warning, nothing in the logs. And the strangest part: if instead of showing all of them you leave a single one hidden, it does propagate. Sync works for every value except one. Emptiness.

The same path's three different "I don't know"s

User state in LoverCast is local-first: it lives on the device and syncs to a Postgres row, one column per state type. On startup it does a pull: it fetches the row and applies it on top of local state.

That pull has to answer three different questions, and all three look a lot like "there's nothing here":

suspend fun fetchRemote(): RemoteUserPreferences? {
    val session = syncSessionStore.getSession() ?: return null
    return syncService.loadPreferences(session.accessToken)
}

One: there's no session, or the cloud isn't answering. It returns null, and the whole pull applies nothing. That's correct: losing coverage can't look the same as having your favorites wiped out.

Two: the row exists but this column has never been written. This user has never touched search history from anywhere, so the column has nothing to say about what's on the phone. Don't touch local.

Three: the column was cleared on purpose. The user cleared their history on the TV. That is an opinion, and it needs to propagate: clear it on the phone too.

Two and three produce the exact same list size — zero — and demand opposite behavior. If your data type can't tell one from the other, you have to pick which of the two cases you're going to get wrong.

Two repos, two answers, same folder

What's interesting about LoverCast is that the same codebase contains both decisions, in the same folder, inheriting from the same base class. Nineteen Synced* repos, all with an applyRemote method that applies the downloaded row on top of local state.

This one distinguishes:

override suspend fun applyRemote(remote: RemoteUserPreferences) {
    // null = column never synced → don't touch local.
    // [] = actually cleared set → propagate the deletion.
    val ids = remote.seriesCompleted ?: return
    local.replaceAllLocal(ids.toSet())
}

And this one collapses:

override suspend fun applyRemote(remote: RemoteUserPreferences) {
    if (remote.liveHiddenIds.isNotEmpty())     local.hideAll(remote.liveHiddenIds)
    if (remote.vodHiddenIds.isNotEmpty())      local.hideAllVod(remote.vodHiddenIds)
    if (remote.seriesHiddenIds.isNotEmpty())   local.hideAllSeries(remote.seriesHiddenIds)
    if (remote.liveCategoryOrder.isNotEmpty()) local.setLiveCategoryOrder(remote.liveCategoryOrder)
    // …and three more just like it
}

hideAll replaces the local set, it doesn't add to it. That's why hiding works, leaving one item hidden works, and leaving zero hidden doesn't: the list arrives empty, isNotEmpty() returns false, and the pull decides the cloud had nothing to say. Same story with category order, with hidden channels, and with channel_order — which does this per category on top of everything, so you can't even return one category to its natural order.

Notice the second block has no comment. There's no decision written down there because none was ever made: isNotEmpty() is what you write when the question "what if it comes back empty?" never got asked. It's the easiest symptom of this bug to spot, and it's not in the logic.

The bug was in the declaration

It's in the type. These columns are the same class of data and they're not declared the same way:

data class RemoteUserPreferences(
    // Nullable on purpose: null = the column was never synced (don't touch local
    // on pull); [] = actually cleared list (propagate the deletion).
    @SerialName("series_completed")    val seriesCompleted: List<String>? = null,
    @SerialName("search_history")      val searchHistory:   List<String>? = null,

    @SerialName("live_hidden_ids")     val liveHiddenIds:   List<String> = emptyList(),
    @SerialName("live_category_order") val liveCategoryOrder: List<String> = emptyList(),
)

Above, List<String>?: the ?: return that distinguishes both cases is possible because the type has a slot to put it in. Below, List<String> = emptyList(): there's no value of that type that means "I don't know." Even if you wanted to distinguish, you have nothing to distinguish with. isNotEmpty() isn't sloppy implementation — it's the only thing you can write with that type.

And that type wasn't chosen in Kotlin either. It's tracing the table:

create table if not exists public.user_preferences (
  user_id               uuid        primary key,
  channel_favorites     jsonb       default null,
  live_hidden_ids       text[]      not null default '{}',
  vod_hidden_ids        text[]      not null default '{}',
  live_category_order   text[]      not null default '{}',
  ...
  series_completed      text[]      default null,
  search_history        text[]      default null,
);

live_hidden_ids and series_completed are the same Postgres type. The only difference between the column that syncs fine and the one that doesn't is a not null default '{}' written into the initial CREATE TABLE, probably in thirty seconds and with the best of intentions: an empty array is more convenient than a null, it saves you checks, you never get a NullPointerException.

And it's true, it does save you all that. What the shortcut doesn't tell you is the price: a not null default '{}' column has a value from the instant the row exists. It has never been empty of information, only empty of elements, and those two things can never be told apart again.

Why an alter table won't fix this

Here's the part that makes this worth a post instead of a ticket.

Dropping the not null is one line. But the day you run it, every row that already exists has '{}' — and there's no way to tell, by looking at that row, whether that {} means "this user has never hidden anything" or "this user hid things and then unhid them all." The information you'd need to decide was never saved. It's not in another table, or in a log, and it can't be inferred: the moment both situations got written with the same value, the difference stopped existing.

So the backfill isn't a SQL problem. It's that you have to choose whose state you're going to break: if you interpret the old {} as "never synced," you're exactly where you started. If you interpret them as "cleared on purpose," the first pull after deploying shows two hundred categories to everyone who had hidden them and hadn't opened the app in a month.

That's what I mean by a one-way door. Nullable → not null is a decision you can revisit. Not null → nullable gives you back the ability to distinguish going forward, but it doesn't give you back the data you already collapsed. In LoverCast, new columns get declared nullable from the start, with the comment in place; the eight old ones are still there, and the cost of fixing them isn't technical.

A write path with nobody on the other end

There's one more detail worth seeing, because it's how these bugs hide from tests.

The push does send the emptiness. When you hit "show all," the repo uploads the local set as is, and {} really does get written into the column. The data travels, arrives, gets saved correctly. There's just nobody reading it: the isNotEmpty() on the other side throws it away.

A write path that works perfectly, whose reader ignores the result, fails nowhere you'd think to look. The push test passes: it checks that what's there gets uploaded. The pull test passes: you feed it two or three fixture elements and it applies them fine. The broken case is the zero-element one, which is exactly the one nobody writes as a test case because it looks like the trivial one.

And in LoverCast this is documented, which is both the most honest and the most uncomfortable part. SYNC.md has a warning saying that if you clear a column in the cloud by hand, set it to [] and not null, because null doesn't propagate the deletion. It's a good warning. What it doesn't say is that for eight of those columns the advice doesn't work: setting [] doesn't propagate anything either, because the reader is isNotEmpty(). The documentation described the tidy half of the system.

Where else this lives

None of the above is specific to Android or Postgres. It's what always happens when a single value has to carry two meanings:

• JSON APIs. A missing field and a field set to [] should be different things in a PATCH, and almost never are. If your deserializer turns "wasn't there" into "empty list" before your logic ever sees it, you've lost the difference at the boundary and you don't get it back inside.
• Forms. "Hasn't filled in this field" and "has cleared it on purpose" send the exact same thing over the wire if an empty input serializes as an empty string. With "no known allergies" and "we haven't asked" in the same box, the difference matters quite a bit.
• Config. A key that isn't set should inherit the default. A key set to empty should override it with empty. Many config systems treat both the same, and that's how you get the "but I set it to zero and it's ignoring me" tickets.
• Caches. "I don't have it cached" and "I have it cached and the result was none" call for opposite things: a query and no query. Collapsing them is how you end up accidentally building a cache that doesn't cache empty results.
• Paginated lists. An empty last page and a swallowed error look a lot alike if all you return is an array.

The way to spot the pattern before you write it is a single question, asked when you declare the field, not when you use it: do absence and emptiness mean the same thing here? If the answer is no, the type needs to be able to say so. If it's yes, write it down in a comment, because whoever comes next is going to assume whatever's convenient for them.

What we take away

Emptiness is a value; absence is the lack of a value. Putting them in the same slot doesn't simplify the model: it takes away its ability to express one of the two, and you don't get to choose which one until a user complains. It happens the same way in any internal tool where someone can clear a field on purpose.

not null default '{}' is a product decision disguised as convenience. It gets written into the CREATE TABLE without a second thought and paid for in the pull, months later, in a different language and a different repository.

An isNotEmpty() in a merge is an unanswered question. It's not always wrong — sometimes "empty means no opinion" is exactly what you want. But if there's no comment next to it saying so, it almost never means it was decided: it means it was never asked. Like the String that was actually an enum I already wrote about, the problem isn't where it fails.

Collapsing information is irreversible. Almost everything in a schema can be changed later. Merging two states into one value can't: you get back the ability to distinguish them going forward, never the data from before. It's the kind of decision you have to make wide awake, and it's thirty seconds up front against a backfill with no correct answer.

In the app this comes down to twelve categories instead of two hundred. But the reason the TV never found out wasn't the TV's fault: it was written into the first version of the table, two years earlier, in a column that couldn't say "I don't know."

---

LoverCast is in production and you can download it. If you have a column that can't tell emptiness from absence and you're deciding what to do about the backfill, write to me.