Case study · Personal project, in progress
A coaching tool built to admit what it doesn't know yet.
Advice about posting on social media is usually generic, undated, and impossible to check — "post at 9am," "use three hashtags," source unknown. This tool takes the opposite approach: every note it gives has to trace back to either a stated platform rule or the account's own measured results. Where neither exists yet, it says nothing rather than guessing convincingly.
Short answerMost engagement advice can't be checked — no date, no source, no sample behind it. This tool only surfaces a note it can justify: either a stated platform rule or a finding from the account's own measured results, carried with the sample size it came from. Five candidate public datasets were audited and rejected before any analysis was written.
The problem
Confident advice, unknown provenance.
The trouble with most engagement advice isn't that it's wrong. It's that there's no way to tell. It arrives with no date, no source, and no sample behind it — and platforms change often enough that a rule which was true three years ago may be actively unhelpful now.
So the design constraint came first: the tool may only surface a note it can justify. In practice that means every claim traces to either a hard platform constraint, like the character limit, or a finding from the account's own reconciled results, carried with the sample size it came from. There is no third category for things that merely sound right.
Before writing any analysis
Five candidate datasets, all rejected.
The fastest way to ship this would have been to train something on a public dataset and present the output confidently. That path got audited and closed instead.
Disqualified for structural reasons
Five public datasets were checked and each ruled out on the merits: no engagement metrics at all, too old to reflect how the platform now works, drawn from a population whose audience and algorithmic treatment don't match, or too small to support any conclusion. A dataset that doesn't measure what you're asking about can't be fixed by using it carefully.
Not available at any sensible price
The account's own developer access carries zero read quota, and current API pricing is pay-per-use with no free read tier. That ruled out pulling history programmatically, so reference data is hand-observed and logged deliberately — small and curated rather than harvested at scale.
No pooling to reach a sample size faster
Own-account results, observed trends, and nominated example accounts stay statistically separate. Merging them would reach a bigger sample sooner, and would repeat exactly the population-mismatch error that disqualified one of the rejected datasets — only self-inflicted this time.
The rule that shapes the output
No single score, and no promises.
It would be easy to average the extracted features into one number and call it a rating. That number would feel useful and mean nothing, because it would imply a precision the underlying data cannot support. So there is no aggregate score. Notes are individual and separately checkable, and they're written as "associated with," never "will get you."
A false "no issues" is worse than a note that turns out wrong. A wrong note is visible and can be argued with. Silence just looks like approval.
That principle came from a real miss: an early version passed a draft that exceeded the character limit without flagging it. The failure wasn't the incorrect advice — it was the confident quiet.
Deciding in advance
What counts as enough data to act on.
The thresholds below were written down before the data existed, specifically so that a promising-looking early pattern couldn't argue its way into the tool later.
Don't look
Anything visible at this size is noise wearing a pattern's clothes.
One question, asked properly
Enough for a single-feature nonparametric check — Spearman or Mann-Whitney rather than Pearson or a t-test, because the outcome distribution is zero-inflated and skewed rather than normal. Effect size and confidence interval reported, language still hedged.
Before data overrides a rule
The bar before collected results are allowed to override a built-in heuristic, or before a multivariate look across roughly ten extracted features is trustworthy at all — on the order of 10–20 observations per feature dimension.
For scale: detecting a moderate correlation at conventional power needs roughly 85 observations; a realistic weak-but-real effect needs closer to 780. At a normal posting cadence, even the 30-example floor is a multi-month wait. That's treated as a constraint to design around, not a reason to lower the bar.
Why this matters beyond one tool
The same discipline applies to your dashboard.
This is a small project about social posts, but the failure mode it guards against is the common one in organizational reporting: a number presented with more confidence than its underlying data earns. A dashboard tile showing a 40% change from eleven records is the same error, and it gets acted on because it looks authoritative.
Deciding what counts as enough evidence before you go looking is the part most reporting work skips. It's also the part that determines whether the resulting numbers can be trusted to make a decision with.
Reporting that overstates what it knows?
If you're not sure whether your numbers support the decisions being made on them, that's worth an outside look.