Test Report — Buzz & Populace

Everything that was run against both products, what it found, and — set out just as plainly — what was never tested. Figures here are read from run artefacts and re-verified commands, not recalled. Where a number could not be sourced, it is marked as not measured rather than estimated.

1 · Headline

A note on the name: the app was called Buzz Buzz while every run in this document was carried out, and was renamed to Buzz on 16 August 2026. The product, the code and the figures are unchanged — only the name is. Run artefacts and commit messages from before that date still say the old one.

The single most useful result in this document is not the clean run. It is that a finished, signed, manually tested application could not create a working account, and that this was found in the first fifteen seconds by software rather than by a person.

The second most useful is that the sixth defect was found on the sign-off run — the run whose whole purpose was to confirm there was nothing left to find.

2 · How the products were tested

Buzz is a multi-user product: a social feed, group chat, presence on a map, and permission rules deciding who may read what. None of that can be proven by one developer with one account, however carefully they tap through every screen. So the primary method was to generate a population and let it use the app through its real API, as genuinely authenticated users, with the app's own permission rules applying.

3 · Populace against Buzz — 9 August 2026

Six simulated drivers, three in Manila and three in Mumbai, against the live backend with 17 tables and 48 row-level-security policies. Each signed up for a real account, set a profile, drove a plausible route, posted, read the feed, liked and commented, opened conversations, sent messages and joined a group. Nobody told Populace where the bugs were.

3.1  The first run — five defects in three and a half minutes

The application had already passed a full manual test of every screen by the person who wrote it. The run found the following:

Defects 4 and 5 were in our own testing tool, and 5 is the one worth dwelling on: an unchecked error is the precise fault Populace exists to catch, and it was sitting in our code. Because the failure was swallowed at signup, the report blamed a later method — it said post failed 14 times when the truth was the profile row was never created. Making that one line throw turned three confusing symptoms into one accurate sentence.

3.2  The re-run after the fixes

All five were fixed and the run repeated against the same backend. Every implemented method was exercised; every account created was deleted afterwards.

Coverage 13/13 methods. Six accounts created, six deleted, nothing left behind. Simulated activity: 6.2 km driven, 22 posts, 59 likes, 22 comments, 21 messages, 2 group joins. Account deletion is included deliberately — it is the path almost nobody exercises and the one regulators ask about.

3.3  Where the capacity ends — 6 to 36 concurrent users

Six runs at increasing concurrency, all of them clean. The zero-failure results are the least interesting thing about them: taken together they show where this backend starts to queue, which is the number worth knowing before real riders arrive rather than after.

Every row counts what that many users actually did, and the 36-user row is no exception: 7,485 calls across twelve methods, zero failures. It reached 36 by way of a 50-user run — fourteen accounts were refused at signup, before the run began, and the run proceeded with the 36 that existed. Those fourteen were provisioning, not load, and they are reported as their own finding below rather than charged to this row. No call failed at 36 concurrent users.

One row needs a pointer: the ten-user figure is the regression re-run of §3.6 (2,030 calls), not the original ten-user run, which made 2,116 calls and has its own detail block below. Both were clean; the later one is quoted here because it ran against the current engine.

Run detail · thirty concurrent users, thirty minutes

The longest and largest run in this document, and the only one to exercise every method the adapter has. Thirty agents for 1,856.7 seconds — just under thirty-one minutes — against a local Supabase stack built that evening from this project's own SQL, including the two migrations added the same day. Read the caveat below before comparing its latencies with anything else here.

332.7 km driven · 1,183 posts · 2,693 likes · 1,054 comments · 1,126 messages · 172 group joins. 0 API failures, 0 transport failures, 0 retries, 0 outages, 0 engine errors, and 30 accounts created and 30 removed, cleanup verified empty afterwards. This is the first run long enough for refreshSession to fire for every agent: all thirty sessions were renewed mid-run and none dropped, which is why coverage reads 13 of 13 here rather than 12.

Run detail · thirty-six concurrent users, ten minutes

The largest run in this document, and the cleanest. It was launched as a fifty-agent run; fifty accounts could not be created, so thirty-six went to work instead, for 931.3 seconds against the isolated cloud test project. Provisioning and the run are reported separately below, because they failed and passed for entirely different reasons.

Provisioning, before the run. createUser was called 50 times: 36 accounts created, 14 refused by Supabase Auth with Request rate limit reached (p50 653 ms, p95 1.3 s across all fifty attempts). This is the signup ceiling described above — a free-tier quota on account creation, not a property of Buzz. The run then began with the 36 accounts that existed.

With the 36 successful createUser calls added, 7,485 successful calls, none of which failed. Every method the app has was exercised: eleven here plus createUser, and listConversations covered by the smoke check beforehand.

Throughput held for the full ten minutes. The engine reports every 24th tick, and the interesting thing is what does not happen — each quarter of the run does roughly the same amount of work as the one before it, so nothing sagged as sessions aged and tables grew:

132.5 km driven · 474 posts · 1,117 likes · 441 comments · 423 messages · 70 group joins. 0 API failures, 0 transport failures, 0 retries, 0 outages, 0 engine errors, and 36 accounts created and 36 removed — cleanup verified empty afterwards. Populace exited non-zero on this run, correctly: it was asked for fifty users and delivered thirty-six, and a tool that reported success there would be worthless. The run itself did not fail.

Run detail · thirty concurrent users, ten minutes

Thirty agents for 943.1 seconds against the isolated cloud test project. All thirty accounts were created without trouble — the signup quota was not reached at this size — and the run returned exit 0.

116.2 km · 403 posts · 984 likes · 397 comments · 389 messages · 55 group joins. Verdict clean: 0 API failures, 0 transport failures, 0 retries, 0 outages, 0 engine errors, 30 accounts created and 30 removed. The 2.8 s reportLocation tail in the first row is the measurement that produced the withdrawn saturation claim — it did not reproduce at thirty-six users, where the same figure came back to 1.4 s.

Run detail · twenty concurrent users, fifteen minutes

Twenty agents for 1,264 seconds against the isolated cloud test project, preceded by clean, doctor and smoke, and followed by clean again. At 21 minutes it is the longest run in this document.

116.3 km · 437 posts · 930 likes · 384 comments · 394 messages · 55 group joins. Verdict clean: 0 API failures, 0 transport failures, 0 retries, 0 outages, 0 engine errors, 13/13 methods, 20 accounts created and 20 removed. At 21 minutes this is also the first run long enough to exercise session refresh properly — all twenty sessions survived it.

Run detail · ten concurrent users, ten minutes

Ten agents across Manila and Mumbai for 809.1 seconds against the isolated cloud test project. clean was run first to prove the target was empty, then doctor and smoke, then the run, then clean again.

39.7 km · 148 posts · 283 likes · 127 comments · 121 messages · 23 group joins. Verdict clean: 0 API failures, 0 transport failures, 0 retries, 0 engine errors, 13/13 methods, 10 accounts created and 10 removed with none stranded.

Run detail · eight concurrent users, ten minutes

The only run at eight agents. Against the isolated cloud test project, not production: doctor confirmed 2 production host(s) denied before it started.

806.8 s · 27.8 km · 78 posts · 233 likes · 84 comments · 106 messages · 17 group joins. Verdict clean: 0 API failures, 0 transport failures, 0 retries, 0 engine errors, 13/13 methods covered, 8 accounts created and 8 removed with none stranded. First run in which openConversation and sendMessage were exercised at volume.

3.4  The local runs, and a sixth defect

Work continued against a local Supabase stack rather than the cloud project, which takes the network out of the measurement. Two further runs, six agents for about five minutes each:

Both reported preclean=0 · run=0 · postclean=0 — clean before, clean during, clean after. The sign-off run was the first exit 0 Populace had ever returned against Buzz.

3.5 · A second backend, and what it exposed

Every run above uses the same backend technology. That made the strongest caveat in earlier versions of this report an open one: Populace had been pointed at exactly one real application, and that application was ours. A testing tool that has only tested its author's own app has been shown to work once, not shown to be portable.

So a second backend was built, deliberately unlike the first on every axis that could hide an assumption, and shipped in the repository as examples/rest-api/ so the result can be reproduced by anyone:

Result: no failures across 430 API calls, 13/13 methods, 6 accounts created and 6 removed. The engine could not tell the two apart.

3.6  Regression run after those changes

The 2,116-call result above was measured on code that has since changed. Buzz was therefore re-run in full against the modified engine: 2,030 API calls, 0 failures, 10 agents, 818.1 s, 13/13 methods, 10 accounts created and 10 removed, verdict clean. Fixing portability did not cost correctness on the original backend.

3.8  The same run over the network, which did not finish

The thirty-user, thirty-minute run above was repeated against the cloud test project the same evening, deliberately, to get the figure a local stack cannot give. It did not complete. It is recorded here because a report that only keeps the runs that worked is not a record.

Every failure was transport: 149 calls refused once the breaker opened, 30 reportLocation timeouts at 20 s, and a handful of fetch failed. Every method that reached the server was clean — like 179, sendMessage 85, comment 72, createUser 30, all at zero.

dns=11.06s  tcp=11.13s  tls=15.44s  total=17.60s

3.9 · The 21 August runs — thirty to three hundred drivers

Seven counted runs in one day against a local Buzz backend, from thirty simulated drivers to three hundred, across thirty cities in seventeen writing systems. 3,174,576 API calls and zero API failures. 740 accounts created, 740 removed, verified by querying the database afterwards rather than by trusting the tool's own report.

The three-hundred run is marked inconclusive because one post call never reached the server — a socket exhausted on the test machine, after 1,316 retried attempts across the run had succeeded. Its API failure count is zero, like every other row. Populace will not label a run clean when it could not make every call, even when nothing in the app went wrong.

What governs every figure above: these runs are against a local Supabase stack over loopback, so no network is included. The same calls measured about 175 ms each against the hosted project before any load at all — a real driver's experience is this cost plus the network, not these numbers.

4 · Populace's own correctness

A testing tool that is wrong is worse than no testing tool, because it is believed. Populace therefore ships a suite that runs the whole product against an in-memory application with no backend of any kind.

$ node src/selftest.mjs
  ✔ without refresh, an expired session poisons the whole run
  ✔ refreshSession keeps a run alive past token expiry
  ✔ an expired run cannot even clean up after itself
  ✔ 0.88% failure rate is still a problem, not 'clean'
  ✔ the verdict line can never contradict the table beneath it
  ✔ a bug inside Populace is kept, not swallowed
  ✔ a run that broke internally is never reported clean
  … 75 checks total

  All checks passed — the engine is app-agnostic and the report is honest.
  exit 0

Count verified by running the suite for this report, not taken from documentation: 75 passing checks, exit code 0.

4.1  The demo adapter run

Populace ships a demo application containing a deliberate row-level-security defect, so that a first-time user can watch the tool find something real. Run on 14 August 2026:

The separation of API failures from transport failures matters: it is what stops a flaky network being reported as a bug in your code.

5 · The shipped Android artefact

6 · Internationalisation

A defect worth recording: the bundled Inter typeface never rendered on Android. It was listed after -apple-system and BlinkMacSystemFont in the font stack, both of which resolve to Roboto on Android — so the font loaded on every device and was used on none of them. Ordering, not availability.

7 · Web properties

Three sites, each measured in a real browser rather than reviewed by eye. Contrast is computed from rendered colours against WCAG AA thresholds — 4.5:1 for normal text, 3:1 for large.

7.1  Defects the measurement found

Every one of these looked acceptable and was not:

• White on the brand orange measured 3.32:1, against the 4.5 required. Pure white on #FF4D17 reaches only 3.35, so no text colour could have fixed it — the surface had to change. This defect was introduced twice, on two different sites, from the same instinct.
• The light theme was unreachable for any visitor whose operating system was set to dark. Palette values were defined in the base rule and overridden in a media query, but not in the explicit light selector, so choosing light kept the dark accents — light violet on a white page, at 2.28:1.
• A near-black scrim survived its background. After a section changed from black to a pale wash, an overlay still painted rgba(5,7,8,.95) over it: grey text on a black smudge.
• Brand colours used as text. Populace's lime measured 1.16:1 on paper and honey gold 2.1:1 on cream. Both are legitimate brand colours and neither is a text colour; each now has a darkened counterpart for type.
• A muted grey at 4.40:1 — close enough to look fine and still fail.
• A dangling custom property. A renamed token left one gradient layer referencing a variable that no longer existed; that layer would have silently vanished.

Three findings during this work were defects in the measuring script rather than the pages — most notably that color-mix() computes to color(srgb 1 1 1 / 0.93), whose 0-to-1 floats were being read as 0-to-255, making white register as near-black. They are recorded here because a measurement you have not questioned is not evidence.

8 · What was not tested

This section is the reason the rest of the report can be trusted. A test report that lists only passes is a marketing document.

9 · Conclusion

Both products do what is claimed of them, within limits this document states rather than hides. Buzz builds, type-checks, ships as a signed artefact, and survived a concurrent multi-user run against its live backend with no failures across 400 calls, and a thirty-six-user run of 7,485 calls with none either. Populace found six genuine defects in a finished application, passes 78 self-tests, and distinguishes a fault in your code from a fault in the network between you.

A seventh defect was found by the owner rather than by any of this, on a handset, after the artefacts were published: every build shipped that day was compiled against a developer's local backend and could not sign in at all. It is the most useful result in the document, because it marks the edge of what everything above actually measures. Populace tests an API. It never opens the app. A client built against the wrong host, with push compiled out, passes every server-side check ever written and still fails the moment a person taps “sign in”.

The honest summary is this: what has been tested has been tested properly, and the largest untested areas are known and written down. The most valuable outcome was not a passing run — it was discovering that an application which had passed a careful manual test by its own author could not create a working account, that three separate symptoms shared one cause, and that the same cause surfaced a third time on the very run intended to sign the work off.

Prepared by Gigzen Private Limited. Every figure in this report is taken from a run artefact or a command re-executed while writing it. Where something was not measured, it says so.