Populace — a simulated population that tests your app for real

Proven, not promised

It found six real bugs in a finished app.

Not a toy example. Buzz is a shipped social application for delivery riders — built, tested by hand, and believed to work. Six simulated people used it for five minutes. A sixth arrived later, on the sign-off run itself. Every run is written up in the full test report, including what was never tested. Every one of these was invisible to single-user testing, because every one of them requires a second person to exist.

5real bugs, first run

318API calls made

6concurrent identities

5minto find them all

Blocker

Nobody could see anybody else's posts

The feed query filtered by the author's own id. With one tester it looked perfect — you always saw your own posts. With six, the app was six private diaries that happened to share a database.

Blocker

Profile updates silently wrote nothing

An INSERT … ON CONFLICT DO UPDATE needs SELECT permission on every column it touches. It had none, so the upsert reported success and changed nothing. No error, no row.

High

Account deletion left the data behind

The auth record went; the posts, photos and messages stayed. The one path almost nobody exercises, and the one regulators ask about first.

High

Messages delivered to the wrong conversation

Thread identity was derived from a pair of ids in inconsistent order, so A→B and B→A were sometimes different threads and sometimes the same one.

Medium

Group membership counted duplicates

No unique constraint on the join table. Rejoining inflated the count, and the number shown to users drifted upward the more they used the app.

“You cannot upsert a column you cannot select.”

And a clean run since

The same six people, the same schema, after the fixes: 627 API calls across twelve endpoints, zero failures, six accounts created and all six removed. Every multi-user path exercised — 88 reads of other people's feeds, 87 likes and 37 comments on other people's posts, 35 private messages between drivers.

Run against a local instance of the production schema, so the latency figures are loopback numbers and are deliberately not quoted here. What it establishes is correctness under concurrency, not speed.

The rule that came out of the run — and the one that had silently broken profile editing for every user of a shipped application.

Is this you

You cannot test this alone, and you know it.

Populace is for anyone shipping software where the second user is the problem. If none of the lines below is your product, you do not need this.

You need it if your app has…

A feed, a chat, or anything one person publishes and another reads
Permission rules deciding who may see what
Presence, read receipts, or realtime fan-out
Counters, memberships, or anything two people can write at once
An account-deletion path you have never really exercised

You do not need it if…

Your product is single-player and always will be
You want a load test — this is a correctness tool, and says so
You want to know whether people want your product. It cannot tell you that, and neither can any other tool

Why not just write the scripts yourself?

You can, and for one flow you probably should. The difference shows up at the fourth flow: a script asserts what you already suspected, whereas a population does what people do and finds what you did not think to assert. Nobody told Populace where the bugs were. It found an upsert that could not read its own column — three times, in three different places, months apart — because six strangers signing up at once is a thing a test script never does.

What it costs

Nothing. Free and open source under AGPL-3.0, on npm as @gigzen/populace. No account, no telemetry, no paid tier.

AGPL means if you offer Populace itself as a hosted service, your version stays open. Using it to test your own product — commercial or not — carries no such obligation.

What it costs you in time

One adapter. Thirteen small methods, two of them required, each answering “how does this happen in my app?”

populace init writes a working REST adapter with every line you need to change marked EDIT. If your API speaks HTTP and JSON, that is an afternoon, not a project.

Install it Read the full test report →

That report lists what was not tested, too. If a tool’s own evidence page has no such section, ask why.

How it works

The engine never learns your app.

Populace knows how to be a person: where they are, when they get bored, who they reply to, when they give up on a slow screen. It knows nothing about your software. Everything app-specific lives in one small adapter you write — and if app logic ever leaks into the engine, Populace has collapsed back into a test script.

Write an adapter

One file. Translate “a person did something” into your API calls. Only createUser is mandatory — implement what you have, and Populace reports honestly on what it therefore could not test.

Run a population

Deterministic identities across real cities, each with their own rhythm, patience and habits. Agent n always gets the same phone number, so re-runs reuse accounts instead of piling up new ones.

Read the evidence

What broke, how often, under how many concurrent users, how slow it got — and, crucially, what was never tested at all.

What you connect it to — and what you don't

Point it here

Your API

The same HTTP endpoints, database and auth your real app talks to — your test environment of them. Six agents sign in as six genuinely different users, so the server has to keep them apart.

Why: every bug found so far lived on the server — a feed query filtered to the wrong user, an upsert with no read permission, a delete that left the rows behind. None was visible from the screen.

Not this

Your .apk or .aab

There is no “drop in your build and go”. A compiled binary is a client; driving it means automating taps on screens, which tests the interface rather than what happens when several people use it at once.

Populace works one layer below that, which is why one adapter covers your Android app, your iOS app and your web app together — they all speak to the same API.

The whole integration is one file

This is a real, working adapter — the Supabase one used for the run that found five bugs, trimmed to its essentials. If your backend speaks HTTP, yours looks like this.

// adapters/my-app.mjs — everything Populace needs to know about your app
export function createAdapter(target) {
  const api = target.url, key = target.key;

  return {
    name: "My App",

    // The one method every adapter must have: make a real account.
    async createUser(person) {
      const r = await fetch(`${api}/auth/v1/signup`, {
        method: "POST",
        headers: { apikey: key, "content-type": "application/json" },
        body: JSON.stringify({ phone: person.phone, password: person.password }),
      });
      if (!r.ok) throw new Error(await r.text());   // a real failure, reported once
      return { id: (await r.json()).user.id, token: "…" };
    },

    // Say something publicly. Populace decides WHEN; you decide HOW.
    async post(user, text) { /* one fetch */ },

    // Read the feed — this is the call that exposed the biggest bug.
    async recentPostsByOthers(user) { /* one fetch */ },

    // Delete the account and its data. Runs automatically after every run.
    async deleteUser(user) { /* one fetch */ },
  };
}

Implement createUser and you can run. Add a method, and Populace starts exercising that behaviour; leave one out, and the report names it under NOT TESTED along with what it would have caught. It never quietly appears more thorough than it was.

The 13-method contract

Implement as many as apply. Coverage is reported as a fraction and every gap is named in the report, so a run can never quietly appear more thorough than it was.

Method	What a person is doing	Required
`createUser`	Signing up for the first time	Required
`deleteUser`	Deleting their account and data	Strongly advised
`signIn`	Coming back later — also makes cleanup read-only	Optional
`setProfile`	Filling in who they are	Optional
`reportLocation`	Moving through the city	Optional
`post`	Saying something publicly	Optional
`recentPostsByOthers`	Reading a feed — the multi-user path	Optional
`like` / `comment`	Reacting to somebody else	Optional
`sendMessage` / `inbox`	Private conversation between two people	Optional
`joinGroup` / `groupMembers`	Membership and counts	Optional
`refreshSession`	Staying signed in past token expiry	Optional

The deliverable

A report you can hand to somebody else.

The live simulation is a demo. The report is the product. It is deliberately conservative: anything unproven is called unproven, and there is no tolerance threshold under which a real bug can hide.

  POPULACE REPORT — Buzz
  test · 6 people · 80.4s
  ────────────────────────────────────────────
  ✖ Problems found:
      · 13 of 318 API calls failed (4.1%).

  YOUR API UNDER 6 CONCURRENT USERS
    method              calls fails    p50    p95
  ✖ recentPostsByOthers    24    24   82ms  120ms
      ↳ 24× returns only the caller's own rows
  ✖ setProfile             18     6  204ms  281ms
      ↳ 6× upsert reported success, wrote nothing
    post                   15     0  138ms  190ms
    like                   39     0   96ms  141ms

  NOT TESTED — adapter implements 11/13
    · joinGroup     would have tested membership counts

  ✔ Cleanup complete — 6 accounts removed.

Failures grouped by shape, not exact text — one bug is one line, not fifty near-identical ones.
Latency per endpoint (p50/p95/p99/max) measured under N genuinely concurrent identities.
Coverage stated honestly. Methods you did not implement are listed with what they would have tested.
Cleanup accounted for. If accounts could not be removed, the report says so rather than claiming success.
Machine-readable JSON for CI, plus a self-contained HTML page — no scripts, nothing fetched — for the people who weren't watching the terminal.
Exits non-zero when problems are found, so a pipeline can gate on it.

Built for real infrastructure

It tells you when the network was the problem.

Staging environments sit behind flaky VPNs. CI runners drop sockets. A testing tool that only works on a perfect connection has not been tested yet — and one that blames your API for a dropped packet will be ignored the first time it cries wolf, along with all the real findings that come after it.

Every call has a deadline

One unresponsive endpoint used to freeze an entire run and produce nothing — the worst failure a testing tool can have, because it doesn't look like a failure. It looks like nothing. Now a slow endpoint becomes a timeout in the report, and the other agents carry on.

Transport failures retry. Yours never do.

A dropped socket is retried with jittered backoff. An error your server actually returned — any status at all — is reported exactly once. Retrying a 500 until it passes would turn your bug into a green tick, which is the worst thing a correctness tool can do.

Retries are visible

A call that only worked on its third attempt is not the same as one that worked first time. Retry counts are printed, and latency is measured on the successful attempt alone so retry time can't quietly inflate your p95.

It gives up on a dead target

After a run of consecutive unreachable calls, Populace concludes the host is gone, stops, and says so — in seconds, rather than grinding out the full duration and handing you nothing.

Three verdicts, because two would be a lie

Verdict	Means	Exit
clean	Every call reached your API and none failed.	`0`
problems-found	Your API returned failures. These are findings about your code.	`1`
inconclusive	Your API never failed — but the run couldn't complete, so nothing was proven. Calling this “clean” would be the most damaging lie the tool could tell.	`1`

Safety

It refuses to touch production.

Populace creates real accounts and writes real rows through your real API. Pointed at production it would put invented people in front of paying customers — that is deception, not testing, and it is painful to unpick afterwards. So the guard refuses in three independent ways, and any one of them is enough to stop a run.

Refusal 01

Environment must opt in

The config has to declare a recognised non-production environment. Never assumed, never inferred.

Refusal 02

Explicit denylist

Name your production hosts once in neverRunAgainst and no later edit can point a run at them. No flag overrides it.

Refusal 03

Loud about the gap

An empty denylist is warned about every single run, because “I forgot to fill it in” is the likeliest version of this mistake.

Cleanup is equally deliberate: agents delete themselves after every run, and populace clean finds anything a crashed run left behind. An identity it could not verify is never reported as absent.

Get it

Download Populace

Two ways in, one engine. The desktop application installs like any other Windows program and needs nothing else on the machine — no Node, no npm, no terminal. The command line is the same code for people who would rather type.

Populace Studio for Windows

Five screens over the engine: start a run, watch the population work, read the verdict, ask what a failure means, check for updates. It runs the same populace command a terminal would and shows you the command it ran, so the window can never claim something the command line would not.

Download for Windows Portable — no install

1.0.0 current version 102 MB installer Windows 10/11 64-bit No Node required

Populace Studio showing a finished run: a clean verdict, 797 API calls, no failures,
and a table of every method with its latency.

A real run against Gitea — an application we did not write.

Windows will warn you, and it is right to

Populace Studio is not code-signed yet — a certificate is bought annually and we have not bought one. So Windows shows “Windows protected your PC.” Choose More info → Run anyway, or don't: an unsigned binary from a stranger deserves the suspicion. Which is why the checksums are here.

Check what you downloaded

# PowerShell
Get-FileHash .\PopulaceStudio-Setup-x64.exe

Setup 7583595375c2aec9fa3dfddf2a1704872c7c0c3271e206ea525f96730cad7bdc
Portable 2ca9832f665ef85ffd2ba81b499f0e2a31257c006d2ccf787cdbebf271e6952b

Or use the command line. Node 18 or newer, and zero runtime dependencies — adapters bring their own, the engine brings none. It runs on macOS and Linux too; the desktop build is Windows-only for now.

Install it

# from npm
npm install -g @gigzen/populace

# see it work against a fake app, no setup
populace demo

# point it at your own app
populace init
populace doctor
populace smoke
populace run --agents 6 --minutes 5

demo runs the whole product against an in-memory app, so you can see exactly what a report looks like before writing a line of adapter code.

Other ways

Download a ZIP

No git required — populace-main.zip. Unzip, then node src/cli.mjs demo.

From source

git clone https://github.com/Shakhtar-Sankur/populace.git, then node src/cli.mjs demo. Same code, no install.

Check what you installed

The scope matters: it is @gigzen/populace. An unrelated package called populace exists and is not this.

Windows

A PowerShell runner ships in examples/buzzbuzz/run-test.ps1 — the Unix one-liner does not work in PowerShell, so there is a native one.

Reference

Commands and configuration

Command	What it does
`populace demo`	Run the whole product against a fake app, immediately
`populace init`	Scaffold a config and a blank adapter
`populace doctor`	Check config, reachability and coverage without running
`populace smoke`	Prove your adapter works — one user, every method once, seconds not minutes
`populace run`	Bring the population to life
`populace clean`	Delete every account a run created
`populace report`	Re-open the report from an earlier run
`populace version`	Version and environment, for bug reports

populace.config.mjs

export default {
  app: "My App",
  adapter: "./adapters/my-app.mjs",

  // Populace only runs against non-production environments, and it checks.
  environment: "test",

  // The safety net that matters most. No flag overrides this.
  neverRunAgainst: ["https://api.myapp.com"],

  // Resilience — see the section above for why each exists.
  timeoutMs:   20000,  // give up waiting on one call
  retries:     3,      // extra tries for calls that never landed
  giveUpAfter: 12,     // unreachable calls before stopping the run

  population: { agents: 8, cities: ["manila", "mumbai"], minutes: 10 },
}

Use it from CI

import { simulate } from "@gigzen/populace";

const report = await simulate({
  configPath: "./populace.config.mjs",
});

if (report.verdict.status !== "clean") {
  process.exit(1);
}

What it will not tell you

Whether people want your product. These people are generated from patterns. They will never surprise you the way a real customer does, they cannot tell you your onboarding is confusing or your pricing is wrong, and they are least accurate for exactly the users least represented online.

Use Populace to prove your app works. Use real people to decide what to build. A report full of green ticks means your API held up — not that anyone wants what you made.