input
{
"file_path": "/tmp/readme-541235c.md"
}result
1 # Track 03: Engineering & Delivery
2
3 Go implementation of all three assignment questions — the DAO layer (Q1), the REST search/retrieve API with authentication (Q2), and the third-party identity connector (Q3) — plus the glue between Q2 and Q3. See root `../CLAUDE.md` for the full assignment text and how this track fits the other four; this file covers only what this track built, what's stubbed, and how.
4
5 ## What's implemented — live, deployed, verified against the running system
6
7 - **Q1 — DAO layer** (`internal/dao`, `internal/dao/postgres`, `internal/dao/sqlite`): `user_profile`/`user_credential` storage behind one `Repository` interface, real implementations for Postgres/CockroachDB and SQLite. Verified against the actually-deployed Postgres database, and in CI against real Postgres and CockroachDB service containers (not mocked) — a cross-backend conformance suite (LT-38) asserts all three backends behave identically, not just that their Go signatures match.
8 - **Q2 — REST API** (`internal/api`, `internal/app`): `POST /profiles/search`, `GET /profiles/{id}`, and a real RFC 6749 client-credentials `POST /auth/token` grant, all behind RS256 JWT verification. Verified live at `https://loginid-takehome.uplifttech.org` with real minted tokens, including rate-limiting, masked-vs-full-read scoping, and pagination.
9 - **Q3 — connector** (`internal/connector`, `cmd/idp-connector`): the third-party IDP connector's own inbound `/auth`/`/identity` contract. Verified in-cluster via `kubectl port-forward` with a real vendor stub backing it — happy path, audience rejection, fail-closed identical-content error responses, a timing floor on the three internal failure branches, and a failure-ratio rate limiter actually firing under a live failed-auth burst, not just asserted in a unit test.
10 - **api-service → idp-connector integration** (`internal/onboarding`, LT-33): api-service's own outbound call to the connector, closing the loop between Q2 and Q3 — its own cached, refreshed OAuth2 client-credentials token; bounded timeouts and retry-with-backoff on connector calls; a generic upward failure on any connector-unreachable condition, with own-token-acquisition failures logged as a distinct class.
11 - CI/CD, containerization, secrets delivery, and database provisioning (04's stories, LT-45/46/47/52) are all Done and live — `pr-check` runs this track's tests against real Postgres and CockroachDB service containers on every PR, not just SQLite.
12
13 ## Deliberately stubbed — stated plainly, not left to infer
14
15 - **The third-party vendor (ABC/XYZ) is an in-process `StubVendorClient`**, not a real HTTP integration. `idp-connector` falls back to it whenever `IDP_ABC_BASE_URL` is unset, and says so explicitly on startup. The assignment names the vendor generically — a real integration was never in scope (`refinement/LT-41.md`'s own stated non-goal).
16 - **The `authorize()` object-level policy hook is a `StopgapAuthorizer`**, a hard-coded, default-deny stand-in — not the real caller-referral datastore `../02-ai-security-architecture/handoff-03-auth.md`'s own design describes. A stated seam, not a hidden gap.
17 - **CockroachDB and SQLite are exercised in CI only, never deployed live.** Only Postgres runs in the actual cluster (LT-52's explicit scope limit) — don't go looking for a running CockroachDB instance anywhere.
18 - **No `POST /profiles` write endpoint exists or is planned.** Q2 is search/retrieve by the assignment's own text; a write path was ruled out as scope creep. Review/demo data enters via a version-controlled, idempotent SQL file (`deploy/demo-data/001-review-profiles.sql`), applied as a documented runbook step, never through the API.
19 - **Rate limiting, the touch counter, and the token issuer's grant limiter are in-process and single-replica-only.** Their own code comments say a shared store (Redis) would be needed for a genuine multi-instance deployment — this project runs `api-service`/`api-service-issuer` at 1 replica with a `Recreate` rollout strategy *specifically* so this simplification holds correctly rather than silently degrading under a rolling update. That constraint exists because of a real bug this project found live (see below), not as a hypothetical caveat.
20 - **The retention sweep (LT-44) is designed and enforced at the schema/DAO level, but not yet operationally running.** `DeleteExpired`/`DeleteProfile` exist on the DAO composite and are tested there directly; nothing yet schedules or calls them against the live database. `../05-data-ops/pii-governance.md`'s retention policy is a real, tracked, backlogged story, not an abandoned requirement.
21
22 ## Package layout
23
24 `cmd/api-service` and `cmd/idp-connector` are both thin wrappers — argument parsing and `main()` only. Route registration and config loading live once, in `internal/app` and `internal/config`, shared by both binaries rather than duplicated per binary (see `decisions/go-layout-debate.md` below). `internal/api` holds Q2's HTTP handlers, JWT middleware, rate limiting, and the token issuer; `internal/connector` holds Q3's inbound handlers and outbound vendor client; `internal/onboarding` holds LT-33's mediating client from `api-service` to the connector; `internal/dao` and its `postgres`/`sqlite` subpackages hold Q1, behind one `Repository` interface `internal/dao/conformance` tests identically across backends.
25
26 ## Test strategy
27
28 Unit-heavy; integration deliberately stubbed where the assignment itself doesn't require full end-to-end coverage. Hand-written interface fakes, not generated mocks, for DAO/handler tests (`decisions/test-double-strategy.md`) — small enough interfaces that a fake struct per test case beats adopting a codegen dependency at this project's scale. The rule that any change to `dao.Repository` (or its sub-interfaces) updates its hand-written fake in the same PR is now a literal checklist line in `.github/pull_request_template.md`, not just prose in a design doc — LT-42's own review found the design doc alone insufficient (a PR landed an interface change without updating its fake, caught only in a later pass), so the checklist line is the actual enforcement site now, and the design doc explains why. Real-backend correctness (Postgres/CockroachDB/SQLite conformance, the CockroachDB serialization-retry path) is proven against real databases, in CI, not mocked — `internal/dao/conformance`'s own suite is the one enforcement site for "do the backends behave alike," distinct from a signature-matching test that proves nothing about behavior.
29
30 ## AI tooling and workflow
31
32 Every implementation story in this track's backlog runs at `sonnet, high` except the connector (`idp-connector`'s `/auth`/`/identity`, Q3), which runs at `opus, high` — the one seat above sonnet in this track, per `../02-ai-security-architecture/planning-approach.md`'s connector-token-lifecycle exception: a subtle bug in that specific code path (a leaked or misscoped vendor token) directly contradicts this project's own threat model in a way most of this track's other code doesn't.
33
34 Beyond model selection, implementation decisions in this track ran through structured adversarial review rather than being decided unilaterally:
35 - `decisions/go-layout-debate.md` — a three-hats debate (optimist/pessimist/pragmatist) on one binary vs. two, ruling to keep two binaries (already published and built against by 04) but merge the *authoring* cost via a shared `internal/app` package, plus a concrete requirement (bounded timeouts, no tight-looping retries on outbound calls) adopted from the pessimist's production-incident pattern-matching.
36 - `decisions/error-semantics.md` — a rotating devil's-advocate pass (Oren Castellan) that closed the sentinel-error set, forced a translation boundary so wrapped DB errors never leak driver text to an API client, and added explicit handling for `context.DeadlineExceeded`/`Canceled`.
37 - `decisions/context-propagation.md` — a second rotating devil's-advocate pass (Nolan Reyes, drawing on production-incident pattern-matching) that scoped multi-statement writes to a transaction under context cancellation, and named which write paths deliberately detach from the inbound context for durability.
38 - `decisions/test-double-strategy.md` — a third rotating devil's-advocate pass (Marisol Ferran) that weighed hand-written fakes against generated mocks; hand-written fakes stood, with the interface-drift risk closed procedurally (the PR-template checklist line above), not by adopting a codegen tool.
39
40 This project is also a broader org-orchestration exercise — a PM agent coordinating five track leads, each running a small hired sub-team with deliberately varied personalities, using patterns like adversarial pairs, red/blue, and structured written debate (`../PLAN.md`, `../CASTING.md`) — the four decisions above are this track's own instance of that same structure, not a separate practice invented for engineering work alone.
41
42 ## Real, live-tested findings
43
44 This track's own delivery process included live acceptance review against the actually-deployed system, not only code review and unit tests — and that process found and helped fix four real, distinct production-shaped bugs, each named individually rather than lumped together:
45
46 1. **A CI coverage gap**, where `pr-check` silently *skipped* every Postgres and CockroachDB conformance test — only SQLite genuinely ran — while still reporting an overall green build. Fixed two ways: real Postgres/CockroachDB service containers added to the CI job, and every affected test fixture changed to fail loudly (not skip) when a required backend DSN is absent specifically in CI.
47 2. **A cross-replica rate-limiting bug** in `api-service`'s own resource-API rate limiter, reproduced by actually crossing its 10 req/min cap live against 2 running replicas — each replica's in-process limiter had no visibility into the other's request count, so the effective cap was silently doubled. Fixed by moving `api-service`/`api-service-issuer` to a single replica with a `Recreate` rollout strategy (04's story), a deliberate scope limit stated plainly above, not a silent workaround.
48 3. **`api-service`'s own Q2 token issuer/verifier wiring shipped with `AUTH_JWT_ISSUER`/`AUTH_JWT_AUDIENCE` unset** in its Kubernetes manifest — minted tokens carried no `iss` and were checked against an empty `aud`, "verifying" successfully for the wrong reason rather than failing loudly. Fixed in the manifest, plus a startup check (`cmd/api-service`) that now refuses to start with either unset, and a URL-shape check on the JWKS/issuer values specifically (a non-empty but garbage value is caught too, not just an absent one).
49 4. **A separate, later total outage in `idp-connector`'s (Q3's) own inbound auth** — its Deployment manifest never set `AUTH_JWKS_URL` at all, so every request, including a genuinely valid token, failed identically with a JWKS-fetch error. Distinct from finding 3: a different service, a different missing variable, found in a separate in-cluster review pass. Fixed the same way: the manifest, plus a startup check in `cmd/idp-connector` itself, plus a systemic test (`internal/config`'s manifest-vs-config test) that now parses every real Kubernetes Deployment manifest and asserts it sets everything its own mode requires — closing this specific class of gap structurally, in CI, rather than relying on a fourth live incident to catch the next occurrence.
50
51 Naming these precisely — which subsystem, which variable, which fix — is itself part of what this AI-workflow note is meant to demonstrate: verification against the live, deployed system, not only against a test suite or a design doc.
52