A US B2B SaaS, post-Series A, had skipped formal QA in the first two years to ship fast — a common pattern that worked until it didn't. Regression bugs were now reaching production weekly, customer NPS was dropping, and two enterprise customers had threatened to churn over reliability issues. The CTO needed a senior QA engineer to build out automation from zero — Playwright + GitHub Actions framework, CI/CD integration, plus a culture shift in the engineering team toward test-first development. The role was IC, not management, but the senior had to bring the existing six engineers along through pairing and code review.
QA hiring is high-volume but low-precision because the role has the widest skill variance of any engineering function. The pool ranges from manual testers who use 'QA engineer' on their resume to senior automation architects who have built test frameworks at multiple companies. The client's challenge was twofold: build the framework AND change the engineering culture so the framework would actually be used. The first is a technical skill; the second is a leadership skill. Most senior QA engineers have one but not both. Engineers who can build a framework but can't get the team to adopt it produce expensive shelfware; engineers who can lead culture change but can't build the framework miss the technical bar. The client had hired a QA manager 14 months earlier who had the leadership skills but not the technical depth — the engagement had ended without a working framework. They didn't want to repeat that mistake.
The funnel had to be wide (because QA-claiming resumes are common) and the filter aggressive (because the dual skill set — build + lead — is rare). 3,188 applicants reached the funnel from our internal database plus three external sources. AI scoring weighted heavily on greenfield-framework-build signal: candidates who had verifiably built a test framework from scratch at a prior role, not just used one. 940 passed the AI score. Senior recruiter screen took that to 220 by verifying greenfield builds through reference calls (so we knew the candidate had owned the framework, not just contributed to it). 85 of 220 completed a 4-hour written assignment: design a Playwright test framework for a small SaaS codebase including CI integration, flake-handling, and a written strategy for getting engineers to write tests. The strategy section was the harder cut — most candidates wrote technical specs but couldn't articulate culture-change tactics. 35 cleared on both dimensions. Live coding (90-minute) focused on debugging flaky tests under time pressure. 18 cleared the technical bar. Cultural interview (60-minute, run by our senior recruiter who'd previously been an engineering manager) probed culture-change aptitude: how do you get a skeptical senior engineer to write a test for their PR? How do you handle pushback when the framework slows down PR cycle time in week one? 12 cleared. The CTO ran final rounds with all 12. The winning candidate (Pune-based, 9 years experience, previously at two SaaS scaleups where she'd built greenfield QA frameworks) had specific stories of converting skeptical engineers — and the references confirmed it.
Offer day 14 at top-of-band. Accepted within 48 hours. Started day 18. The engineer built out the Playwright framework over the first 90 days with a deliberate adoption strategy: she paired with each of the six existing engineers on writing their first test, sat in on every code review, and ran a weekly 'flake clinic' to drain the regression backlog. Automation coverage on the critical user flows hit 90% by day 90 — significantly above the 60% target the CTO had set. Regression bugs dropped 65% in the next quarter; the two enterprise customers who'd threatened churn signed renewal contracts. Customer NPS rose 12 points over six months. The engineer is still with the team 13 months in; converted to full-time at month seven with a Staff QA title. The engineering team's test-first culture (a phrase the CTO had been trying to instill for 18 months) actually emerged after the new hire's first 90 days. Two more engineering hires through us in the following six months (a Senior Backend and a Staff DevOps).
Two-hour call with the CTO. Reviewed the three engineering anti-patterns that had created the regression backlog (no integration tests, no PR-gate on staging deploys, no flake-tracking). Locked our screening rubric.
QA is a high-volume role with wide skill variance. AI scoring filtered hard on greenfield-framework-build signal — not 'has used Selenium' but 'has built a test framework from scratch and rolled it out across an engineering team.'
85 candidates completed a 4-hour assignment: design a Playwright test framework for a small SaaS codebase including CI integration, flake-handling, and a strategy for getting engineers to write tests. 35 cleared.
Top 12 went to the CTO for final rounds. Selected for both technical depth and culture-change aptitude. Offer day 14. Started day 18. Automation coverage hit 90% within 90 days of start.
For role intersections that combine technical depth with culture-change leadership, the screening has to be explicit about both dimensions. Most QA searches we see fail because the rubric tests only the technical skill, then the hire builds shelfware. We now require a written culture-change section in the assignment for any role where adoption matters more than build (QA, DevOps platform engineering, design systems). The second lesson: pairing as an adoption strategy beats edict every time. The hire's most effective lever in this engagement wasn't the framework she built; it was the time she spent pairing with each existing engineer on their first test. Skeptics convert through doing, not through being told.