EstablishedStrong evidence

Experimentation and metrics

Defines success metrics and runs A/B tests to decide whether a launch actually moved the product.

Also called: A/B testing, experimentation, product metrics, KPIs, success metrics

How the score was computed

News signal0.31: News signal 0.31: 2 verified article(s) mention this skill, weighted by how recent they are, source type, and evidence level (0.93 of 3 needed for a full signal). Strong means at least 0.4.
JD signal56%: JD signal 0.56: 9 of 16 job descriptions require it and 0 prefer it (preferred counts 0.5). Strong means at least 35%.
Momentum+0.83: Momentum +0.83: 1 article(s) in the last 30 days vs 1 in the 180 days before, compared per day.

In the news 4

Emerging trends in AI agent evaluation
AI as Normal Technology · Blog · Oct 19, 2026
“This work requires both technical and sociotechnical evaluation”
Strong evidenceAI-generated · Shift Extraction
Evaluation of agents is expanding beyond lab benchmarks to in-context and sociotechnical studies of how agents affect human work, with real-world usage data and open-world evaluations.
Coordinate product launches with Datadog
Datadog · Primary source · Sep 8, 2026
“connects launch planning, instrumentation, experimentation, user experience analysis, and reporting together”
Strong evidenceAI-generated · Shift Extraction
Datadog Product Analytics now has a Launches workflow that links launch planning, instrumentation, experimentation, user experience analysis and reporting in one place.
Coordinate product launches with Datadog
Datadog · Primary source · Sep 8, 2026
“define what they need to measure before release”
Strong evidenceAI-generated · Shift Extraction
Teams can define launch questions and a tracking plan before release, with instrumentation changes reviewed in a pull request before rollout starts.
Coordinate product launches with Datadog
Datadog · Primary source · Sep 8, 2026
“A rollout can begin with employees or a small percentage of traffic”
Strong evidenceAI-generated · Shift Extraction
Rollouts use feature flags for progressive exposure, starting with employees or a small share of traffic, and run through experiments comparing treatment and control.

In job postings 9

Anthropicrequired
Technical Program Manager, Inference · Mid-Senior (7+ yrs) · captured Oct 27, 2026
“define the productivity metrics that matter, find the bottlenecks”
CoreWeaverequired
Senior Technical Program Manager - Fleet Delivery & Capacity Management · Senior · captured Oct 27, 2026
“define meaningful metrics, and guide dashboard development”
Nurorequired
Lead Technical Program Manager, AI Platform · Lead (people manager) · captured Oct 27, 2026
“Use metrics to identify bottlenecks and improve reliability”
Figmarequired
Technical Program Manager - AI Tooling · Mid (5+ yrs) · captured Oct 27, 2026
“define measurable outcomes and provide clear reporting on delivery health”
Coinbaserequired
Technical Program Manager, CX Intelligence & AI · Senior (10+ yrs) · captured Oct 27, 2026
“define north-star success metrics, and set quarterly targets for shared KRs”
Airbnbrequired
Senior Manager, Technical Program Management (Search & AI Foundations) · Senior Manager · captured Oct 27, 2026
“Familiar with A/B testing, incremental delivery and deployment.”
Databricksrequired
Staff Technical Program Manager- Security · Staff · captured Oct 27, 2026
“Define and track security metrics (KPIs, OKRs)”
Waymorequired
Technical Program Manager - Onboard Systems, ML Modeling · Mid (5+ yrs) · captured Oct 27, 2026
“key performance indicators (KPIs) to leadership and stakeholders”
Relativity Spacerequired
Staff Technical Program Manager (Hardware) · Staff · captured Oct 27, 2026
“Establish goals and metrics to monitor program performance”

Waiting for an editor (not counted yet)

Coordinate product launches with Datadog: Bits AI drafts the launch context, measurement questions, tracking plan and experiment steps for teams to edit.
- Extractor confidence 0.55 is below 0.7
Coordinate product launches with Datadog: Launches shows Session Replay and RUM data for users exposed to the flag, broken down by device, screen size or country, to catch segment-specific defects.
- Extractor confidence 0.65 is below 0.7