Context
Large open-source contribution events struggle with fair, scalable PR evaluation. Manual difficulty tagging (L1/L2/L3) is subjective, inconsistent, and vulnerable to bias.
Goal
Design and deploy a fully automated, deterministic PR scoring engine capable of evaluating thousands of pull requests across dozens of repositories without manual intervention.
Engineering highlights
✨ Evaluated 14,500 Pull Requests
Processed and deterministically scored over 14,500 merged PRs during the event without manual intervention.
✨ Configured Across ~100 Repositories
Deployed as a GitHub App installed across nearly 100 independent open-source projects, enabling centralized yet repository-isolated scoring.
✨ Handled 2,500+ Contributors
Supported large-scale participation with thousands of contributors submitting PRs concurrently across multiple repositories.
✨ Deterministic & Auditable Scoring (No AI)
Implemented fully rule-based scoring using objective signals (LOC, files changed, density, test additions, structural impact), ensuring transparency and zero hidden bias.
✨ Idempotent & Replay-Safe Processing
Ensured each PR is scored exactly once, preventing duplicate webhook deliveries or retries from inflating contributor scores.
Impact
- 100+ repositories integrated.
- 2,500+ contributors evaluated.
- 14,500 merged PRs processed and scored.
- Zero manual difficulty assignments required.
Outcomes
Transformed PR evaluation from a subjective, manual bottleneck into a deterministic, scalable, and fully auditable system — restoring fairness and trust across a large open-source event.
Developer Tooling • Open Source Infrastructure • Event Systems
ECWoC Sentinel – Fair & Transparent PR Scoring System
Design and deploy a fully automated, deterministic PR scoring engine capable of evaluating thousands of pull requests across dozens of repositories without manual intervention.
Event-driven backend system using GitHub webhooks, Redis-backed job queues, and a deterministic rule engine to score merged PRs across 100+ repositories.