Tidox Radar

LLM Evaluation & Benchmarking

Frameworks for evaluating, red-teaming, and benchmarking LLMs and agent pipelines. · 166 items · 13 this week ▼ 3 · 0 new this week

  1. 1
    ninjahawk/livenerf
    Benchmark for tracking model capability after release
    GitHub · ★ 1,646 · Python · first seen 2026-09-22 · hn · reddit
    heat 13.2
    3 / 25 mentions
  2. 2
    bottomless/agent-duel ▲ rising
    Desktop app for blind comparisons between coding agents.
    GitHub · ★ 30 · TypeScript · first seen 2026-10-08 · hn
    heat 8.4
    7 / 10 mentions
  3. 3
    infrixo-systems/rag-evaluation-starter
    Evaluate RAG pipelines against golden sets with multiple metrics.
    GitHub · ★ 1 · Python · first seen 2026-10-24 · hn · reddit
    heat 4.8
    0 / 5 mentions
  4. 4
    cloudishai/ai-deploy-bench ▲ rising
    Benchmark AI agents building and deploying web apps across platforms.
    GitHub · ★ 16 · Python · first seen 2026-10-14 · hn · reddit
    heat 4.6
    3 / 6 mentions
  5. 5
    imxinchengyou/cacheverifier
    Semantic cache hit verification research with conformal control.
    GitHub · ★ 2 · Python · first seen 2026-07-30 · hn · reddit
    heat 3.2
    0 / 11 mentions
  6. 6
    gauge-sh/gauge-cli
    Open-source CLI for hillclimbing agent skills, docs, and MCP servers.
    GitHub · ★ 10 · TypeScript · first seen 2026-10-20 · hn
    heat 1.8
    0 / 2 mentions
  7. 7
    coverage-cat/consumer-agent-testing-tool
    Surrogate testing framework for evaluating consumer AI agents and APIs.
    GitHub · ★ 14 · Python · first seen 2026-10-22 · hn
    heat 1.7
    0 / 2 mentions
  8. 8
    la5u/parsimony
    Measures how much code AI coding agents add from benchmarks.
    GitHub · ★ 13 · Python · first seen 2026-10-10 · hn · reddit
    heat 1.7
    0 / 3 mentions
  9. 9
    facebookresearch/swe-sweep
    Benchmark measuring how many bugs LLMs find and fix in codebases.
    GitHub · ★ 214 · Python · first seen 2026-10-01 · hn · reddit
    heat 1.5
    0 / 2 mentions
  10. 10
    slee-persis/gvs5h
    Five Qwen models matching Claude performance on coding benchmarks.
    GitHub · ★ 1,157 · Python · first seen 2026-08-28 · hn · reddit
    heat 1.4
    0 / 6 mentions
  11. 11
    yakoub-droid/scientific_logic_auditor-
    Web platform benchmarking prompts across multiple LLMs with an AI judge evaluator.
    GitHub · ★ 1 · first seen 2026-10-30 · reddit
    heat 1.3
    0 / 2 mentions
  12. 12
    tarunagarwal1981/wobbly
    Metamorphic testing framework for finding wrong LLM answers without answer keys.
    GitHub · ★ 6 · Python · first seen 2026-10-04 · reddit
    heat 1.2
    0 / 2 mentions
  13. 13
    uka-135/deepseek-long-thread-record
    Pseudonymised six-day DeepSeek conversation record archived on Zenodo.
    GitHub · first seen 2026-11-01 · hn
    heat 1.1
    0 / 1 mentions
  14. 14
    bmiedlar/nullprime
    Dataset testing if AI models know when they cannot judge primality.
    GitHub · ★ 1 · Python · first seen 2026-10-31 · hn
    heat 1.0
    0 / 1 mentions
  15. 15
    google-deepmind/gbench
    Evaluation and serving performance benchmark for Gemma and open LLMs.
    GitHub · ★ 4 · Python · first seen 2026-10-31 · hn
    heat 1.0
    0 / 1 mentions
  16. 16
    tianjun588/llm-web-scraping-benchmark
    Live web scraping and proxy benchmark for LLMs and RAG pipelines.
    GitHub · ★ 2 · Python · first seen 2026-10-31 · hn
    heat 1.0
    0 / 1 mentions
  17. 17
    facebookresearch/programbench
    Benchmark testing whether language models can rebuild programs from scratch.
    GitHub · ★ 979 · Python · first seen 2026-10-30 · hn
    heat 1.0
    0 / 1 mentions
  18. 18
    navitallabs/navital-gym
    Auto-graded benchmark environments for AI agents on Indian-reality tasks.
    GitHub · ★ 2 · Python · first seen 2026-10-30 · hn
    heat 1.0
    0 / 1 mentions
  19. 19
    matheus-c-feitosa/lowbit-atlas
    Low-bit LLM CPU compatibility and performance benchmark atlas
    GitHub · ★ 2 · Python · first seen 2026-10-19 · hn · reddit
    heat 0.9
    0 / 2 mentions
  20. 20
    keerthirajan58/bugbout
    Benchmark to compare which coding agent best fixes bugs in your codebase.
    GitHub · ★ 2 · Python · first seen 2026-10-29 · hn
    heat 0.9
    0 / 1 mentions
  21. 21
    hemanth1999k/mistrial
    Reliability testing tool for flaky LLM judges across runs and rubrics.
    GitHub · ★ 7 · Python · first seen 2026-10-27 · hn
    heat 0.8
    0 / 1 mentions
  22. 22
    psinglap/websearch-api-eval
    Web search API evaluation and benchmarking for AI agents.
    GitHub · HTML · first seen 2026-10-27 · hn
    heat 0.8
    0 / 1 mentions
  23. 23
    assemblyai/voice-standard-bench
    Open-source benchmarking framework from AssemblyAI.
    GitHub · ★ 7 · Go · first seen 2026-10-14 · hn · reddit
    heat 0.7
    0 / 2 mentions
  24. 24
    princejain756/agent-trials
    Interview your coding agent with sandboxed task suite and scoring.
    GitHub · ★ 2 · TypeScript · first seen 2026-10-14 · hn · reddit
    heat 0.7
    0 / 2 mentions
  25. 25
    0xguenther/agent-write-path-runs
    Benchmark dataset for AI agent reliability with timeout failures.
    GitHub · ★ 1 · JavaScript · first seen 2026-10-07 · hn
    heat 0.7
    0 / 2 mentions
  26. 26
    google-deepmind/detectorscope
    Build human-auditable AI text detectors with mechanistic grounding.
    GitHub · ★ 302 · Python · first seen 2026-10-24 · hn
    heat 0.7
    0 / 1 mentions
  27. 27
    soniqo/voice-tau2-bench
    Voice agent benchmark with local speech inference on Apple Silicon.
    GitHub · Python · first seen 2026-11-01 · reddit
    heat 0.7
    0 / 1 mentions
  28. 28
    danial-ui6/reythun-iaa-sample-
    Quality report methodology for a 200-item RLHF harmlessness preference dataset.
    GitHub · first seen 2026-10-23 · hn
    heat 0.7
    0 / 1 mentions
  29. 29
    deepsense-ai/eda-benchmark
    Benchmark for evaluating language models.
    GitHub · ★ 18 · Python · first seen 2026-10-23 · hn
    heat 0.7
    0 / 1 mentions
  30. 30
    mohsen1/chicken-bench
    Benchmark comparing decision APIs playing Crossy Road.
    GitHub · ★ 1 · Python · first seen 2026-10-23 · hn
    heat 0.7
    0 / 1 mentions
  31. 31
    node-q/hbench
    Benchmark runner for AI coding agent harnesses.
    GitHub · ★ 10 · Rust · first seen 2026-10-31 · reddit
    heat 0.7
    0 / 1 mentions
  32. 32
    mauricekleine/nonobench
    LLM reasoning benchmark built on nonogram puzzles with 80 models.
    GitHub · ★ 10 · TypeScript · first seen 2026-10-04 · reddit
    heat 0.6
    0 / 2 mentions
  33. 33
    sauersml/watching
    Experiments on language models detecting read-only activation probes.
    GitHub · ★ 65 · Python · first seen 2026-10-20 · hn
    heat 0.6
    0 / 1 mentions
  34. 34
    two-delta/evoweb
    Benchmark for coding agents on evolving web applications.
    GitHub · ★ 13 · Python · first seen 2026-10-20 · hn
    heat 0.6
    0 / 1 mentions
  35. 35
    segmently-ai/context-recall
    Benchmark measuring how well context tools surface needed docs for coding agents.
    GitHub · ★ 1 · Python · first seen 2026-10-28 · reddit
    heat 0.6
    0 / 1 mentions
  36. 36
    aipdv/fightbench
    Replay-verified benchmark for AI decision models using Mortal Kombat.
    GitHub · ★ 2 · Python · first seen 2026-10-19 · hn
    heat 0.6
    0 / 1 mentions
  37. 37
    voygr-tech/sf-from-memory
    Benchmark of six LLMs drawing San Francisco from memory
    GitHub · Python · first seen 2026-10-19 · hn
    heat 0.6
    0 / 1 mentions
  38. 38
    ibragim-bad/evals-from-sessions
    Turn coding agent sessions into private evals and cost comparisons.
    GitHub · ★ 77 · Shell · first seen 2026-10-18 · hn
    heat 0.5
    0 / 1 mentions
  39. 39
    fstandhartinger/jevbench
    Benchmark and leaderboard for AI decision models measuring accuracy and calibration
    GitHub · ★ 326 · Python · first seen 2026-09-22 · hn · reddit
    heat 0.5
    0 / 5 mentions
  40. 40
    yadu9989/memstrata-lme500-harness
    Model-selectable LongMemEval-S verifier with immutable data and explicit scoring.
    GitHub · Python · first seen 2026-10-16 · hn
    heat 0.5
    0 / 1 mentions
  41. 41
    tako88/coding-bench
    Benchmark with five hard coding tasks for LLM evaluation.
    GitHub · ★ 1 · Python · first seen 2026-10-24 · reddit
    heat 0.5
    0 / 1 mentions
  42. 42
    blackwaltz0818/ai-assistant-accuracy-test
    Test if paid AI assistants answer small company questions correctly.
    GitHub · Python · first seen 2026-10-02 · hn
    heat 0.5
    0 / 2 mentions
  43. 43
    namanxdev/rag-eval-harness
    RAG evaluation harness measuring retrieval quality on legal contract queries.
    GitHub · ★ 2 · Python · first seen 2026-10-02 · hn
    heat 0.5
    0 / 2 mentions
  44. 44
    zie1ony/jev-talks
    Experiment checking Jev's talking skills
    GitHub · ★ 6 · Python · first seen 2026-09-30 · hn
    heat 0.4
    0 / 2 mentions
  45. 45
    dogukanurker/benchkit
    Benchmark local LLMs with real evaluation suites and actual scores
    GitHub · ★ 35 · Python · first seen 2026-10-22 · reddit
    heat 0.4
    0 / 1 mentions
  46. 46
    jordimarsal/ggufbench
    Agentic benchmark for local GGUF models
    GitHub · Python · first seen 2026-10-22 · reddit
    heat 0.4
    0 / 1 mentions
  47. 47
    composo-ai/dishwasherbench
    Dishwasher-loading game to benchmark language models in browser.
    GitHub · ★ 73 · TypeScript · first seen 2026-10-13 · hn
    heat 0.4
    0 / 1 mentions
  48. 48
    mr-september/artificial-analysis-weights
    Artificial Analysis Intelligence Index vs parameter count for local LLMs.
    GitHub · ★ 2 · JavaScript · first seen 2026-10-21 · reddit
    heat 0.4
    0 / 1 mentions
  49. 49
    bespokelabsai/nimble
    Local typed decisions, contrastive data curation, and model evaluation.
    GitHub · ★ 2,275 · Python · first seen 2026-09-21 · hn
    heat 0.4
    0 / 3 mentions
  50. 50
    abduldattijo/ai-slop-detection-reproduction
    Reproducible code for detecting AI-written movie reviews experiment.
    GitHub · ★ 4 · Python · first seen 2026-10-12 · hn
    heat 0.4
    0 / 1 mentions
  51. 51
    zhulinchng/jevper
    TypeSafe Jev-shaped classification wrapper over OpenAI-like clients.
    GitHub · ★ 19 · Python · first seen 2026-09-23 · hn
    heat 0.4
    0 / 3 mentions
  52. 52
    alitenes2020-sys/repository-name-llama-hallucination
    Benchmark comparing hallucination versus abstention in open-weight LLMs.
    GitHub · Python · first seen 2026-10-11 · hn
    heat 0.4
    0 / 1 mentions
  53. 53
    kyle-pena-nlp/beelinebench
    Benchmark measuring decision model navigation efficiency.
    GitHub · ★ 3 · Python · first seen 2026-10-11 · hn
    heat 0.4
    0 / 1 mentions
  54. 54
    vigneshm1995personal/skillgate
    Evaluate agent skills and compare models before deployment
    GitHub · ★ 1 · Python · first seen 2026-10-19 · reddit
    heat 0.4
    0 / 1 mentions
  55. 55
    unprofane/scunthorpe-test
    Profanity filter benchmark where blocking Scunthorpe counts against you.
    GitHub · TypeScript · first seen 2026-10-17 · reddit
    heat 0.3
    0 / 1 mentions
  56. 56
    balasubramaniyan17/experimentguard
    A/B test analyzer that validates results and identifies issues.
    GitHub · ★ 2 · Python · first seen 2026-10-08 · hn
    heat 0.3
    0 / 1 mentions
  57. 57
    edgedelta/project-arena
    Kubernetes incident benchmark for testing AI investigation tools.
    GitHub · ★ 13 · Python · first seen 2026-10-08 · hn
    heat 0.3
    0 / 1 mentions
  58. 58
    felladrin/cat-goric-game-benchmark
    Cat Goric game played by classifier LLM with harness bridge.
    GitHub · JavaScript · first seen 2026-10-16 · reddit
    heat 0.3
    0 / 1 mentions
  59. 59
    sunkenintime/which-ai
    Benchmark comparing AI model design and UI capabilities.
    GitHub · ★ 140 · TypeScript · first seen 2026-10-16 · catalog
    heat 0.3
    0 / 1 mentions
  60. 60
    sweduel/swe-duel
    Ranks frontier coding models through adversarial Red-Blue duels.
    GitHub · ★ 7 · Python · first seen 2026-10-16 · reddit
    heat 0.3
    0 / 1 mentions
  61. 61
    viridianaiqualityassurance/viridian-assurance-research
    Research on AI evaluation validity and benchmark reliability.
    GitHub · Python · first seen 2026-10-07 · hn
    heat 0.3
    0 / 1 mentions
  62. 62
    ggz14/betterbench
    Real-world percentile benchmark for LLM inference servers.
    GitHub · ★ 76 · Python · first seen 2026-08-06 · reddit
    heat 0.3
    0 / 2 mentions
  63. 63
    mupt-ai/self-bench
    Benchmark coding agents on private repository evals.
    GitHub · ★ 34 · TypeScript · first seen 2026-08-14 · hn
    heat 0.3
    0 / 2 mentions
  64. 64
    alexellis/rigmark
    Benchmark tool for local AI used by coding agents.
    GitHub · ★ 26 · Python · first seen 2026-10-06 · hn
    heat 0.3
    0 / 1 mentions
  65. 65
    alymoursy/hieraticbench
    Benchmark for AI reading ancient Egyptian hieratic script.
    GitHub · ★ 4 · TypeScript · first seen 2026-10-06 · hn
    heat 0.3
    0 / 1 mentions
  66. 66
    illate/parity
    Statistical tool for comparing cheaper models to current ones.
    GitHub · Python · first seen 2026-10-06 · hn
    heat 0.3
    0 / 1 mentions
  67. 67
    johnmu/codex-reasoning-token-experiment
    Codex reasoning token comparison between API and ChatGPT Pro.
    GitHub · ★ 4 · Python · first seen 2026-10-06 · hn
    heat 0.3
    0 / 1 mentions
  68. 68
    simonether/skill-placebo
    Placebo-controlled benchmark of coding agent skills
    GitHub · ★ 22 · Shell · first seen 2026-10-06 · hn
    heat 0.3
    0 / 1 mentions
  69. 69
    sorami-consulting-au/llm-tokens-after-cancel
    Journals and analysis for tokens after cancel report.
    GitHub · Python · first seen 2026-10-14 · reddit
    heat 0.3
    0 / 1 mentions
  70. 70
    itsloganmann/rare-failure-eval
    Agent evaluation experiment on rare failures and selection errors.
    GitHub · ★ 1 · Python · first seen 2026-10-05 · hn
    heat 0.3
    0 / 1 mentions
  71. 71
    optimizer077/paper-review-harness
    CVPR review harness with reviewer personas and protocols.
    GitHub · ★ 11 · Python · first seen 2026-10-13 · reddit
    heat 0.3
    0 / 1 mentions
  72. 72
    tap222/assay-evals
    Behavioral regression testing for AI apps with pytest plugin.
    GitHub · ★ 23 · Python · first seen 2026-10-13 · reddit
    heat 0.3
    0 / 1 mentions
  73. 73
    joonfjp/16agents
    Personality test designed for AI agents.
    GitHub · JavaScript · first seen 2026-10-04 · hn
    heat 0.3
    0 / 1 mentions
  74. 74
    zhebrak/brokenml
    Benchmark of 46 broken ML workloads for agent evaluation.
    GitHub · ★ 1 · Python · first seen 2026-10-12 · reddit
    heat 0.3
    0 / 1 mentions
  75. 75
    talhatahir-10xe/attention-galaxy
    Browser visualization of language model attention and predictions.
    GitHub · ★ 1 · JavaScript · first seen 2026-10-11 · reddit
    heat 0.2
    0 / 1 mentions
  76. 76
    routsom/causeval
    Statistically rigorous causal evaluation for LLM apps with confidence intervals.
    GitHub · ★ 3 · Python · first seen 2026-10-02 · hn
    heat 0.2
    0 / 1 mentions
  77. 77
    scalingintelligence/kernelbench
    Benchmark testing LLMs writing GPU kernels from Torch to CUDA.
    GitHub · ★ 1,307 · Jupyter Notebook · first seen 2026-10-02 · hn
    heat 0.2
    0 / 1 mentions
  78. 78
    poojithdevan4d/inferpilot
    Evidence-first vLLM screening and controlled capacity analysis for inference optimization.
    GitHub · ★ 3 · Python · first seen 2026-10-01 · hn
    heat 0.2
    0 / 1 mentions
  79. 79
    samuelrojas-dev/agent-cost-bench
    Reproducible denial-of-wallet benchmark for tool-using LLM agents.
    GitHub · ★ 1 · Python · first seen 2026-10-01 · hn
    heat 0.2
    0 / 1 mentions
  80. 80
    cellaflow/cellaflow-benchmarks
    Cellaflow benchmarks for testing and evaluation.
    GitHub · TypeScript · first seen 2026-10-09 · reddit
    heat 0.2
    0 / 1 mentions
  81. 81
    microsoft/thinkingbox
    Framework for defining MCP tool servers and evaluating agents.
    GitHub · ★ 111 · Python · first seen 2026-10-09 · reddit
    heat 0.2
    0 / 1 mentions
  82. 82
    dalek2point3/emulate-write-evaluation
    200-prompt evaluation of Emulate Write text generation model
    GitHub · ★ 3 · TeX · first seen 2026-09-30 · hn
    heat 0.2
    0 / 1 mentions
  83. 83
    tomerglick57/jevstiller
    Distill a repeated Jev classification task into local model
    GitHub · ★ 71 · Python · first seen 2026-09-30 · hn
    heat 0.2
    0 / 1 mentions
  84. 84
    nacloos/baba-is-ai
    Code for the Baba Is AI benchmark paper on breaking rules to beat benchmarks.
    GitHub · ★ 51 · Python · first seen 2026-10-08 · reddit
    heat 0.2
    0 / 1 mentions
  85. 85
    AI快站模型质量检测
    Web tool for testing OpenAI-compatible API model quality and generating reports.
    catalog · chinese-indie-dev · first seen 2026-10-08 · catalog
    heat 0.2
    0 / 1 mentions
  86. 86
    adarshent/spnda
    Epistemic uncertainty quantification and hallucination detection for LLMs.
    GitHub · ★ 33 · Python · first seen 2026-09-05 · hn · reddit
    heat 0.2
    0 / 3 mentions
  87. 87
    aiobjectives/tttc-light-js
    LLM analysis tool for collective discourse and decision-making improvement.
    GitHub · ★ 63 · TypeScript · first seen 2026-10-21 · hn
    heat 0.2
    0 / 1 mentions
  88. 88
    secondstate-ai/finance-agents-benchmark
    Benchmark for AI agents doing financial due diligence
    GitHub · ★ 5 · Python · first seen 2026-09-28 · hn
    heat 0.2
    0 / 1 mentions
  89. 89
    ankit-aglawe/tinyjev
    Tiny decision model answering choice score and questions with calibrated probabilities
    GitHub · ★ 28 · Python · first seen 2026-09-22 · hn
    heat 0.2
    0 / 2 mentions
  90. 90
    apolinario/decision-index
    Reproduce Jev decision-model benchmark suite locally.
    GitHub · ★ 83 · Python · first seen 2026-10-06 · reddit
    heat 0.2
    0 / 1 mentions
  91. 91
    gabeorlanski/scb-problems
    SlopCodeBench problems dataset for evaluating coding models.
    GitHub · ★ 14 · Python · first seen 2026-10-20 · hn
    heat 0.2
    0 / 1 mentions
  92. 92
    deadatreides/llm-measurements
    Lab notebook with 232 local-LLM experiment reports and harnesses.
    GitHub · ★ 2 · Python · first seen 2026-10-05 · reddit
    heat 0.2
    0 / 1 mentions
  93. 93
    neurono-ml/typed-lm
    Single-pass semantic routing for LLMs with typed decision API.
    GitHub · ★ 10 · Rust · first seen 2026-09-26 · hn
    heat 0.2
    0 / 1 mentions
  94. 94
    jackjusko/sftmill
    Off-policy synthetic data distillation engine for LLMs.
    GitHub · ★ 20 · Python · first seen 2026-10-04 · reddit
    heat 0.2
    0 / 1 mentions
  95. 95
    gazelle93/decision-models-under-pressure
    Eight decision models tested under pressure, bias and adversarial conditions.
    GitHub · ★ 4 · Python · first seen 2026-09-25 · hn
    heat 0.2
    0 / 1 mentions
  96. 96
    ntlm1686/your-language-model-is-already-a-decision-model
    Compare unfinetuned Qwen with Jev on decision accuracy and calibration.
    GitHub · ★ 7 · Python · first seen 2026-09-25 · hn
    heat 0.2
    0 / 1 mentions
  97. 97
    obielin/reliopt
    Reliability-constrained multi-objective optimization for LLM programs.
    GitHub · ★ 1 · Python · first seen 2026-09-25 · hn
    heat 0.2
    0 / 1 mentions
  98. 98
    wwoosshh/entail
    Checks model config matches inference engine behavior for transformers and vLLM.
    GitHub · ★ 3 · Python · first seen 2026-09-25 · hn
    heat 0.2
    0 / 1 mentions
  99. 99
    mohduwaish59/brier
    Typed calibrated decisions from open LLMs using next-token probabilities.
    GitHub · ★ 4 · Python · first seen 2026-10-03 · reddit
    heat 0.2
    0 / 1 mentions
  100. 100
    ikaleio/lm-detector
    Model fingerprint detection with shared reference bank.
    GitHub · ★ 1,342 · TypeScript · first seen 2026-09-24 · hn
    heat 0.2
    0 / 1 mentions
  101. 101
    tommkruix/agentrulebench
    Benchmark testing AI agents against architectural rules.
    GitHub · ★ 3 · JavaScript · first seen 2026-09-08 · hn
    heat 0.2
    0 / 2 mentions
  102. 102
    alexiush/fleet
    Distributed test-time scaling algorithm for LLM workflows.
    GitHub · Jupyter Notebook · first seen 2026-10-02 · reddit
    heat 0.2
    0 / 1 mentions
  103. 103
    deepansh-saxena/jev-guardrails
    Comparing LLM-as-judge vs TypeSafe Jev for agent guardrails.
    GitHub · ★ 3 · Python · first seen 2026-09-23 · hn
    heat 0.2
    0 / 1 mentions
  104. 104
    gracejackson-sudo/quant-delta-predictor
    Calibrated risk envelope for LLM quantization with uncertainty flagging.
    GitHub · ★ 4 · Python · first seen 2026-09-23 · hn
    heat 0.2
    0 / 1 mentions
  105. 105
    kmcheung12/sniffslop
    AI text detection using Jev in browser.
    GitHub · ★ 1 · JavaScript · first seen 2026-09-23 · hn
    heat 0.2
    0 / 1 mentions
  106. 106
    yididev/jev-benchmark
    Rubric-based classification benchmark comparing Jev against Claude and OpenJev.
    GitHub · ★ 3 · Python · first seen 2026-09-23 · hn
    heat 0.2
    0 / 1 mentions
  107. 107
    lossfunk/authority-bias
    Study of language models abandoning correct answers when sources disagree.
    GitHub · ★ 1 · Python · first seen 2026-10-01 · reddit
    heat 0.2
    0 / 1 mentions
  108. 108
    instax-dutta/sysone-bench
    Independent benchmark comparing System One decision models Laya versus Jev
    GitHub · ★ 6 · Python · first seen 2026-09-22 · hn
    heat 0.1
    0 / 1 mentions
  109. 109
    pulse-energy-eu/slopshape
    Verification artifacts for identifying AI-generated commercial web content.
    GitHub · ★ 75 · Python · first seen 2026-09-22 · hn
    heat 0.1
    0 / 1 mentions
  110. 110
    selmar/typesafe-jev-calibrate-for-code-review
    Calibrating Jev decision model for code review tasks.
    GitHub · ★ 3 · Python · first seen 2026-09-22 · hn
    heat 0.1
    0 / 1 mentions
  111. 111
    cocodot2026/cocodot-llmprobe
    CLI to verify and benchmark LLM APIs for model downgrades
    GitHub · ★ 3 · Python · first seen 2026-09-30 · catalog
    heat 0.1
    0 / 1 mentions
  112. 112
    cocodot LLM 降智检测
    Detect LLM API model degradation or swapping
    catalog · chinese-indie-dev · first seen 2026-09-30 · catalog
    heat 0.1
    0 / 1 mentions
  113. 113
    0xsero/model-toolkit
    Model evaluation and compression tools with benchmarks and reproducible evidence.
    GitHub · ★ 4 · Python · first seen 2026-09-21 · hn
    heat 0.1
    0 / 1 mentions
  114. 114
    pokertools-arena/pokertools-arena.github.io
    Browser-first AI poker benchmark for testing models in autonomous tournaments.
    GitHub · ★ 2 · JavaScript · first seen 2026-09-21 · hn
    heat 0.1
    0 / 1 mentions
  115. 115
    tomosu.ai
    Score repositories with production reliability index and merge recommendations
    catalog · free-for-dev · first seen 2026-09-29 · catalog
    heat 0.1
    0 / 1 mentions
  116. 116
    chr-kelly/jev-cookbook
    Runnable question sets and eval harness for TypeSafe Jev with CLINC150 results.
    GitHub · ★ 1 · Python · first seen 2026-09-20 · hn
    heat 0.1
    0 / 1 mentions
  117. 117
    harnessrouter/systemoneharness
    Harness for running System One models locally or on HarnessRouter cloud.
    GitHub · ★ 207 · Python · first seen 2026-09-20 · hn
    heat 0.1
    0 / 1 mentions
  118. 118
    openlayer-ai/jevals
    Agent evals and guardrails as Jev decisions running locally or in loop.
    GitHub · ★ 110 · Python · first seen 2026-09-20 · hn
    heat 0.1
    0 / 1 mentions
  119. 119
    harveyai/harvey-labs
    Benchmark evaluating agent capabilities for legal work support.
    GitHub · ★ 1,547 · Python · first seen 2026-10-12 · hn
    heat 0.1
    0 / 1 mentions
  120. 120
    tashonbraganca/messy-docs-bench
    Benchmark for AI models on business documents
    GitHub · Python · first seen 2026-09-28 · reddit
    heat 0.1
    0 / 1 mentions
  121. 121
    michaelasper/benchmarks
    Reproducible coding agent benchmark reports.
    GitHub · ★ 10 · Python · first seen 2026-07-31 · hn · reddit
    heat 0.1
    0 / 5 mentions
  122. 122
    hansenq/nav-evals-public
    Benchmark evaluating LLMs on driving directions accuracy.
    GitHub · TypeScript · first seen 2026-10-11 · hn
    heat 0.1
    0 / 1 mentions
  123. 123
    gi-dellav/anypick
    LLM selection tool filtering by capabilities and pricing.
    GitHub · ★ 4 · Python · first seen 2026-09-08 · hn · reddit
    heat 0.1
    0 / 2 mentions
  124. 124
    ed-is-ai/featherbench
    Lightweight framework for building custom LLM benchmarks.
    GitHub · ★ 4 · Python · first seen 2026-08-07 · hn
    heat 0.1
    0 / 8 mentions
  125. 125
    UGI Leaderboard
    AI ERP writing and censors benchmark leaderboard.
    catalog · fmhy · first seen 2026-09-26 · catalog
    heat 0.1
    0 / 1 mentions
  126. 126
    gauravapiscean/agentic-kv-cache
    Reproducing agentic KV-cache policy claims on real Claude Code session traces.
    GitHub · ★ 24 · Python · first seen 2026-09-10 · hn
    heat 0.1
    0 / 2 mentions
  127. 127
    grafana/o11y-bench
    Benchmark for observability tasks using conversational agents.
    GitHub · ★ 76 · Python · first seen 2026-10-08 · hn
    heat 0.1
    0 / 1 mentions
  128. 128
    sregym/sregym
    Gym environment testing if AI agents can resolve production incidents.
    GitHub · ★ 391 · Python · first seen 2026-10-08 · hn
    heat 0.1
    0 / 1 mentions
  129. 129
    sankahq/bench
    Sanka migration benchmark for Django and FastAPI.
    GitHub · Python · first seen 2026-09-23 · reddit
    heat 0.1
    0 / 1 mentions
  130. 130
    Epoch AI
    Aggregated AI model benchmark results and comparisons.
    catalog · fmhy · first seen 2026-09-23 · catalog
    heat 0.1
    0 / 1 mentions
  131. 131
    sriramvarun0636/vasool
    Revenue-recovery agent and evaluation harness for Razorpay.
    GitHub · ★ 1 · Python · first seen 2026-09-01 · hn
    heat 0.1
    0 / 2 mentions
  132. 132
    coroot/rca-lab
    Kubernetes lab reproducing production incidents for testing.
    GitHub · ★ 38 · Go · first seen 2026-08-14 · hn
    heat 0.1
    0 / 2 mentions
  133. 133
    llm-as-a-verifier/llm-as-a-verifier
    LLM-as-Verifier framework for fine-grained agent feedback.
    GitHub · ★ 3,341 · Python · first seen 2026-08-18 · hn
    heat 0.1
    0 / 2 mentions
  134. 134
    polmanas1998-star/membench
    Benchmark for memory layers with statistical rigor and controls.
    GitHub · ★ 2 · Python · first seen 2026-09-06 · reddit
    heat 0.1
    0 / 2 mentions
  135. 135
    sierra-research/tau2-bench
    Benchmark for tool-agent-user interaction in real-world domains.
    GitHub · ★ 2,368 · Python · first seen 2026-10-04 · hn
    heat 0.1
    0 / 1 mentions
  136. 136
    Tripstitch
    Benchmarks for evaluating AI trip planning models and agent pipelines.
    catalog · fmhy · first seen 2026-09-20 · catalog
    heat 0.1
    0 / 1 mentions
  137. 137
    ashwinugale/tracelint
    Deterministic linter for agent runs without LLM judge.
    GitHub · ★ 12 · Python · first seen 2026-08-18 · hn
    heat 0.1
    0 / 2 mentions
  138. 138
    nicia-ai/admission-decision-eval
    Decision model evaluation on knowledge-base write-admission task.
    GitHub · JavaScript · first seen 2026-10-02 · hn
    heat 0.1
    0 / 1 mentions
  139. 139
    abundant-ai/incident-arena
    20-task SRE incident repair benchmark for Harbor.
    GitHub · ★ 6 · Python · first seen 2026-09-30 · hn
    heat 0.1
    0 / 1 mentions
  140. 140
    mukundzha/crilio
    CI gate for AI prompt testing and regression prevention.
    GitHub · ★ 2 · Python · first seen 2026-08-24 · hn
    heat 0.1
    0 / 2 mentions
  141. 141
    ammaar-alam/minebench
    Minecraft-style voxel benchmark for AI model comparison.
    GitHub · ★ 343 · TypeScript · first seen 2026-07-22 · hn · reddit
    heat 0.1
    0 / 3 mentions
  142. 142
    partyfly/tokenizer-fairness-2026
    Reproduction and extension study on tokenizer fairness across multilingual datasets.
    GitHub · HTML · first seen 2026-09-27 · hn
    heat 0.1
    0 / 1 mentions
  143. 143
    duckdblabs/db-benchmark
    Reproducible benchmark of database-like operations.
    GitHub · ★ 186 · R · first seen 2026-09-25 · hn
    heat 0.1
    0 / 1 mentions
  144. 144
    handshake-ai-research/studentbench
    Reproduce all empirical results from the StudentBench tutoring evaluation paper.
    GitHub · ★ 13 · Python · first seen 2026-09-25 · hn
    heat 0.1
    0 / 1 mentions
  145. 145
    joshbolding/ggufaudit
    Audit tool checking GGUF file quantization types and accuracy.
    GitHub · ★ 2 · Python · first seen 2026-08-28 · reddit
    heat 0.1
    0 / 2 mentions
  146. 146
    smarr/are-we-fast-yet
    Language implementation performance comparison benchmark suite.
    GitHub · ★ 420 · Java · first seen 2026-09-25 · hn
    heat 0.1
    0 / 1 mentions
  147. 147
    kantahayashiai/jev-does-not-play-dice
    Experiments on Jev probability calibration and uncertainty.
    GitHub · ★ 4 · JavaScript · first seen 2026-09-24 · hn
    heat 0.1
    0 / 1 mentions
  148. 148
    cocodot LLM 降智检测
    LLM API detection tool for model fraud and downgrade.
    catalog · chinese-indie-dev · first seen 2026-09-09 · catalog
    heat 0.1
    0 / 1 mentions
  149. 149
    Arena TTS
    Text-to-speech model benchmarks and rankings.
    catalog · fmhy · first seen 2026-09-09 · catalog
    heat 0.1
    0 / 1 mentions
  150. 150
    learnerlabs-ai/replications
    Learner Labs experimental reports data and replication tools
    GitHub · first seen 2026-09-22 · hn
    heat 0.0
    0 / 1 mentions
  151. 151
    nari-labs/benchmarks
    Open-source benchmarking tool from Nari.
    GitHub · ★ 11 · Python · first seen 2026-08-21 · hn · reddit
    heat 0.0
    0 / 2 mentions
  152. 152
    logan-markewich/jeff
    Self-hosted drop-in replacement for TypeSafe's Jev powered by GliFormer.
    GitHub · ★ 313 · Python · first seen 2026-09-21 · hn
    heat 0.0
    0 / 1 mentions
  153. 153
    thomeras/agent_detective
    Eval framework for multi-agent systems with OTEL traces and blame analysis.
    GitHub · ★ 3 · Python · first seen 2026-07-29 · hn
    heat 0.0
    0 / 4 mentions
  154. 154
    samchon/evidence-benchmark-results
    Evidence graph benchmark results and data.
    GitHub · ★ 2 · TypeScript · first seen 2026-08-22 · reddit
    heat 0.0
    0 / 2 mentions
  155. 155
    ifixai-ai/ifixai
    Independent auditing framework for evaluating AI agent behavior.
    GitHub · ★ 42,372 · Python · first seen 2026-08-09 · hn
    heat 0.0
    0 / 2 mentions
  156. 156
    dreamfast/abliterlitics
    Forensic analysis of LLM ablation techniques.
    GitHub · ★ 59 · Python · first seen 2026-07-26 · reddit
    heat 0.0
    0 / 2 mentions
  157. 157
    run-llama/extractbench
    Benchmark for schema-guided enterprise document extraction.
    GitHub · ★ 115 · Python · first seen 2026-08-11 · hn · reddit
    heat 0.0
    0 / 2 mentions
  158. 158
    eugr/llama-benchy
    Benchmarking tool for LLM inference backends.
    GitHub · ★ 766 · Python · first seen 2026-08-14 · reddit
    heat 0.0
    0 / 2 mentions
  159. 159
    ctgt-inc/lineage-eval
    Benchmark for evaluating selective censorship in language models.
    GitHub · ★ 9 · Python · first seen 2026-07-30 · hn
    heat 0.0
    0 / 4 mentions
  160. 160
    braveann011/ai-halo-valuation-bias
    Valuation bias study across multimodal LLMs.
    GitHub · ★ 3 · Python · first seen 2026-07-28 · hn
    heat 0.0
    0 / 3 mentions
  161. 161
    sunblaze-ucb/exploitgym
    Benchmark for evaluating AI agents' exploit development abilities.
    GitHub · ★ 1,242 · C · first seen 2026-07-23 · hn
    heat 0.0
    0 / 5 mentions
  162. 162
    david-g-3654/homebench
    Benchmark local LLMs with TUI leaderboard for speed and quality.
    GitHub · ★ 59 · Python · first seen 2026-08-04 · hn
    heat 0.0
    0 / 2 mentions
  163. 163
    iamfaham/agentsnap
    Deterministic snapshot testing framework for AI agents.
    GitHub · ★ 4 · Python · first seen 2026-07-29 · hn · reddit
    heat 0.0
    0 / 2 mentions
  164. 164
    jacquardlabs/gauntlet
    Independent auditing judges for artifact validation.
    GitHub · ★ 7 · Python · first seen 2026-08-14 · hn
    heat 0.0
    0 / 2 mentions
  165. 165
    hkust-nlp/llm-compression-intelligence
    Research showing compression represents intelligence linearly.
    GitHub · ★ 153 · Python · first seen 2026-08-11 · hn
    heat 0.0
    0 / 2 mentions
  166. 166
    tanishq-dubey/macos-laguna-s2.1
    Community benchmarks for Poolside Laguna S 2.1.
    GitHub · ★ 121 · Python · first seen 2026-07-22 · hn
    heat 0.0
    0 / 2 mentions