Tidox Radar

Local LLM Inference & Runtimes

Running LLMs on-device: llama.cpp frontends, model managers, local inference engines. · 282 items · 25 this week ▼ 10 · 0 new this week

  1. 1
    niko1221/strata
    One-click install for running Qwen3.8 on consumer hardware
    GitHub · ★ 37,610 · C++ · first seen 2026-09-28 · hn · reddit
    heat 18.1
    3 / 45 mentions
  2. 2
    nace-ai/kapitsa ▲ rising
    Apple Silicon optimized inference for Qwen and Nemotron
    GitHub · ★ 579 · C++ · first seen 2026-10-19 · hn · reddit
    heat 16.8
    4 / 19 mentions
  3. 3
    jesdga95/basalt ▲ rising
    Local inference engine for Qwen3.8-Flash-Next on NVIDIA Blackwell GPUs.
    GitHub · ★ 413 · C++ · first seen 2026-10-10 · reddit
    heat 11.5
    4 / 13 mentions
  4. 4
    mw00/project-maya
    Run massive MoE models privately on your GPUs.
    GitHub · ★ 2,368 · Cuda · first seen 2026-10-11 · hn · reddit
    heat 10.6
    2 / 11 mentions
  5. 5
    zagitta/gpu-unlock
    Unlocks full AI speed on GeForce 20/30/40 series cards.
    GitHub · ★ 14,110 · C · first seen 2026-10-21 · catalog · hn · reddit
    heat 8.3
    0 / 15 mentions
  6. 6
    t-crypt/berth ▲ rising
    GPU scheduler leasing card to one inference engine.
    GitHub · ★ 266 · Go · first seen 2026-10-11 · reddit
    heat 8.0
    2 / 11 mentions
  7. 7
    davide-troiani/moeshift
    Memory-efficient Mixture-of-Experts LLM inference optimized for Apple Silicon.
    GitHub · ★ 19 · C++ · first seen 2026-10-30 · hn · reddit
    heat 6.5
    0 / 7 mentions
  8. 8
    jakeatx/llamampere ▲ rising
    llama.cpp fork with significantly improved performance on Ampere NVIDIA GPUs.
    GitHub · ★ 207 · C++ · first seen 2026-10-30 · reddit
    heat 5.6
    2 / 5 mentions
  9. 9
    valeriodolci/capy-flash
    Run Qwen3 MoE models on one or two RTX 50-series GPUs with CPU offloading.
    GitHub · ★ 128 · C++ · first seen 2026-10-15 · hn · reddit
    heat 4.3
    1 / 7 mentions
  10. 10
    meshapplied/goethe
    Serve models fast on your own GPU infrastructure.
    GitHub · ★ 43 · Rust · first seen 2026-10-13 · hn
    heat 3.8
    1 / 3 mentions
  11. 11
    deepseek-ai/deepinfer
    High-performance large-scale inference framework for DeepSeek models.
    GitHub · ★ 24,803 · Python · first seen 2026-10-20 · hn · reddit
    heat 3.7
    0 / 9 mentions
  12. 12
    incoai/splash
    Local inference engine for Apple silicon with Metal acceleration.
    GitHub · ★ 2,012 · C++ · first seen 2026-09-23 · hn · reddit
    heat 3.6
    1 / 5 mentions
  13. 13
    indras-mirror/xolotl
    Run DeepSeek-V4.1 552B locally on RTX 4090 with 90GB RAM.
    GitHub · ★ 432 · C++ · first seen 2026-10-17 · reddit
    heat 3.4
    1 / 4 mentions
  14. 14
    prabanta-dev/janas
    Run LLMs bigger than RAM on CPU or GPU faster than llama.cpp.
    GitHub · ★ 563 · C · first seen 2026-10-12 · hn · reddit
    heat 3.0
    1 / 5 mentions
  15. 15
    kouhxp/gutsy
    Local decision model with calibrated probabilities via Jev-style API.
    GitHub · ★ 81 · HTML · first seen 2026-10-01 · hn
    heat 2.9
    0 / 13 mentions
  16. 16
    alainnothere/llama.cpp
    LLM inference engine in C and C++.
    GitHub · ★ 1 · C++ · first seen 2026-08-16 · hn
    heat 2.6
    1 / 9 mentions
  17. 17
    betelgeuse-ai/betelgeuse
    Custom fork optimising mixture-of-experts inference on a 12 GB VRAM GPU.
    GitHub · ★ 70 · C++ · first seen 2026-10-31 · reddit
    heat 2.3
    0 / 3 mentions
  18. 18
    tensorrails/warmctl
    GPU checkpoint/restore for vLLM snapshot warmed-up engines into OCI images.
    GitHub · ★ 30 · Go · first seen 2026-10-12 · hn · reddit
    heat 1.9
    0 / 2 mentions
  19. 19
    ishekkhal/bitrun
    Zero-compile runner for ternary LLMs with wattage telemetry.
    GitHub · ★ 8 · Go · first seen 2026-10-13 · hn · reddit
    heat 1.9
    0 / 5 mentions
  20. 20
    gufo-org/gufo
    Open-source AI engine for AMD Strix Halo processors.
    GitHub · ★ 1,020 · C++ · first seen 2026-10-08 · hn · reddit
    heat 1.9
    0 / 5 mentions
  21. 21
    exo-explore/exo
    Run frontier AI models locally.
    GitHub · ★ 48,147 · Python · first seen 2026-07-28 · hn
    heat 1.7
    0 / 6 mentions
  22. 22
    johnfink8/sfnx
    CUDA engine splitting MoE LLM layers across V100 cards for inference.
    GitHub · ★ 49 · Python · first seen 2026-10-26 · reddit
    heat 1.7
    0 / 3 mentions
  23. 23
    whoreson/picolm
    Pure C and GPU inference for multiple open LLM models.
    GitHub · ★ 15 · C · first seen 2026-09-03 · hn
    heat 1.6
    0 / 4 mentions
  24. 24
    llamastash/llamastash
    Terminal-native local LLM orchestrator proxy.
    GitHub · ★ 281 · Rust · first seen 2026-10-06 · reddit
    heat 1.5
    0 / 4 mentions
  25. 25
    armbues/mlx-decision
    Run decision models locally on Apple Silicon with MLX.
    GitHub · ★ 12 · Python · first seen 2026-10-24 · reddit
    heat 1.5
    0 / 2 mentions
  26. 26
    unslothai/llama.cpp
    Efficient LLM inference in C/C++.
    GitHub · ★ 251 · C++ · first seen 2026-08-21 · hn · reddit
    heat 1.4
    1 / 4 mentions
  27. 27
    wici-ai/inferssd
    Run models larger than RAM or VRAM by streaming MoE experts from SSD.
    GitHub · ★ 211 · C++ · first seen 2026-10-30 · hn
    heat 1.4
    0 / 2 mentions
  28. 28
    sergqwer/strata-nvfp4
    Qwen 125B MoE in NVFP4 on single RTX card with CPU experts.
    GitHub · ★ 211 · C++ · first seen 2026-10-13 · reddit
    heat 1.3
    0 / 3 mentions
  29. 29
    cobanov/ps5lm
    Run GGUF models natively on jailbroken PS5 consoles.
    GitHub · ★ 780 · C++ · first seen 2026-10-14 · hn · reddit
    heat 1.2
    0 / 3 mentions
  30. 30
    fareedkhan-dev/kimi-k3-in-c
    Kimi K3 running on single CPU in portable C99.
    GitHub · ★ 9,275 · C · first seen 2026-08-02 · hn · reddit
    heat 1.2
    0 / 5 mentions
  31. 31
    ggml-org/llama-macos
    cosy macOS home for running LLMs locally.
    GitHub · ★ 1,717 · Swift · first seen 2026-08-12 · hn
    heat 1.2
    0 / 4 mentions
  32. 32
    beamivalice/sushi
    High quality quant model engine for Apple Silicon using STQ.
    GitHub · ★ 391 · Zig · first seen 2026-10-19 · hn · reddit
    heat 1.1
    1 / 2 mentions
  33. 33
    srkrz23/vramkit
    VRAM compression for LLM inference with lossless weight codec.
    GitHub · ★ 4 · Python · first seen 2026-10-17 · hn
    heat 1.1
    0 / 2 mentions
  34. 34
    eagle-hq/eagle-4
    Official inference code for EAGLE-4 speculative decoding.
    GitHub · ★ 1,097 · Python · first seen 2026-10-20 · hn · reddit
    heat 1.0
    0 / 2 mentions
  35. 35
    skeggsguy/flash-next-ssd
    Run Qwen Flash Next model on Mac Mini M5.
    GitHub · ★ 17 · C++ · first seen 2026-10-05 · reddit
    heat 1.0
    0 / 4 mentions
  36. 36
    achyudh/llama.cpp
    LLM inference in C/C++.
    GitHub · ★ 80 · C++ · first seen 2026-10-17 · hn · reddit
    heat 1.0
    0 / 3 mentions
  37. 37
    introspective-diffusion/dlmrt
    Single-request inference engine for block-diffusion language models.
    GitHub · ★ 17 · Cuda · first seen 2026-10-16 · reddit
    heat 1.0
    0 / 3 mentions
  38. 38
    megasoft1978/strim-engine
    Swift/MLX inference engine and agent harness for Qwen3 on Apple Silicon.
    GitHub · ★ 2 · Swift · first seen 2026-10-30 · hn
    heat 1.0
    0 / 1 mentions
  39. 39
    spinfinc/spinf-serve
    Bulk classification and inference on one GPU with single-pass content scoring.
    GitHub · ★ 20 · Python · first seen 2026-10-30 · hn
    heat 1.0
    0 / 1 mentions
  40. 40
    tensorninja/ninfer-4090
    High-speed LLM inference engine for large models.
    GitHub · ★ 15 · C++ · first seen 2026-08-26 · hn
    heat 1.0
    0 / 4 mentions
  41. 41
    wallawalla47/infernix
    Single-GPU inference engine for Qwen3.8 on RTX 5090.
    GitHub · ★ 177 · C++ · first seen 2026-10-10 · reddit
    heat 0.9
    0 / 2 mentions
  42. 42
    solvingsteve/gevva0
    Cascading decision gateway with logit scoring via llama.cpp
    GitHub · ★ 11 · Python · first seen 2026-09-28 · hn · reddit
    heat 0.9
    0 / 3 mentions
  43. 43
    syv-ai/hyperqwen
    Serve large Qwen models fast on single GPUs with vLLM.
    GitHub · ★ 2,473 · Python · first seen 2026-10-02 · reddit
    heat 0.9
    0 / 2 mentions
  44. 44
    heterodoxin/abliteratedoutthebox
    llama.cpp fork performing abliteration during inference.
    GitHub · ★ 160 · C++ · first seen 2026-10-21 · reddit
    heat 0.9
    0 / 2 mentions
  45. 45
    kekzl/imp
    Run large AI models privately on NVIDIA RTX graphics cards with fast inference.
    GitHub · ★ 124 · C++ · first seen 2026-10-27 · hn
    heat 0.8
    0 / 1 mentions
  46. 46
    magnitudedev/magnitude
    Inference engine auto-tuning kernels for 2x speed over Ollama
    GitHub · ★ 7,655 · Rust · first seen 2026-09-30 · hn
    heat 0.8
    0 / 9 mentions
  47. 47
    ekzhang/resin
    Experimental library for building LLM inference engines with CUDA support.
    GitHub · ★ 359 · Python · first seen 2026-10-26 · hn
    heat 0.8
    0 / 1 mentions
  48. 48
    theoleecj/semif-openjev
    Semantic conditional execution from open models on 3090
    GitHub · ★ 4,916 · Python · first seen 2026-09-28 · hn
    heat 0.8
    0 / 2 mentions
  49. 49
    joakimpalm-zen/xyntetik-runner
    Local LLM inference engine with reproducible receipts.
    GitHub · ★ 30 · C · first seen 2026-08-25 · hn · reddit
    heat 0.8
    0 / 3 mentions
  50. 50
    aymen94/neurondump
    Inspect and edit local LLM internals with 3D exploration and patching.
    GitHub · ★ 15 · Rust · first seen 2026-10-25 · hn
    heat 0.8
    0 / 1 mentions
  51. 51
    juraj-jumic/dspark-local-bench
    Benchmark DSpark speculative decoding on consumer GPUs.
    GitHub · ★ 3 · Python · first seen 2026-10-25 · hn
    heat 0.8
    0 / 1 mentions
  52. 52
    peonist-ai/halogen-flash-server
    Local inference server exposing OpenAI and Anthropic APIs for AMD.
    GitHub · ★ 1,176 · Shell · first seen 2026-09-18 · hn · reddit
    heat 0.7
    0 / 4 mentions
  53. 53
    blockwayz/agens-engine
    Fast local inference engine for Agens Volundr
    GitHub · ★ 10 · C++ · first seen 2026-10-18 · reddit
    heat 0.7
    0 / 2 mentions
  54. 54
    anonnerdfrenzy/survival-llm
    Free offline AI survival assistant with sourced answers running locally.
    GitHub · ★ 1,218 · Python · first seen 2026-10-27 · hn
    heat 0.7
    0 / 2 mentions
  55. 55
    nace-ai/llama.cpp
    E-DLM working fork of llama.cpp inference engine.
    GitHub · ★ 4 · C++ · first seen 2026-11-01 · reddit
    heat 0.7
    0 / 1 mentions
  56. 56
    ngxson/wllama
    WebAssembly binding enabling browser-based llama.cpp LLM inference.
    GitHub · ★ 1,355 · TypeScript · first seen 2026-11-01 · reddit
    heat 0.7
    0 / 1 mentions
  57. 57
    thefahien/elpis-on-7900xtx
    Qwen 27B EXL3 inference recipe and benchmarks on AMD 7900 XTX.
    GitHub · ★ 4 · Python · first seen 2026-11-01 · reddit
    heat 0.7
    0 / 1 mentions
  58. 58
    qxoticai/qxotic
    Run LLM inference securely within GraalVM enterprise Java runtime environments.
    GitHub · ★ 186 · Java · first seen 2026-09-15 · hn
    heat 0.7
    0 / 4 mentions
  59. 59
    justvugg/lumabri
    Run large AI models across distributed computers and servers.
    GitHub · ★ 210 · C · first seen 2026-08-09 · hn
    heat 0.7
    0 / 3 mentions
  60. 60
    llm-d/llm-d
    High-performance distributed LLM inference on Kubernetes.
    GitHub · ★ 5,121 · Python · first seen 2026-10-23 · hn
    heat 0.7
    0 / 1 mentions
  61. 61
    apollo-mg/project-apollo
    Home lab for local LLM inference experiments on RDNA4 and Tesla P100 GPUs.
    GitHub · ★ 4 · Python · first seen 2026-10-31 · reddit
    heat 0.7
    0 / 1 mentions
  62. 62
    bpx03/qengine
    C++/CUDA inference engine for Qwen 27B INT4 on 3x RTX 3090.
    GitHub · ★ 31 · Cuda · first seen 2026-10-31 · reddit
    heat 0.7
    0 / 1 mentions
  63. 63
    divagr18/qwenpu
    LLM running on NPU inside an LLM weights.
    GitHub · ★ 86 · Python · first seen 2026-10-31 · reddit
    heat 0.7
    0 / 1 mentions
  64. 64
    proprietarylegal/deepseek-v4.1-flash-2xdgx-spark-x86-offload
    DeepSeek-V4.1-Flash on DGX Spark with x86 offload achieving 2,606 tok/s.
    GitHub · ★ 2 · Python · first seen 2026-10-31 · reddit
    heat 0.7
    0 / 1 mentions
  65. 65
    asf0/libstrix
    HIP kernels making vLLM up to 31x faster on AMD Strix Halo GPUs.
    GitHub · ★ 16 · Python · first seen 2026-10-30 · reddit
    heat 0.6
    0 / 1 mentions
  66. 66
    asf0/strix-vllm
    Optimized vLLM fork for AMD Strix Halo with heterogeneous KV-cache and fast inference.
    GitHub · ★ 4 · Python · first seen 2026-10-30 · reddit
    heat 0.6
    0 / 1 mentions
  67. 67
    fronteagle739/streaming-llama-stack
    Streaming and disaggregated LLM inference stack optimized for low memory with fast SSDs.
    GitHub · ★ 8 · C · first seen 2026-10-30 · reddit
    heat 0.6
    0 / 1 mentions
  68. 68
    leonxger/deepseek-colab
    Run uncensored DeepSeek-V4.1-Flash IQ2_XS on Google Colab with CUDA.
    GitHub · ★ 9 · Python · first seen 2026-10-30 · reddit
    heat 0.6
    0 / 1 mentions
  69. 69
    q8atnight/strata-dual-3090
    Runs Strata LLM across two RTX 3090 GPUs with NVLink peer splitting.
    GitHub · ★ 5 · C++ · first seen 2026-10-30 · reddit
    heat 0.6
    0 / 1 mentions
  70. 70
    anthonybudd/llama.cpp-wasm-webgpu
    LLM running in browser via llama.cpp WASM with WebGPU.
    GitHub · ★ 5 · HTML · first seen 2026-10-05 · reddit
    heat 0.6
    0 / 3 mentions
  71. 71
    generated-computers/locals
    Run many AI agents simultaneously on Mac, up to 3x faster.
    GitHub · ★ 445 · Python · first seen 2026-10-21 · hn
    heat 0.6
    0 / 1 mentions
  72. 72
    tara-vision/taraengine
    Private OpenAI-compatible local inference engine built on llama.cpp for various models.
    GitHub · ★ 1 · Go · first seen 2026-10-21 · hn
    heat 0.6
    0 / 1 mentions
  73. 73
    zw5/ane-fieldfare
    Fast local inference for Gemma 4 26B model.
    GitHub · ★ 16 · Swift · first seen 2026-10-21 · hn
    heat 0.6
    0 / 1 mentions
  74. 74
    stew675/llama-cpp-rdna-boosts
    High-speed RDNA/HIP patches for llama.cpp with numerics assurance.
    GitHub · ★ 231 · Shell · first seen 2026-10-17 · hn · reddit
    heat 0.6
    0 / 2 mentions
  75. 75
    flashml-org/freetoken
    Run massive models locally on desktop.
    GitHub · ★ 15,294 · Python · first seen 2026-08-21 · hn · reddit
    heat 0.6
    0 / 8 mentions
  76. 76
    mushroom-systems/lichen
    Local API-compatible replacement for Jev TypeSafe System One model.
    GitHub · ★ 59 · Python · first seen 2026-09-23 · hn
    heat 0.6
    0 / 7 mentions
  77. 77
    staylamebro/backburner
    Run 27B model faster using iPhone GPU over USB-C cable.
    GitHub · ★ 1,147 · Python · first seen 2026-10-03 · hn · reddit
    heat 0.6
    0 / 2 mentions
  78. 78
    vx-lang/glm-vx
    GLM-5.3 serving engine with experimental Vx CPU kernels.
    GitHub · ★ 4 · Python · first seen 2026-10-20 · hn
    heat 0.6
    0 / 1 mentions
  79. 79
    warpfront/hipfire
    RDNA-native LLM inference engine in Rust for AMD GPUs.
    GitHub · ★ 782 · Rust · first seen 2026-10-01 · hn
    heat 0.6
    0 / 3 mentions
  80. 80
    lxe/dsx
    DeepSeek-V4.1-Flash on dual RTX 3090 with expert deferral
    GitHub · ★ 88 · Rust · first seen 2026-10-19 · hn · reddit
    heat 0.6
    0 / 2 mentions
  81. 81
    tin-hat/qwen-tin
    Streams MoE experts from NVMe for low-memory inference
    GitHub · ★ 3 · C++ · first seen 2026-10-19 · hn
    heat 0.6
    0 / 1 mentions
  82. 82
    ag88/llama.cpp-model-runner
    Simple shell script launcher for llama.cpp server models.
    GitHub · Shell · first seen 2026-10-27 · reddit
    heat 0.6
    0 / 1 mentions
  83. 83
    geobellingham/pumpkin
    Easy tool for running AI models locally on your machine.
    GitHub · ★ 11 · Python · first seen 2026-10-27 · reddit
    heat 0.6
    0 / 1 mentions
  84. 84
    ninnix/q36
    Local Qwen inference engine for Vulkan and Metal.
    GitHub · ★ 77 · C · first seen 2026-08-04 · reddit
    heat 0.5
    0 / 2 mentions
  85. 85
    cuppatea1983/leviathan
    Local LLM inference engine with CAD and coding automation capabilities.
    GitHub · ★ 3 · Python · first seen 2026-10-18 · hn
    heat 0.5
    0 / 1 mentions
  86. 86
    kmic-68/llama.cpp
    llama.cpp optimized for 2x Tesla P100 with 2x decode speed.
    GitHub · ★ 26 · C++ · first seen 2026-10-18 · hn
    heat 0.5
    0 / 1 mentions
  87. 87
    ahtx/kindi
    Custom LLM inference engine for low-cost GPUs.
    GitHub · ★ 112 · Python · first seen 2026-10-26 · reddit
    heat 0.5
    0 / 1 mentions
  88. 88
    classevelabs/rai
    CPU-only LLM inference with hand-written AVX2 kernels in Rust.
    GitHub · ★ 14 · Rust · first seen 2026-10-02 · hn · reddit
    heat 0.5
    0 / 2 mentions
  89. 89
    lithos-ai/lithos-metal
    High-performance LLM inference kernels for Apple silicon.
    GitHub · ★ 331 · Python · first seen 2026-10-17 · hn
    heat 0.5
    0 / 1 mentions
  90. 90
    inordic/cmp170hx-vllm-qwen38-flash-next
    Qwen3.8-Flash on dual GPUs with vLLM pipeline parallelism.
    GitHub · ★ 3 · Python · first seen 2026-10-25 · reddit
    heat 0.5
    0 / 1 mentions
  91. 91
    maxdevv/lean-qwenflash
    Run Qwen3.8-Flash with 262k context on 8GB RAM and 16GB VRAM.
    GitHub · ★ 62 · Python · first seen 2026-10-25 · reddit
    heat 0.5
    0 / 1 mentions
  92. 92
    moontato/strata-orin
    Strata LLM inference port for NVIDIA Jetson Orin with CUDA optimization.
    GitHub · C++ · first seen 2026-10-25 · reddit
    heat 0.5
    0 / 1 mentions
  93. 93
    sybil-solutions/glm-flash-lite
    Run GLM-5.3-Flash on single RTX 3090 with elastic GPU cache.
    GitHub · ★ 348 · Python · first seen 2026-10-25 · reddit
    heat 0.5
    0 / 1 mentions
  94. 94
    beehive-lab/jitllm
    High-performance LLM inference for JVM on GPUs.
    GitHub · ★ 292 · Java · first seen 2026-10-24 · hn
    heat 0.5
    0 / 2 mentions
  95. 95
    icefog72/ik_llama.cpp
    llama.cpp fork with additional quants and improved performance.
    GitHub · ★ 3 · C++ · first seen 2026-10-24 · reddit
    heat 0.5
    0 / 1 mentions
  96. 96
    theivanr/llama-on-legacy-gpu
    High-performance llama.cpp patches for legacy GPUs.
    GitHub · ★ 6 · first seen 2026-10-24 · reddit
    heat 0.5
    0 / 1 mentions
  97. 97
    shifulegend/project-zero
    CPU-optimized LLM inference engine in C.
    GitHub · ★ 54 · C · first seen 2026-07-22 · reddit
    heat 0.5
    0 / 4 mentions
  98. 98
    theroyallab/tabbyapi
    API server for Exllama with OpenAI compatibility.
    GitHub · ★ 1,504 · Python · first seen 2026-08-17 · reddit
    heat 0.5
    0 / 3 mentions
  99. 99
    recursive-logic-core/driftbreak
    Local VRAM governor with deterministic state recovery for AI systems.
    GitHub · Python · first seen 2026-09-01 · hn
    heat 0.5
    0 / 3 mentions
  100. 100
    leonccaa/revolta
    High-throughput inference server for multi-GPU NVIDIA V100 with KV offload.
    GitHub · ★ 29 · Python · first seen 2026-10-23 · reddit
    heat 0.5
    0 / 1 mentions
  101. 101
    nero7991/llm.vhdl
    VHDL Qwen LLM inference engine on FPGA fabric.
    GitHub · ★ 44 · VHDL · first seen 2026-10-04 · hn · reddit
    heat 0.4
    0 / 2 mentions
  102. 102
    mohammadhaishemkhawaja/hashyy
    177B MoE model at 11.5 tok/s on 12GB GPU via expert streaming.
    GitHub · ★ 23 · Python · first seen 2026-10-03 · hn · reddit
    heat 0.4
    0 / 2 mentions
  103. 103
    ryan75195/proposed-api-catalogue
    Benchmark vLLM servers with plain-English performance reports
    GitHub · first seen 2026-10-22 · reddit
    heat 0.4
    0 / 1 mentions
  104. 104
    theimmortalpython/pklm-sandbox
    Sandbox for PKLM Core with Hugging Face logits processing
    GitHub · ★ 1 · Python · first seen 2026-09-28 · hn
    heat 0.4
    0 / 2 mentions
  105. 105
    barretts/floatie
    Local Laguna S 2.1 Q3 inference engine development.
    GitHub · C++ · first seen 2026-10-13 · hn
    heat 0.4
    0 / 1 mentions
  106. 106
    seuros/mdma
    LLM inference engine with custom kernels and exact memory admission.
    GitHub · ★ 18 · Rust · first seen 2026-10-13 · hn
    heat 0.4
    0 / 1 mentions
  107. 107
    openbmb/llama.cpp-omni
    Omni inference in C/C++ for local LLM runtime.
    GitHub · ★ 41 · C++ · first seen 2026-10-21 · reddit
    heat 0.4
    0 / 1 mentions
  108. 108
    recursive-oss/rec
    Lightning-fast inference engine for local LLMs.
    GitHub · ★ 258 · Python · first seen 2026-10-21 · reddit
    heat 0.4
    0 / 1 mentions
  109. 109
    happyme531/open_rknpu2_lm_infer
    Open-source high-performance LLM inference engine for RK3588 NPU.
    GitHub · ★ 59 · Rust · first seen 2026-10-20 · reddit
    heat 0.4
    0 / 1 mentions
  110. 110
    ohgeebtw/geezuz-qwen3.8-27b-dual-gpu
    Multi-GPU CUDA kernel for Qwen3.8-27B with speculative decoding.
    GitHub · ★ 16 · Cuda · first seen 2026-10-20 · reddit
    heat 0.4
    0 / 1 mentions
  111. 111
    raymondhuang210129/llama.cpp-adaptive-kv-streaming
    LLM inference engine in C and C++.
    GitHub · ★ 465 · C++ · first seen 2026-08-31 · hn · reddit
    heat 0.4
    0 / 2 mentions
  112. 112
    alexbrandes/cmp170hx-qwen38-flash-next
    Qwen3.8-Flash-Next on 2x CMP 170HX with 226 tok/s.
    GitHub · ★ 10 · Python · first seen 2026-10-19 · reddit
    heat 0.4
    0 / 1 mentions
  113. 113
    dominikbucko/qwen38-flash-next-2x3090
    Qwen3.8-Flash-Next optimized for dual RTX 3090 with uncensored mode
    GitHub · ★ 131 · Python · first seen 2026-10-19 · reddit
    heat 0.4
    0 / 1 mentions
  114. 114
    publicexcess/ijah
    Raw-metal inference engine for Qwen3.8-Flash-Next on Apple M5
    GitHub · ★ 11 · Python · first seen 2026-10-19 · reddit
    heat 0.4
    0 / 1 mentions
  115. 115
    publicexcess/splish
    Fork of Splash tuned for Apple M5 GPUs
    GitHub · ★ 32 · C++ · first seen 2026-10-19 · reddit
    heat 0.4
    0 / 1 mentions
  116. 116
    wegotomars/tinkerbox
    Inference for specialized models on Apple Neural Engine via Core ML
    GitHub · ★ 12 · Python · first seen 2026-10-19 · reddit
    heat 0.4
    0 / 1 mentions
  117. 117
    argonautlabsai/argodrive
    Run large mixture-of-experts language models efficiently directly from solid-state drives.
    GitHub · ★ 6 · Python · first seen 2026-09-15 · hn
    heat 0.4
    0 / 4 mentions
  118. 118
    bertholomus/deepseek-v4.1-tensorfold-tp2-2xgb10
    DeepSeek-V4.1-Flash on 2x GB10 with TensorFold TP2 engine
    GitHub · ★ 60 · Python · first seen 2026-10-18 · reddit
    heat 0.4
    0 / 1 mentions
  119. 119
    blockwayz/llama.cpp
    llama.cpp fork with Agens Volundr model support
    GitHub · ★ 2 · C++ · first seen 2026-10-18 · reddit
    heat 0.4
    0 / 1 mentions
  120. 120
    elevatormusic/deepseek-v4.1-flash-1017-exl3-tensorfold-2x-dgx-spark
    DeepSeek model optimized for 2x NVIDIA DGX Spark with TensorFold acceleration.
    GitHub · ★ 72 · Python · first seen 2026-10-18 · reddit
    heat 0.4
    0 / 1 mentions
  121. 121
    foldl/chatllm.cpp
    Pure C++ implementation for real-time LLM chat on CPU and GPU.
    GitHub · ★ 938 · C++ · first seen 2026-10-18 · reddit
    heat 0.4
    0 / 1 mentions
  122. 122
    guregodevo/janis
    Local LLM inference engine written in pure Go.
    GitHub · Go · first seen 2026-11-01 · hn
    heat 0.4
    0 / 1 mentions
  123. 123
    vnk8071/unified-ai-runtime
    Unified C API for all inference runtimes including llama.cpp and whisper.cpp.
    GitHub · ★ 4 · C · first seen 2026-10-18 · reddit
    heat 0.4
    0 / 1 mentions
  124. 124
    sqliteai/warp
    Stream model weights from NVMe for inference.
    GitHub · ★ 2,527 · C · first seen 2026-08-14 · hn
    heat 0.3
    0 / 6 mentions
  125. 125
    loopai-hq/pulsar
    Fast inference engine for Qwen models on Apple Silicon GPUs.
    GitHub · ★ 10 · C++ · first seen 2026-10-09 · hn
    heat 0.3
    0 / 1 mentions
  126. 126
    msuiche/weightless
    Projective refusal steering without loading model weights.
    GitHub · ★ 126 · Python · first seen 2026-10-17 · reddit
    heat 0.3
    0 / 1 mentions
  127. 127
    nicorobinreal/llmspeed
    Preview local LLM speed before buying hardware.
    GitHub · ★ 166 · TypeScript · first seen 2026-10-17 · reddit
    heat 0.3
    0 / 1 mentions
  128. 128
    wormley/b70_quad
    Recipes for running LLMs on 4 Intel Arc B70 cards.
    GitHub · Python · first seen 2026-10-17 · reddit
    heat 0.3
    0 / 1 mentions
  129. 129
    deepanwadhwa/opendecision
    Lightweight CPU-friendly open-source decision model alternative to Jev.
    GitHub · ★ 114 · Python · first seen 2026-09-19 · hn
    heat 0.3
    0 / 5 mentions
  130. 130
    fabiogreter/lily-qwen3.8-flash-next
    Qwen3.8-Flash-Next optimized for Apple Silicon M5 or newer.
    GitHub · ★ 12 · Rust · first seen 2026-10-08 · hn
    heat 0.3
    0 / 1 mentions
  131. 131
    aparajith24/nano-vllm
    Mini vLLM inference engine implementation in PyTorch.
    GitHub · ★ 9 · Python · first seen 2026-10-16 · reddit
    heat 0.3
    0 / 1 mentions
  132. 132
    bodhi37/qwen3.8-flash-12gbram-32gbvram-ssd-recipe
    Qwen 3.8 Flash optimized for 12GB VRAM and 32GB RAM.
    GitHub · ★ 21 · Shell · first seen 2026-10-16 · reddit
    heat 0.3
    0 / 1 mentions
  133. 133
    bodhi37/strata
    Experimental Strata fork for Qwen 3.8 inference.
    GitHub · ★ 7 · C++ · first seen 2026-10-16 · reddit
    heat 0.3
    0 / 1 mentions
  134. 134
    knapcio/deepseek-v4.1-flash-4x-dgx-spark-tp4
    Tuned TP4 serving profile for DeepSeek V4.1 Flash on DGX Spark.
    GitHub · ★ 85 · Python · first seen 2026-10-16 · reddit
    heat 0.3
    0 / 1 mentions
  135. 135
    mattjcoles/lgtmaybe
    Any model with no API keys, maybe a merge.
    GitHub · ★ 183 · Python · first seen 2026-10-16 · reddit
    heat 0.3
    0 / 1 mentions
  136. 136
    jadidbourbaki/llama.cpp
    Port or fork of llama.cpp for running LLMs in C/C++.
    GitHub · ★ 6 · C++ · first seen 2026-09-27 · hn
    heat 0.3
    0 / 3 mentions
  137. 137
    cyqlelabs/flux
    Tool to run LLMs locally across GPU and CPU.
    GitHub · ★ 8 · Rust · first seen 2026-10-07 · hn
    heat 0.3
    0 / 1 mentions
  138. 138
    yavarb/moefit
    Run large MoE models on Apple Silicon with expert paging.
    GitHub · ★ 3 · Python · first seen 2026-10-07 · hn
    heat 0.3
    0 / 1 mentions
  139. 139
    brainchillz/r9700-tp4-allreduce
    One-shot P2P all-reduce kernel for 4x Radeon AI PRO R9700 GPUs with vLLM.
    GitHub · ★ 1 · HIP · first seen 2026-10-15 · reddit
    heat 0.3
    0 / 1 mentions
  140. 140
    henrique-simoes/llama.cpp
    llama.cpp fork adding DFlash2 speculative decoding with multi-GPU tensor splitting.
    GitHub · C++ · first seen 2026-10-15 · reddit
    heat 0.3
    0 / 1 mentions
  141. 141
    jdrews/vllm-mxfp4
    Fastest vLLM build optimized for AMD Radeon R9700 GPUs.
    GitHub · Python · first seen 2026-10-15 · reddit
    heat 0.3
    0 / 1 mentions
  142. 142
    lkarlslund/ninfer6000
    High-performance single-GPU inference for selected model checkpoints.
    GitHub · ★ 12 · C++ · first seen 2026-10-15 · reddit
    heat 0.3
    0 / 1 mentions
  143. 143
    mybigday/llama.rn
    React Native binding for llama.cpp to run LLMs on Android and iOS.
    GitHub · ★ 1,064 · C · first seen 2026-10-15 · reddit
    heat 0.3
    0 / 1 mentions
  144. 144
    neuromaniacmd/affinity
    Fork adding DeepSeek V4/V4.1 Flash support for dual Radeon AI PRO GPUs.
    GitHub · ★ 1 · C++ · first seen 2026-10-15 · reddit
    heat 0.3
    0 / 1 mentions
  145. 145
    afshinm/laya-mps
    Run Jev-style typed decisions locally on Mac with low RAM usage.
    GitHub · ★ 25 · Python · first seen 2026-09-21 · hn
    heat 0.3
    0 / 2 mentions
  146. 146
    fesens/opentpu
    Open-source AI accelerator running LLMs on FPGA hardware.
    GitHub · ★ 883 · Verilog · first seen 2026-10-06 · hn
    heat 0.3
    0 / 1 mentions
  147. 147
    sf-stav/velogb10
    GB10-optimized inference engine
    GitHub · ★ 124 · Rust · first seen 2026-10-06 · hn
    heat 0.3
    0 / 1 mentions
  148. 148
    cosmonomic/pulsarlm
    LLM inference engine that forgets old tokens for speed.
    GitHub · ★ 4 · C++ · first seen 2026-10-14 · reddit
    heat 0.3
    0 / 1 mentions
  149. 149
    gitshop77/gyrostrata
    Strata engine support for Gyro-M on 12GB VRAM.
    GitHub · ★ 14 · Python · first seen 2026-10-14 · reddit
    heat 0.3
    0 / 1 mentions
  150. 150
    jitllm/jitllm
    Inference engine adapting to hardware to run models faster.
    GitHub · ★ 57 · Go · first seen 2026-10-14 · reddit
    heat 0.3
    0 / 1 mentions
  151. 151
    sootaugur/qwen-mla-vllm
    vLLM plugin for training-free MLA retrofits of Qwen models.
    GitHub · Python · first seen 2026-10-14 · reddit
    heat 0.3
    0 / 1 mentions
  152. 152
    zeuss5/cuda-exl3
    EXL3 CUDA kernels for Blackwell with vLLM plugin.
    GitHub · ★ 7 · Python · first seen 2026-10-14 · reddit
    heat 0.3
    0 / 1 mentions
  153. 153
    giveen/ninfer-ext
    Forked inference engine optimized for speed.
    GitHub · ★ 26 · C++ · first seen 2026-10-13 · reddit
    heat 0.3
    0 / 1 mentions
  154. 154
    he-be/turbo-fieldfare
    Run Gemma 4 26B inference in about 2GB RAM on Apple Silicon.
    GitHub · ★ 2 · Swift · first seen 2026-10-27 · hn
    heat 0.3
    0 / 1 mentions
  155. 155
    mozilla-ai/llamafile
    Distribute and run LLMs with a single file.
    GitHub · ★ 26,539 · C++ · first seen 2026-10-27 · hn
    heat 0.3
    0 / 1 mentions
  156. 156
    naklitechie/diskformer.js
    Run MoE models larger than GPU in browser tab.
    GitHub · ★ 2 · JavaScript · first seen 2026-10-13 · reddit
    heat 0.3
    0 / 1 mentions
  157. 157
    quartermaster-labs/quartermaster
    Local AI inference server hot-swapping models behind unified OpenAI-compatible API.
    GitHub · ★ 33 · Go · first seen 2026-10-13 · reddit
    heat 0.3
    0 / 1 mentions
  158. 158
    valeriodolci/vramble
    GPU lease manager queuing local services wanting exclusive access.
    GitHub · ★ 10 · Python · first seen 2026-10-13 · reddit
    heat 0.3
    0 / 1 mentions
  159. 159
    mizorewww/laya-coreml
    Local Laya typed decisions on Apple Core ML with fast inference.
    GitHub · ★ 1,623 · Python · first seen 2026-09-20 · hn
    heat 0.3
    0 / 3 mentions
  160. 160
    asf0/strix-llama.cpp
    LLM inference engine in C and C++.
    GitHub · ★ 3 · C++ · first seen 2026-10-12 · reddit
    heat 0.3
    0 / 1 mentions
  161. 161
    halo-box/strix-llama.cpp
    LLM inference engine in C and C++.
    GitHub · ★ 251 · C++ · first seen 2026-10-12 · reddit
    heat 0.3
    0 / 1 mentions
  162. 162
    ledissolution/gewell
    Gemma 4 inference engine for local model execution.
    GitHub · ★ 9 · Cuda · first seen 2026-10-12 · reddit
    heat 0.3
    0 / 1 mentions
  163. 163
    lukolszewski/vllm
    High-throughput memory-efficient inference and serving engine for LLMs.
    GitHub · ★ 2 · Python · first seen 2026-10-12 · reddit
    heat 0.3
    0 / 1 mentions
  164. 164
    mrinohk/qwen36-p100-patches
    Tesla P100 llama.cpp patches doubling Qwen3.5 decode and prompt speed.
    GitHub · ★ 4 · Shell · first seen 2026-10-12 · reddit
    heat 0.3
    0 / 1 mentions
  165. 165
    shinbunbun/llama-cpp-p100-patches
    llama.cpp performance patches developed on Tesla P100 GPU.
    GitHub · ★ 57 · Nix · first seen 2026-10-12 · reddit
    heat 0.3
    0 / 1 mentions
  166. 166
    tc-mb/llama.cpp-omni
    Omni multimodal inference in C/C++ for local execution.
    GitHub · ★ 314 · C++ · first seen 2026-10-12 · reddit
    heat 0.3
    0 / 1 mentions
  167. 167
    rdaum/eider
    Performant neural network inference experiment.
    GitHub · ★ 27 · Rust · first seen 2026-07-19 · hn · reddit
    heat 0.3
    0 / 13 mentions
  168. 168
    labscommunity/cascadia
    Run any model on Intel silicon with distributed inference.
    GitHub · ★ 52 · Rust · first seen 2026-08-12 · hn
    heat 0.3
    0 / 2 mentions
  169. 169
    ninjapear/gufo-qwen3.6-35b-a3b-q6dense
    3000 tok/s prefill fork for Qwen 3.6 35B on Strix Halo.
    GitHub · ★ 13 · C++ · first seen 2026-10-03 · hn
    heat 0.3
    0 / 1 mentions
  170. 170
    cortexist/exllamav3
    Optimized quantization and inference library for local LLMs.
    GitHub · ★ 3 · Python · first seen 2026-10-11 · reddit
    heat 0.2
    0 / 1 mentions
  171. 171
    giveen/decapybara
    Open-source C++ and CUDA reconstruction of MegaCapybara inference server.
    GitHub · ★ 2 · C++ · first seen 2026-10-11 · reddit
    heat 0.2
    0 / 1 mentions
  172. 172
    extraltodeus/pendulum
    Personality tuning and control vectors for local GGUF models.
    GitHub · ★ 5 · Python · first seen 2026-10-10 · reddit
    heat 0.2
    0 / 1 mentions
  173. 173
    yibie/strata-mlx
    Qwen3.8-Flash-Next on Apple Silicon using MLX engine and GGUF files.
    GitHub · ★ 110 · Python · first seen 2026-10-10 · reddit
    heat 0.2
    0 / 1 mentions
  174. 174
    ckcool6/gguf_libra
    LLM client design based on forgetting mechanism for memory management.
    GitHub · ★ 2 · JavaScript · first seen 2026-10-01 · hn
    heat 0.2
    0 / 1 mentions
  175. 175
    sktime/tserve
    Local inference server for time series foundation models.
    GitHub · ★ 19 · Python · first seen 2026-10-01 · hn
    heat 0.2
    0 / 1 mentions
  176. 176
    microsoft/windowsml
    Microsoft's high-performance local AI inference framework for Windows.
    GitHub · ★ 49 · PowerShell · first seen 2026-10-09 · hn
    heat 0.2
    0 / 2 mentions
  177. 177
    omgitsbase/llmash
    Ollama-compatible LLM server on tuned llama.cpp.
    GitHub · ★ 37 · C++ · first seen 2026-09-07 · hn
    heat 0.2
    0 / 3 mentions
  178. 178
    scosman/cursed_browser
    Offline AI assistant running local LLMs on Android and Windows
    GitHub · ★ 282 · Swift · first seen 2026-10-22 · hn
    heat 0.2
    0 / 1 mentions
  179. 179
    syv-ai/qwen38-27b-rtx3090
    Optimized Qwen 3.8 inference on single RTX 3090.
    GitHub · ★ 2,473 · Python · first seen 2026-08-15 · hn · reddit
    heat 0.2
    0 / 11 mentions
  180. 180
    pragmatwice/jeva.cpp
    llama.cpp fork with JEV-compatible decision API
    GitHub · ★ 4 · C++ · first seen 2026-09-28 · hn
    heat 0.2
    0 / 1 mentions
  181. 181
    halfvulpes/deepinfer-sm80
    DeepInfer implementation optimized for SM80 Ampere architecture.
    GitHub · ★ 91 · Python · first seen 2026-10-20 · hn
    heat 0.2
    0 / 1 mentions
  182. 182
    peonist-ai/halogen-server
    Fast Qwen 27B inference on AMD Strix Halo GPUs
    GitHub · ★ 151 · Python · first seen 2026-10-06 · hn
    heat 0.2
    0 / 2 mentions
  183. 183
    Can I Run AI Locally
    Determine which AI models you can run locally.
    catalog · fmhy · first seen 2026-10-06 · catalog
    heat 0.2
    0 / 1 mentions
  184. 184
    cactus-compute/needle
    On-device foundation model for tiny devices with tool calling and embeddings.
    GitHub · ★ 14,434 · Python · first seen 2026-08-10 · hn · reddit
    heat 0.2
    0 / 6 mentions
  185. 185
    5p00kyy/infermeld
    AMD Vulkan and NVIDIA CUDA inference kit using llama.cpp.
    GitHub · ★ 1 · Python · first seen 2026-10-05 · reddit
    heat 0.2
    0 / 1 mentions
  186. 186
    blockwayz/agens-sglang
    SGLang with Agens Volundr model support and reasoning budget.
    GitHub · ★ 1 · Dockerfile · first seen 2026-10-05 · reddit
    heat 0.2
    0 / 1 mentions
  187. 187
    willswire/swift-newt
    On-device multiple-choice answers with Swift and Core AI.
    GitHub · ★ 3 · Swift · first seen 2026-09-26 · hn
    heat 0.2
    0 / 1 mentions
  188. 188
    eelgaev/strata-ac922
    Qwen3.8-Flash-Next on IBM AC922 unified memory NVLink system.
    GitHub · ★ 11 · C++ · first seen 2026-10-04 · reddit
    heat 0.2
    0 / 1 mentions
  189. 189
    jiuyue0820/strata
    One-click Qwen3.8-Flash-Next local inference for consumer hardware.
    GitHub · C++ · first seen 2026-10-04 · reddit
    heat 0.2
    0 / 1 mentions
  190. 190
    tomtsai28/pulsar-asm
    Pure x86-64 assembly LLM engine for Gemma-2B in 5KB code.
    GitHub · ★ 1 · HTML · first seen 2026-10-04 · reddit
    heat 0.2
    0 / 1 mentions
  191. 191
    argonautlabsai/deltafin
    Large model streaming from SSDs on Apple Silicon.
    GitHub · ★ 99 · Rust · first seen 2026-09-08 · hn
    heat 0.2
    0 / 5 mentions
  192. 192
    neurall/llama.cpp
    Llama.cpp fork scaling 2-20x on multi-GPU partial VRAM.
    GitHub · ★ 16 · C++ · first seen 2026-09-25 · hn
    heat 0.2
    0 / 1 mentions
  193. 193
    therezor/cardputer-ai
    Fully offline 8M param LLM chatbot on ESP32-S3 Cardputer.
    GitHub · ★ 126 · C++ · first seen 2026-09-25 · hn
    heat 0.2
    0 / 1 mentions
  194. 194
    daragontech/dlaya
    Fast local Laya decision model binding for Delphi and Pascal.
    GitHub · ★ 5 · Pascal · first seen 2026-10-03 · reddit
    heat 0.2
    0 / 1 mentions
  195. 195
    daragontech/liblayax
    Cross-platform C API library for in-process Laya decision model.
    GitHub · ★ 11 · C++ · first seen 2026-10-03 · reddit
    heat 0.2
    0 / 1 mentions
  196. 196
    daragontech/llaya
    Lua binding for in-process Laya decision model inference.
    GitHub · ★ 5 · C · first seen 2026-10-03 · reddit
    heat 0.2
    0 / 1 mentions
  197. 197
    daragontech/rlaya
    Safe Rust binding for in-process Laya decision model.
    GitHub · ★ 3 · Rust · first seen 2026-10-03 · reddit
    heat 0.2
    0 / 1 mentions
  198. 198
    lkarlslund/laya.cpp
    C++ inference for Laya typed decisions with multi-backend support.
    GitHub · ★ 129 · C++ · first seen 2026-10-03 · reddit
    heat 0.2
    0 / 1 mentions
  199. 199
    xanpavle/rocmfix
    Tool to auto-detect AMD GPUs and configure local AI settings.
    GitHub · ★ 17 · Python · first seen 2026-09-19 · hn
    heat 0.2
    0 / 2 mentions
  200. 200
    jadidbourbaki/bobcat
    Inference engine optimized for Apple silicon.
    GitHub · ★ 5 · Rust · first seen 2026-10-02 · reddit
    heat 0.2
    0 / 1 mentions