ShikumiMiner AST features fail cross-project

ShikumiMiner shows that a Clang LibTooling pipeline can extract AST and CFG features from C++ local LLM inference codebases and produce a per-project distribution across seven study-defined implementation categories. Whether those categories are stable enough to classify a function in a project the model has not seen — that it does not establish.

Design

What was measured	Baseline / comparator	Sample	Conditions
Function-level category assignment (7 labels, multi-label)	Manual source-derived reference labels	250 + 80 = 330 functions (two disjoint random samples)	Random Forest, Gini impurity, bootstrap sampling, √p features per split, min_samples_leaf=5, balanced_subsample weighting, seed 42, threshold selected from {0.02…0.80} by max F1 on OOB
AST-only vs CFG-only vs fused AST+CFG	Each configuration compared to the others	7,945 distinct fused feature vectors (identical vectors collapsed)	5-fold stratified cross-validation
Cross-project generalization	Held-out project vs. model trained on the other nine	Varies by project (3 to 3,577 functions)	Leave-one-project-out (LOPO); same RF hyperparameters
Category presence overlap between projects	Pairwise set intersection over the 7 categories	10 repositories	Jaccard index on category sets
Feature–complexity association	—	10,807 functions with valid CFGs	Spearman rank correlation, 10 features

The ten repositories span from gpt2.cpp (1 source file, 5 functions) to llama.cpp (330 source files, 4,846 functions). Clang/LLVM 22.1.8 handles parsing; scikit-learn 1.9.0 handles classification. Parse rates run from 86.8% (gemma.cpp) to 100% (gpt2.cpp, minchatgpt.cpp, InferLLM).

Results

Manual validation. On the 250-function random sample, macro F1 is 0.57, micro F1 is 0.53 (74 TP, 69 FP, 60 FN). Complete agreement across all seven categories: 145 of 250 functions (58%). The 95% bootstrap CI for micro F1 is [0.47, 0.60]. A second disjoint sample of 80 functions gives macro F1 0.39, micro F1 0.45. Pool both (n=330) and micro F1 lands at 0.51, 95% CI [0.46, 0.57].

Ablation. Three feature configurations, 5-fold stratified CV, 7,945 distinct vectors:

• AST-only: macro F1 0.40, micro F1 0.46
• CFG-only: macro F1 0.22, micro F1 0.24
• Fused AST+CFG: macro F1 0.39, micro F1 0.45

Fused does not beat AST-only. Marginally lower on both metrics. No minimum detectable effect or power analysis is reported, so the near-identical scores (0.40 vs 0.39 macro) cannot be read as equivalence. Per-category, adding CFG features reduces F1 for Advanced Sampling (0.36→0.34), Memory Management (0.44→0.40), and Training Pipeline (0.20→0.18). Interactive Inference drops from 0.58 to 0.57; Model Loading and Validation holds at 0.55.

CFG-only is bad across the board. Training Pipeline: F1 0.02. Advanced Sampling: 0.12. No power analysis is reported for this arm either, given the 7,945-vector sample and 5-fold split.

Leave-one-project-out. Average macro F1 across the ten LOPO folds: 0.112. Average micro F1: 0.167. minchatgpt.cpp gets the highest micro F1 at 0.318 (37 functions held out), then llama.cpp at 0.274 (3,577 functions held out). gpt2.cpp: 0.000 on both. Three functions in the held-out set — underpowered to detect anything.

Jaccard similarity. Four repositories (llama.cpp, gemma.cpp, LMDeploy, DeepSpeed-FastGen) contain all seven categories; pairwise Jaccard similarity is 1.00. Four others (ONNX Runtime GenAI, OpenVINO GenAI, InferLLM, SGLang) share the same six categories with Training Pipeline absent — also 1.00 among themselves. Cross-group: 6/7 ≈ 0.86. minchatgpt.cpp (5 categories) scores ≈ 0.83 against six-category repos, ≈ 0.71 against seven-category repos. gpt2.cpp (2 categories): ≈ 0.14 against seven-category repos.

Spearman correlations. Across the ten features, correlations with function complexity range from 0.20 to 0.30. Strongest: function-call count (ρ_s = 0.30). Weakest: validation-path count (ρ_s = 0.20). Only 77 of 10,807 functions carry training evidence — the Training Pipeline category is constrained to a very small subset.

Reported by 2609.02789 functions· scroll →

Conditions around the comparison

scroll →

5

min_samples_leaf

42

random seed

7,945 distinct vectors

distinct vectors for ablation

250 functions

manual validation sample

Where the result stops

The LOPO evaluation is the load-bearing limitation. A macro F1 of 0.112 means the classifier performs near chance on a project it was not trained on. The study-specific categories (defined by the authors, not drawn from an existing ontology) make this worse: a reader cannot map the seven labels to a pre-existing code-architecture taxonomy without doing the mapping work themselves.

Three conditions the study was underpowered to resolve rather than tested and cleared:

• Small held-out sets. gpt2.cpp (3 functions) and minchatgpt.cpp (37 functions) — too small to distinguish a true F1 of 0.05 from 0.30. The 0.000 result for gpt2.cpp is a null, not evidence of zero generalization.
• CFG contribution. The ablation’s near-identical AST-only and fused scores (Δ macro F1 = 0.01) fall within the noise of a 5-fold split on 7,945 vectors. No confidence interval on the difference is reported.
• Training Pipeline category. 77 of 10,807 functions carry training evidence. The per-category F1 (0.18–0.20) is estimated from a very small effective sample. The CFG-only drop to 0.02 is consistent with a null rather than a specific failure mode.

Scope is also narrow: C++ source in local LLM inference/deployment codebases only. No mixed-language projects, no server-side serving code in Python or Go, no training codebases.

What to do differently

Nothing yet. Classification accuracy is too low, cross-project transfer too weak, to use ShikumiMiner’s output as a structural signal in a codebase review or code-generation evaluation. The Jaccard analysis is descriptive but not actionable — it confirms that most of these repos implement a similar set of subsystems, which is already evident from reading their READMEs. The feature correlation data (Spearman 0.20–0.30) is too weak to use as a complexity heuristic.

If you are building a tool that needs to identify function roles in a C++ inference codebase, the practical takeaway: AST structure alone carries most of the signal, CFG adds nothing measurable. But even that AST-only signal (macro F1 0.40 in-sample) does not transfer to a new project without retraining. This is consistent with the broader pattern that code-structure classifiers are highly distribution-dependent, as we noted when Tiel-Coder’s apparent gaps turned out to depend on unstated precision in the evaluation harness.

---

Method: this note was drafted by qwen/qwen3.8-27b from a single source — the arXiv abstract for 2609.02789. Before publication an automated gate re-checked every extracted claim against the source document (74 claim(s) and 119 quantity(ies) verified) and every claim-shaped number in the draft against that evidence (1 derived from it, no rows from our own tables were supplied to the draft). The studio has not re-run 2609.02789’s benchmarks; figures attributed to it are its own.

FAQ

Can I use the seven categories as a checklist for my own C++ inference engine?

No. The categories are defined by the authors for this specific study and are not grounded in a published code-architecture taxonomy. Their utility depends on the classifier assigning them correctly, and the LOPO results (macro F1 0.112) show that it does not assign them correctly on unseen projects. You would be applying a label scheme with no demonstrated transfer.

Does the ablation rule out CFG features for code classification more broadly?

No. The ablation is specific to multi-label classification of functions in C++ LLM inference codebases into seven particular categories, using a Random Forest with the stated hyperparameters, on 7,945 distinct feature vectors. The near-zero difference between AST-only and fused (0.40 vs 0.39 macro F1) is within the resolution of a 5-fold split. No CI on the difference or power analysis is reported. It does not rule out CFG usefulness for other classifiers, other languages, or other label sets.

Why is the LOPO result so much worse than the manual validation?

The manual validation samples 250 or 80 functions from the full pool and compares model predictions against human labels — the model was trained on data that includes those functions' source projects. LOPO trains on nine projects and tests on the tenth. The model must generalize across codebases that differ in scale (5 to 4,846 functions), parse rate (86.8% to 100%), and category composition (2 to 7 of 7 present). The drop from micro F1 0.51 (pooled in-sample) to 0.167 (LOPO average) reflects this distribution shift, not a measurement error.

What would it take to make this useful for evaluating a codebase I maintain?

Retrain the classifier on your own codebase (or a closely related set), validate against your own ground-truth labels, and confirm the seven categories map to something you actually care about. The paper's contribution is the pipeline (Clang LibTooling → AST/CFG feature extraction → RF classification) and the demonstration that AST features carry most of the discriminative signal in this domain. The specific model, thresholds, and category definitions are not transferable as-is.