这种重新表述变成了“我应该删除哪些障碍？”进入一个干净的优化：找到 M 个块的集合，其移除使得能量 xᵀH⁰x 最小化，前提是恰好移除 N 个块中的 M 个。从数学上讲，这是一个有约束的二元优化问题；从物理上讲，它是一个伊辛玻璃，一个具有守恒磁化强度的全耦合自旋系统（固定数量的移除块起到固定总自旋的作用）。我们建立的关键属性是，这种能量是下游质量的有力代表：自旋系统的低能量状态对应于高性能的修剪模型。最小化能量和最大化基准分数成为相同的搜索。
块选择变成一个约束二元优化问题，相当于寻找伊辛玻璃的低能态；每个解决方案都说明要删除 N 个块中的哪 M 个块。右：我们将耦合变量 α 插入到每个块的残差路径中以构建 Hessian。资料来源：论文图 1。
这是实用的原因是成本。 Hessian 矩阵，即完整的耦合集，仅根据小型校准数据集的前向和后向传递计算一次。之后，评估任何候选配置都是一次廉价的能量计算，无需运行实际模型，更不用说对其进行基准测试了。而且由于耦合不依赖于压缩目标，因此可以重复使用相同的 Hessian 矩阵来求解许多不同的 M 值。
Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
Hugging Face Blog 发布了这条动态，详情请查看官方发布。
正文 · 来源原文
Our latest paper, LLM Compression by Block Removal with Constrained Binary Optimization, takes that correspondence literally. We reformulate block selection as a constrained binary optimization (CBO) problem that maps directly onto an Ising glass, a disordered spin system with all-to-all interactions and a fixed number of "up" spins. The energy of that spin system turns out to be a strong, cheap proxy for how well the pruned model will actually score on benchmarks, which means we can rank a huge number of candidate configurations without benchmarking any of them, and hand the hard instances to the same classical and quantum-inspired solvers we use elsewhere at Multiverse. The payoff in the deep-compression regime is large: at 50% compression of Llama-3.3-70B-Instruct, we gain almost 23 percentage points on MMLU over the best competing block-removal method.
Most existing block-removal methods score each block on its own, then remove the ones that look least important, using magnitude, sensitivity, or "block influence" heuristics. In physics terms these are mean-field methods: they treat each block as if its contribution were independent of the others, the way mean-field theory replaces a spin's neighbors with a single averaged field. A related shortcut is to only ever remove a single consecutive run of blocks, which keeps the problem small but throws away most of the search space.
想要完整的技术细节，包括泰勒展开式推导、QUBO 映射、求解器基准、校准数据集消融和完整的结果表？阅读有关 Hugging Face 的完整论文，或联系我们的团队讨论将其应用到您自己的模型中。该代码在 github.com/CompactifAI/Block_removal_through_constrained_binary_optimization 上开源。
The trouble is that blocks are not independent, any more than spins in a real magnet are. Whether removing block 20 hurts the model depends on whether you also removed block 19 or block 24, an interaction, or coupling, between the two decisions. As models get deeper and more heterogeneous, ignoring those couplings leaves quality on the table, especially when you want to remove a lot of blocks at once. What you really want is to search over combinations of blocks while accounting for how they interact, but the number of combinations grows exponentially, so brute force looks hopeless. This is precisely the regime, exponentially large configuration spaces with pairwise couplings, where the tools of statistical physics earn their keep.
We attach a binary variable to each transformer block: 0 means keep it, 1 means remove it, just like a spin that can point down or up. Then we do a second-order Taylor expansion of the model's loss with respect to those variables, which produces an (approximate) Hessian matrix. The diagonal of that Hessian is how much each block matters on its own; the off-diagonal entries are exactly the pairwise couplings between blocks, the many-body physics that mean-field methods throw away.
That reformulation turns "which blocks should I remove?" into a clean optimization: find the set of M blocks whose removal minimizes the energy xᵀH⁰x, subject to removing exactly M of the N blocks. Mathematically this is a constrained binary optimization problem; physically it is an Ising glass, an all-to-all coupled spin system with conserved magnetization (the fixed number of removed blocks plays the role of a fixed total spin). The key property we establish is that this energy is a strong proxy for downstream quality: low-energy states of the spin system correspond to high-performing pruned models. Minimizing energy and maximizing benchmark score become the same search.
Block selection becomes a constrained binary optimization problem, equivalent to finding low-energy states of an Ising glass; each solution says which M of N blocks to delete. Right: the coupling variable α we insert into each block's residual path to build the Hessian. Source: paper Figure 1.
The reason this is practical is cost. The Hessian, i.e. the full set of couplings, is computed just once, from forward and backward passes on a small calibration dataset. After that, evaluating any candidate configuration is a single cheap energy calculation, no need to run the actual model, let alone benchmark it. And because the couplings don't depend on the compression target, the same Hessian can be reused to solve for many different values of M.
For most models the configuration space is large but still checkable. Because computing one energy is so cheap, we brute-force it on a single GPU, checking up to tens of billions of spin configurations. A few million take seconds; the hardest tractable case here, removing 8 of Llama-3.3-70B's 80 blocks (about 29 billion configurations), took roughly two days.
Beyond that the exact approach breaks down, and this is where casting the problem as an Ising glass pays off a second time. In its equivalent QUBO form (the constraint absorbed into a penalty term), the exact same task can be handed to the highly optimized classical, quantum, and quantum-inspired solvers built for this class of Hamiltonian, the machinery of quantum annealing, QAOA, tabu search, and specialized branch-and-bound. We find that an open-source tabu solver reliably reaches the lowest-energy states in seconds, even on the hardest cases we can verify against brute force. So the method scales to models where enumerating configurations is out of the question, using solvers that are squarely in Multiverse's domain.
There's a subtle but important point here, and it runs against the usual grain of optimization. Normally a CBO or annealing solver is judged by whether it finds the true ground state. We don't actually need the ground state. What we need is a fast way to generate a handful of good low-energy states, and that is a far easier bar, which is why lightweight solvers work so well for us and why we can afford to run several of them.
The energy is a strong proxy for quality, but not a perfect one, so the single lowest-energy state isn't always the best model. This turns out to be a feature, not a bug: once the Hamiltonian is set up, reading off the ground state and the low-lying excited states is essentially free, giving a spectrum of high-quality candidate prunings to try rather than one fragile answer. Exploring excited states, not just the ground state, is itself an area of active physics research, and it maps neatly onto what practitioners actually need here.
A concrete example: for Llama-3.1-8B-Instruct at 16/32 blocks removed, most of the top states cut blocks toward the end of the model, as prior work would expect. But the 17th excited state is the first to propose removing a block near the beginning of the model, and after light retraining that configuration outperforms the ground state across several benchmarks. That directly disproves the common assumption that the best pruning is one consecutive chunk of middle-or-late blocks, and it shows why respecting the full many-body structure of the problem pays off.
Left: which blocks each of the 20 lowest-energy states removes (red = removed). Right: the 17th excited state, which removes an early block, beats the ground state on several benchmarks after retraining. The best model is an excited state, not the ground state. Source: paper Figure 2.
Across Llama-3.1-8B-Instruct, Qwen3-14B, and Llama-3.3-70B-Instruct, our method (CBO) is on par with or better than state-of-the-art block-removal baselines, and the gap widens as compression gets more aggressive.
The clearest win is deep compression of Llama-3.3-70B-Instruct, evaluated without retraining. Up to 24 of 80 blocks removed, CBO is roughly on par with block influence. But at 32/80 and 40/80, it pulls decisively ahead, with an almost 23-point MMLU advantage at the deepest setting, where it beats the baseline on every benchmark we tested. For Qwen3-14B at 12/40 removed, CBO leads MMLU by about 10 points. At lighter compression the methods are comparable, which is expected: the couplings matter most when you're cutting deep.
At 40/80 (50% depth), CBO holds MMLU near 77 while the strongest baseline falls to the mid-50s. Source: paper Table 2.
Block removal gets much harder on modern heterogeneous architectures, where different block types are interleaved, and the Ising formulation doesn't care: a coupling is a coupling regardless of what kind of block sits at each site. To stress-test that, we applied the method to NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, a hybrid model that interleaves Mamba2, attention, and mixture-of-experts (MoE) layers in a non-uniform pattern, without any retraining.
Nothing about our formulation assumes a homogeneous stack, so it transfers directly. Removing 2–3 MoE layers or 2 attention layers, CBO finds configurations that beat block influence on AIME25 and GPQA. The results also confirm that redundancy in these hybrid models is real but unevenly distributed: some expert layers are far more disposable than others, and the method's ability to search the coupled configuration space is what locates the good cuts. Even here, the pattern from the dense models holds, the best configuration is often an excited state rather than the ground state.
Reframing a messy machine-learning problem as an Ising Hamiltonian, then solving it with the classical and quantum-inspired optimization machinery built for physics, is squarely in Multiverse's wheelhouse, it's the same instinct that runs through our compression stack. And block removal composes with the rest of that stack, quantization, low-rank/SVD compression, width pruning, and knowledge-distillation-based healing, so it slots into a larger pipeline rather than competing with it.
Want the full technical details, including the Taylor-expansion derivation, the QUBO mapping, the solver benchmarks, the calibration-dataset ablations, and the complete results tables? Read the full paper on Hugging Face, or get in touch with our team to talk about applying this to your own models. The code is open-sourced at github.com/CompactifAI/Block_removal_through_constrained_binary_optimization.
