You have hit the plateau that every serious optimization effort eventually reaches, and the honest answer is that your next meaningful win will not come from stacking more pruning on top of FP16. The tools you have already applied are table stakes now, and the fact that they are not moving the needle is not a failure on your part. It is a signal that the easy wins are gone, and the next phase requires a deliberate choice between fundamentally different strategies rather than another pass with the same set of levers.
Low-rank factorization, like SVD or LoRA-style compression, sounds appealing because it promises to shrink the model without touching the weights you have already optimized. But in practice, post-training low-rank methods rarely deliver the kind of gains you are looking for after FP16 and pruning have already removed the redundancy that these techniques are designed to exploit. The model has already been squeezed; the remaining structure is dense with information that does not compress gracefully into a lower-rank approximation without significant retraining. If you want to go down this path, you need to plan for fine-tuning as part of the process, not as an afterthought. Distillation, on the other hand, is not about compressing the existing weights. It is about training a smaller model to mimic the behavior of the original, and that is a different beast entirely. It can produce a model that is dramatically smaller and faster, but it requires a solid training pipeline, a good amount of data, and a willingness to accept that the student will not replicate the teacher perfectly.
Quantization, particularly INT8 or INT4 with methods like GPTQ or AWQ, is likely your most practical next step, but it comes with its own trade-offs. You have already seen that FP16 gives you a 2× reduction, and moving to INT8 could get you close to another 2× on top of that, with INT4 pushing even further. The catch is that aggressive quantization can degrade accuracy, and the hardware support for these formats varies widely. TensorRT and FlashAttention are worth exploring if you control the deployment environment, because they can unlock speed gains that no amount of model compression will give you. These are not competing approaches; they are complementary. The real question is not which single technique is best, but which combination fits your specific constraints around accuracy, latency, and memory.
The practical path forward is to stop chasing a single silver bullet and instead run a small, structured experiment. Pick one quantization method, apply it to your current model, and measure the accuracy drop alongside the speed and size improvements. Do the same with a distilled student model, but only if you have the data and compute budget to do it properly. Low-rank methods should be lower on your list unless you are willing to retrain. The models that ship in production today are rarely the result of one clever trick; they are the product of a series of measured trade-offs. Start with the quantization experiment because it is the cheapest to run and gives you immediate data. Then, based on those numbers, decide whether the accuracy hit is acceptable or whether distillation is worth the extra effort. That decision will tell you more than any further theorizing about what might work.