Speculative Decoding.

#8
by Sinaya - opened

Can you train speculative decoding for this model (and 4b one)? It would really become a local beast with that 2-3 X speed bump.

It would be soooo good.

I would Recommend Medusa (Just imagine that banger name: Aggressive-Medusa lol)... Unless there are better newer methods I'm not aware of.

I have already done the 2b and 4b models :) Feel free to check them out.

I noticed the 2b and 4b models but they have lower quality responses naturally because they are smaller models. What I am referring to is a method that without reducing the performance of a model makes it faster.

For example, with Medusa we can make the 9b model faster than the 4b model (with the small sacrifice of the 9b model taking up ~2 percent more VRAM than before)

I have 16GB VRAM so it's enough to run the 9b model but I get only 30 tps. With speculative decoding that could go up to 60 or 75 tps.

It would be completely lossless with zero drop in response quality.

Sign up or log in to comment