David Belton PRO

This model does not contain MTP layers ; you need to run at non-MTP.

As of this writing:

There are pipeline (issues as well as optimizations) issues still currently, and it is not widely supported in some AI Apps.

Specifically:
Ggufs:

• Imatrix is not yet supported for MTP.
• Not all AI apps have updated to support it -> result -> MTP ggufs do not work at all.
• Misc issues with speed still being worked on.

Training is compounded by number of experts in the model, which adds a serious level of time to the training.
Even 1000 samples [small!] takes 6-12 hrs.
Consider 31B dense , same samples, 30-60 minutes.

I will add to the list; may wait for specific Heretic and/or tuned version.

I already have a 43B-A3B version running in the lab ; however tuning these sparse moe models take a lot more work/time and ahh... detail. AND a lot more VRAM!!! [can't compress these atm, so BF16 required => 100 GB+ ]

I may make a Q6 high and/or a Q8 Hybrid and/or Q8 "HI".
Imatrix does not have any affect on Q8 or BF16 ; unless the other tensors in the model are set at Q6 or lower.

A Q8 "HI" is a special case; where one or more tensors/layers are set at BF16.

Currently working with Qwen 3.5/6 35B-A3B in the lab ; learning the "quirks" ; still a ways to go.

RE: 16-18 B ; yes, something running in the lab right now. (Gemma 4).

Also can make Qwen 3's (Version 3) moes like Llama3.2-8X3B as well ; I have some of these at my repo too.
I have built a few GPT-OSS ; and some 12B [mistral nemo] as well as mistral nemo "large" 15-17Bs...

A lot of options ;

Maybe in the future ; atm still learning/addressing quirks with these new Gemmas.
Google released three different arch structure here : "E", "MOE", and 31B dense.

Also plans to create larger Gemma 4s too ; which may work better for specific applications and/or work better period.
These are in the plans for next week.

UPDATE:
https://huggingface.co/DavidAU/gemma-4-E4B-it-The-DECKARD-Expresso-Universe-HERETIC-UNCENSORED-Thinking

Exceeds Gemma4 26B-A4B in critical benchmarks.

Training a Gemma 4 Reap 19B-A4B right now ; should be done tomorrow, then testing.

RE: FRanken merge 26B-A3B ; yes, just need to make a map for Mergekit ; this is also in progress.

RE: Claudes ; depends on how reap turns out.
There are a lot of updates still in progress with Unsloth/Llamacpp RE: Gemma 4s atm too ;
There are also some dataset issues to address when training with Gemma 4s.

NOTE:
Just finished a number of fine tunes on Gemma 4's E4B ; which is a MOE LIKE model. These will release in the next day or so ; pending final testing.

UPDATE:
All of these are now up; and can be downloaded.
Awaiting quants.

RE: 13B:
=> one is upscaled + trained, the other is merge of two 9Bs fine tunes (and upscaled).

They are hidden as of this writing (undergoing private testing), awaiting final metrics / eval.
If they "pass" ; they will be made public.

These will be active within 24-48 hrs pending results.

Currently have full running 13B (GLM 4.7 Flash) - which is very strong ; and experimental 21Bs of Qwen 3.5.
These are trained.

These are in testing, and access is limited as of this writing.

As for MOEs:
This is a little more complicated as scripting must be written for Mergekit to "moe together" 0.8B, 2B, 4B, 9Bs etc etc.
A draft (by me) has been completed to do this; but not tested/debugged yet.

No time line here ; too many variables.

RE 35B moes ; it is possible to address this in a different way ; but I have not tried it yet.
This is a different approach than REAP.