Haven't had any issues with colours fading, but then I'd always paint porous filler with watered PVA (white) woodglue (2 coats watered 3:1 works well), then begin your paintjob on top of that.
That fading issue might be an essential part of the paint base getting sucked away into the very dry porous material and not sticking around at the surface to do it's job.

I don't use pva on the denser boards as I've found the more vinyl-like surface tends to separate from the pva which tends to flake.

I do tend to inset a lot of oddments, bark and cork for cliff/weathered textures, bits of old root for trees, small stones (helps living at the seashore). When a block of DAS or similar clay starts to get stiff and look past it's best it soon turns into fancy figure bases, 28mm roofing, field boundaries and will now be rough small terrain features like sea cliffs, rows of houses and castle/harbour parts. It's amazing just how much random future project useful stuff a fist sized mass of that stuff can turn into and it's (almost) cheap as dirt.