Nothing is 100% crazy, but in my experience with Kei trucks and vans - they're optimized for dense city work. Small diameter tires and powertrains optimized for efficient operation in city traffic on relatively flat metro streets. Nothing is impossible with the swift and frequent application of money, but when I try to tally all the changes you'd need to get such a vehicle setup for even relatively light overlanding/soft-road travel, it feels like a dead-end.
If you want to aim for small, there are other good choices - a tin-top Suzuki 4x4 isn't much bigger than a Kei truck, but has an engine and gearing at least theoretically capable of sustained highway travel and a low-range transfer case for soft-road/off-road work. They also have a TON of support in the aftermarket and enthusiast communities, making any needed repairs or upgrades much more likely to be (a) cheap and (b) well understood, so you're not re-inventing the wheel all the time.