With a dumb frame buffer there will be very little work getting X11 going.
There's already lots of support in the MIT code for 8 bit deep dumb
frame buffers. It'll just be a matter of reading the keyboard and mouse
properly (I think - I could be completely wrong).
There's also 32 bit server code (from DEC) in the contrib hierchy.
Libraries could be a challenge to port. Not because its very difficult
or machine dependant, but all kinds of correct decisions need to be made.
For example, if I were to port linux to an 4000, should off_t etc be
64 bit? If so, should the library and compile environment have support
for a 32 bit environment and a 64 bit one (by having the 32 bit syscall
lib functions convert their args and returns to 64 bit).
Take a look at the various papers from Convex ('92 Winter USENIX proceedings),
BSD CSRG, etc. on supporting Big Files.