Probably because the metadata is in the wrong place (ie, not at the beginning of the file).
I downloaded two programs to move the metadata; MetaData Mover (http://rndware.info/products/metadata-mover.html) and MP4 Fast Start (http://www.datagoround.com/lab/), both of which killed my mp4 files, leaving them with 67kb per file. Perhaps my mp4 files were encoded in a strange way? They’re recorded using a fairly standard video camera.
In any case, after a long time of searching for the solution, turns out I only needed to do the following:
apt-get install mpeg4ip mp4creator -optimize myfile.mp4
In my case I have a client that needs to be able to upload streaming videos to her WordPress site; she now just needs to upload the video files directly from her video camera, click a little button I’ve put on the WordPress site to search for unoptimized files and optimize them. Excellent.