A forum hinted me to the software MP4Box, which I was able to install by installing the gpac bundle. From there, it was simply a matter of...
MP4Box a.mp4 -cat b.mp4 -out result.mp4
a is input, with -cat saying that we should concatenate b to the end of a.
This worked well enough for me, but since the video codec world is strange and mysterious, your results may vary.