I had this happen to me as well. I had just started out, I thought I was doing well up until then because I had only lost one person, but then I set the order to make 4 mills. I think I went down to about 4-5 people after doing that.
They'd start making the mill, then they'd get hungry, but they wouldn't stop hauling the stuff to the mill, so they died halfway through it*. I only had apples and pears for them to eat then, I don't know when they ate or what tasks they did before started doing the mill, but I didn't have them ordered to do anything else at the time.
*That created a whole new issue. They die, but then you can't click the "___ starved to death" message to see where they died to destroy the corpse (I don't bother with that until I have at least some armor first). I forgot about it and just focused on the town area for a while, then when I scanned the map for something, I saw I had about 10 Ghosts of Past Heroes roaming around. I nearly choked on my water then, heh. I can't seem to get the 3 heroes I've got running around to do anything about them either.

