A subnet mask refers to the practice of subnetting, which is, from my understanding of it, a way of dividing one network into smaller logical networks. A subnet mask is used to tell the network devices (whether it be the computer or any routers, modems, etc.) what addresses are local (belongs to this network), and what addresses are remote (belongs to that network).
So, if a computer's IP address is 192.168.1.104, and its subnet mask is 255.255.255.0, then the computer (and every other device attached to the same network) will assume that every IP on that computer's local network will be in the format 192.168.1.xxx, with xxx being the only part that would vary. Likewise, if the subnet mask is 255.255.0.0, then the computer will assume that every IP address in its local network will be in the format of 192.168.xxx.*xxx*.
Subnetting a network is not really useful in a home network. It is mainly used on larger networks (with 255+ computers) to reduce unnecessary network activity.