Your horse can be ridden to travel faster than you can on foot. You automatically obtain a horse when you buy a Stable from Robin (10,000 gold, 100 hardwood*, 5 iron bars).
Horses in Stardew require no upkeep. They need no food, water, medicine, or petting. If you leave your horse elsewhere, it will return to its stable safely overnight.
* Hardwood is easiest found in the Secret Forest where you can harvest 12 pieces a day.