The president called for fewer non-working holidays even as he's spent 22% of his second presidential term on a golf course