Most of the calories we burn daily do not derive from exercise or from the thermic effect of food, but rather from the automatic and largely unseen operations of our bodies that are collectively called "basal metabolism". It is the energy used to "mind the store" and it is substantial. The pie chart below illustrates the relative contributions of metabolism, exercise and thermic food effects to total calorie output in a typical human being.
This is why exercise can seem so frustrating. Even if you do a lot of it, the calories burned from it are small in comparison to metabolism (and therefore to total daily calorie INTAKE).