First, let me preface this by saying that you are free to play the game however you wish in the way that brings you enjoyment.
That being said, it is the developer's intention that players create teams for each element so that you have elemental advantage for any given foe's element. Focusing on just one element is fine for early-game content and story events. But if you intend on sticking with a single element party, you will eventually reach a point where not only will your debuff success be reduced, but your damage will be significantly reduced as well when using a non-superior element.
In regards to same-element battles, you are essentially making battles harder for yourself by using Mithra, a weak summon, in place of anything else that improves your damage.