In every battle, it shows your party's HP on the top right, and the enemy's HP (and I think their Element?) on the top left. As the enemy's HP goes down, the green circle next to their name will change colors, from green to blue to yellow to red.
When it turns red, you can try to capture that enemy. (All party members have their own "capture" skill, so you can keep trying with everybody until you succeed.)
Sometimes the circle will turn red but the enemy will still have enough HP that it's harder to capture (or at least, it seems that way to me.) So you can take a chance and attack the enemy 1 or 2 more times, and then try to capture again (as long as you don't kill the enemy in the process.)
Also, when you capture an enemy they go into your "roster" on the bottom left corner. If you click the picture of the guy with what looks like the sun behind him, you can release an enemy. Doing so will change your party's element for the remainder of the battle. (I'm not really clear on how or why you'd want to change the element so if anybody has good information on when to do this I'd appreciate it.)
NOTE: The "roster" makes it look like you can only hold 6 enemies at a time - but this is not the case. (I currently have 95 enemies - or "Onigoes" - in my inventory.)
Along with releasing enemies in battle, you also need to use them to craft different items and equipment (which will be covered in a separate topic).
At the end of battle, you get Party EXP, Energy, and if you captured any enemies, Hunter EXP.
Party EXP levels up your party, and as your level increases you learn more skills.
Energy gained just replenishes your energy.
Hunter EXP is important, because as your Hunter rank goes up you can (among other things) store a higher maximum amount of energy! The starting max is 1,000 energy, but with each increase of your Hunter rank the amount goes up.