There are major and minor elements at the bottom-left of your screen, right? In order to have a chance of the monster wanting to join you, you have to make sure that at least 4 (I think) of the minor elements are the same as the major element. You can change the minor element by casting--if you can't cast any magic of the same element as the major element, then you can change the major element by casting at least 3 of the same element.
If you end the battle that way, the monster might stay behind and then you'll have a chance to make it join your party. You can use a monster you already have to help if it can but regardless it's not a given that the new monster will join.
...hopefully that made sense (or is right). XD I haven't really touched the game in a month; I should really go finish it.
