You didn't break the capture system, or atleat, I don't think so.
You were probably trying to support shadow Pokémon, and in doing so, messed up the capture system.
Let me give you an example.
Does your pokeball in items.txt look like this?
4,POKeBALL,Poké Ball,3,400,A device for catching wild Pokémon. It is thrown like a ball at the target. It is designed as a capsule system.,0,2,3,
If so, change the 3 to a 0.
the last 3, that is.