You can challenge a player by clicking their name (the names are on the bar to the side) and hitting the "challenge" button. If a player challenges you, a notification will appear on the screen that they sent a challenge; the battle will begin when you hit "accept". As for the doubles format, i think smogon doubles would work but im not too knowledgeable in this area, i know its different from vgc though, if you are used to that.
http://www.smogon.com/forums/threads/the-beginners-guide-to-pok%C3%A9mon-showdown.3496279/
This link might help you if youre still confused. Lmk if you have anymore questions!