This is actually a good question. I would say that any game that is virtual and prompts some sort of interaction is a video game. After all, the title is pretty self explanatory. There was a long time where I found myself questioning whether Visual Novels counted as games. After all, as is the case with video games themselves, their title is pretty self explanatory. In a lot of ways, Visual Novels are just another form of visual entertainment, a sort of alternative to reading and watching a show (and, similarly, is a mesh of both of them). And I think that's true in some cases, but I don't think Visual Novel as a whole describes a non-video game media.
For instance, there are some Visual Novels that can go from beginning to end with absolutely no player input. In this case, you're on the outside. Much like a book or a television show, you are watching a story unfold, and you can do nothing to impact it other than when to pause and progress. At the same time, there are other Visual Novels, such as Ace Attorney, Zero Escape (999 and Virtue's Last Reward), and Time Hollow, that pride themselves on their player input. These are no doubt games, and in these games, the player feels almost synonymous with the protagonist, to the point that many of the protag's actions are dictated by the player (though they still have thoughts and personalities of their own). These are more gameplay heavy examples, though.
VNs like School Days, Fate/Stay Night, and Steins;Gate are the best of both worlds. They present themselves as Visual Entertainment but give the player control over the story and the actions of the player in the form of choices. These are games as well, because, at the end of the day, I would say that it is input, or larger, choice that makes a video game a video game.
People might take this to mean that only games that make the player make large, world-changing in-game decisions are games. While those decisions are a matter of choice, I'm referring to much more casual decisions. For instance, in Mario, you can decide to go straight to the end of the level. But maybe you don't want to. Maybe you want to jump on some goombas, or maybe you want to do what you can to get the highest score? Speedrun? Sure. It's these factors that dictate the fun in games. You aren't forced to do anything, and the way you do things is entirely up to you because you have an effect on whether the game progresses or not.
And Visual Novels are largely a subset of the Adventure genre, and it's greatly apparent. Adventure games are known for their many, many, MANY endings, often in the form of death. Because you are making a choice to...say, ignore the item that was in a room that you need later to avoid death (a staple of the Sierra games). Now, this may make you think that Phoenix Wright, where the answer to almost everything is precise and specific, is less of a game because there is only one way to go about it. I mean sure, it has a great story and characters, but what makes this VN/Point and Click a game by my definition?
Well, let's take a look at Final Fantasy XIII. FFXIII is something of a black sheep in the series, and it is the cardinal example of a developer stripping the player of choice because they don't quite get it. Most of the game takes place in a line, with very little derivation. Until the endgame, every place you go to and every story decision you make is planned. Even the battle system is very "Press A to Continue". What's more, it prides itself on its presentation, lore, and story. In many ways, it mirrors the example of a non-game VN. But FFXIII is a game. In the subjective sense, it is a game because the player does have the choice to do things even if they are in no way compelled to do them. Not in the sense of exploration but in battle input and shopping. In the objective sense, the game has customization that can be used to vary the party formation, items that can be used to avoid death, and the player can do things other than automatic attacks to kill an enemy that might otherwise be troublesome. It is through those "troublesome" foes that I shall make my point.
What these two games have in common is that despite the fact that they don't push for choice, there are options. They require the player to think, and unless they go to a guide, that thought process is vital to the progression of the game, even if there is only one answer and one end result. Because despite the fact that the player doesn't have direct control of the story's path, they do have control over its progression. It requires their input and their attention, and that's what makes them more than just visual entertainment alternatives or minor interactive media. Visual Novels are yet another story-telling media, as are video games. But I think, for the two to be synonymous, the player must be given the privilege of choice to have some control over the story's progression or the facets that make it up. And this goes for all genre, not just VNs. Simulators, other Adventure Games, etc.
