The button to delete a game save can be found on the main screen.
The player can either resume the game by clicking on the chimney or click the little cross at the bottom to delete that save.
There is an alternative way of deleting the saves if you are having trouble.
The data for all saved games can be found in the appdata folder.
C:\Users\<Your Username>\appdata\Roaming\Little Inferno\profiles.bin