yes it's this I guess, the problem is that after the battle vs ganon link returned in the past, the timeline without link in this way is the future one when link was sent to the past..it's really strange and interesting. also, when link has sent back, in that time the game doesn't say if there is ganon or there isn't because it was defeated in the future...another idea is that all the timeline without link continue just for seven years while link is in the temple, and in these 7 years we have LttP, OoA/S etc... then, after 7 years the hero of time wakes up and defeats ganon...