No matter how good any ages of MLP has been, there was always something about it that it shouldn't be.
Age 1: G1
This show was more than you'd expect for the first TV series based on a line of toys aimed at little girls. It begins with a dark and menacing villain named Tirek who is surprisingly violent and ends up like many Disney villains around that time. Though animated movies would do this, it was rare for TV cartoons in the 1980's to.
Yet, obviously this show wasn't that popular or else people would have remembered it in the time of Friendship is Magic and used it to say MLP can be good. Obviously no one gave it a chance coz the line of toys had a reputation around the name.
Age 2: My Little Pony Tales
Here MLP lost all appeal to older viewers and only little kids would like it.
Age 3: G3
I don't think I need to comment here.
Age 4: Friendship is Magic (Part 1)
MLP becomes better than G1. Intentional appeal to older viewers is thrown in in the form of pop culture references, comedy, and themes only we could appreciate.
On that note, when the fandom was created, it represented acceptance. People could say they loved G3 without getting a hard time. Love and tolerate! John de Lancie and the others constantly mentioned this point on their documentary. It changed people's outlook for the better and helped them a great deal. It made me more accepting of other people's interests and got me close to happiness for a time.
Yet, because of the line of toys and the previous two shows, most people not in the fandom would judge those who were and get quite nasty to them, forcing many of us to hide from all but those we really trust. (Though I do have a talent for explaining its popularity without getting any skepticism. Only my brother didn't believe me right away, but today he's trying to convince me to not give up on Season 7.)
Age 5: Friendship is Magic (Part 2)
The writers are intentionally throwing more stuff in for the fandom, and more of the world is becoming more aware and accepting of the fandom.
But, ever since Season 3 the fandom has been shrinking and filling up with people who complain about every little thing, and in some cases, belittle the opinions of others. (I even heard a theory - though I can't confirm it so don't quote me on that - that anti-bronies have joined the fandom to become part of something big, but the show hasn't improved their attitude)
Likewise, I don't get that positive energy I used to, though it's still one of my favorite shows.
And, eventually the loss of some of its best writers caused the show to slowly decline in quality. It's not QUITE Spongebobbed yet but... well, I already ranted and raved about that.
At this point, I've given up hope that MLP will ever have a perfect time.
Perhaps the reason some people dislike lighthearted shows like MLP is that it's just a reminder of a world they long for that doesn't, and never will, exist. (And even Equestria isn't perfect, like the Pokemon world it's a partially-tamed deathworld)