Double Diamond knew about the rules and reality of Our Town. However, he also sometimes thought he knew Starlight Glimmer. Her smile stuck with him longer than most ponies, and for a while he tried to tell himself that it was not what he thought it was. The way she lead the town took turns in his mind until it was the mare he started to think about. Daydreams were dead in Our Town, but Double Diamond’s thoughts strayed to what Starlight Glimmer would be like as a mare instead of a beacon.
Eventually, after much turmoil he never wanted to backtrack through, he hid that he thought about what it could be like to love Starlight Glimmer. He justified this to himself with reminders that he only thought about this sometimes. The Starlight Glimmer of his thoughts was an unequal mare, which was bad, but she was harmless. After all, she was no more than the meaningless fancy of dreams. Everypony in Our Town smothered any dreams, if they had any left. Double Diamond knew he would too.
Our Town was a small place, and there were no rumors. Privacy was non-existent. Everything changed, which made it so different from any life in Equestria proper. Starlight Glimmer had a way of learning things, and that was how she learned the first secret of Our Town: Double Diamond was in love with her. He even told her himself, when they were alone.
And then they kept having time together, and Double Diamond kept trying to piece together the mare that might be Starlight Glimmer. Hers was the house that did not change, the only piece of uncomfortable constance in Our Town, and it was there Double Diamond worked towards building a relationship with Starlight Glimmer. His one secret of Our Town became half of the sole secret, and they had a shadow of what would be dates and get-togethers in that house.
That is, until one day Starlight Glimmer decided there was a different house she wanted Double Diamond to go to.
And she never told him why.