Tom Ford's Painful Memories of Childhood Bullying: 'I Was Absolutely Tortured'

For more from Tom Ford, watch the full episode of The Jess Cagle Interview, available now on the new People/Entertainment Weekly Network (PEN). Go to PEOPLE.com/PEN, or download the PEN app on Apple TV, Roku Players, Amazon Fire TV, Xumo, Chromecast, iOS and Android devices.

Tom Ford may have a Jay Z song named after him now, but he wasn’t always so popular.

During an appearance on The Jess Cagle Interview, the 55-year-old fashion mogul and filmmaker spoke about growing up in Texas and dealing with bullying as a child.

“As a kid in the ’60s, I wasn’t great at football, I was not great at team sports, I wasn’t great with my BB gun, which in Texas soon turns into a .410 shotgun and then turns into something else,” he tells PEOPLE and Entertainment Weekly‘s editorial director. “I wasn’t interested in those things and so I was teased a lot.”

Instead, Ford had already developed an interest in fashion, which didn’t sit well with his peers. “As a kid I had a very, I think you’re born the way you’re going to be and I came out dressed in a little blazer and little loafers,” he recalls. “When I was in the second grade I didn’t want to carry a book bag, I carried an attaché case. Of course I was tortured. I was absolutely tortured.”

When Ford headed to college in New York City, he found out what made him feel so ostracized — he realized he was gay.

Despite all of his success, those painful childhood memories are still with him to this day. Ford says it took him a long time to deal with the suffering, and he still flashes back to those vulnerable times.

“I think I shoved them to the back of my mind and didn’t really confront them. I think that they’re very present to date,” he says of the memories. “Still to this day if I walk past a group of kids, they can be 8 years old playing soccer, and that ball comes towards me, I panic because, ‘My God, I have to kick that ball and they’re all going to laugh because I’m not great at soccer.’ It’s a sort of instant panic.”

Still, Ford says he had a happy childhood. At home, his parents accepted him — and continue to support him — for who he is.

When asked what Ford would tell his 7-year-old self, the advice was something he tells himself to this day.

” ‘Relax,’ but that’s what I would whisper in my own ear now,” he says. ” ‘Relax, it’s going to be okay. Relax.’ “