“You can’t be a normal person at that point,” the former Philadelphia Eagle said during a recent podcast appearance

Jason Kelce is reflecting on the pros and cons of his family’s fame and how his wife, Kylie Kelce, feels about all of it.

Jason, 36, spoke candidly about the level of attention he and Kylie, 32, have had to grow accustomed to amid his brother Travis’s relationship with Taylor Swift during an episode of Andrew Santino’s Whiskey Ginger podcast.

“Travis and Taylor have taken it to another level and that’s a whole other side of it,” the former Philadelphia Eagles center said.

Jason added, “Kylie and I think we have it bad, and then we go hang out with one of them for a second. I don’t even know how to, like, you can’t be a normal person at that point.”

The father of three said that fame “gets to be too much at times” and that Kylie worries about their daughters being in the public eye.

“I think hat she’s a little bit uncomfortable with, I guess, how much it’s become and our family’s out there, like — people know what our kids look like,” Jason said, noting that they allowed it for his Kelce documentary on Prime Video in 2023.

Jason said he and Kylie have since “tried to retract” the amount of publicity on their daughters, but acknowledged that it’s “part of the territory” amid his family’s fame.

However, the beloved couple is always happy to interact with the fans who support them and act respectfully. “More often than not, it’s such an awesome interaction whenever you’re doing that stuff and the vast majority of people are beyond just respectful and appreciative,” Jason said on the podcast.

He also noted that his family is “in a position of great fortune” and that the fans are a big reason behind their success. “It’s all because of these people that are really, really big fans,” Jason said.

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer , from celebrity news to compelling human interest stories.

The retired NFL star’s comments about fame comes shortly after he and Kylie were harassed by a New Jersey woman who wanted a photo with them while they were out on a dinner date.

Margate City Mayor Michael Collins issued an apology to the couple after the incident went viral on social media.

“On behalf of the City of Margate, I’d like to formally apologize to Jason and Kylie Kelce for the experience they had in Margate City over the holiday weekend,” the Mayor wrote in his statement. “As a father with young children, I know as well as anyone the importance of a date night with your wife and would like to offer a redo date night with dinner on me,” Collins added.

Jason Kelce Says It’s ‘Weird’ Not Gearing Up for Practice After NFL Retirement: ‘I’m Still Getting Used to It’

The woman, later identified as Andreé Goldberg, apologized to Jason and Kylie in a statement obtained by ABC affiliate WPVI-TV

“In a heated moment, I said things that were out of character for me and that I regret and for that, I am sorry,” Goldberg said. “My anger, and my actions, are not who I am, and certainly not indicative of the welcoming community of Margate.”