Kim and Kanye at Odds Over Trump and More as Friends Encourage Them to 'Hold On': Source

When Kanye West arrived at Trump Tower to meet with President-elect Donald Trump on Tuesday morning, it was the latest in a dramatic series of events that has put stress on his relationship with wife Kim Kardashian West.

But a source close to the couple tells PEOPLE exclusively that their political differences are the least of their worries. “That’s so far down the list of what they’re dealing with at this point,” says the source. “Lots of couples have political differences; that’s not a deal breaker. They’re dealing with much more important things than who each of them supported for President.” However, an insider adds that the reality star “currently has no imminent plans to file for divorce.”

The source tells PEOPLE that there have been several stressors in the marriage. West, 39, was only recently discharged from the hospital where he spent over a week being treated for exhaustion. The rapper’s hospitalization came less than two months after Kardashian West’s own traumatic experience, in which she was robbed at gunpoint and had millions of dollars worth of jewelry stolen from her in Paris.

The dueling crises have taken a toll on their marriage. “Kim needed him after the robbery,” says the source. “She needed her husband. But that was the beginning of his issues, so he wasn’t there for her like he probably should have been. So Kim was trying to be supportive of him, but she needed his support at that time.”

“She had to be the stronger one,” the source continues, noting that she was trying to give some normalcy to the couple’s two children, North, 3½, and Saint, 1. “She was going through a lot. And you know that the Kardashians are very close and very supportive of each other, and they got her through it, but there are times where a wife needs her husband.”

RELATED VIDEO: Kanye West Resurfaces as a Blond Following Hospitalization

As the couple is navigating the rough waters, people in their inner circle are encouraging the couple to keep working on the marriage. “We’re telling them to hold on, to get through this,” says the source. “They both need to understand that adversity is what will make them stronger as a couple.”

Those close to the couple believe that some good can come from the latest challenges. “If they can get through this season, they will create an unbreakable bond,” says the source. “If anyone can do it, they can.”