It Ends with Us Justin Baldoni's Clash Goes Past Blake Lively: 'Cast Will Have Nothing to Do with Him' (Exclusive)

The issues brewing behind the scenes of It Ends with Us surpass alleged creative differences between the film's director Justin Baldoni and star Blake Lively, a source tells PEOPLE.

"All is not what it seems," a set source tells PEOPLE. "There is much more to this story. The principal cast and [author] Colleen Hoover will have nothing to do with him."

Reps for Lively, 36, and Baldoni, 40, did not respond to PEOPLE's requests for comment.

It Ends With Us opened to a strong $50 million at the domestic box office over the weekend, with Lively starring as Lily Bloom, a florist who falls in love with a neurosurgeon named Ryle Kincaid (Baldoni), and a toxic relationship forms. The cast also includes Jenny Slate, Brandon Sklenar, Hasan Minhaj, Isabela Ferrer, Alex Neustaedter, and more.

Rumors of discord behind the scenes grew prior to the film's release as Baldoni did most of his press solo, and did not pose for pictures alongside the cast at the film's New York premiere. Fans were also quick to spot that Lively, Hoover and Slate do not follow Baldoni back on Instagram.

Reps for Slate, Sklenar, Minhaj and Ferrer, Neustaedter and Hoover did not respond to requests for comment.

Following a Hollywood Reporter story on Aug. 8 reporting there "was a fracture among the filmmakers in the post-production process, wherein two different cuts of the movie emerged," a source close to Baldoni attributed the discord to "creative differences" that had been "overblown" to PEOPLE Monday, claiming Lively "wants to direct the next movie, sidelining him and securing the rights from Colleen."

But there has not been a discussion yet of a second movie, says a Hollywood source.

Baldoni told Variety his production company Wayfarer Studios has the option on adapting It Starts with Us but "I haven’t even begun to think that far ahead. I’m trying to stay in the present as much as I can."

Reports of Baldoni's alleged behavior surfaced Aug. 9, when the Daily Mail published a report alleging Baldoni was "chauvinistic" and "borderline abusive" on the set of It Ends with Us. The same day, Page Six reported Baldoni created an “extremely difficult” atmosphere behind the scenes for the entire cast.

Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

At the premiere in New York City on Aug. 6, Baldoni suggested to Entertainment Tonight that Lively should helm a sequel. "I think that there are better people for that one," he said. "I think Blake Lively's ready to direct. That's what I think."

Lively, who is also a producer on It Ends with Us, has spoken of putting her touches on the film's music, costumes, dialogue and more.

She told BBC at the London premiere, from which Baldoni was absent, "I really feel like we delivered a story that's emotional and it's fun, but also funny, painful, scary, tragic and it's inspiring and that's what life is, it's every single color."

It Ends with Us is now in theaters.