Craig Conover Reveals the Most Difficult Chapters to Write in His New Book 'Pillow Talk'

Craig Conover isn't holding back.

To celebrate the release of his debut memoir Pillow Talk: What's Wrong with My Sewing? on Tuesday, Conover, 33, tells PEOPLE about becoming his most "vulnerable" and "transparent" self while writing — even while discussing difficult topics from his life.

"So much of my life has played out on screen and I've always lived my life being as honest as I can be," the Southern Charm star says. "I didn't want to write to write a book that said 'This is what actually happened,' I wanted to write a book that said 'This is what also happened.' "

Pillow Talk follows Conover from his Delaware upbringing and early love for sewing to his days starring in one of Bravo's hit reality TV shows and launching his successful business, Sewing Down South. While the book certainly celebrates the highs of his journey, Conover does not shy away from discussing the low points as well, including his experience with bullying as a child and his battle with Adderall addiction.

But Conover said perhaps the most difficult chapter to write was the one chronicling his relationship with his ex-girlfriend Naomie Olindo.

"I really looked in the mirror for the first time," he says. "I've been honest with myself about the breakup but there were so many layers to it."

Conover and Olindo dated for three years before breaking up in 2017. Their relationship drama was previously featured on Southern Charm and season 6 focused on the fallout of their split.

Looking back, Conover says he didn't realize how his Adderall addiction impacted his relationship.

"I was able to not address it for a long time, but writing this book was basically one long therapy session of finally telling the truth to myself," he recalls. "There were a lot of things about our relationship that people didn't know about."

Fortunately, Conover didn't have to tell their story alone. "At the time, she and I were just in a great place so my writer was able to interview her and have a very honest talk with her," he says.

"I didn't think it was appropriate to tell our story without her involvement," he adds.

Olindo is set to return to the upcoming season of Southern Charm — a move Conover previously said he was in full support of.

"I was actually a big proponent of it," he said during a recent appearance on Watch What Happens Live. "She came clean about everything. She apologized to everyone that she needed to."

Asked whether he received an apology as well, Conover said: "Yep, she said everything that everyone has always wanted her to say to me."

While chatting with PEOPLE, Conover teased the upcoming season, calling Olindo a "great new addition" and hinting at plenty of "explosions" among the cast.

PILLOW TALK: What's Wrong with My Sewing? hits bookshelves on March 29. It was published simultaneously as an audiobook from Simon & Schuster Audio, read by Conover.