André Leon Talley on What The Devil Wears Prada Got Wrong About Anna Wintour

"Anna Wintour would never walk in and throw down her coat and handbag on a desk. No."

André Leon Talley is passionately explaining that The Devil Wears Prada—the 2008 movie starring Meryl Streep and inspired by Wintour—was not rooted altogether in fact. "At Vogue, girls did not run down the halls in stiletto heels into Ms. Wintour’s office. No. They got it so wrong." By the way, the 71-year-old North Carolina native reveals the filmmakers almost cast him in the movie—as himself. "Someone called me and said, ‘Would you come and audition?’ And I said, 'No! Goodbye!' "

The celebrated writer and fashion editor (whose new memoir, The Chiffon Trenches, is out May 19) began working at Vogue in 1983 and left in 2013. He quickly rose to creative director, promoted by Wintour to her right hand. "Meryl Streep did a great job in the movie," Talley concedes. "It was a combination of Anna Wintour and [the late Harper’s Bazaar editor] Liz Tilberis. "But the man who played me?" He means Stanley Tucci. "No! It did not reflect the real world of Vogue."

If there’s anyone who knows the real world of Vogue and the real Anna Wintour, it’s Talley. You might have seen him in actual documentaries about the magazine, including The September Issue or The Gospel According to André. But his new memoir digs much deeper. The Chiffon Trenches chronicles his improbable rise from the front porch of his grandmother’s home in Durham, North Carolina to the front rows of fashion. He also does a good amount of "unpeeling the onion about her," he says of his longtime boss and mentor, Wintour.

For more than 30 years, their bond was symbiotic—and oftentimes rooted in silence. "We didn’t have to speak," he says. (They really were close. His colleague Grace Coddington once remarked, "Andre is the only person who’s seen Anna in her underwear." Talley laughs this off while acknowledging he’s been in many gown fittings with her.) "I knew what she was thinking, without words. She doesn’t say many words." Eventually Wintour didn’t utter any at all. Their relationship fell apart over the last decade and he, well, fell out of Vogue.

Talley is clearly in pain. "There was a divide and then an earthquake," he says of the fallout—centered around the 2018 discontinuation of his emceeing the glittering arrivals at the Met Gala. His booming, precise voice cracks as he says, "I felt, at a certain moment, she could not articulate it to me. Something happened." Wintour seemed to once again allow silence do the heavy lifting. "I had suddenly become too old, overweight and uncool for Anna Wintour," he says. "I don’t think she understands what she does to people." He adds that he has "huge emotional and psychic scars" from their relationship.

RELATED VIDEO: We Finally Know What Anna Wintour Thought of Meghan Markle's Wedding Dress

A source close to Wintour tells PEOPLE exclusively: "Anna considered André a friend for over 30 years and naturally was saddened by the way he chose to portray many aspects of their friendship, but he is of course entitled to tell it as he remembers it. She wishes him the best."

"I love her," Talley says. He knows the reaction to the book so far has indicated he may feel otherwise. "People see my book as a vengeful, bitchy tell-all. It is not. My book is in many ways as a love letter to Anna Wintour."