© Provided by PeopleDonald Miralle/Getty James Caan and Hugh Hefner in 2003
James Caan once had an extended stay at the Playboy Mansion.
Back in 2003, the actor - who died Wednesday at the age of 82 - told Esquire, "To get over my divorce, I got a prescription to live at the Playboy Mansion for a while."
James was previously married four times: first to Dee Jay Mattis, from 1961 to 1966, then to Sheila Ryan in 1976. He later tied the knot with Ingrid Hajek in 1990. The two divorced in 1995, and he wed Linda Stokes later that year.
The actor filed for divorce from Stokes in 2005, but the two later reconciled. James and Stokes were in the middle of a divorce as of December 2016, according to TMZ.
"I have four wives and five kids. I apparently don't know the difference [between sex and love]," James joked in his 2003 Esquire interview.
Never miss a story - sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.
RELATED: The Playboy Mansion Has Finally Sold for $100 Million
James' son Scott Caan spent time at the Mansion with his father when James briefly lived there during Scott's childhood.
In a 2010 interview with PEOPLE, James recalled with a laugh, "I'd have Scott bring me girls."
"No one could say no to a cute little boy," he added of Scott, now 45.
Scott's mother is Ryan, who died in 2012 at age 60.
RELATED: James Caan Remembered by Adam Sandler, Rob Reiner and More: "Best of the Best"
A tweet from James' official account Thursday read, "It is with great sadness that we inform you of the passing of Jimmy on the evening of July 6."
"The family appreciates the outpouring of love and heartfelt condolences and asks that you continue to respect their privacy during this difficult time," the tweet added.
A cause of death was not made immediately available.
James is survived by sons Scott, James, Jacob and Alexander, as well as daughter Tara.
Read the original article on People