Jason Isaacs was the perfect Lucius Malfoy in the Harry Potter films, but in a new interview with Digital Spy, the actor admitted that he was “mightily pissed off” after being asked to audition for the role. In fact, he almost turned it down.

“I didn’t audition for Lucius; I auditioned for Gilderoy Lockhart,” he explained. Afterward, they asked him to read for another part — one that he knew he didn’t want. (The role of Gilderoy Lockhart, the Defense Against the Dark Arts Professor, eventually went to Kenneth Branagh.)

“I was about to go play Captain Hook [in 2003’s Peter Pan] and didn’t want to do two children’s [movie] villains,” Isaacs told Digital Spy of why he was against playing the nefarious wizard.

Despite his hesitation, he did the reading for Lucius Malfoy — and clearly, he nailed it. But when he got home, he immediately informed his agent that it wasn’t going to happen. “I said to my agent, ‘When they call, tell them no, thank you very much, I am about to go play Captain Hook.'”

Thankfully (for Isaacs and fans of the Harry Potter films), the actor’s agent wasn’t about to take no for an answer. At least not that quickly. He encouraged his client to take the weekend to mull it over, but Isaacs didn’t want to even do that. “I said, ‘No, I don’t want to think about it — it’s fun and lovely and flattering but I am not playing two children’s villains!'”

Over the weekend, though, he had a change of heart. “[E]verybody I knew called me — nieces, nephews, godchildren, and then the parents,” Isaacs told Digital Spy. “They all tried to persuade me to take the job — not because they cared about me but because they wanted to visit the set! So, I took the job and thank God I did!”

Isaacs appeared in six of the Harry Potter films — and he’s open to doing more if the opportunity ever presents itself. “If [JK Rowling] called me and went, ‘I want to do a Lucius Malfoy spinoff film,’ she wouldn’t get to the middle of her sentence before I’ve said yes and left my house! But I am not holding my breath for that call.”

For what it’s worth, we would LOVE to see a Lucius Malfoy spinoff movie.

Did you love (or love to hate) Jason Isaacs as Lucius Malfoy? Let us know @BritandCo!

(Photo via John Phillips/Getty Images for BFI + Stuart C. Wilson/Getty Images for Warner Bros. Studio Tour London)