Grant worked with Fellowes on his Oscar-winning 2001 film Gosford Park, in which he played a footman at a stately home, and he was surprised when he wasn't invited to be a part of the hugely successful period drama.
However, Grant remains hopeful he will be asked to appear in the show's upcoming fourth season.
He tells Britain's The Independent magazine, ""I have no idea (why I wasn't asked to be in Downton Abbey). I know Julian, but I was never asked. I've never been up for it. Maybe next series.""
Fellowes won the Best Original Screenplay Academy Award for his work on Gosford Park, and one of the film's stars, Dame Maggie Smith, went on to head the cast of Downton Abbey.