You know 7 cards. Your 2 and the 5 on the board. (The number of other players does not change this.) There are 45 other cards and 9 are Diamonds. 1 in 5 is 20%. He has 2 cards so 2 in 5 is 40%

Of course the action of the hand should give you some idea of what he might have. Did he play like he was on a draw? Was he agressive until the 4th Diamond hit. Is he a bluffer or trapper? That should tell you more than worrying about pot odds.