The underground man is extremely self conscious and his fear of others judging him permeates his interactions with his classmates. He views himself as an outsider to them (which I believe he is), not just isolated from them, but on a completely different plane of thinking. He seems to have a fundamentally different view of his earliness to the dinner than the others.