One Piece Li Lufei said "not him" when he first saw Blackbeard. What do you mean "not him" here?

Lz, let me help you analyze it from beginning to end.

Luffy Sauron and Nami came to this island to inquire about the empty island. On the island, they met a man riding a horse. Both man and horse are sick. Then I met a boxing champion who was making trouble in the street. Then I met Blackbeard at the hotel. After Blackbeard left the hotel, Bellamy and his party came to the hotel, laughed at Luffy's empty island fantasy and beat him up. Luffy and others left the hotel and met Blackbeard. Blackbeard told Luffy that empty islands exist and people's dreams will never end. Nami listened to Blackbeard and told Luffy to leave, and asked Luffy, "What might he know about the empty island? Who is he? " . Luffy replied, "I don't know, and it's not him." Sauron went on to say, "It's probably them."

These words mean that those people were with this man (black beard) before.