Can I fly directly from Xiamen to Hong Kong without a Hong Kong and Macao pass but only a passport?

Don't listen to the ticket sellers. They just want to make money. Just sell them to you. They don't care whether you can leave Xiamen Airport or not, and you haven't signed an agreement with them. You can't leave Xiamen airport, you can get a full refund. If you only have a passport and a ticket to Hong Kong, you will not be allowed to leave the country at Xiamen Airport.

The legal way to go to Hong Kong is limited to holding a pass to and from Hong Kong and Macao, and at the same time holding a valid Hong Kong tourist endorsement.

(3) The legal way to come to Hong Kong with a passport is to transit through Hong Kong to a third country, or from a third country to the mainland through Hong Kong. To transit a third country through Hong Kong, you need a third country visa on your passport and an air ticket from Hong Kong Airport to a third country. There is no visa on the passport, and you need a ticket from Hongkong Airport to Indonesia or Maldives, which are all landing visa countries recognized by our national border inspection department.