Can I go to Taiwan with an ordinary passport?

No

Because Taiwan and Mainland China do not officially recognize each other as countries

So

when you come to Taiwan, we have to be on your side Apply for a Taiwan Entry Permit (issued by mainland China) and ask a travel agency to help you apply for a Taiwan Permit (issued by Taiwan)

We from Taiwan cannot use our passports to go to the mainland and must apply for a Taiwan Compatriot Permit from the mainland