I assume folks who advocate for a transaction tax don't actually want less transactions, rather they want more "real" transactions and a less artificially volatile market place.
Unfortunately, a transaction tax would create the exact opposite of that situation. The hypothesis that if there were a transaction tax there would be less transactions is incorrect. What would happen is that transaction quantities would get bigger in order to overcome the new added cost. These larger transactions would magnify the risk at play in the market place. This in turn would raise the reward for being able to pull out of quotes faster and/or to misrepresent the riskiness of your trading strategy.
So a transaction tax would actually incentivize more "false" liquidity and work to the betterment of companies that are more risky.
What HFT systems actually do, is allow firms to "pay" for priority of an order at a price level, by investing in network infrastructure/algorithms. If you want remove that advantage the easiest way would be a system where you transparently pay for priority of an order. The system with the least likely negative side impacts of this would be making arbitrary price level sizes. That is, instead of quoting down only the penny level, let people quote arbitrary (or some fixed but very small) decimals of a penny. That way if you really want to pay up for priority, you can just increment your order slightly and actually pay for the privilege.
We actually have a system similar to what you describe in your last paragraph with the three "inverted" US equity exchanges where liquidity providers pay a fee and liquidity takers receive a rebate. It means that shares posted to the inverted exchanges are cheaper (net of fees) to aggressors than shares posted to other exchanges, so liquidity takers with a smart order router will look first to the inverted exchanges before the other exchanges.
Unfortunately, a transaction tax would create the exact opposite of that situation. The hypothesis that if there were a transaction tax there would be less transactions is incorrect. What would happen is that transaction quantities would get bigger in order to overcome the new added cost. These larger transactions would magnify the risk at play in the market place. This in turn would raise the reward for being able to pull out of quotes faster and/or to misrepresent the riskiness of your trading strategy.
So a transaction tax would actually incentivize more "false" liquidity and work to the betterment of companies that are more risky.
What HFT systems actually do, is allow firms to "pay" for priority of an order at a price level, by investing in network infrastructure/algorithms. If you want remove that advantage the easiest way would be a system where you transparently pay for priority of an order. The system with the least likely negative side impacts of this would be making arbitrary price level sizes. That is, instead of quoting down only the penny level, let people quote arbitrary (or some fixed but very small) decimals of a penny. That way if you really want to pay up for priority, you can just increment your order slightly and actually pay for the privilege.