Nearly 400,000 self-employed people on low incomes are being penalised for filing their tax returns late when most of them don't owe any tax.
Penalties can build up until people owe thousands of pounds, according to Tax Policy Associates.
The non-profit called for penalties to be cancelled if HMRC finds people have no taxable income.
HMRC said new rules would mean people who sometimes miss a filing deadline would not be penalised.
Tax Policy Associates revealed that Conservative Party chairman Nadhim Zahawi paid a penalty over previously unpaid tax while he was chancellor.
The boss of HMRC told MPs on Thursday there were no penalties for "innocent" tax errors.
Nearly 12 million self-employed people need to fill out self-assessment tax returns each year.
Through Freedom of Information requests to HMRC, Tax Policy Associates found that nearly 400,000 people earning less than £13,000 received a penalty for not filing a tax return on time between 2018 and 2020.
The vast majority of those had no tax to pay anyway because they earned less than the tax-free personal allowance - which was £12,500 in 2019/20 or £11,850 in 2018/19.
The lower people's income, the more likely they are to be penalised, the not-for-profit firm said.
Fines of £100 are automatically issued to people if they fail to file an online return by 31 January each year.
Tax Policy Associates founder Dan Neidle said: "It's astonishing and unjust that hundreds of thousands of people are being charged penalties for late filing of their tax returns, when in fact their income is so low they have no tax to pay."
For most of those affected, the £100 penalty represents more than half their weekly income.
Penalties can build up if they a not paid, and after a year, can be more than £1,600.
While recipients of the fines can appeal to HMRC on the grounds that their income is too low, tax agents say many simply pay the fines, unaware that they shouldn't have to pay them.
Mr Neidle said HMRC was acting in good faith but needed to realise the impact its heavy fines were having on people who may not realise they earn too little to owe any tax.
The data also revealed that the penalties are imposed disproportionately on the bottom 30% of the population by income, who earn less than £13,000 a year.
On the bottom tenth, the proportion receiving penalties is more than twice the level in higher income brackets.
"HMRC has sophisticated computerised systems. And those systems have not been distinguishing between people who have tax to pay and haven't filed and people who have no tax to pay and haven't filed," said Mr Neidle.
"I don't think it's right to have a system which puts the onus on people in low income to fill out forms for HMRC to avoid penalties.
"What should happen is that if you're late filing, and then it turns out you have no tax to pay, your penalty should be cancelled. That is how it used to work many years ago and I think that's how it should work."
"Parliament needs to change the law and make it to the penalties are cancelled or refunded if in fact, you've no tax to pay," he added.
HMRC said automatic penalties "are necessary for the efficient functioning of HMRC as it means public services can be funded on time".
"We are reforming penalties so taxpayers who occasionally miss the filing deadline will not face financial penalties," a spokesperson said.
HMRC said that 95% of people pay their tax on time.
"We encourage anyone who does not need to file a return to tell HMRC, the steps of how to do so are included in reminders sent direct to customers every year," the spokesperson added.
Have you been wrongly fined? Share your experiences by emailing firstname.lastname@example.org.
Please include a contact number if you are willing to speak to a BBC journalist. You can also get in touch in the following ways:
WhatsApp: +44 7756 165803
If you are reading this page and can't see the form you will need to visit the mobile version of the BBC website to submit your question or comment or you can email us at HaveYourSay@bbc.co.uk. Please include your name, age and location with any submission.