Firms that shift profits to lower tax jurisdictions cost Africa $38bn (£25bn) a year, says a report produced by a panel he heads.
"Africa loses twice as much money through these loopholes as it gets from donors," Mr Annan told the BBC.
It was like taking food off the tables of the poor, he said.
Between 2010 and 2012 five under-priced mining concessions were sold in "highly opaque and secretive deals" in the Democratic Republic of Congo, depriving the country of $1.3bn in revenues, double DR Congo's health and education budgets combined.
In Zambia between 2005 and 2009, 500,000 copper mine workers were paying a higher rate of tax than major multinational mining firms.