Source: "Epidemies, Famines, and Population in the Cape Verde Islands, 1580-1900.", by K. David Patterson  (The International Journal of African Historical Studies) Patterson's paper presents a table with population estimates and the most severe famines. The data is collected from many different resources and has been critically looked at. Still there are many uncertainties about these estimates. However it looks like the best source available and thus it is really worth sharing.
(Scale for Population on the left, for Famines on the right)