Technically alcohol is a poison but I should imagine that plenty of us consume that. Aspartame is only a poison to those with an intolerance by definition as a poison is any substance, normally chemical, that when absorbed by a living organism causes illness, paralysis or death.
Any substance when consumed in large amounts, even water, could fit this definition. Anyone remember the girl (Leah Betts) who died through drinking too much water after taking ecstasy? It was the water that killed her due to brain swelling. Drinking too much water can also reduce the electrolytes in your blood and cause you to become sluggish and and reduce muscle effectiveness, essentially ill. Granted you would have to drink gallons and gallons. But its possible.
Aspartame has been proven safe to use. In foodstuffs and drinks. In reasonable quantities. Yes if you consume ridiculous amounts OR if you have a particular sensitivity then it may cause you a problem. My daughter has a nut allergy. So I don't give them to her but I don't go around telling everyone else they are dangerous and to not eat them. Not do I not eat them. Clearly that only applies to those who either eat large amounts and get sick or those who also have an allergy.
Surely this applies to everything, whether chemical, animal, vegetable or mineral? Everything in moderation.