You keep bringing up people's starving "affecting me" what sort of effect are you hinting at? They produce nothing, they trade nothing so they are not an economic factor (or they would not be starving) so it's not as if their buying or not buying is going to affect me.
I'm not hinting, I am stating it directly. Millions starving will affect you (and me) in many ways.
First, everyone in an economy participates in it. That means the poor, even though they have little, buy things. Think Wal-Mart. If millions are starving, they cannot buy things. That affects companies that sell, and it affects companies that make things. It leads to more unemployment, which exacerbates the problem. At a very basic level, it is important that people keep consuming.
Second, as I've pointed out, people who are literally starving are dangerous. They may be religious and they may have morals, but few will starve quietly to death in a corner while there is food for their family in the local grocery store. Given a high enough misery index, people will begin to do what they think they have to in order to survive. I was alluding to this when I joked about breaking in to your house and eating you; sorry, I should have been more clear. History is replete with examples of small societies (stranded parties of people) who turned to murder and cannibalism to survive. Law, morals and religion go out the window when actual life and death are on the line. But whether people will break into your house and eat you, or just start swarming local stores and simply taking what they want, as social order breaks down, it will indeed affect you. It will affect all of us.
You speak of Somalia and other nations like that where starvation is rampant. You're right, that mostly doesn't affect us in a manifest way. The people who live there do not take part in our local economy, nor in the global economy in a major way. Our economies are not tied together by trade, and the Somalis, for example, don't really have an alternative to starving, when there is nowhere near them where they could go and steal food either.
As I said, I am not an economist, nor am I an expert on economies. What I do know is pretty simple stuff, like the fact that economies require companies to make goods, people to sell goods, and people to buy goods. If there is not a balance between them, they will seek equilibrium by reducing what they do to match the others. Fewer buyers means fewer manufacturers, which means fewer employees, and so on. I used to believe it was best if government did not attempt to apply a steadying hand to that balance. I still believe the lightest of touches is all that is required, too much is as destructive as too little. And yes, that light touch is in the form of taxes that take from you (and me) and give it to the undeserving. I believe that is actually of benefit to me as long as the balance is kept.