I hung out with a friend from college at March Meeting. He tried to provoke a debate about why should we study condensed matter theory when there are areas of science that have a much closer connection to human life (say biophysics, genetics, and other areas of life sciences). He tried to argue that taxpayers are subsidizing our hobby projects when that money could be spent on medical advances and such.
My first rebuttal was the following. Condensed matter physics is still very relevant to technology. There have been some exciting breakthroughs in negative index materials, integrated photonics, and nanotransistors. And of course, there is the elusive dream of achieving a room temperature superconductor and finding a theory to explain high T_c superconductors (both developments will probably come together). Our society needs to be ready for the next technological revolution and I guarantee you when it happens, we will need all those condensed matter physicists and their expert knowledge.
His response was that technology isn't as wonderful as it seems. It dramatically improves the quality of life for the upper third of society, but the lower 2/3 of society don't experience any gains at all. Without any real background knowledge, I will say that's probably correct. Society is constructed so that by default, rich people get all the benefits. It is very difficult to find a job that doesn't reinforce this structure unless you devote your life to missionary or charity work.
My rebuttal to his second response is the following. I think that a life of service is a wonderful thing, but that it should not be forced on anyone. It should be a calling (my history teacher didn't think high school students should be required to do community service). As a scientist, I believe truth is the highest law and therefore one should be true to his/her character. The noblest thing to do is to follow your calling faithfully and energetically.
The best way to understand this idea is to read Anna Karenina by Leo Tolstoy. Levin is a well-educated, weathly land owner. He is constantly torn between becoming a politician and a voice for the common man and his love of the simple farming life. His wife, Kitty, has a similar conflict. She tries to be a nurse, but realizes that she just wants to be a good mother. Tolstoy seems to say that there is nothing wrong with aspiring to a simple life; the important thing is to do what you feel in your heart is true. In stark contrast to Levin and Kitty is the doomed affair between the married Anna Karenina and an army officer named Vronsky. These two characters are dishonest in their relationship and therefore they are "punished" at the end of the novel.
You don't have to beat yourself if you don't want to spend all your time helping the old, sick and poor. There is plenty of good that can be done in your own small sphere of influence.