Here's why you should be worried about this.
1. A lot of Americans break the law, but very few actually get arrested for it. You probably think that you have nothing to hide, right? Remember that stuff that you bought online but didn't pay the sales tax on? Remember writing $0 when your IT-40 asked you to calculate the sales tax for these purchases? PRISM remembers what you bought online, and the Indiana Department of Revenue remembers what you reported.
2. But maybe you are 100% in compliance with all federal and state laws. Maybe you're utterly confident that the government has bigger fish to fry, like terrorists, drug lords, and conservative activists. You still should be worried about data collected on you. If the government can't keep PRISM a secret, it can't keep collected data a secret either.
3. This isn't going to make us safer. There are always going to be ways to send messages that can't be traced. Mail a letter with the wrong return address. Post a cat video with a hidden code. ("Calico if by land. Tabby if by sea") And do you think all those spam messages are really trying to sell you something?
4. Private citizen who volunteer information are law enforcement's best resource for gathering information on dangerous criminals. Anything the government does to antagonize citizens makes them less of a resource.