This has been a controversial question that many marketers and SEO's have been asking themselves and the experts for a while now, and even more since Google's latest caffeine update.

The answer is YES - Google always has been picky about the types of websites that link to yours.  In a nutshell, only links from quality, "relevant" websites will help your rankings on Google. They, no doubt, need to be from websites that are not spammy, nor link farms, and are somewhat related to yours.  Furthermore, what we're seeing now after the caffeine update is that links from unrelated sites can actually hurt your rankings.

Prior to Google's caffeine update (the 2010 revamp of it's algorithim) having links from spammy and bad quality sites could never HURT your rankings - it just wouldn't HELP them.  Based on our research, and after many discussions with experts in our field, we think that this has changed in the 2010 caffeine update.  It does appear now that links now can actually harm your rankings. Now, there is no way to prove this 100% - but we've done some studies and believe this to be the case.

What does this mean?

1).  You need to WATCH OUT for your competition.  If this theory is true, that means your competitors could sabatage your rankings by having spammy unrelated sites linking back to yours (or even buying links). To be blunt, the SERP's could easily turn into a battlefield where SEO companies could now sabatage, or atleast downgrade the rankings of any website.

2).  You need to do a thorough link analysis of your website and idenfity low quality, spammy links. In the past it helped you - but now it can harm you. Attempt to contact these webmasters to get the low quality links removed, if possible.

3). Quality is definitely more important than Quantity.  In Google's eyes, 10 links from quality, relevant websites will help your rankings more than 100 links from bad quality, irrelevant websites. So focus on quality links, even if they come in smaller numbers. It is worth it in the long-run.