In terms of produce, there are several ways you can reduce the load:

1. Look at the list of the "Dirty Dozen" published by the Environmental Working Group. This is a list of the most pesticide laden produce on the market and, as a side, they also list the least pesticide laden produce as well, then make sure you avoid all the "Dirty Dozen" and only buy those organic.
2. During the spring/summer time frame a lot of local farms in many areas will sell produce direct to consumers but you have to opt-in with them and spots fill up fast (Note: this is also known as Community Supported Agriculture). Essentially, you get a basket of produce per week of whatever the farmer harvests that week. Generally, these tend to be organic or in the process of converting to organic (hence pesticide free) and will generally cost you less compared to if you buy the same from a store. The only thing to note is that you get what you get and you do not really have any control on what you can demand.
3. Look out for local farmers markets and shop there as opposed to at a store, you may get better prices that way, but YMMV.
4. Look out at your local grocery for on-sale items in the organic section. Sometimes, for in-season produce, you may be able to get organic at cheaper prices then conventional. You won't believe how many times I have seen folks simply walk past the organic section to the conventional and buy some conventional item and pay more for it because they did not notice that the same organic product was on sale.
5. Finally, focus on eating in-season produce since this will be more readily available and hence priced appropriately.

In terms of organic dry goods your best bet is to shop online. In many cases you will be able to beat paying taxes and if you buy enough then you may be able to offset shipping costs and still come out ahead (and keep in mind online shopping has come a long way so many merchants now offer very reasonable flat shipping fees no matter how much you buy). Also look out for sale items in stores, stores will rotate on-sale items and you can pick up extra for non-perishables when on sale.

[EDIT] I would also like to add that growing your own is another great way to save.