From the public address, anyone can see the amount of bitcoins in that address.

For example:
https://blockchain.info/address/1PVypwAvvGcXAfVvxT67yrSARMCDfKzKfy


The private address is like your public adress's 'password' that you use to send bitcoins.

If I understand you correctly the answer is: no your bills could still be counterfeited.

I think you are looking for something like firmcoin. These are digital bills that can confirm for the user that they have the private key to an adress without showing it, even offline.
https://bt.irlbtc.com/view/232898.0

Creating a counterfeit of most any paper item is trivial unless expensive anti-counterfeit measures are taken.  Here's an example of the challenge to making paper hard to replicate:
 - http://rt.com/usa/100-dollar-bills-misprinted-496

This is such a challenge that even community/local currency issuers such as Brixton Pound have gone to using electronic (mobile/SMS) ledgers:
 - http://www.youtube.com/watch?v=qa9Bqrs9yAQ

Even tactics like putting a hologram sticker to protect the private key (as is employed with the Casascius physical coin has) been discovered to be vulnerable:
 - http://codinginmysleep.com/casascius-physical-bitcoins-cracked-at-defcon

The problem comes with intending the item (paper, coin, whatever) to circulate.  

What does work well are single-use vouchers.  When buying a MoneyPak at a convenience store, that piece of paper that is issued contains a code that can be redeemed online.   This same approach where a one-time code is issued is how ticket-in/ticket-out (TITO) equipment at casinos work.   Other e-gift cards and other vouchers/coupons work with the same concept.   The code is revealed to the customer but the intention is that the code will be redeemed instead of the paper with the code being transferred to someone else as an exchange of value.

The single-use e-code/voucher approach used by MoneyPak/UKash/CashU/etc. all are regulated financial products (as there is an issuer of the code) and issuing a private currency would bring the regulators:
 - http://www.the-star.co.ke/news/article-128667/cbk-investigates-bangla-pesa

So even if one wanted to go that route, there are hurdles beyond the technological.

Wow, that's actually a lot, I wasn't aware about that!
The fee system to keep the miners running might even be cheaper than printing money, after all.

Had never heard about this Brixton pound either, reading about it now, fascinating story...

Many thanks for the links!