Disclaimer: I'm an information security professional at $DAYJOB. A separate computer can certainly be worthwhile to use exclusively for sensitive transactions, though I'd add at least one element: a separate email account used only for secure transactions, that will never be used to originate email. Something like a dropbox; an account vendors can use reach you if they need to send a notification, but nothing you'll ever exchange messages with. Since email accounts are pretty much free, feel free to segregate your online persona, as the compromise of one account will limit damage potential to services linked to that account.
The 'leaving it unplugged all the time except when in use' part can backfire, as most modern operating systems require the ability to call home to mama, or they stop working. They expect to be able to get updates and do other maintenance for the operating system and other crapware that they load on the machine before you buy it.
Another option is a 'live CD/DVD'. Download the .iso (disk image), burn it to media (CD or DVD), and boot off it. Basically, it's an operating system that is run completely from a CD or DVD, that writes NO files to the hard drive, and cannot be modified, as the media is read-only. If you absolutely need to, you can use a thumb drive to store temporary files, but this increases your risk slightly. In layman's terms, it permits you use a second (or as many as you like) operating system on the computer you already own. It is considerably slower because CD/DVD reader speeds are much slower than a hard drive. If you have a relatively modern machine (<=3 years old) they generally run well if you have sufficient patience. Older hardware may require a 'lightweight' distro. If you really like the new operating system after you try it, many have the option of permanent installation on the same computer if you have sufficient hard drive space.
The 'live CD/DVD' and private email account(s) is most definitely my preference from a security standpoint. Here's the live distro that I prefer:
http://www.debian.org/CD/live/ Give it a try, and if you need help, drop me a line if you have trouble addressing it by reading the FAQ:
http://www.debian.org/CD/faq/About the only choice I'd recommend against for secure online communication is your cell phone. Security on Android (and Apple too) is abysmal, and you're at the mercy of any software developer that has software on your phone/device.
**edit: Just a note that the 'live' distro mentioned above is free, as in speech and beer.