If you don't want to join the numerous government departments in loosing personal data (yours and others ... apparently even just full name and address is considered "worth knowing" to identify thieves) then you will want to start encrypting your data. The simplest way to use your data on a Windows PC with the confidence that when you shut down your PC the data is safe is to use "on the fly" encryption software. One "favorite" product for this in many quarters is TrueCrypt. Which lets you create a strongly encrypted file that you mount as a "virtual hard drive" on your PC. Just put your data there and when you logout the drive is unmounted and then when you login again you start TrueCrypt and mount that "file" to get your drive full of data back.
If you want to send people e-mails with interesting information in it, you should encrypt that information as it goes out over the web. There are many ways of doing this using certificates and keys but the two main ways are GPG/PGP and S/MIME. S/MIME is supported by many mail clients but requires you to get a certificate from somewhere. Go to Thawte (https://www.thawte.com/cgi/personal/contents.exe) to get a free e-mail certificate (read the documentation for you mail client on how to get the certificate and key into the mail client for use - many mail clients support this such as Thunderbird and Outlook). GPG/PGP is an arguably more popular free variety of encryption. To use GPG to do encryption (think of GPG as the free version of PGP). Try something like the popular gpg4win (http://www.gpg4win.org/). With GPG you just create a key pair and send out the public version, you don't have to go get a certificate from a certificate authority like Thawte. This is why, until Thawte starting doing free mail certificate, GPG was far more popular. Also, the way GPG works, you can either integrate it with some mail client, or use it separately which includes the ability to encrypt any file you have on your PC (so it's a sensible choice for encrypting data to put on a CD and send to someone who you have a GPG public key for).
For a detailed explanation and overview of technologies go to the appropriate web page. But basically, you want (for either kind of encryption GPG or S/MIME) a private Key which allows you to decrypt/sign your mails and a a public key or certificate that you send out to people so they can encrypt stuff to send to you safe in the knowledge that only you can decrypt and read it.
If you want to use good passwords for everything, and not have to worry about remembering them. Then use PasswordSafe. It runs on windows, has some "techy" tools for Linux if every anyone needs to. Several large companies have "verified" it for use via their security departments (VERY BIG companies - in case you worry how safe free software is).
Secure Data Erasure:
If you have an old PC, just use DBAN (http://dban.sourceforge.net/) to create a CD that when you boot a PC with it will give you the ability to completely obliterate all your data using up to military grade erasure. A simple random data overwrite is fairly safe and much quicker, but if you dn't mind leaving the machine running overnight clobbering your data then go for the full guttman wipe.
If you just want to be sure files you delete are actually gone from your hard disk then try Eraser (http://www.heidi.ie/eraser/). This is most likely wanted once you have copied all your data to a TrueCrypy volume and now want to delete the old copies from your computer in a way you can be confident can't be recovered.
Some Hints and Notes:
Apple Mac users
i Just use the FileVault in place of TrueCrypt and store your data in your home directory
ii Just use your KeyChain in place of PasswordSafe
iii Just use the Finder's "Secure Empty Trash" function to safely and securely delete your files from your trash folder.
iv Yes, it is that simple thanks to the apple mantra of "it should just work"
v For GPG check out http://macgpg.sourceforge.net/
i BACKUP YOUR DATA - Sounds obvious, but too many people miss it and with Encryption you MUST make sure you know your passwords
ii READ THE MANUAL - Backing up some encrypted data isn't as simple as copying a file so make sure you follow instructions on backing up encrypted data.
iii DID YOU REALLY WANT TO BACKUP THE UN-ENCRYPTED DATA - Bear this in mind, you may want to, or may not, but make sure you know which you have done.
iv HAVE A TRANSITION PHASE - You're data has been un-encrypted for ages and your password have been written on a peice of paper for ages right ?. So don't setup passwordsafe and TrueCrypt and immediately delete all your old records and data. Put the old stuff to one side (or on a CD/DVD) and start using passwordsafe and TrueCrypt with the warm comfy knowledge you have a fallback if things go horribly wrong.
i With both products above, all of your security now relies on the quality of just one or two passwords. So make them good.
ii One suggestion is to use a pass phrase not a password. Pick something memorable that isn't too likely o be though of by someone else (i.e. Quotes from Shakespeare as they are easy to verify if you aren't sure, or song lyrics (e.g "Fat bottomed girls you make the rockin' world go round")
iii Another suggestion is to make thing a little harder to guess by using some kind of quote or enclosure to your pass phrase. For example, actually include the quotes in the above pass phrase. Or maybe put brackets around your pass phrase or some other random placeholder like zeros (e.g 0Fat bottomed girls you make the rockin' world go round0 ). These make any attempt to brute force a pass phrase much harder as you have to get the right quote (there are quote databases available on the web) and the right surrounding place holders.
i The reason to use the above and not other products, is primarily that they are free open software so if you need to use them on several PCs you don't have to buy more copies. Also, you can encourage friends and colleagues to use them (hey, they're free guys and girls) and then have a common shared pool of experience for using them that's not so dry and boring to use as documentation online or this e-mail (i.e. chat's down the pub or on the golf course).
i I actually have and use both S/MIME (Mac supports this out of the box) and GPG via a GPGMail plugin. There is nothing wrong anyone doing this as long as you don't use both S/MIME andn GPG for the same message as it will confuse mail clients and make "signed" e-mails show as "broken". This is because once you have signed the mail with one encryption technology, signing it with the next change the message and so "breaks" the validity o the signature with the first.
ii Some issues can be experienced with problems reading signed mails that are old, and obviously you will need to keep copy of old expired keys to read mails you sent or received in the past. (GPG is slightly better at avoiding this problem than S/MIME is). This is a result of certificates and keys expiring. Generally you can still read the mails but they will show up with an error stating that the certificate or key they were encrypted with is expired.
iii Obviously if you want to mail something to a fellow TrueCrypt user (or old fshioned person if you include the link to the TrueCrypt web site), you could always put the files in a TrueCrypt volume, mail that as an attachment and then phone them with the password (or have an agreed password you "share" for such purposes).
Obviously this is just recommendations of software from the web, use it at your own risk !!!! I don't take responsibility :-).