All of Bethesda's games have a similar problem - they're really difficult to get started with because the choices are so vast, it can get overwhelming. Do I go here first, or there? Will I lock myself out of X because I advanced the Y quests too quickly? What skills should I be prioritizing?
Skyrim's better than some similar games at this (I'm looking at you, Oblivion, with your weird level progression that could leave your character broken and underpowered in the middle of the game...), but I still find myself having "geez, I wish I'd known this, or I would have done that earlier" moments. I've got a few friends still yet to play the game who were asking for advice, so I've put together this short (and hopefully spoiler-free) guide.
Character Build & Leveling
I tend to play a sneaky character who also has significant magical aptitude, so keep that in mind. The combat in these games tends to be a bit one-note, so I prefer to backstab wherever possible. I followed the guide in this G.SE question as far as skills and perks go. I found Sneak to be an important tree, as well as Blacksmithing, Enchanting, and Alchemy. Most of the other trees I neglected perk-wise, except to take the first tier "x% bonus" perks when they unlocked.
When you level up, you'll be given the option to choose Health, Magic, or Stamina to increase. I found Health to be critical, and having enough Magic to cast the higher level spells is quite useful. Stamina increases carry weight, and sprint distance, but I didn't find it to be nearly as critical.
Sneak levels up when you remain undetected around hostiles, but it gains massive XP when you perform a successful sneak attack. In the early going, it's really hard to get sneak attack hits, but I've found that often you'll end up travelling through Dragur crypts in this game. The Dragur tend to stay dormant until you've walked past them, so if you learn to tell the ones that will wake up apart from the others, you can sneak next to them and backstab them before they wake up. In the early going, you can sometimes gain an entire level of sneak per backstab. The high backstab damage also means that you'll be leveling one-handed pretty quickly as well.
For blacksmithing, alchemy, and enchanting, you'll want to create expensive items from the cheapest ingredients you can find. For blacksmithing, take pelts off of animals you kill, and turn them into leather and leather strips. You can buy iron bars from most blacksmiths, and a combination of iron bar + leather strip makes an iron dagger. You can also take leather + 2x leather strips and make a set of leather bracers. If you buy raw materials from the vendors and sell back the finished goods, you'll likely lose money, but you'll be gaining skill quickly.
Money's not terribly hard to come by when you're a master alchemist, however. In the fields around Whiterun you can find several farms (which have bales of wheat you can harvest) plus many mountain flowers and butterfly wings for harvesting. Putting Blue Mountain Flower with either Wheat or Butterfly Wings (not the blue variant, mind you) will create a Restore Health potion, which is essential in the early going. Alchemy vendors frequently stock these ingredients, and Blisterwort, which also restores health.
When you're to the point where you're doing alchemy for profit (and, by extension, raise your alchemy rank considerably), head to Solitude and visit the kitchen on the first floor of Castle Dour. Here you'll find a couple of Giant's Toes you can steal. These respawn every few days, and can be mixed with a Blue Mountain Flower + a Blue Butterfly Wing to make an extraordinarily valuable (but not particularly useful) potion. You can also occasionally find Giant's Toes at alchemy vendors, and be sure to pick them up if you can. Also, any potion that has invisibility (try Vampire Dust + Charus Eggs) or Paralyze is extraordinarily profitable.
Enchanting requires more effort, planning, and supplies. Snap up any soul gems you find (Dwemer ruins in particular are loaded with them), and buy smaller gems from mages. The court mages in the major holds tend to stock small gems frequently. You'll need to learn soul trap, or have a weapon enchanted with a short duration soul trap spell. If you're just trying to rank-up this skill, you can kill mudcrabs along the river bank near Whiterun, or hunt wolves in the wilderness to fill your gems. Then, enchant the iron daggers and/or leather bracers you're making for the Blacksmithing skill. The most profitable enchantments tend to be the kind that boost your combat skills (ie, fortify 2-handed) or that add draining/paralyze effects to weapons. You can then sell the enchanted items and buy more soul gems to repeat the process. Having Azura's Star can be helpful - as you're traveling about, watch for pilgrims who will tell you where the shrine is. Just like in previous games, this artifact is a soul gem that is not consumed during enchanting.
On the magic side of things, I focused on Illusion and Conjuring. Getting Illusion to 75 unlocks Invisibility, which is very nice as a sneaky mage. Plan to cast a lot of illusion spells, though. I cast Muffle constantly around enemies, and threw Rally-type spells around on groups of friendlies whenever possible. Pick up the Silent Casting perk ASAP, which effects all spells and shouts. Conjuring is good for the "conjure ally" spells, which allow you to divert enemy attention and get some "free hits" in even though you're maintaining stealth.
The first thing you want to prioritize when you start playing is taking the main quest line to Whiterun. You may feel a bit underpowered for the quests there, but they're surprisingly more doable than you'd think. Your reward will be a follower/pack mule, plus the opportunity to purchase a house. This follower is useful in the early going, although if you want to train Sneak you might occasionally want to leave her behind. Focus saving your cash so you can get the house in Whiterun, as it's the earliest available and you'll want it to store your junk.
Get used to the layout of the market area, as there are several shops here that are good to visit frequently. The alchemy shop here is a good starting point for the budding alchemist - if you make friends with the owner, you can take (without sneaking/stealing/etc!) quite a few of her ingredients for free, which makes early potion making cheap and easy. The blacksmith shop is close to the south gate, and you can buy and sell both outside and inside the shop. This dual inventory means that you've got 2x the blacksmithing materials to work with.
Inside Dragonreach, behind the Jarl's throne, there's a map room. If you touch the flags, your map will update, giving you the location of some of the larger settlements in the game. Also note the court wizard's office, as you can buy and sell magical items here. He's also got an enchanter and an alchemy workbench, which are both very useful. As I mentioned above, the area right around Whiterun is loaded down with alchemy ingredients. Harvest here frequently, and you'll always have health potions available.
Joining the Guilds
At this point, you may wish to visit Winterhold and join the Mages' Guild, and Riften to join the Thieves' Guild. Being a member of the Mages' Guild means that you can buy spell books from all the different magic disciplines, and the Thieves' guild has a fence who will buy anything (including stolen items) and has a high amount of gold. Plus, she tends to stock a decent amount of lockpicks. You can also run randomly generated quests at the Thieves' guild, which can be easy money. The "Numbers Job" and "Heist Job" quests are both very easy and very profitable. Don't advance the guild quest lines too far though, since as a mage/thief you may want to use the rewards, and the enchantments you get are based on your level.
Once you've done this, I'd suggest playing the Companions' quest line to completion, which is essentially the Fighter's Guild from previous games. This quest line starts in Whiterun, so it should be easy to find. The rewards aren't awesome, but it's a good opportunity to get some combat experience with some strong helpers. If you can, buy the Healing Hands spell, which will allow you to bring allies back into combat when they run out of health and yield.
From here, you have several choices. I'd recommend advancing the main quest line to the point where the Throat of the World is a location you can fast travel to, and you've learned a particular shout that was thought lost. These events give you some powerful tools for dealing with dragons, which you'll encounter frequently enough that it's worth the trouble.
At this point, it's probably safe to engage in the other guild quest lines. The Thieves' guild and Mages' guild questlines are the most rewarding, but you can also join the Dark Brotherhood if you so desire. There's a civil war brewing, and you can feel free to participate if you so choose, although I found this more tedious than anything else, and I did it just for the achievements. Of course, the amount of other side quests, daedric quests, and other interesting areas to poke your nose into is near unlimited.