The Hammer of the Emperor!
First and foremost, welcome to the Guard!
All the posts before me give good advice, but I will try to expand on it as much as I can.
I highly suggest you buy the battleforce as one of your first purchases. It gives you enough units to either form a SINGLE platoon (1 troop choice minus the company command squad) or two Veteran squads (2 troop choices) and a company command squad. The heavy weapon squad is important for your siege, being able to bring different types of weapons to the party. The sentinel is also a welcome addition. If cash is a big factor then you might consider buying multiples of this.
Going off of GW the battle force saves you roughly $40, not including shipping. Which is almost enough to buy any other model.
HQ: The IG boasts a nice selection of HQ choices, each has their own use and style of play.
Command Squad: For your siege army I recommend bring at least one of these to the party. The commander can issue out orders as well as bring along some cronies that help out your army in multiple ways, from getting reserves in faster, slowing enemy reserves, or dropping down some ordnance on the enemy (Master of Ordnance might be a good call for you).
Commissar/Lord Commissar/Commissar Yarrick: Badass is an easy way to summarize these. Good for keeping your men from running, and can help out in the fight as well. Nothing can ruin a siege more than the sieger running away lol.
Primaris Psyker: Useful and devestating. I usually use this unit for my assault list, might be useful to have to fight off deepstrikers who wanna rough up your siege force. Force weapon, need I say more?
Ministorum Priest and Techpriest Enginseer: Good specialists in their fields, but for a siege they might not be all too vital. The Ministorum priest might be useful if you wanna throw a squad into close combat when an enemy that is getting too close. The enginseer would be a better choice of the two. Have him stay close to your vehicles in case they get busted up.
Ogryns: Big, dumb, and powerful. You might want to hold off on buying these highly expensive units in both points and $$$.
Ratlings: Pen and rend FTW baby! More accurate than regular guardsmen and can infiltrate. For your siege style army I suggest you use these. They may be sub-human, but they are good shots and cooks! Watch your personal items around them though.... *shifty eyes*
Psyker battle squad: Nothing like having a group of choir boys melting you with lightning. Good unit, but I don't think they will work well in your army. Buy this unit later on if you are still interested.
Storm Troopers: Not too expensive, better armoured, better shots, and their weapons and can cause a Space Marine to hide in a building like a little girl. They are capable of different roles in your army and are useful in every way. Bring at least one squad, they won't let you down.
Marbo: AKA the great, great great, great great, great great, great great, great great, great great, great great, great great, great great, great great, great great, great great, great great, great great, great great, great great, great grandson of Solid Snake and Rambo. If you want something removed from existence, call this man in. Low point cost and scary as hell. For your army, I would say bring him if you wanna take out an enemy unit you absolutely hate. The last thing I have to say about Marbo is "He's behind you!"
Infantry Platoon: You need to buy enough units to either make a large platoon or multiple smaller ones. I recommend these for your army opposed to veterans. They are capable of adapting to any job. You can specialize each squad within them or make them cannon fodder. Also, you can set them up to bring a heavy weapon team in each squad. This is a nice way to get cheap heavy weapons (in some cases half off on the point cost).
Heavy Weapon Squads: These are vital for a siege. Mortars are good for shelling the enemy without having to see them, heavy bolters are good for taking out infantry, the autocannons can handle infantry and lighter armor, lascannons are great for armor or heavy troops. A tip though, it may be tempting, but don't mix your weapons in your squads. Not doing this can help you avoid those pesky "dang, I should have fired that weapon at that guy instead" issues. No use shooting a lascannonand a mortar at a Land Raider =P. Again, if you are not too worried about your infantry squads moving around a lot, throw a heavy weapon team in each squad and get these weapons at discount prices! SUNDAY SUNDAY SUNDAY! ALL LASCANNONS MUST GOOOOOOOO!!!!!!
Special Weapons Squad: These guys are great for specialized tasks. Being able to bring heavy hitting or versatile weapons is extremely useful. Not to mention you can set up sniper teams with these guys. yet again Pen and rend FTW baby!
Conscripts: Cannon fodder to the extreme. Low cost low output. These guys come in cheap and you can bring 50 per platoon if you wanted. Whiteshields are useful at slowing down an advancing army, helping protect your more...useful units. Commander Chenkov FTW! "Forward, you dogs!"
Veteran Squads: Better shots and can have better armor. They cost more than normal squads and don't output too much more. More for specialized armies. I don't think you will have much use for these.
Penal Legion Squad: Criminals and shitbags. They have some cool abilities but like the veterans are more for specialized armies.
Chimera Transport: Transports are useful, but in my opinion they are more of an assault unit than a siege type. Having a command squad in one would be good to help protect your commander so he can keep belting out orders. But I suggest you only buy one when you first start off, if you are interested in the unit.
Rough Riders: These guys are probably best used in an assault army, great for quick hit and runs. But I don't think you are going for that.
Sentinels: As I said before, they are versatile, armored, and fast. What more could you want? Plasma Cannon? Sounds like a deal to me =D
Hellhound/variants: Mobile death for infantry and vehicles alike. They have to get relatively close to use, but they are devestating none the less. Kind of an assault unit as well, so you might not want to invest in them too much for your army.
Valkryie/Vendetta: Fast, deadly, and feared. My Vendetta is the bane of my friends existence. They make it a point to try and take it down as fast as they can, but rarely are able to. You can say they have a personal vendetta against my Vendetta =P. Wordplay is fun lol. Fast transport with an small assortment of weapons. Excellent for popping armor from across the board. Could fit right in with your siege army nicely.
Leman Russ/variants: Bring at LEAST one, simple as that. Cheap, powerful, and multitalented. The thing that got me to switch from Space Marines to Imperial Guard was how versatile the guard is (also me being in the Florida Army National GUARD, helped too lol). The Leman Russ has seven variations. These things are deadly in every sense of the word. You can customize these bad boys to handle any threat. You can use weapons from battle cannons and demolisher cannons to gatlings guns. If you really wanna spend some points on one of these you can outfit this beauty with six weapons (this is including one Hunter-killer missile). I can not praise this vehicle enough. It is a major threat to anyone.
Hydra Flak Tank Battery: If you have a problem with skimmers, get one of these, other than that not any real reason to get one.
Ordnance Battey: Yet again, the guard has an impressive array of variations. There are four different types of ordnance you can bring. From the heavy hitting long range Basilisk, to the short range heavy mortar of the Griffon. These are essential for your siege army. Try and get three of these for your army if you wanna play larger games.
Manticore Missile Laucher: Rain down death from the sky. Only has four shots but those four shots will not be a waste, then you have mobile cover =P. You might wanna bring one of these for your army.
Deathstrike Missile Launcher: Hell unleashed, provided it makes it in time to fire. Takes a bit of time to fire, but once it does, watch as your opponents will is crushed.
As for all the special characters, they are all preference when it comes to what you need out of them. My recomendations for a siege style army when it comes to these characters would be.....
Chenkov, Pask, Castellan, Bastonne, and maybe Harker. But these are mostly for splurging. =P
You didn't specify your spending capabilities, but I assume since you can't afford the Kriegs right now you don't have a large spending limit here.
Minimum of 50 guardsmen, three heavy weapon squads, a chimera, 1-3 sentinels, minimum of one Leman Russ, and 1-3 basilisks.
Now the best way of getting this starting force would be to buy the following. (this is assuming you bought from Games Workshop)
3x Cadian Battleforce (75 infantry, 9 Heavy Weapon Teams, and 3 Sentinels)
1x Leman Russ
= Approximately $415
If you buy the 3 battleforces you will be able to use 15 of the units as the Company Command Squad and the two Platoon Command Squads needed to form an army. The remaining 60 infantry can be used in various ways. By forming two platoons comprised of 3 squads each you can exchange two guardsmen from each squad in a single platoon and get a full spread out weapons team in at half point price (varies on what weapons you use). Then throw the other two weapons teams in at normal price, then use the now jobless six guardsmen as a special weapons team. That is just an idea for you. You have hundreds of variations you can do with this set up.
Once you get these I would suggest buying ratlings, storm troopers, and a Valkryie. With the setup provided you can get a feel for the majority of the weapons available and see what you like before you buy any additional models. Minus the storm troopers Hot-shot lasguns (L33T). Don't forget to throw in some commissars if you have a problem with your squads breaking.
I hope this long and drawn out post can help you decide on what to buy. I absolutely love the Imperial Guard and can't get enough of them. They have far more options on lists than any other army. I know you will enjoy playing the Guard as much as I do. Just remember, the Imperial Guard is not a cheap army, but it is worth it if you use them properly. I would love to hear about what you end up buying. Again, welcome to the Guard and may the light of the Emporer guide you!