Just did the trip to Erlian to get the exit stamp, here's a quick braindump.

1) Getting there: there are daily sleeper buses. Several from a bus station that is on the south 3rd ring rd. There is also a bus from a station near north 3rd ring road and Deshengmen Wai avenue. That bus leaves daily around 4:30, and the owner can be reached for reservation 8783 0592. All buses cost 180yuan and take about 12 hours. Going there, the bus was late (because of some snow halfway), so we departed Beijing around 7:15 and arrived in Erlian at 6:15.
There are also trains as discussed.

2) On arrival. Erlian is a small and not very useful city, but it has ample facilities to make the wait for the return bus bearable. Taxis around town cost 5yuan, taxis to the border cost 10 yuan. There are several internet bars in the vicinity (with your back to the train station, turn left, then right at the first intersection, i saw at least 3 internet bars on that street). The Pacific hotel has what seems to be the only public Western-style toilet in town. There is a "dinosaur" museum which I didn't have time to check. There are two buses - route #1 goes to bus and train stations, and also almost to the border. Bus #2 goes to the dinosaur museum. For those who find themselves really attached to the place, hotels can be had for 50 per night (or cheaper for crappy ones). There is a market with the usual beads and such about 2 blocks away from the bus station. Nothing special there, except perhaps stuffed birds.

3) The border crossing. The guards at both Chinese and Mongolian side are quite adamant that you cannot cross on foot. There are jeeps that will take you to Mongolia and back for 40-50 yuan each way. Or if you are a cheap bastard like me, you can try to swing it for free. I hitched a ride through the first gate to the Chinese border (about 200 meters) then once i got my exit stamp, I tried to go right back through the other door. Unfortunately, someone with big stars on his epaulets had nothing to do so he checked my passport for mongolian stamp and sent me to Mongolia. I think this can be worth a try because on my second visit (with a stamp from Mongolia) the bored guy was not there, and the regular people in the booth didn't even look at anything except the visa. Moving on, hitch another ride (perhaps on a truck) through the second gate, walk about half-way to the Mongolian border. Lines here are bigger but people don't mind foreigners queuing Chinese-style (jumping to the front of everyone). At both borders, there are at least some people who speak English and/or Russian. Get out, go through the hole in the fence, and come back. Duty-free prices are comparable to Beijing retail. Hitch another ride to China, there are trucks and buses going through. On the chinese side, you can just walk out of the building and walk the 200 meters to the border gate. There is a money collection going on as you enter the arrival hall, but I didn't pay nor was asked to. Arrival forms are available.

Overall, quite a painless experience. I was at the border at around 8:40 and was back in China at about 10:20, keeping in mind that I argued every step of the way and waited to hitch a ride, so your crossing will probably be faster. Also I just needed an exit stamp and not a new visa.
I heard that the border is open from 8:30 in the morning until 6pm, but no official info to corroborate.
China Mobile signal is available all the way to Mongolia.

4) Getting back.
There are some trains, but very few and far in between. I made a photo of the train schedule if anyone is interested. The only daily train is at 8am to Jining. There are sleeper buses departing at 2:30, 3, 3:30 and 4pm, also 180rmb. I took the 2:30 one and was in Beijing about 12 hours later - there was a lhuge traffic jam and we took some weird detour. These buses go to the far-away station to the south 3rd ring rd.

Summary.

Time: bus to bus, about 31 hours, door-to-door about 34 hours.
Money: round trip bus 360, 2 taxis in Erlian 20, total 380 not counting food.
Impression: Not the most comfortable way to travel, but quite bearable. And considering a trip to anywhere else would be about 2000 or more, quite cheap.