One solution here is to mark several rooms for "small-group chat (recommended size: 5)" and/or just suggest that people try joining rooms with less people.
I also like @skola2015 suggestion of ending the Unhangout with a final call to join rooms for people who want to have an extended chat.
(Of course, people can set up their own hangouts... But, one of the benefits of a centrally-facilitated course is providing a catalyst for people to try things they might not otherwise do.)
Re: Text chat
Agreed, several chat tabs could be good. One for on-topic, one for analyzing the course or meta conversation, maybe one for troubleshooting. If that's not easy to do, I could provide a page with a web IRC client for support/backchannel conversation, or we could use another real-time platform like Rizzoma.com (based on the old Google Wave) to have parallel conversation. (Though Riz has a bit of a learning curve as well.)
Personally, I prefer traditional text chat (IRC-style, or like what's in Unhangouts now) to Twitter chats. But, of course, a lot of people like Twitter as well, and it has some advantages. No reason we can't do both, and even have people quote remarkable lines from either one in the other.
Re: pre-recorded videos
I think pre-recording makes sense when the content is pretty much scripted. However, maybe it would also make sense to release the videos early (maybe the morning of the Unhangout) for those who want to watch ahead of time, and come prepared with questions, a sense of what what will be covered, and more attention to give to the other elements of the Unhangout (chat, breakout sessions.)
The question of editing vs. recording live is an interesting one. It definitely adds more work to edit, especially if there's nobody with editing skills around... but it's worth considering for "course content" style videos, as opposed to more dynamic conversations.
And then, asking people to watch course content videos ahead of time might make the "live session" part more interactive with what people are typing in chat.