There's a significant degree of variance in the culture/etiquette of different BJJ schools, but on average things tend to be significantly less formal than in most Japanese arts.
Some schools expect you to bow went entering or leaving the mat as you would in Judo, but these are in the minority. It's reasonably common to bow at the start and/or end of class, but when that happens it's generally just a quick line up and bow following the instructor's lead (as opposed to a more involved ritual such as "kneel, bow to the instructor, bow to the picture of the founder, bow to the flag, blah, blah, blah" that you might find in some Japanese arts). Generally if you just follow the example of the students around you you'll be fine.
Instructors are typically referred to by their first name or "coach." In some schools the instructor might be called "Professor" (which is just Portuguese for "teacher" or "sensei", but this is less common.
At my gym, the common procedures look something like the following:
Class begins with the instructor calling everyone in attendance to circle up, we're getting started. Typically we'll go through some light warm ups, then the instructor will demonstrate 2-3 techniques and have students pair up to practice those techniques. After that comes rolling (live newaza sparring, most commonly) or some other live drill (such as pass/sweep, trying to escape a bad position, or fighting for takedowns). When starting to roll with a new partner*, usually we begin with a handslap and fist bump. At the end of class, students line up according to rank, the teacher and students bow to each other, and then the teacher walks down the line giving each student and handshake and hug. The student at the head of the line follows the leader so that everybody ends up shaking everybody's hand.
The above is typical at our school, but not set in stone. Sometimes the teacher might add a bow-in at the start of class. Sometimes we'll skip the bow out. Sometimes we'll mix up the class format a bit. Generally you'll be okay if you just go with the flow and watch the folks around you.
*(In some schools the instructor will tell you who to roll with, in other schools you just grab whoever is nearby. Some schools have a rule that lower belts aren't allowed to ask black belts to roll, which I find obnoxious.)
(BTW - we're not much on proclaiming "Oss" at our school. I've been to schools where everybody does that and it seems kind of silly to me.)