raspberrypi/mmu/README

See the top level README file for more information on documentation
and how to run these programs.

This example demonstrates MMU basics.

So what an MMU does or at least what an MMU does for us is it translates
virtual addresses into physical addresses as well as checking access
permissions.  And lastly it gives us tighter control over cachable
regions.

So what does all of that mean?  Well so far, FROM THE ARM'S PERSPECTIVE
we have been using physical addresses.  We know from looking at the
Broadcom manual for this chip that the actual physical addresses
for peripherals for example as far as the manual is concerned are at
addresses that start with 0x7E, but in the ARM's address space we use
0x20.  So there is already an address translation going on somewhere.
But from the ARM's address bus perspective the 0x20... addresses are
the physical addresses as is the memory we use starting at 0x00000000.

So this address for example

#define ARM_TIMER_CTL 0x2000B408

we consider to be a physical address within the arms address space, as
is the address 0x00008000 where we assume our program is loaded before
the GPU lets the ARM start.

Basically ignore the man behind the curtain, you generally dont deal
with this, the ARM is usually the main processor and the memory system
is designed around it rather than what we have in this chip.

So physical addresses are the addresses that are used on the ARM's
address bus when accessing memory or peripherals.  When we power up the
MMU is off and the addresses we use when we write programs are physical
addresses.  But the MMU sits in the middle, when it is turned on then
the addresses we program with are considered virtual addresses, the
MMU converts them into physical addresses and the physical address
goes out on the address bus.  So we could for example program the
MMU to have the virtual address 0x00000000 map to the physical address
0x00100000 for example.  Now we cannot have any address map to any
address we cannot have 0x01234 map to 0x45678 for example, it doesnt
work that way.  If it did we would need a directory of addresses that
is larger than the amount of memory we have, if we wanted to convert
any address to any address we would need a look up table 4GBytes in
size if any byte address could be any other byte address.

What really happens is we can break up the space into blocks and the
whole block is virtualized somewhere else so for example we will in
this example have the virtual address range 0x00200000 to 0x002FFFFF
access the physical 0x00000000 to 0x000FFFFF range.  Hopefully this
will make sense soon...

As with all baremetal programming, wading through documentation is
the bulk of the job.  Definitely true here, with the unfortunate
problem that ARM's docs dont all look the same from one Archtectural
Reference Manual to an other.  We have this other problem that we
are techically using an ARMv6 (architecture version 6) but when
you go to http://infocenter.arm.com and look at the Reference Manuals
there is an ARMv5 and then ARMv7 and ARMv8, but no ARMv6.  Well
the ARMv5 manual is actually the original ARM ARM, that I assume they
realized couldnt maintain all the architecture variations forever,
so the perhaps wisely went to one ARM ARM per rev.  With respect to the
MMU, that started in ARMv5 and with ARMv6 there were some changes
made but it still has a backwards compatible mode such that programs
that use the MMU (linux for example) dont necessarily need an overhaul
every version.  So you can look at the various architectural reference
manuals or sometimes technical reference manuals for specific cores
and see descriptions of the MMU tables and addressing but the
part I mentioned as unfortunate is that the drawings and descriptions
dont have the same look and feel.  They have the same basic content
though.

I am mostly using the ARMv5 Architectural Reference Manual.  Possibly
an older one than the one on ARMs page.  ARM DDI0100I.  Where the I is
the rev of that ARM ARM.  The ARMv5 ARM does show ARMv6 stuff in
particular with respect to them MMU, so it is probably the right
manual for this processor, although you could use the ARMv7 and be
careful to ignore features added in v7.

So there are blocks they call sections and blocks they call pages. If we
were to simply take every possible address and make a look up table
and the contents of the table are the physical address, we could then
translate any virtual address to any physical address, but it would
take up to 4GBytes for that table.  (and we would have to access
everything as bytes since a scheme like that would allow the four
bytes in an instruction or other word sized access to be in up to
four different physical places)  That is not exactly what happens
but it is along the same path.  Instead of taking the entire address
and having a look up table, we take the top bits of the address and
that goes into the first level translation table.  Basically bits
31:20 (bits 31 down to 20 or perhaps think of it as address>>20) are
added (orred) to the base address for this table we have to prepare.
The contents of the table are not necessarily the replacement bits, but
the way we are using it they are.

The ARM documentation talks about sections and pages, perhaps this is
not the intended distiction, but with sections the first level
translation table contains both the replacement bits (will describe
what that means in a second) and the permission and other control bits.
For a page, the first level translation table contains an offset to
a second level translation table, a second table.  The combination of
bits in that first table and second table serve to describe the
access permissions, and replacement bits.

So with what I am telling you so far with the addition of saying that
we will mostly be talking about 1MByte sections, that means that
I can have a virtual address of 0x1230ABCD, virtual being the address
that I write my software to use, and have that get converted by the
MMU to the address 0x4560ABCD.  Basically the address bits 31:20 I can
change in the MMU using a 1MByte section.  Further those upper address
bits which are 0x123 in this example are used to look up an entry
in the first level descriptor table, and that entry contains the bits
0x456 as well as some other bits for permissions and cache control.
Assuming the permissions and such are okay the MMU then simply replaces
the 0x123 with 0x456 causing our 0x1230ABCD address to actually
access 0x4560ABCD.  The lower 20 bits, for a 1MByte section have
to be the same in the virtual and physical address.  So only some
of the upper bits are replaced.

Now maybe you can see why there are blocks or chunks of memory that
are virtualized, the lower address bits are not modified between
the virtual and physical, basically a whole block of memory space
aligned on some power of 2.  And the other thing to understand now
is that because the translation table ultimately contains the
replacement bits for the bits used to look up into the table,  Depending
on how many permission and other control bits we want the number
of replacement bits left over in a 32 bit word are limited.  But if
we were to have a second table, then between the first and second
tables we have 64 bits so when we have a bunch of bits to replace
meaning we have a smaller block of memory being virtualized somewhere
else, we will need the secondary table.  

So you may be thinking that we have a chicken and egg problem, but we
dont.  We want to access something at some address, that act causes
the MMU to access the translation tables which are at some address
in memory, now if the MMU had to go through the MMU, you would have
that chicken and egg problem.  You dont the MMU does not use virtual
addresses it is all physical addresses, it doesnt send itself through
itself.  But this does mean that we have to carve out some amount
of memory for the MMU translation tables.  The pictures imply this
can vary but as far as we are concerned all of the MMU tables, first
level has to fit within 16Kbytes.

So we can be looking at the same picture I took a couple of pages
out of the ARM manual and put them in this repo as a postscript, if
on linux then no big deal your pdf reader will/should also read
postscript (postscript is like assembly and pdf is simply the machine
code for that assembly, assuming unencrypted, with free tools you can
generally go back and forth between pdf and ps).  Atril, evince, etc
can display this, gsview and others like it will work on both windows
and Linux.  section_translation.ps is the name of the file.

The picture on the second page is where we want to start, and a
picture is worth a thousand words, and although this is verbose already
hopefully I wont have to spend too many more words on this picture.

The first thing the picture is telling us is that there is a
base address somewhere that we tell the MMU about that is the base
address for our translation table memory, where are primary and
secondary translation tables live.  This is important SBZ means should
be zero, the lower 14 bits assuming X is zero, must be zero so we
must choose an address that has the lower 14 bits zero.  I have chosen
0x00004000 which just barely makes that requirement.  I assume
that my program is loaded into the ARM address 0x8000, I will need
to have some exception handlers at 0x0000, but 0x4000 to 0x8000 is
not being used (I have my stack elsewhere).

So we have a base address for our translation table.  So lets do the
conversion mentioned above of virtual 0x1230ABCD to physical 0x4560ABCD.
What they are calling a modified virtual address is our...virtual
address the address we write in our program on the processor side
of the MMU.  So that is the 0x1230ABCD address.  We break that address
up into its two parts, the Table Index which is 0x123 and the section
index which is the 0x0ABCD part.  The next thing down is the address
of the first level descriptor.  So they take the 12 bits of index
shift those left two so it makes a word address and add that to the
translation tables base address.  In this case 0x123<<2 = 0x48C and
our base address of 0x00004000 gives us 0x0000448C.  Now the descriptors
are all physical addresses the MMU doesnt use the MMU to access the
MMU tables.  So we read the 32 bit entry at the address we computed
and we get the first level descriptor.  The first thing we look at
in the first level descriptor are the lower 2 bits.  If those bits are
a 0b10 then this is a section, the other bit patterns are documented
not far below these pages in the manual.  The first of the two pages
I have here shows the 0b10 in those lower bits and also says that
to be a 1MB descriptor we need bit 18 to be a zero, and so we will.
The MMU now knowing this is a 1MB first level descriptor then it checks
the other bits not shown on either of these pages but we will cover,
for access permissions, if we have not violated any permissions then
it takes the upper 12 bits of the descriptor and tacks those on top
of the lower 20 bits of our virtual address to make the physical address
and then the MMU sends that down the pipe and we do our memory/peripheral
access.

These pictures in whatever form show the virtual to physical translation
but we as MMU programers need to go from physical to virtual, if after
we turn the MMU on we still want to be able to access the UART for
example will will have to have an entry so that we can control and
allow the access using the access control permissions.  Hopefully you
have figured out that we can replace those 12 bits with whatever 12
bits we want, including the same 12 bits.  Why would we use the MMU
to replace some address bits with the same address bits!  Remember the
MMU is not only there to remap memory space, but it is also there to
allow for control over access permissions and to allow control over
caching.  Separate controls for each page or section.  So working
backward we want to have our uart which is in the section 0x20200000
be available to us after the MMU is enabled.  It really makes it so
much easier if we have the virtual match the physical for peripherals
and actually this example starts off with virtual matching physical
for all the sections we care about.  So we need 0x202.... to result
in 0x202.  So our translation table entry is 0x202 based or
table_base + (0x202<<2).  And the data at that address needs to be
0x202xxxxx with the lower two bits a 0b10.  And the rest of the
bits such that it just works.

So now we have to chat a bit about that.  The "other" bits are the
domain, the TEX bits and the C and B bits.  The C bit is the simplest
one to start with that means Cacheable.  For peripherals we absolutely
dont want them to be cached.  Lets say for example we are polling a
register in the uart to see if the tx buffer is empty so we can
send another character, so we read that register a bunch of times
until some control bit indicates tx buf is empty.  Well if the cache
were on the first time we read that register its value gets cached
then the next time we get the cached value not the real value, if all
we are doing is polling and we dont evict that cached value then all
we will ever see is the stale, cached, regsiter value, if that
value did not show that tx buff was empty, then we will never see
the indication when it changes.  So never make a peripherals space
cacheable.  This is a good place to point out the purpose fo an MMU
again cache control.  Right now we can see that the MMU even with
virtual = physical, allows us to turn on the data cache, but gives
us control that we can mark perhipheral address spaces as not
cacheable.

The b bit, means bufferable, as in write buffer.  Something you may
not have heard about or thought about ever.  It is kind of like a cache
on the write end of things instead of read end.  It is a thing somewhere
between the processor and the memory that tells the processor, let me
take that write information and deliver it for you, you can keep
doing other stuff.  Now writes in general are "fire and forget".  When
you perform a write both the address and data are known, in general
the memory controller can and depending on the design, will, take the
address and data and tell the processor, I will go and do that for you
you keep processing.  Well that works fine as an optimization for the
first write, but eventually the write has to end up in the slow
main memory.  So if you do two or a bunch of writes in a row the
processor gets the optimization on the first one but the second one
has to wait for the first and the processor ends up waiting.  Well
further down if you were to have a small buffer that could hold more
than one write in flight at a time, and allow the processor to get
this optimization for more than just one write cycle but maybe many
or several then for situations where the processor is doing random
writes, you probably can gain some speed.  A good place to use this
is when you have the cache on, as a cache line is not just one
word or whatever wide, it can be several words of data, so when you
have a cache miss, need to read a cache line, but you dont have an
open spot and need to evict someone from the cache that multi-word
eviction can go into the write buffer, allowing the cache to do
the cache line read.  But if the write buffer is not there or not
enabled then everyone has to wait for that cache line eviction
to make room for the cache line fill to then finally send the
read data back to the processor. Now do we want to enable the write
buffer for peripherals?  Well probably not, even though the arm
manual may show a combination with B on that means device access.  Lets
take the generic write buffer case and not necessarily an ARM one.
The write buffer absorbs some number of write accesses for the processor
so the processor can continue excuting and not have to wait for a
slow memory transaction to complete.  So the processor is operating
ahead of the writes the program thinks have completed.  So maybe we
poll the uart status register, it says the tx buf is empty, we write
a byte, which lands in the buffer behind some other writes, we then
have another byte to send, we read the status register, if the reads
and writes are not serialized meaning if the reads take a separate
path from the writes, then it is possible that the write of our first
byte is stuck in the write buffer waiting on other writes, so the write
has not hit the uart, the txbuf still shows empty, the next read
of the status register shows empty so we send another byte, but
eventually the two writes hit but there is only room for one.  So we
probably dont want to use write buffering in general with peripeherals
unless we are sure we know how the hardware works and we dont have these
race conditions.

Now the TEX bits you just have to look up and there is the rub there
are likely more than one set of tables for TEX C and B, I am going
to stick with a TEX of 0b000 and not mess with any fancy features
there.  Now depending on whether this is considered an older arm
(ARMv5) or an ARMv6 or newer the combination of TEX, C and B have
some subtle differences.  The cache bit in particular does enable
or disable this space as cacheable.  You still independently need
to turn on the instruciton and data caches and need an if cacheable
and the cache is on for the access type within that section, then it
will cache it...So we set tex to zeros to just keep it out of the way.

Lastly the domain bits.  Now you will see a 4 bit domain thing and
a 2 bit domain thing.  These are related.  There is a register in
the MMU right next to the translation table base address register this
one is a 32 bit register that contains 16 different domain definitions.

The two bit domain controls are defined as such.

0b00 No access Any access generates a domain fault
0b01 Client Accesses are checked against the access permission bits in the TLB entry
0b10 Reserved Using this value has UNPREDICTABLE results
0b11 Manager Accesses are not checked against the access permission bits in the TLB
entry, so a permission fault cannot be generated

For starters we are going to set all of the domains to 0b11 dont check
cant fault.  What are these 16 domains though?  Notice it takes 4 bits
to describe one of 16 things.  The different domains have no specific
meaning other than that we can have 16 different definitions that we
control for whatever reason.  You might allow for 16 different
threads running at once in your operating system, or 16 different
types of software running (kernel, application, ...) you can mark
a bunch of sections as belonging to one parituclar domain, and with a
simple change to that domain control register, a whole domain might
go from one type of permission to another, from no checking to
no access for example.

Since I usually use the MMU in bare metal to enable data caching on ram
I set my domain controls to 0b11, no checking and I simply make all
the MMU sections domain number 0.

So we end up with this simple function that allows us to add first level
descriptors in the MMU translation table.

unsigned int MMU_section ( unsigned int vadd, unsigned int padd, unsigned int flags )
{
    unsigned int ra;
    unsigned int rb;
    unsigned int rc;

    ra=vadd>>20;
    rb=MMUTABLEBASE|(ra<<2);
    ra=padd>>20;
    rc=(ra<<20)|flags|2;
    PUT32(rb,rc);
    return(0);
}

So what you have to do to turn on the MMU is to first figure out all
the memory you are going to access, and make sure you have entries
for that.  Now if you do the math, 12 bits off the top are the
first level index, that is 4096 things, times 4 bytes per that is 16KBytes
thus the reason for an alignment on 16K.  Now one solution you might
simply do is fill the whole 16K with 1MByte sections that allow full
uncached access...Basically completely map the virtual to physical
one to one.  I didnt do that, I was a little more concervative on the
clock cycles, not that that really matters here...For this example I
wanted to have the memory we are really using around 0x00000000 and
then some entries I can play with to show you the MMU is working and
then the entries for the peripherals I am using.

    MMU_section(0x00000000,0x00000000,0x0000|8|4);
    MMU_section(0x00100000,0x00100000,0x0000);
    MMU_section(0x00200000,0x00200000,0x0000);
    MMU_section(0x00300000,0x00300000,0x0000);
    //peripherals
    MMU_section(0x20000000,0x20000000,0x0000); //NOT CACHED!
    MMU_section(0x20200000,0x20200000,0x0000); //NOT CACHED!

I didnt need to cache that first section, but did, will leave it up
to you to do a read performance test of some sort to determine if the
cache when enabled does make it faster.

So once our tables are setup then we need to actually turn the
MMU on.  Now I cant figure out where I got this from, and I have
modified it in this repo.  According to this manual it was with the
ARMv6 that we got the DSB feature which says wait for either cache
or MMU to finish something before continuing.  In particular when
initializing a cache to start it up you want to clean out all the
entries in a safe way you dont want to evict them and hose memory
you want to invalidate everything, mark it such that the cache lines
are empty/available.  not mentioned yet but the MMU has a mini cache
that it uses for things it has looked up, think about every access we
do through the MMU, imagine if it had to do walk the descriptor tables
every single read or write could require two more reads from the
table.  So there is this TLB which caches up the last N number of
descriptor table lookups.  Well like cache memory on power up, the
tlb might be full of random bits as well, so we need to invalidate
that too.  Then this dsb thing comes in, we do the dsb instruction
to tell the processor to wait for the cache subsystem and MMU subsystem
to finish wiping their internal tables before we go forward and
turn them on and try to use them.

After we invalidate the cache and tlb, and you may be asking why are
we messing with the cache?  Well the MMU gets us access to the data
cache since we need the MMU to distinguish ram from peripherals before
generically turning on the data cache.  Second in the ARM the MMU
enable bit and the cache enable bits are in the same register so it
makes sense to just do cache enabling and MMU enabling in one function
call.

So after the DSB we set our domain control bits, now in this example
I have done something different, 15 of the 16 domains have the 0b11
setting which is dont fault on anything, manager mode.  I set domain
1 such that it has no access, so in the example I will change one
of the descriptor table entries to use domain one, then I will access
it and then see the access violation.  there are two registers that
hold the translation table base address, I program them both, not
sure what the difference is, why there are two...

Understand I have been runnign on ARMv6 systems without the DSB for
some time and it just works, so maybe that is dumb luck...

Now I can start the MMU.  This code relies on the caller to set
the MMU enable and I and D cache enables.  This is because this
is derived from code where sometimes I turn things on or dont turn
things on and wanted it generic.


.globl start_MMU
start_MMU:
    mov r2,#0
    mcr p15,0,r2,c7,c7,0 ;@ invalidate caches
    mcr p15,0,r2,c8,c7,0 ;@ invalidate tlb
    mcr p15,0,r2,c7,c10,4 ;@ DSB ??

    mvn r2,#0
    bic r2,#0xC
    mcr p15,0,r2,c3,c0,0 ;@ domain

    mcr p15,0,r0,c2,c0,0 ;@ tlb base
    mcr p15,0,r0,c2,c0,1 ;@ tlb base

    mrc p15,0,r2,c1,c0,0
    orr r2,r2,r1
    mcr p15,0,r2,c1,c0,0

    bx lr

I am going to mess with the translation tables after the MMU is started
so I assume we have to invalidate when a table entry changes so that
just in case the old one is cached up in the tlb, we can force the
read of the new one by invalidating all the tlbs.


.globl invalidate_tlbs
invalidate_tlbs:
    mov r2,#0
    mcr p15,0,r2,c8,c7,0  ;@ invalidate tlb
    mcr p15,0,r2,c7,c10,4 ;@ DSB ??
    bx lr

So the program starts by putting a few things in memory spaced
apart such that they will be in different sections when the
MMU is turned on.  We write then read those back.


DEADBEEF
00045678
00145678
00245678
00345678

Now the MMU is turned on with these sections mapped with virtual =
physical.

00045678
00145678
00245678
00345678

Nothing magical yet.  But now we start to swizzle things around, two
of the spaces are swapped 0x001...addresses point at 0x003 and vice
versa.  0x002 points at 0x000...And the output confirms that, we didnt
write anything to memory, just played games with what physical address
comes from what virtual.

00045678
00345678
00045678
00145678

And then the icing on the cake, one section is marked as domain 1
instead of domain 0, domain 1 was set for 0b00 no access so when we
touch that domain we should get an access violation.
                                                                         
00045678                                                                        
00000010                                                                        

How do I know what that means with that output.  Well from my blinker07
example we touched on exceptions (interrupts).  I made a generic test
fixture such that anything other than a reset prints something out
and then hangs.   In no way shape or form is this a complete handler
but what it does show is that it is the exception that is at address
0x00000010 that gets hit which is data abort.  So figuring out it was
a data abort (pretty much expected) have that then read the data fault
status registers, being a data access we expect the data/combined one
to show somthing and the instruction one to not.  Adding that
instrumentation resulted in.
                                                                            
00045678                                                                        
00000010                                                                        
00000019                                                                        
00000000                                                                        
00008110                                                                        
E5900000                                                                        
00145678           

Now I switched to the ARM1176JZF-S Technical Reference Manual for more
detail and that shows the 0x01 was domain 1, the domain we used for
that access. then the 0x9 means Domain Section Fault.

The lr during the abort shows us the instruction, which you would need
to disassemble to figure out the address, or at least that is one
way to do it perhaps there is a status register for that.

The instruction and the address match our expectations for this fault.