I am working on an Opteron 2384. I have a guest VM running on KVM with NPT enabled. I am seeing issues with inconsistency of memory look ups after modifying an NPT shadow page entry in a simple test of double mapping. I believe that the hardware is caching Guest physical to host physical entries in what is called a Nested TLB (information found here http://developer.amd.com/wordpress/media/2012/10/NPT-WP-1%201-final-TM.pdf). Is there a way to flush Nested TLB entries? No documentation exists from AMD. Intel has an invept instruction to do this.
My test is as follows.
From the guest, I created a kernel module that 1) gets two free pages, 2) writes a 1 to byte 0 on page 1 and writes a 2 to byte 0 of page 2, then 3) hypercalls to kvm passing the physical pages of both page 1 and page 2.
From the KVM, I walk page the shadow NPT page tables to get the host physical frame of page 1 and page 2. Then I write the physical frame number of page 1 to the page table entry for page 2. I then flush the entire tlb (__flush_tlb_all(),kvm_mmu_flush(tlb(vcpu), kvm_flush_remote_tlbs(vcpu->kvm)). The effect is that the entry in the NPT shadow page tables for page 2 now points to the host physical frame of page 1.
Back in the guest, after the hypercall I flush the TLB using __flush_tlb_all() and just to be sure reading and writing to the cr3. I then read the 0 byte on page 1 and page 2. I expect both to return 1 as the shadow pages have been update to point both guest physical pages to the same host physical page. However, I see that reading page 2 returns a 2.
Please advise on how to solve this issue as there is a clear inconsistency of accessing memory even after the TLBs are adequately flushed.
There may be concerns about caching issues. I have disabled caching with set_memory_uc() on each of the guest pages and cleared the caches with wbinvld from both the guest and host.
03:29 PM by