Topic Title: AMD Nested TLB Flushing - How do you do it? Consistency errors being observed
Topic Summary: Question on how to flush Nested TLB Entries
Created On: 09/04/2014 10:14 AM
Status: Post and Reply
Linear : Threading : Single : Branch
Search Topic Search Topic
Topic Tools Topic Tools
View similar topics View similar topics
View topic in raw text format. Print this topic.
 09/04/2014 10:14 AM
User is offline View Users Profile Print this message

Author Icon
thepalindrome
Peon

Posts: 3
Joined: 09/04/2014

I am working on an Opteron 2384. I have a guest VM running on KVM with NPT enabled. I am seeing issues with inconsistency of memory look ups after modifying an NPT shadow page entry in a simple test of double mapping. I believe that the hardware is caching Guest physical to host physical entries in what is called a Nested TLB (information found here http://developer.amd.com/wordpress/media/2012/10/NPT-WP-1%201-final-TM.pdf). Is there a way to flush Nested TLB entries? No documentation exists from AMD. Intel has an invept instruction to do this.

My test is as follows.

From the guest, I created a kernel module that 1) gets two free pages, 2) writes a 1 to byte 0 on page 1 and writes a 2 to byte 0 of page 2, then 3) hypercalls to kvm passing the physical pages of both page 1 and page 2.

From the KVM, I walk page the shadow NPT page tables to get the host physical frame of page 1 and page 2. Then I write the physical frame number of page 1 to the page table entry for page 2. I then flush the entire tlb (__flush_tlb_all(),kvm_mmu_flush(tlb(vcpu), kvm_flush_remote_tlbs(vcpu->kvm)). The effect is that the entry in the NPT shadow page tables for page 2 now points to the host physical frame of page 1.

Back in the guest, after the hypercall I flush the TLB using __flush_tlb_all() and just to be sure reading and writing to the cr3. I then read the 0 byte on page 1 and page 2. I expect both to return 1 as the shadow pages have been update to point both guest physical pages to the same host physical page. However, I see that reading page 2 returns a 2.

Please advise on how to solve this issue as there is a clear inconsistency of accessing memory even after the TLBs are adequately flushed.

Edit:

There may be concerns about caching issues. I have disabled caching with set_memory_uc() on each of the guest pages and cleared the caches with wbinvld from both the guest and host.



Edited: 09/04/2014 at 03:29 PM by thepalindrome
 09/05/2014 10:16 AM
User is offline View Users Profile Print this message

Author Icon
thepalindrome
Peon

Posts: 3
Joined: 09/04/2014

I have moved my code and tests to an Intel machine with EPT. My test works as expected. This leads me to believe the AMD processor is not properly managing the TLB or there is a bug in KVM. My guess is the former as I have modified KVM to flush all host and guest entries on VMRUN and thus all TLBs should be cleared before reading in step 3.

Are there any amd reps that can comment?

 09/09/2014 03:00 PM
User is offline View Users Profile Print this message

Author Icon
neo5555
Case Modder

Posts: 944
Joined: 08/13/2008

This maybe of some help...

 

http://www.dailytech.com/Understanding++AMDs+TLB+Processor+Bug/article9915.htm



-------------------------

Pics of my rig
i7 4770k, custom water cooling, 8Gig Corsair Dom 2133 9-11-10-27, ASrock Z87 OC Formula, SilverStone ST-1500 PSU, Creative X-Fi Platinum, Logitech Z-5500D , 2xHIS 7970 CF, Asus VW246H Monitor, Win 7 64Bit HP, Corsair 800D Case

 09/10/2014 07:31 AM
User is offline View Users Profile Print this message

Author Icon
thepalindrome
Peon

Posts: 3
Joined: 09/04/2014

Originally posted by: neo5555 This maybe of some help...

 

 

 

http://www.dailytech.com/Understanding++AMDs+TLB+Processor+Bug/article9915.htm

 

I believe I have addressed this bug by disabling TLB Filtering which can be found in the init_amd code. So I believe this is a separate issue.

Statistics
86654 users are registered to the AMD Support and Game forum.
There are currently 2 users logged in.

FuseTalk Hosting Executive Plan v3.2 - © 1999-2014 FuseTalk Inc. All rights reserved.