From ralf@linux-mips.net Thu Aug 1 16:13:43 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id QAA00753; Thu, 1 Aug 2002 16:13:43 +0200 (MET DST) Received-Date: Thu, 1 Aug 2002 16:13:43 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpda00707; Thu Aug 1 16:13:33 2002 Received: from dea.linux-mips.net (shaft17-f3.dialo.tiscali.de [62.246.17.3]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g71DP8M16238 for ; Thu, 1 Aug 2002 15:25:09 +0200 (CEST) Received: (from ralf@localhost) by dea.linux-mips.net (8.11.6/8.11.6) id g71DP0129522; Thu, 1 Aug 2002 15:25:00 +0200 Date: Thu, 1 Aug 2002 15:25:00 +0200 From: Ralf Baechle To: "Maciej W. Rozycki" Cc: linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes Message-ID: <20020801152500.A31808@dea.linux-mips.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: ; from macro@ds2.pg.gda.pl on Mon, Jul 29, 2002 at 04:29:35PM +0200 On Mon, Jul 29, 2002 at 04:29:35PM +0200, Maciej W. Rozycki wrote: > The following patch fixes all the places the default caching policy is > used but various local hacks are coded. Also the sc coherency algorithm > is configured for R4k processors which was previously left as set (or not) > by the firmware. A side effect is is created > and all conditional CPU options are set somehow. Tested on an R4400SC > (for both MIPS and MIPS64) and on an R3400. > > Admittedly, CONF_CM_DEFAULT is defined in a bit weird way, but I couldn't > figure any better one that wouldn't result in a serious but unnecessary > header bloat. If anyone has a better idea, please share any suggestions > here. > > OK to apply? Looks mostly right except that the code in config-shared.in which deciedes if a system is coherent is wrong. Basically it assumes every R10000 or every uniprocessor system is non-coherent and that's wrong. As coherency between CPUs and for DMA I/O is basically the same thing I've changed your code from the use of CONFIG_CPU_CACHE_COHERENCY to CONFIG_NONCOHERENT_IO which did already exist; I don't think we need another config symbol to handle this. Will apply once I did a few test builds and patches the whole thing into 2.5 ... Ralf From macro@ds2.pg.gda.pl Thu Aug 1 17:24:26 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id RAA06337; Thu, 1 Aug 2002 17:24:26 +0200 (MET DST) Received-Date: Thu, 1 Aug 2002 17:24:26 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd006333; Thu Aug 1 17:24:23 2002 Received: from delta.ds2.pg.gda.pl (macro@delta.ds2.pg.gda.pl [213.192.72.1]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g71FOLM17705 for ; Thu, 1 Aug 2002 17:24:22 +0200 (CEST) Received: from localhost by delta.ds2.pg.gda.pl (8.9.3/8.9.3) with SMTP id RAA08910; Thu, 1 Aug 2002 17:24:43 +0200 (MET DST) Date: Thu, 1 Aug 2002 17:24:43 +0200 (MET DST) From: "Maciej W. Rozycki" To: Ralf Baechle cc: Carsten Langgaard , linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [patch] MIPS64 R4k TLB refill CP0 hazards In-Reply-To: <20020731223158.A6394@dea.linux-mips.net> Message-ID: Organization: Technical University of Gdansk MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII On Wed, 31 Jul 2002, Ralf Baechle wrote: > I doublechecked the docs for the R4700 as well - just one cycle needed > between a tlbw and eret. After looking at the generated assembly I discovered the handlers don't fit in 128 bytes. They didn't crash since I have modules disabled for now, so the vmalloc path didn't get hit and the user path happened to fit, but it was pure luck. The path got hit before I fixed a bug in gas though -- that's the explanation of the false cache error exceptions I used to observe. Here is a temporary corrected version I use now. It works, but ultimately the chosen handler should get copied somewhere above KSEG0+0x200. Also the "non_vmalloc" path looks bogus -- k0 gets loaded with a random value from (k1) (if it doesn't happen to fault again), k1 retains the pointer and then both get loaded into the TLB -- intriguing... I believe the patch should get applied for now to avoid surprises. OK? Maciej -- + Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: macro@ds2.pg.gda.pl, PGP key available + patch-mips-2.4.19-rc1-20020726-mips64-tlbex-r4k-2 diff -up --recursive --new-file linux-mips-2.4.19-rc1-20020726.macro/arch/mips64/mm/tlbex-r4k.S linux-mips-2.4.19-rc1-20020726/arch/mips64/mm/tlbex-r4k.S --- linux-mips-2.4.19-rc1-20020726.macro/arch/mips64/mm/tlbex-r4k.S 2002-07-25 02:57:02.000000000 +0000 +++ linux-mips-2.4.19-rc1-20020726/arch/mips64/mm/tlbex-r4k.S 2002-07-31 23:20:21.000000000 +0000 @@ -5,6 +5,7 @@ * * Copyright (C) 2000 Silicon Graphics, Inc. * Written by Ulf Carlsson (ulfc@engr.sgi.com) + * Copyright (C) 2002 Maciej W. Rozycki */ #include #include @@ -22,8 +23,7 @@ * After this macro runs we have a pointer to the pte of the address * that caused the fault in in PTR. */ - - .macro LOAD_PTE2, ptr, tmp + .macro LOAD_PTE2, ptr, tmp, kaddr #ifdef CONFIG_SMP dmfc0 \ptr, CP0_CONTEXT dmfc0 \tmp, CP0_BADVADDR @@ -32,8 +32,8 @@ dmfc0 \tmp, CP0_BADVADDR dla \ptr, pgd_current #endif - bltz \tmp, kaddr - ld \ptr, (\ptr) + bltz \tmp, \kaddr + ld \ptr, (\ptr) dsrl \tmp, (PGDIR_SHIFT-3) # get pgd offset in bytes andi \tmp, ((PTRS_PER_PGD - 1)<<3) daddu \ptr, \tmp # add in pgd offset @@ -48,6 +48,35 @@ daddu \ptr, \tmp .endm + + /* + * Ditto for the kernel table. + */ + .macro LOAD_KPTE2, ptr, tmp, not_vmalloc + /* + * First, determine that the address is in/above vmalloc range. + */ + dmfc0 \tmp, CP0_BADVADDR + dli \ptr, VMALLOC_START + + /* + * Now find offset into kptbl. + */ + dsubu \tmp, \tmp, \ptr + dla \ptr, kptbl + dsrl \tmp, (PAGE_SHIFT+1) # get vpn2 + dsll \tmp, 4 # byte offset of pte + daddu \ptr, \ptr, \tmp + + /* + * Determine that fault address is within vmalloc range. + */ + dla \tmp, ekptbl + sltu \tmp, \ptr, \tmp + beqz \tmp, \not_vmalloc # not vmalloc + .endm + + /* * This places the even/odd pte pair in the page table at the pte * entry pointed to by PTE into ENTRYLO0 and ENTRYLO1. @@ -59,6 +88,7 @@ dmtc0 \pte1, CP0_ENTRYLO1 # load it .endm + .text .set noreorder .set mips3 @@ -66,105 +96,93 @@ __INIT .align 5 -FEXPORT(except_vec0) +LEAF(except_vec0) .set noat PANIC("Unused vector called") 1: b 1b nop +END(except_vec0) + + /* + * TLB refill handler for the R4000. + * Attention: We may only use 32 instructions / 128 bytes. + */ .align 5 LEAF(except_vec1_r4k) .set noat - dla k1, pgd_current - dmfc0 k0, CP0_BADVADDR - ld k1, (k1) - bltz k0, vmaddr - dsrl k0, (PGDIR_SHIFT-3) # get pgd offset in bytes - andi k0, ((PTRS_PER_PGD - 1)<<3) - daddu k1, k0 # add in pgd offset - dmfc0 k0, CP0_BADVADDR - ld k1, (k1) # get pmd pointer - dsrl k0, (PMD_SHIFT-3) # get pmd offset in bytes - andi k0, ((PTRS_PER_PMD - 1)<<3) - daddu k1, k0 # add in pmd offset - dmfc0 k0, CP0_XCONTEXT - andi k0, 0xff0 # get pte offset - ld k1, (k1) # get pte pointer - daddu k1, k0 - ld k0, 0(k1) # get even pte - ld k1, 8(k1) # get odd pte - dsrl k0, 6 # convert to entrylo0 - dmtc0 k0, CP0_ENTRYLO0 # load it - dsrl k1, 6 # convert to entrylo1 - dmtc0 k1, CP0_ENTRYLO1 # load it - nop # Need 2 cycles between mtc0 - nop # and tlbwr (CP0 hazard). - tlbwr - nop - eret -vmaddr: - dla k0, handle_vmalloc_address + dla k0, handle_vec1_r4k jr k0 nop END(except_vec1_r4k) - + + __FINIT + + .align 5 +LEAF(handle_vec1_r4k) + .set noat + LOAD_PTE2 k1 k0 9f + ld k0, 0(k1) # get even pte + ld k1, 8(k1) # get odd pte + PTE_RELOAD k0 k1 + b 1f + tlbwr +1: nop + eret + +9: # handle the vmalloc range + LOAD_KPTE2 k1 k0 invalid_vmalloc_address + ld k0, 0(k1) # get even pte + ld k1, 8(k1) # get odd pte + PTE_RELOAD k0 k1 + b 1f + tlbwr +1: nop + eret +END(handle_vec1_r4k) + + __INIT + /* * TLB refill handler for the R10000. * Attention: We may only use 32 instructions / 128 bytes. */ - .align 5 LEAF(except_vec1_r10k) + .set noat + dla k0, handle_vec1_r10k + jr k0 + nop +END(except_vec1_r10k) + + __FINIT + + .align 5 +LEAF(handle_vec1_r10k) .set noat - LOAD_PTE2 k1 k0 + LOAD_PTE2 k1 k0 9f ld k0, 0(k1) # get even pte ld k1, 8(k1) # get odd pte PTE_RELOAD k0 k1 nop tlbwr eret -kaddr: - dla k0, handle_vmalloc_address # MAPPED kernel needs this - jr k0 - nop - END(except_vec1_r10k) - - __FINIT - .align 5 -LEAF(handle_vmalloc_address) - .set noat - /* - * First, determine that the address is in/above vmalloc range. - */ - dmfc0 k0, CP0_BADVADDR - dli k1, VMALLOC_START - - /* - * Now find offset into kptbl. - */ - dsubu k0, k0, k1 - dla k1, kptbl - dsrl k0, (PAGE_SHIFT+1) # get vpn2 - dsll k0, 4 # byte offset of pte - daddu k1, k1, k0 - - /* - * Determine that fault address is within vmalloc range. - */ - dla k0, ekptbl - sltu k0, k1, k0 - beqz k0, not_vmalloc - - /* - * Load cp0 registers. - */ +9: # handle the vmalloc range + LOAD_KPTE2 k1 k0 invalid_vmalloc_address ld k0, 0(k1) # get even pte ld k1, 8(k1) # get odd pte - -not_vmalloc: PTE_RELOAD k0 k1 nop tlbwr eret - END(handle_vmalloc_address) +END(handle_vec1_r10k) + + + .align 5 +LEAF(invalid_vmalloc_address) + .set noat + PANIC("Invalid kernel address") +1: b 1b + nop +END(invalid_vmalloc_address) From macro@ds2.pg.gda.pl Thu Aug 1 18:04:55 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id SAA09821; Thu, 1 Aug 2002 18:04:55 +0200 (MET DST) Received-Date: Thu, 1 Aug 2002 18:04:55 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd009819; Thu Aug 1 18:04:50 2002 Received: from delta.ds2.pg.gda.pl (macro@delta.ds2.pg.gda.pl [213.192.72.1]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g71G4nM18154 for ; Thu, 1 Aug 2002 18:04:49 +0200 (CEST) Received: from localhost by delta.ds2.pg.gda.pl (8.9.3/8.9.3) with SMTP id SAA09605; Thu, 1 Aug 2002 18:05:17 +0200 (MET DST) Date: Thu, 1 Aug 2002 18:05:17 +0200 (MET DST) From: "Maciej W. Rozycki" To: Ralf Baechle cc: linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes In-Reply-To: <20020801152500.A31808@dea.linux-mips.net> Message-ID: Organization: Technical University of Gdansk MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII On Thu, 1 Aug 2002, Ralf Baechle wrote: > Looks mostly right except that the code in config-shared.in which deciedes > if a system is coherent is wrong. Basically it assumes every R10000 or > every uniprocessor system is non-coherent and that's wrong. As coherency > between CPUs and for DMA I/O is basically the same thing I've changed your > code from the use of CONFIG_CPU_CACHE_COHERENCY to CONFIG_NONCOHERENT_IO > which did already exist; I don't think we need another config symbol to > handle this. Will apply once I did a few test builds and patches the > whole thing into 2.5 ... Huh? Coherent caching mode can be used for a few processors only, namely R4[04]00MC and presumably SB1 (inferred from the sources), i.e. the ones that support the interprocessor coherency protocol. If you know of any other processor that supports the protocol, I'd be pleased to see a reference to a spec -- I hoped someone, possibly you, would fill the missing bits when I proposed the patch a month ago. Nobody bothered, though, sigh... I see your changes are broken conceptually, as the caching mode for the TLB should be inferred from the CPU configuration in the first place and not the system selection (actually it should be best selected ath the run time). Hence I'd invert the flag, since most systems are non-coherent, and only permit it for certain processors. Using a non-coherent configuration for an UP system that supports coherency (do SGI IP27 and SiByte SB1250 have another agent in the chipset that may issue coherent requests regardless of the number of processors started?) results in a performance hit only due to superfluous invalidations, but using a coherent configuration for a processor/system that doesn't support it may lead to a hard to debug hang with no apparent reason (as I wrote previously, even NMI/Reset stopped working on my system -- I had to hit the power switch). I'll cook another patch to fix what got broken. Maciej -- + Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: macro@ds2.pg.gda.pl, PGP key available + From ralf@linux-mips.net Thu Aug 1 18:49:46 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id SAA13273; Thu, 1 Aug 2002 18:49:46 +0200 (MET DST) Received-Date: Thu, 1 Aug 2002 18:49:46 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd013271; Thu Aug 1 18:49:37 2002 Received: from dea.linux-mips.net (shaft16-f78.dialo.tiscali.de [62.246.16.78]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g71GnYM18596 for ; Thu, 1 Aug 2002 18:49:34 +0200 (CEST) Received: (from ralf@localhost) by dea.linux-mips.net (8.11.6/8.11.6) id g71GnTM23294; Thu, 1 Aug 2002 18:49:29 +0200 Date: Thu, 1 Aug 2002 18:49:29 +0200 From: Ralf Baechle To: "Maciej W. Rozycki" Cc: linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes Message-ID: <20020801184929.B22824@dea.linux-mips.net> References: <20020801152500.A31808@dea.linux-mips.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: ; from macro@ds2.pg.gda.pl on Thu, Aug 01, 2002 at 06:05:17PM +0200 On Thu, Aug 01, 2002 at 06:05:17PM +0200, Maciej W. Rozycki wrote: > Huh? Coherent caching mode can be used for a few processors only, namely > R4[04]00MC and presumably SB1 (inferred from the sources), i.e. the ones > that support the interprocessor coherency protocol. If you know of any > other processor that supports the protocol, I'd be pleased to see a > reference to a spec -- I hoped someone, possibly you, would fill the > missing bits when I proposed the patch a month ago. Nobody bothered, > though, sigh... R10000. > I see your changes are broken conceptually, as the caching mode for the > TLB should be inferred from the CPU configuration in the first place and > not the system selection (actually it should be best selected ath the run > time). Hence I'd invert the flag, since most systems are non-coherent, > and only permit it for certain processors. Back in time I prefered CONFIG_NONCOHERENT_IO over CONFIG_COHERENT_IO because the noncoherent case needs additional code and in general I'm trying to reduce the number of the #if !defined conditionals for easier readability. The R10000 is our standard example why looking at the processor type doesn't work. It's used in coherent mode in IP27 but in coherent mode but in coherent mode in IP28 or IP32. Otoh I don't know of any system that supports coherency but also is being used with non-coherent processors. > Using a non-coherent > configuration for an UP system that supports coherency (do SGI IP27 and > SiByte SB1250 have another agent in the chipset that may issue coherent > requests regardless of the number of processors started?) Yes. That's how coherency is working - all agents have to support coherent requests or coherency simply won't work. So basically we'd be trully picky we'd have to verify that all agents, processor and other support coherency but just using the system type seems to be sufficient. > results in a > performance hit only due to superfluous invalidations, but using a > coherent configuration for a processor/system that doesn't support it may > lead to a hard to debug hang with no apparent reason (as I wrote > previously, even NMI/Reset stopped working on my system -- I had to hit > the power switch). Using a non-coherent mode on IP27 may result in nice, hard to trackdown bus errors. Ralf From macro@ds2.pg.gda.pl Thu Aug 1 19:13:46 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id TAA16686; Thu, 1 Aug 2002 19:13:46 +0200 (MET DST) Received-Date: Thu, 1 Aug 2002 19:13:46 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd016684; Thu Aug 1 19:13:38 2002 Received: from delta.ds2.pg.gda.pl (macro@delta.ds2.pg.gda.pl [213.192.72.1]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g71HDbM18936 for ; Thu, 1 Aug 2002 19:13:37 +0200 (CEST) Received: from localhost by delta.ds2.pg.gda.pl (8.9.3/8.9.3) with SMTP id TAA10809; Thu, 1 Aug 2002 19:14:05 +0200 (MET DST) Date: Thu, 1 Aug 2002 19:14:05 +0200 (MET DST) From: "Maciej W. Rozycki" To: Ralf Baechle cc: linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes In-Reply-To: <20020801184929.B22824@dea.linux-mips.net> Message-ID: Organization: Technical University of Gdansk MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII On Thu, 1 Aug 2002, Ralf Baechle wrote: > R10000. OK. Any specs anywhere? > Back in time I prefered CONFIG_NONCOHERENT_IO over CONFIG_COHERENT_IO > because the noncoherent case needs additional code and in general I'm > trying to reduce the number of the #if !defined conditionals for easier > readability. Hmm, what's wrong with "#ifndef"? Not much less readable than "#ifdef", IMO. > The R10000 is our standard example why looking at the processor type doesn't > work. It's used in coherent mode in IP27 but in coherent mode but in > coherent mode in IP28 or IP32. Otoh I don't know of any system that > supports coherency but also is being used with non-coherent processors. Yep, I suppose so, but the first criterion should be the CPU anyway. Basically: 1. Does the CPU support coherency? 2. If so, does the system? I'm going to express it this way in the config script. > > Using a non-coherent > > configuration for an UP system that supports coherency (do SGI IP27 and > > SiByte SB1250 have another agent in the chipset that may issue coherent > > requests regardless of the number of processors started?) > > Yes. That's how coherency is working - all agents have to support coherent > requests or coherency simply won't work. So basically we'd be trully > picky we'd have to verify that all agents, processor and other support > coherency but just using the system type seems to be sufficient. Well, inferring from docs and my experience it's not needed. A system may simply require areas used for DMA to be marked as non-coherent by CPUs. Often uncached accesses are used to prevent spoiling the caches anyway. > > results in a > > performance hit only due to superfluous invalidations, but using a > > coherent configuration for a processor/system that doesn't support it may > > lead to a hard to debug hang with no apparent reason (as I wrote > > previously, even NMI/Reset stopped working on my system -- I had to hit > > the power switch). > > Using a non-coherent mode on IP27 may result in nice, hard to trackdown bus > errors. Weird, but I accept it as a fact. Still a bus error expresses more than a hang. ;-) -- + Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: macro@ds2.pg.gda.pl, PGP key available + From ralf@linux-mips.net Thu Aug 1 19:18:16 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id TAA19132; Thu, 1 Aug 2002 19:18:16 +0200 (MET DST) Received-Date: Thu, 1 Aug 2002 19:18:16 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd019128; Thu Aug 1 19:18:14 2002 Received: from dea.linux-mips.net (shaft16-f78.dialo.tiscali.de [62.246.16.78]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g71HICM19184 for ; Thu, 1 Aug 2002 19:18:12 +0200 (CEST) Received: (from ralf@localhost) by dea.linux-mips.net (8.11.6/8.11.6) id g71HI4I23624; Thu, 1 Aug 2002 19:18:04 +0200 Date: Thu, 1 Aug 2002 19:18:04 +0200 From: Ralf Baechle To: "Maciej W. Rozycki" Cc: Carsten Langgaard , linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [patch] MIPS64 R4k TLB refill CP0 hazards Message-ID: <20020801191804.D22824@dea.linux-mips.net> References: <20020731223158.A6394@dea.linux-mips.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: ; from macro@ds2.pg.gda.pl on Thu, Aug 01, 2002 at 05:24:43PM +0200 On Thu, Aug 01, 2002 at 05:24:43PM +0200, Maciej W. Rozycki wrote: > After looking at the generated assembly I discovered the handlers don't > fit in 128 bytes. They didn't crash since I have modules disabled for > now, so the vmalloc path didn't get hit and the user path happened to fit, > but it was pure luck. The path got hit before I fixed a bug in gas though > -- that's the explanation of the false cache error exceptions I used to > observe. Ouch. It was a known problem but we simply ignored it for a while as that handler just overwrites the cache error handler which normally should be used extremly rarely, if at all. The problem is somewhat itching by now as we're supporting the SB1 core which in it's revision one may throw spurious cache errors, so the handler is actually used ... Ralf From ralf@linux-mips.net Thu Aug 1 19:58:36 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id TAA23594; Thu, 1 Aug 2002 19:58:36 +0200 (MET DST) Received-Date: Thu, 1 Aug 2002 19:58:36 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd023592; Thu Aug 1 19:58:27 2002 Received: from dea.linux-mips.net (shaft16-f78.dialo.tiscali.de [62.246.16.78]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g71HwPM19526 for ; Thu, 1 Aug 2002 19:58:25 +0200 (CEST) Received: (from ralf@localhost) by dea.linux-mips.net (8.11.6/8.11.6) id g71HwKg24078; Thu, 1 Aug 2002 19:58:20 +0200 Date: Thu, 1 Aug 2002 19:58:20 +0200 From: Ralf Baechle To: "Maciej W. Rozycki" Cc: linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes Message-ID: <20020801195820.F22824@dea.linux-mips.net> References: <20020801184929.B22824@dea.linux-mips.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: ; from macro@ds2.pg.gda.pl on Thu, Aug 01, 2002 at 07:14:05PM +0200 On Thu, Aug 01, 2002 at 07:14:05PM +0200, Maciej W. Rozycki wrote: > > R10000. > > OK. Any specs anywhere? techpubs.sgi.com should have a somewhat older manual (must have!) and www.necel.com as well. The geniouses at NEC stripped the description of the cache instruction from their manual so it you really want both manuals. > > Back in time I prefered CONFIG_NONCOHERENT_IO over CONFIG_COHERENT_IO > > because the noncoherent case needs additional code and in general I'm > > trying to reduce the number of the #if !defined conditionals for easier > > readability. > > Hmm, what's wrong with "#ifndef"? Not much less readable than "#ifdef", > IMO. Just a small detail. Nest conditions several times and the spaghetti starts :-) > Basically: > > 1. Does the CPU support coherency? > > 2. If so, does the system? > > I'm going to express it this way in the config script. Have fun expressing if a R4000 variant supports coherency :-) You can't if you don't want to introduce even more R4000 types or subtypes. > Well, inferring from docs and my experience it's not needed. A system > may simply require areas used for DMA to be marked as non-coherent by > CPUs. Often uncached accesses are used to prevent spoiling the caches > anyway. None such MIPS system known where this is a sensible mode of operation - and I've hacked quite a number of platforms. Anyway, if there were such systems they'd either have to be considered as coherent or as non-coherent. Our current model doesn't permit any finer grained configuration and unless such a system actually exists I don't think we should introduce one. Btw, I've seen a fairly new system in which the non-coherent bits in some agent are not working at all - it's easier to implement that way ... > > Using a non-coherent mode on IP27 may result in nice, hard to trackdown bus > > errors. > > Weird, but I accept it as a fact. Still a bus error expresses more than > a hang. ;-) Not so weired. The system is still operating in coherent mode; that also means the directory caches still think they know where a particular memory address is cached. It's possible to operate an IP27 in non-coherent mode for I/O only by also reconfiguring the whole chipset but that turned out to be always slower and harder to get the software correct, so it's not used except for debugging with a logic analyzer; details are mindbogglingly complex. Ralf From macro@ds2.pg.gda.pl Thu Aug 1 20:48:57 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id UAA26603; Thu, 1 Aug 2002 20:48:57 +0200 (MET DST) Received-Date: Thu, 1 Aug 2002 20:48:57 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd026601; Thu Aug 1 20:48:51 2002 Received: from delta.ds2.pg.gda.pl (macro@delta.ds2.pg.gda.pl [213.192.72.1]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g71ImoM20282 for ; Thu, 1 Aug 2002 20:48:50 +0200 (CEST) Received: from localhost by delta.ds2.pg.gda.pl (8.9.3/8.9.3) with SMTP id UAA12348; Thu, 1 Aug 2002 20:49:18 +0200 (MET DST) Date: Thu, 1 Aug 2002 20:49:18 +0200 (MET DST) From: "Maciej W. Rozycki" To: Ralf Baechle cc: linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes In-Reply-To: <20020801195820.F22824@dea.linux-mips.net> Message-ID: Organization: Technical University of Gdansk MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII On Thu, 1 Aug 2002, Ralf Baechle wrote: > techpubs.sgi.com should have a somewhat older manual (must have!) and > www.necel.com as well. The geniouses at NEC stripped the description of > the cache instruction from their manual so it you really want both manuals. Thanks for the reference. > > Hmm, what's wrong with "#ifndef"? Not much less readable than "#ifdef", > > IMO. > > Just a small detail. Nest conditions several times and the spaghetti > starts :-) Well, that should be avoided whether the condition is positive or negative. Comments after "#else", "#elif" and "#end" might help a bit if a discipline is kept. > > Basically: > > > > 1. Does the CPU support coherency? > > > > 2. If so, does the system? > > > > I'm going to express it this way in the config script. > > Have fun expressing if a R4000 variant supports coherency :-) You can't > if you don't want to introduce even more R4000 types or subtypes. Who said I don't want to? ;-) Not a big deal at this stage. > None such MIPS system known where this is a sensible mode of operation - > and I've hacked quite a number of platforms. Anyway, if there were such > systems they'd either have to be considered as coherent or as non-coherent. Thanks for the clarification. > Our current model doesn't permit any finer grained configuration and unless > such a system actually exists I don't think we should introduce one. Sure, but a one point we'll have an option to select the model at the run time anyway. At least this is one of goals I'd like to see fulfilled in the future. -- + Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: macro@ds2.pg.gda.pl, PGP key available + From macro@ds2.pg.gda.pl Fri Aug 2 10:36:50 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id KAA10935; Fri, 2 Aug 2002 10:36:50 +0200 (MET DST) Received-Date: Fri, 2 Aug 2002 10:36:50 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd010933; Fri Aug 2 10:36:45 2002 Received: from delta.ds2.pg.gda.pl (macro@delta.ds2.pg.gda.pl [213.192.72.1]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g728aiM24584 for ; Fri, 2 Aug 2002 10:36:44 +0200 (CEST) Received: from localhost by delta.ds2.pg.gda.pl (8.9.3/8.9.3) with SMTP id KAA24499; Fri, 2 Aug 2002 10:36:58 +0200 (MET DST) Date: Fri, 2 Aug 2002 10:36:57 +0200 (MET DST) From: "Maciej W. Rozycki" To: Carsten Langgaard cc: Ralf Baechle , linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes In-Reply-To: <3D4A4191.DF5EFFC4@mips.com> Message-ID: Organization: Technical University of Gdansk MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII On Fri, 2 Aug 2002, Carsten Langgaard wrote: > The Malta board is a system that both run coherent and non-coherent, so I would > prefer, that we either make the coherency a configuration option or make it > possible to determine at run time. The latter, definitely. -- + Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: macro@ds2.pg.gda.pl, PGP key available + From ralf@linux-mips.net Fri Aug 2 12:16:20 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id MAA12388; Fri, 2 Aug 2002 12:16:20 +0200 (MET DST) Received-Date: Fri, 2 Aug 2002 12:16:20 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd012386; Fri Aug 2 12:16:16 2002 Received: from dea.linux-mips.net (shaft17-f126.dialo.tiscali.de [62.246.17.126]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g72AGDM25034 for ; Fri, 2 Aug 2002 12:16:14 +0200 (CEST) Received: (from ralf@localhost) by dea.linux-mips.net (8.11.6/8.11.6) id g72AG0H03001; Fri, 2 Aug 2002 12:16:00 +0200 Date: Fri, 2 Aug 2002 12:16:00 +0200 From: Ralf Baechle To: Thiemo Seufer Cc: "Maciej W. Rozycki" , linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes Message-ID: <20020802121600.A27824@dea.linux-mips.net> References: <20020801152500.A31808@dea.linux-mips.net> <20020801184929.B22824@dea.linux-mips.net> <20020801170649.GB15334@rembrandt.csv.ica.uni-stuttgart.de> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <20020801170649.GB15334@rembrandt.csv.ica.uni-stuttgart.de>; from ica2_ts@csv.ica.uni-stuttgart.de on Thu, Aug 01, 2002 at 07:06:49PM +0200 On Thu, Aug 01, 2002 at 07:06:49PM +0200, Thiemo Seufer wrote: > [snip] > > Back in time I prefered CONFIG_NONCOHERENT_IO over CONFIG_COHERENT_IO > > because the noncoherent case needs additional code and in general I'm > > trying to reduce the number of the #if !defined conditionals for easier > > readability. > > > > The R10000 is our standard example why looking at the processor type doesn't > > work. It's used in coherent mode in IP27 but in coherent mode but in > > coherent mode in IP28 or IP32. Otoh I don't know of any system that > > JFTR: non-coherent mode in IP28 or IP32. Yep, you're right. I hope everybody concluded that from my non-coherent sentence ;-) From ralf@linux-mips.net Fri Aug 2 12:38:20 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id MAA13491; Fri, 2 Aug 2002 12:38:20 +0200 (MET DST) Received-Date: Fri, 2 Aug 2002 12:38:20 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd013489; Fri Aug 2 12:38:15 2002 Received: from dea.linux-mips.net (shaft17-f126.dialo.tiscali.de [62.246.17.126]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g72AcDM25307 for ; Fri, 2 Aug 2002 12:38:14 +0200 (CEST) Received: (from ralf@localhost) by dea.linux-mips.net (8.11.6/8.11.6) id g72Ac5a03412; Fri, 2 Aug 2002 12:38:05 +0200 Date: Fri, 2 Aug 2002 12:38:05 +0200 From: Ralf Baechle To: Carsten Langgaard Cc: "Maciej W. Rozycki" , linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [update] [patch] linux: Cache coherency fixes Message-ID: <20020802123805.A3329@dea.linux-mips.net> References: <20020801152500.A31808@dea.linux-mips.net> <20020801184929.B22824@dea.linux-mips.net> <3D4A4191.DF5EFFC4@mips.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <3D4A4191.DF5EFFC4@mips.com>; from carstenl@mips.com on Fri, Aug 02, 2002 at 10:23:53AM +0200 On Fri, Aug 02, 2002 at 10:23:53AM +0200, Carsten Langgaard wrote: > The Malta board is a system that both run coherent and non-coherent, so I > would prefer, that we either make the coherency a configuration option or > make it possible to determine at run time. Definately the latter then. The price to pay is small and I don't think forcing the use to change the kernel when just changing CPU modules is acceptable. Ralf From ralf@linux-mips.net Fri Aug 2 13:06:10 2002 Received: (uucp@localhost) by guadalquivir.fnet.fr (8.8.8/97.02.12/Guadalquivir); id NAA14735; Fri, 2 Aug 2002 13:06:10 +0200 (MET DST) Received-Date: Fri, 2 Aug 2002 13:06:10 +0200 (MET DST) Received: from indus.fnet.fr(192.134.188.133) via SMTP by guadalquivir.fnet.fr, id smtpd014733; Fri Aug 2 13:06:06 2002 Received: from dea.linux-mips.net (shaft17-f74.dialo.tiscali.de [62.246.17.74]) by indus.fnet.fr (8.11.3nb1/8.11.3/$RCSfile$) with ESMTP id g72B64M25637 for ; Fri, 2 Aug 2002 13:06:04 +0200 (CEST) Received: (from ralf@localhost) by dea.linux-mips.net (8.11.6/8.11.6) id g72B5qj03625; Fri, 2 Aug 2002 13:05:52 +0200 Date: Fri, 2 Aug 2002 13:05:52 +0200 From: Ralf Baechle To: Carsten Langgaard Cc: "Maciej W. Rozycki" , linux-mips@fnet.fr, linux-mips@oss.sgi.com Subject: Re: [patch] MIPS64 R4k TLB refill CP0 hazards Message-ID: <20020802130552.C27824@dea.linux-mips.net> References: <20020731223158.A6394@dea.linux-mips.net> <20020801191804.D22824@dea.linux-mips.net> <3D4A519B.3EF5459@mips.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.2.5.1i In-Reply-To: <3D4A519B.3EF5459@mips.com>; from carstenl@mips.com on Fri, Aug 02, 2002 at 11:32:18AM +0200 On Fri, Aug 02, 2002 at 11:32:18AM +0200, Carsten Langgaard wrote: > > > After looking at the generated assembly I discovered the handlers don't > > > fit in 128 bytes. They didn't crash since I have modules disabled for > > > now, so the vmalloc path didn't get hit and the user path happened to fit, > > > but it was pure luck. The path got hit before I fixed a bug in gas though > > > -- that's the explanation of the false cache error exceptions I used to > > > observe. > > > > Ouch. It was a known problem but we simply ignored it for a while as that > > handler just overwrites the cache error handler which normally should be > > used extremly rarely, if at all. The problem is somewhat itching by now > > as we're supporting the SB1 core which in it's revision one may throw > > spurious cache errors, so the handler is actually used ... > > > > Maybe it's time for some intelligent check for the size of these exception > routine. Easy trick at compile time which will just inflate the object code a little bit: .align 5 LEAF(except_vec1_r4k) [...] END(except_vec1_r4k) .org except_vec1_r4k + 0x80 This will result in an assembler error if the body of the except_vec1_r4k function is bigger than 0x80. Ralf