From linux-mm-www-bounce@nl.linux.org Wed Dec 13 17:56:24 2006 Received: from localhost ([127.0.0.1] helo=humbolt) by humbolt.nl.linux.org with esmtp (Exim 4.22) id 1GuXOx-00053Y-Ib; Wed, 13 Dec 2006 17:56:15 +0100 Received: with ECARTIS (v1.0.0; list linux-mm-www); Wed, 13 Dec 2006 17:56:12 +0100 (CET) Received: from bay0-omc1-s5.bay0.hotmail.com ([65.54.246.77]) by humbolt.nl.linux.org with esmtp (Exim 4.22) id 1GuXOX-0004wf-RK for linux-mm-www@nl.linux.org; Wed, 13 Dec 2006 17:55:50 +0100 Received: from hotmail.com ([64.4.19.46]) by bay0-omc1-s5.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2668); Wed, 13 Dec 2006 08:41:39 -0800 Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Wed, 13 Dec 2006 08:41:39 -0800 Message-ID: Received: from 64.4.19.200 by by109fd.bay109.hotmail.msn.com with HTTP; Wed, 13 Dec 2006 16:41:37 GMT X-Originating-IP: [149.117.164.28] X-Originating-Email: [timberdogsw2@hotmail.com] X-Sender: timberdogsw2@hotmail.com From: "Chip Kerchner" To: linux-mm-www@nl.linux.org Bcc: Subject: Problems with hugepage read speed Date: Wed, 13 Dec 2006 08:41:37 -0800 Mime-Version: 1.0 Content-Type: text/plain; format=flowed X-OriginalArrivalTime: 13 Dec 2006 16:41:39.0980 (UTC) FILETIME=[9037ECC0:01C71ED5] Received-SPF: X-ecartis-version: Ecartis v1.0.0 Sender: linux-mm-www-bounce@nl.linux.org Errors-to: linux-mm-www-bounce@nl.linux.org X-original-sender: timberdogsw2@hotmail.com Precedence: bulk Reply-to: linux-mm-bugs@nl.linux.org List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: X-List-ID: List-subscribe: List-owner: List-post: List-archive: X-list: linux-mm-www I recently added hugepage support to my program and am seeing a strange behavior. When I write data to a hugepage, I am seeing about a 100% improvement versus normal pages. But when I read data from a hugepage, I am seeing anywhere from 15-35% decrease in speed versus normal pages. I was wondering if anoyone can help me resolve this problem or at least point me to someone that can? I am running my tests on a Woodcrest machine (Core Duo). I have setup my machine to have 1024 hugepage pages, each of which is 2MB. Here is a listing my code for reference: #define HUGE_PAGE_SIZE (2048 * 1024) void **memPtr; size_t size; int shmid; memPtr = NULL; /* Handle hugepage size requests */ size = (size + HUGE_PAGE_SIZE - 1) & -HUGE_PAGE_SIZE; if ((shmid = shmget(IPC_PRIVATE, size, (SHM_HUGETLB | IPC_CREAT | SHM_R | SHM_W))) != -1) { memPtr = shmat(shmid, (void *)SHM_ADDR, SHM_RND); if (memPtr == (void **)(-1)) { memPtr = NULL; } shmctl(shmid, IPC_RMID, NULL); } if (0 != errno) { // printf("errno %d %ld\n", errno, size); // shmdt((const void *)memPtr); errno = 0; memPtr = NULL; } #if 1 if (NULL != memPtr) { printf("SUCCESS!!! %p\n", memPtr); { ticks p3ticks; char *ptr; void **newPtr, **readPtr; int i; p3ticks = getticks(); memset(memPtr, 0, size); p3ticks = getticks() - p3ticks; printf("Huge write ticks %12llu size = %9ld\n", p3ticks, size); newPtr = memalign(align, size); p3ticks = getticks(); memset(newPtr, 0, size); p3ticks = getticks() - p3ticks; printf("Normal write ticks %12llu size = %9ld\n", p3ticks, size); free(newPtr); readPtr = malloc(size); ptr = (char *)readPtr; for (i = 0; i < size; i++) { *ptr++ = (char)(rand() & 0xff); } p3ticks = getticks(); memcpy(readPtr, memPtr, size); p3ticks = getticks() - p3ticks; printf("Huge read ticks %12llu size = %9ld\n", p3ticks, size); free(readPtr); newPtr = memalign(align, size); readPtr = malloc(size); ptr = (char *)readPtr; for (i = 0; i < size; i++) { *ptr++ = (char)(rand() & 0xff); } p3ticks = getticks(); memcpy(readPtr, newPtr, size); p3ticks = getticks() - p3ticks; printf("Normal read ticks %12llu size = %9ld\n", p3ticks, size); free(newPtr); free(readPtr); } } #endif } if (NULL == memPtr) { memPtr = memalign(align, size); } Here is the output from my program: SUCCESS!!! 0x2a9a000000 Huge write ticks 6513570 size = 6291456 Normal write ticks 12884967 size = 6291456 Huge read ticks 11098737 size = 6291456 Normal read ticks 8490375 size = 6291456 SUCCESS!!! 0x2ac8800000 Huge write ticks 47046024 size = 35651584 Normal write ticks 73629720 size = 35651584 Huge read ticks 71786745 size = 35651584 Normal read ticks 61150383 size = 35651584 uname -a Linux woodcrest 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 2006 x86_64 x86_64 x86_64 GNU/Linux grep -i huge /proc/meminfo 23:HugePages_Total: 1024 24:HugePages_Free: 1024 25:Hugepagesize: 2048 kB Chip Kerchner _________________________________________________________________ WIN up to $10,000 in cash or prizes – enter the Microsoft Office Live Sweepstakes http://clk.atdmt.com/MRT/go/aub0050001581mrt/direct/01/ -- Linux-mm-www: http://linux-mm.org/ website maintenance list Archive: http://mail.nl.linux.org/linux-mm-www/ Development: linux-mm@kvack.org From linux-mm-www-bounce@nl.linux.org Wed Dec 13 22:19:46 2006 Received: from localhost ([127.0.0.1] helo=humbolt) by humbolt.nl.linux.org with esmtp (Exim 4.22) id 1GubVr-0006XW-2o; Wed, 13 Dec 2006 22:19:39 +0100 Received: with ECARTIS (v1.0.0; list linux-mm-www); Wed, 13 Dec 2006 22:19:35 +0100 (CET) Received: from bay0-omc2-s41.bay0.hotmail.com ([65.54.246.177]) by humbolt.nl.linux.org with esmtp (Exim 4.22) id 1GubVK-0006MS-FG for linux-mm-www@nl.linux.org; Wed, 13 Dec 2006 22:19:06 +0100 Received: from hotmail.com ([64.4.19.44]) by bay0-omc2-s41.bay0.hotmail.com with Microsoft SMTPSVC(6.0.3790.2668); Wed, 13 Dec 2006 13:12:10 -0800 Received: from mail pickup service by hotmail.com with Microsoft SMTPSVC; Wed, 13 Dec 2006 13:12:09 -0800 Message-ID: Received: from 64.4.19.200 by by109fd.bay109.hotmail.msn.com with HTTP; Wed, 13 Dec 2006 21:12:08 GMT X-Originating-IP: [149.117.164.28] X-Originating-Email: [timberdogsw2@hotmail.com] X-Sender: timberdogsw2@hotmail.com In-Reply-To: From: "Chip Kerchner" To: linux-mm-www@nl.linux.org Bcc: Subject: RE: Problems with hugepage read speed Date: Wed, 13 Dec 2006 13:12:08 -0800 Mime-Version: 1.0 Content-Type: text/plain; format=flowed X-OriginalArrivalTime: 13 Dec 2006 21:12:09.0903 (UTC) FILETIME=[5A01A3F0:01C71EFB] Received-SPF: X-ecartis-version: Ecartis v1.0.0 Sender: linux-mm-www-bounce@nl.linux.org Errors-to: linux-mm-www-bounce@nl.linux.org X-original-sender: timberdogsw2@hotmail.com Precedence: bulk Reply-to: linux-mm-bugs@nl.linux.org List-help: List-unsubscribe: List-software: Ecartis version 1.0.0 List-Id: X-List-ID: List-subscribe: List-owner: List-post: List-archive: X-list: linux-mm-www >One additional notable item is that I tried turning of hugepages and >instead used pure share memory and saw both a slowdown in reads and writes >(probably expected). This would imply that there is a bug/problem with the >shared memory approach of hugepages. Any ideas? > >// if ((shmid = shmget(IPC_PRIVATE, size, (SHM_HUGETLB | IPC_CREAT >| SHM_R | SHM_W))) != -1) { > if ((shmid = shmget(IPC_PRIVATE, size, (IPC_CREAT | SHM_R | >SHM_W))) != -1) { > > >SUCCESS!!! 0x2a9b52b000 >Huge write ticks 15018903 size = 6291456 >Normal write ticks 13445280 size = 6291456 >Huge read ticks 11167425 size = 6291456 >Normal read ticks 8458803 size = 6291456 > > >Chip Kerchner > > >>I recently added hugepage support to my program and am seeing a strange >>behavior. When I write data to a hugepage, I am seeing about a 100% >>improvement versus normal pages. But when I read data from a hugepage, I >>am seeing anywhere from 15-35% decrease in speed versus normal pages. I >>was wondering if anoyone can help me resolve this problem or at least >>point me to someone that can? >> >>I am running my tests on a Woodcrest machine (Core Duo). I have setup my >>machine to have 1024 hugepage pages, each of which is 2MB. >> >>Here is a listing my code for reference: >> >>#define HUGE_PAGE_SIZE (2048 * 1024) >> >> void **memPtr; >> size_t size; >> int shmid; >> >> memPtr = NULL; >> >> /* Handle hugepage size requests */ >> >> size = (size + HUGE_PAGE_SIZE - 1) & -HUGE_PAGE_SIZE; >> if ((shmid = shmget(IPC_PRIVATE, size, (SHM_HUGETLB | IPC_CREAT | >>SHM_R | SHM_W))) != -1) { >> memPtr = shmat(shmid, (void *)SHM_ADDR, SHM_RND); >> if (memPtr == (void **)(-1)) { >> memPtr = NULL; >> } >> >> shmctl(shmid, IPC_RMID, NULL); >> } >> >> if (0 != errno) { >>// printf("errno %d %ld\n", errno, size); >>// shmdt((const void *)memPtr); >> errno = 0; >> memPtr = NULL; >> } >> >>#if 1 >> if (NULL != memPtr) { >> printf("SUCCESS!!! %p\n", memPtr); >> { >> ticks p3ticks; >> char *ptr; >> void **newPtr, **readPtr; >> int i; >> >> p3ticks = getticks(); >> memset(memPtr, 0, size); >> p3ticks = getticks() - p3ticks; >> printf("Huge write ticks %12llu size = %9ld\n", p3ticks, >>size); >> newPtr = memalign(align, size); >> p3ticks = getticks(); >> memset(newPtr, 0, size); >> p3ticks = getticks() - p3ticks; >> printf("Normal write ticks %12llu size = %9ld\n", p3ticks, >>size); >> free(newPtr); >> readPtr = malloc(size); >> ptr = (char *)readPtr; >> for (i = 0; i < size; i++) { >> *ptr++ = (char)(rand() & 0xff); >> } >> p3ticks = getticks(); >> memcpy(readPtr, memPtr, size); >> p3ticks = getticks() - p3ticks; >> printf("Huge read ticks %12llu size = %9ld\n", p3ticks, >>size); >> free(readPtr); >> newPtr = memalign(align, size); >> readPtr = malloc(size); >> ptr = (char *)readPtr; >> for (i = 0; i < size; i++) { >> *ptr++ = (char)(rand() & 0xff); >> } >> p3ticks = getticks(); >> memcpy(readPtr, newPtr, size); >> p3ticks = getticks() - p3ticks; >> printf("Normal read ticks %12llu size = %9ld\n", p3ticks, >>size); >> free(newPtr); >> free(readPtr); >> } >> } >>#endif >> } >> >> if (NULL == memPtr) { >> memPtr = memalign(align, size); >> } >> >>Here is the output from my program: >> >>SUCCESS!!! 0x2a9a000000 >>Huge write ticks 6513570 size = 6291456 >>Normal write ticks 12884967 size = 6291456 >>Huge read ticks 11098737 size = 6291456 >>Normal read ticks 8490375 size = 6291456 >>SUCCESS!!! 0x2ac8800000 >>Huge write ticks 47046024 size = 35651584 >>Normal write ticks 73629720 size = 35651584 >>Huge read ticks 71786745 size = 35651584 >>Normal read ticks 61150383 size = 35651584 >> >>uname -a >>Linux woodcrest 2.6.9-34.ELsmp #1 SMP Fri Feb 24 16:56:28 EST 2006 x86_64 >>x86_64 x86_64 GNU/Linux >> >>grep -i huge /proc/meminfo >>23:HugePages_Total: 1024 >>24:HugePages_Free: 1024 >>25:Hugepagesize: 2048 kB >> >>Chip Kerchner >> >>_________________________________________________________________ >>WIN up to $10,000 in cash or prizes – enter the Microsoft Office Live >>Sweepstakes http://clk.atdmt.com/MRT/go/aub0050001581mrt/direct/01/ >> >> >>-- >>Linux-mm-www: http://linux-mm.org/ website maintenance list >>Archive: http://mail.nl.linux.org/linux-mm-www/ >>Development: linux-mm@kvack.org >> > >_________________________________________________________________ >All-in-one security and maintenance for your PC. Get a free 90-day trial! >http://clk.atdmt.com/MSN/go/msnnkwlo0050000002msn/direct/01/?href=http://clk.atdmt.com/MSN/go/msnnkwlo0050000001msn/direct/01/?href=http://www.windowsonecare.com/?sc_cid=msn_hotmail > > _________________________________________________________________ Get free, personalized commercial-free online radio with MSN Radio powered by Pandora http://radio.msn.com/?icid=T002MSN03A07001 -- Linux-mm-www: http://linux-mm.org/ website maintenance list Archive: http://mail.nl.linux.org/linux-mm-www/ Development: linux-mm@kvack.org