<<  >> (p.23)
    Author Topic: Mining rig extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS]  (Read 169565 times)

    • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 07, 2012, 09:41:09 PM
     #441
    Quote from: DBordello on May 07, 2012, 09:37:52 PM
    Have you enabled IOMMU in the BIOS?  It is my understanding that is necessary for PCIe pass-thru.
    IOMMU is for AMD; I have an Intel platform so I have VT-d. And yes, it is enabled. PVE kernel panics when I try to hotplug the devices to the virtual machine, and the virtual machine won't boot when the devices are added to the config file when it isn't booted.
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 07, 2012, 09:50:31 PM
     #442
    Quote from: irritant on May 07, 2012, 09:47:45 PM
    have you tried ESX(i)? i think it may be a little bit more user friendly, vCenter management software, just install it to an USB stick, and you should be able to create VMS (if you have some storage repository), after you have done that, you can (try to) add devices with the configuration manager (vcenter), you can do that from the start but i always check if it is working the normal way...   

    why did you choose PVE? you feel more comfortable with it?
    It was a recommendation, because it is based on a pre-setup KVM kernel. I heard ESXi has a limitation of 4 devices though?
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • DiabloD3
    Legendary
    *
    Offline Offline

    Activity: 1162
    Merit: 1000


    DiabloMiner author


    View Profile WWW
    May 07, 2012, 10:05:52 PM
     #443
    Quote from: rjk on May 07, 2012, 09:41:09 PM
    Quote from: DBordello on May 07, 2012, 09:37:52 PM
    Have you enabled IOMMU in the BIOS?  It is my understanding that is necessary for PCIe pass-thru.
    IOMMU is for AMD; I have an Intel platform so I have VT-d. And yes, it is enabled. PVE kernel panics when I try to hotplug the devices to the virtual machine, and the virtual machine won't boot when the devices are added to the config file when it isn't booted.

    No. IOMMU is a industry term for it. VT-d is an implementation of IOMMU, AMD, Sun, and IBM also offer implementations of it.
    DiabloMiner

  • DiabloD3
    Legendary
    *
    Offline Offline

    Activity: 1162
    Merit: 1000


    DiabloMiner author


    View Profile WWW
    May 07, 2012, 10:19:15 PM
     #444
    Quote from: irritant on May 07, 2012, 10:10:37 PM
    http://en.wikipedia.org/wiki/IOMMU#IOMMU_in_relation_to_virtualization    Wink

    AMD has published a specification for IOMMU technology in the HyperTransport architecture.[1] Intel has published a specification for IOMMU technology as Virtualization Technology for Directed I/O, abbreviated VT-d.[2] Information about the Sun IOMMU has been published in the Device Virtual Memory Access (DVMA) section of the Solaris Developer Connection.[3] The IBM Translation Control Entry (TCE) has been described in a document entitled Logical Partition Security in the IBM eServer pSeries 690.[4] The PCI-SIG has relevant work under the terms I/O Virtualization (IOV)[5] and Address Translation Services (ATS).

    So, are we done playing wikipedia lawyer?
    DiabloMiner

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 07, 2012, 11:15:21 PM
     #445
    Quote from: jcpham on May 07, 2012, 11:10:09 PM
    Anything new to report? If you get a kernel panic on a linux guest during boot or just can't get a linux guest to boot, I'd be interested in knowing if a similarly configured windows VM does the same.I think we had figured out about the device id's for your devices last.
    It was the host that panicked, not the guest. The output is on the previous page.

    Maybe ill try a windoz guest and see what happens.
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 08, 2012, 01:28:47 AM
     #446
    I'm on my phone so it makes it a little hard to elaborate on all the points, but I'll try to remember all of them.

    Yes there is onboard video, it doesn't show in that list because - grepped for "Cypress XT" not "VGA". When I grepped for VGA, I got several screens of output and didn't sift through it thoroughly.

    I have tried hotplugging and coldplugging different devices. Yes, I would assume that the host is trying to use one of the cards and that could cause a conflict. I just had a thought - would it be possible that the host is glomming on all the cards just because they are there? I should change my BIOS to boot the internal video first and use it as primary display.

    When hotplugging, the host (dom0) was what panicked. When coldplugging, the guest (domU) either flashed the VNC terminal on and off repeatedly (when a single device was configured) or showed a black screen and froze (no networking or anything) when all cards were assigned.

    Unfortunately development is kind of on hold because of time constraints (weekender) and because these fans at 100% are damn annoying.

    I need to get the following list of stuff done:
    Custom bracket - need to find a small fabricator near me
    Copper bus bar - determine correct size, layout, and method of attaching conductors
    PSU - figure out how to turn it on
    PCIe 6/8 pin connectors - obtain a bunch and figure out how to hook them to the bus bars (ring/spade terminals? solder? clamp?)
    Air channel/duct - needed so that the air doesn't escape past the heatsinks over the top - also a job for the fabricator - needs to be slotted so the upper level PCIe extenders can pass thru
    Fan contoroller - wire it in once it arrives and set up the control software (to be written, coder is lined up for the job)

    Damn, I wish it was as easy as throwing some money at it and putting it together like Legos. Tongue Problem is, I'm kinda weak when it comes to the software and scripting/programming side of things. Once it's done, does anyone wanna buy it and have the distinction of owning the worlds fastest, most complicated GPU based rig? Grin It will need a 7KW power outlet and cooling. Grin
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 08, 2012, 01:42:59 AM
     #447
    Quote from: irritant on May 08, 2012, 01:37:16 AM
    one quick note, after what you said i remember you have to unbind the pci device first (logical right), have you done that? i cant recall having read it(but that doesnt say too much)

    4. unbind device from host kernel driver (example PCI device 01:00.0)
     Load the PCI Stub Driver if it is compiled as a module
           modprobe pci_stub
     lspci -n
    locate the entry for device 01:00.0 and note down the vendor & device ID 8086:10b9
            ...
           01:00.0 0200: 8086:10b9 (rev 06)
           ...
     echo "8086 10b9" >sys/bus/pci/drivers/pci-stub/new_id
    echo 0000:01:00.0 >sys/bus/pci/devices/0000:01:00.0/driver/unbind
    echo 0000:01:00.0 >sys/bus/pci/drivers/pci-stub/bind

    http://www.linux-kvm.org/page/How_to_assign_devices_with_VT-d_in_KVM


    in xen it was something with pciback

    http://wiki.xensource.com/xenwiki/Assign_hardware_to_DomU_with_PCIBack_as_module

    really not sure though..



    Having to unbind it from the host first makes perfect sense, I didn't really think of that. I'll have to try those steps and see how I get on.
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 08, 2012, 10:19:58 PM
     #448
    Quote from: rjk on May 08, 2012, 01:42:59 AM
    Having to unbind it from the host first makes perfect sense, I didn't really think of that. I'll have to try those steps and see how I get on.
    Played with it a little more.
    Unbinding and rebinding to PCI_stub seemed to work (Is this persistent? If not, how do I make it save these settings between reboots?), and I was able to attach a device to a VM. However, X segfaulted with error 11. I was able to lspci and see the video card from inside the VM though. PVE doesn't seem to like having more than 2 PCI devices per VM, and if you add more to the VM config, it simply removes them for you.

    I tried installing a new VM with the cards pre-attached, and the bootdisk just showed a blank screen and went no further. I tried installing Windows 7 x64, but it kept asking for a driver for the CDROM which I didn't have or know where to get.
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 08, 2012, 10:55:01 PM
     #449
    What's the smallest Linux install that I can have that will mine at full speed, and how little RAM can I assign to it? If I have to have 2 devices per VM, that would mean 17 VMs in a dual-GPU situation, which would suck. And, since I have only 8GB of RAM, I will have to have fewer than 512MB RAM per VM. I don't know how much RAM PVE uses for Dom0, but if I assume 1GB (like XenServer), that leaves me with 7GB divided by 17 VMs, equals ~420MB RAM per VM. I suppose that should be enough for something that is stripped down, but I doubt it would run BAMT with all its monitoring and stuff.
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • Gomeler
    Hero Member
    *****
    Offline Offline

    Activity: 697
    Merit: 500



    View Profile
    May 09, 2012, 01:06:38 AM
     #450
    Quote from: irritant on May 08, 2012, 11:03:11 PM
    LinuxCoin maybe? although it doesn't say how much RAM you need

    http://www.linuxcoin.co.uk/wiki/index.php/Headless_Linuxcoin

    Quote
    The disk image contains the AMD APP SDK, ATI Catalyst & Drivers, poclbm & phoenix miner, Bitcoin, and more.

    the Xeon E3-1225  seems to have just 4 cores & 4 threads .. and i think the host wants one , so maximum 3 virtual machines possible, each getting a thread?

    You can overprovision the CPU on a VM host. So long as you don't peg the CPU cores on all your VMs, the hypervisor will load balance your CPUs to provide enough CPU power for your VMs. The major obstacle is always ram and storage IOPs. Luckily ram is dirt cheap.

  • Gomeler
    Hero Member
    *****
    Offline Offline

    Activity: 697
    Merit: 500



    View Profile
    May 09, 2012, 07:27:41 AM
     #451
    If ram becomes an issue I'd suggest just trying 8 GB dimms. It may work, I've seen more bizarre things than that.

  • terrytibbs
    Hero Member
    *****
    Offline Offline

    Activity: 560
    Merit: 501



    View Profile
    May 09, 2012, 02:37:29 PM
     #452
    Quote from: jcpham on May 09, 2012, 02:32:18 PM
    luke-jr is virtualizing gpu's with kvm. I'll try to dig up that info or have a conversation with him or see how he likes this thread.
    Just don't mention religion.

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 09, 2012, 02:37:54 PM
    Last edit: May 09, 2012, 02:51:27 PM by rjk
     #453
    Quote from: jcpham on May 09, 2012, 02:32:18 PM
    luke-jr is virtualizing gpu's with kvm. I'll try to dig up that info or have a conversation with him or see how he likes this thread.
    You know, I did discuss it with him months ago, but that was only the most basic bare minimum of information, because I didn't have compatible hardware. I'll need to grep my IRC logs to see if he gave out any technical information. All I remember was that it was 1 GPU and that it kept crashing.

    EDIT: I found some stuff, and I know there is more, but I can't be bothered to find it all right now.
    Code:
    Dec 26 19:54:12 <rjk2> gmaxwell: i thought there was a driver limit of 8 gpus maximum
    Dec 26 19:54:28 <gmaxwell> Driver. Yup.
    Dec 26 19:54:35 <gmaxwell> So run multiple copies of the driver. :)
    Dec 26 19:54:55 <gmaxwell> (according to luke the ati drivers work in kvm with the pci remapping stuff)
    Dec 26 19:55:35 <gmaxwell> so in _theory_ you could start two VMs, map 4 cards to each and run 16 GPUs on a system.
    Dec 26 19:55:45 <gmaxwell> (presuming you could plug in that many)
    Dec 26 19:55:58 <gmaxwell> (and presuming your motherboard doesn't catch fire)
    Dec 26 19:57:38 <rjk2> I have been trying to find someone that has actually been able to mine successfully in a virtualized environment
    Dec 26 19:57:48 <gmaxwell> luke-jr:
    Dec 26 19:57:52 <rjk2> i know it ought to work
    Dec 26 19:58:16 <luke-jr> gmaxwell: only with motherboards/CPUs that support it
    Dec 26 19:58:44 <rjk2> vt-d, and iommu right
    Dec 26 19:58:49 <luke-jr> VT-d in my case
    Dec 26 19:58:54 <rjk2> but does it work well?
    Dec 26 19:58:55 <rjk2> stable?
    Dec 26 19:58:58 <luke-jr> also, Intel pulled a big scam earlier this year
    Dec 26 19:59:02 <rjk2> because i want to try it
    Dec 26 19:59:09 <luke-jr> well, my Radeon still manages to crash my host system sometimes
    Dec 26 19:59:17 <rjk2> hmm
    Dec 26 19:59:25 <gmaxwell> rjk2: you get diminishing returns packing more cards onto one system. ::shrugs::
    Dec 26 19:59:26 <luke-jr> usually when I try to reinitialize it
    Dec 26 19:59:30 <luke-jr> which is mostly playing around
    Dec 26 19:59:38 <luke-jr> if I knew a proper way to do that, it might be OK
    Dec 26 20:00:09 <rjk2> what i wanted to do was use intel vt-d with  built-in video powering my desktop environment, and discrete cards mining
    Dec 26 20:00:23 <luke-jr> rjk2: tht's what I do
    Dec 26 20:00:24 <rjk2> z68 based
    Dec 26 20:00:46 <rjk2> luke-jr: sounds good, using kvm or xen?
    Dec 26 20:01:21 <luke-jr> KVM
    Dec 26 20:01:47 <luke-jr> Z68 = no VT-d
    Dec 26 20:01:53 <rjk2> wait what
    Dec 26 20:02:05 * catalase_ is now known as catalase
    Dec 26 20:02:08 <rjk2> i know the K procs with overclocking don't have it
    Dec 26 20:02:17 <rjk2> but z68 doesn't either?
    Dec 26 20:02:20 <luke-jr> nope
    Dec 26 20:02:26 <luke-jr> http://ark.intel.com/compare/52816,52812
    Dec 26 20:02:31 <rjk2> damn it
    Dec 26 20:02:37 <luke-jr> return it :P
    Dec 26 20:03:13 <rjk2> is that what "embedded options available" means? VT-d?
    Dec 26 20:03:19 <luke-jr> no
    Dec 26 20:03:26 <luke-jr> Intel® Virtualization Technology for Directed I/O (VT-d)
    Dec 26 20:03:37 <luke-jr> it's exclusive to Q67
    Dec 26 20:03:38 <rjk2> oh, i see its down lower
    Dec 26 20:03:41 <luke-jr> at least for Sandy Bridge
    Dec 26 20:03:45 <rjk2> hmm
    Dec 26 20:04:01 <luke-jr> originally it wasn't
    Dec 26 20:04:03 <rjk2> man that is lame
    Dec 26 20:04:08 <luke-jr> but Intel bugged, and decided the solution was to fix their specs
    Dec 26 20:04:20 <luke-jr> ie, they said the bug was the advertising VT-d on the others
    Dec 26 20:04:31 <gmaxwell> doh! that sucks.
    Dec 26 20:04:34 <luke-jr> yep
    Dec 26 20:04:38 <luke-jr> but they also bugged the SATA
    Dec 26 20:04:46 <luke-jr> so I returned my H67 and got a Q67
    Dec 26 20:04:48 <luke-jr> :D
    Dec 26 20:04:53 <rjk2> lol too true >.>
    Dec 26 20:04:58 <luke-jr> though it was VERY difficult to get Q67 at the time
    Dec 26 20:07:34 <rjk2> luke-jr: have you had the opportunity to test IOMMU (AMD) systems in the same manner yet?
    Dec 26 20:08:21 <luke-jr> no
    Dec 26 20:08:36 <luke-jr> I think someone did though, and found they had to do 1 VM per card
    Dec 26 20:08:42 <rjk2> lolwut
    Dec 26 20:08:44 <luke-jr> I've only tested 1 on VT-d
    Dec 26 20:09:18 <luke-jr> the annoying thing is, if the card crashes, I pretty much have to reboot
    Dec 26 20:09:33 <luke-jr> in theory, I *should* be able to 'remove' the card and power cycle just it
    Dec 26 20:09:39 <luke-jr> but I haven't figured out how in practice
    Dec 26 20:09:51 <luke-jr> even if I remove and rescan, it doesn't get reset
    Dec 26 20:10:09 <luke-jr> upgrading fglrx *might* help
    Dec 26 20:10:28 <luke-jr> since newer versions support hot-switching between IGP and discrete on laptops
    Dec 26 20:10:58 <luke-jr> (the error I was getting found only Google results talking about that)
    Dec 26 20:11:37 <luke-jr> latest fglrx *seems* to be working just as well as the good old 2.1 now tho
    Dec 26 20:12:19 <luke-jr> ie, no CPU hogging, and full 308 MH/s
    Dec 26 20:12:48 <luke-jr> I don't have a sane way to reproduce the card crashing though, so no idea how many months until I can confirm it reinitializes
    Dec 26 20:12:48 <rjk2> is that with one card?
    Dec 26 20:12:51 <luke-jr> yes
    Dec 26 20:13:04 <rjk2> i built myself a mini-itx system, jsut to do that
    Dec 26 20:13:04 <luke-jr> also, OpenCL is a memory hog :/
    Dec 26 20:13:37 <luke-jr> rjk2: note that PCI bus, and probably PCI-E extenders, will be all-or-nothing to VMs
    Dec 26 20:13:49 <luke-jr> ie, you might be unable to isolate single cards in separate VMs

    And, the issue requiring a restart that Luke-jr is discussing was worked around later in a different chat (put the machine to sleep, and bring it back out, instead of rebooting).
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • ice_chill
    Sr. Member
    ****
    Offline Offline

    Activity: 336
    Merit: 250


    View Profile
    May 18, 2012, 11:12:04 PM
     #454
    Any update ? This was the most interesting mining non-FPGA thread.

    Is the Rig all running now ?

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 19, 2012, 01:20:05 AM
     #455
    Quote from: ice_chill on May 18, 2012, 11:12:04 PM
    Any update ? This was the most interesting mining non-FPGA thread.

    Is the Rig all running now ?
    Not really, I just got the fan controller but haven't had a chance to install it. I'd like to do it tomorrow, but that's the day of the state auction, and I sure as heck am not going to miss that. http://www.das.ohio.gov/Divisions/GeneralServices/Surplus/WarehouseNextAuction.aspx Grin

    I also only have 13 cards, so I need to get some more. I guess I'm kind of waffling around, hoping that a pile of 7990s drop in my lap sometime soon, although that is rather unlikely lol.
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!

  • Garr255
    Legendary
    *
    Offline Offline

    Activity: 938
    Merit: 1000


    What's a GPU?


    View Profile
    May 19, 2012, 02:22:23 AM
     #456
    Ahh I really wish I could go to that. Piles of computers!!
    “First they ignore you, then they laugh at you, then they fight you, then you win.”  -- Mahatma Gandhi

    Average time between signing on to bitcointalk: Two weeks. Please don't expect responses any faster than that!

  • 1l1l11ll1l
    Legendary
    *
    Offline Offline

    Activity: 1274
    Merit: 1000


    View Profile WWW
    May 19, 2012, 03:31:03 AM
     #457
    I work just down the street from our statue university. Their surplus outlet is always packed with equipment. 3 story building, 1st floor is almost completely dedicated to computers and servers.
    happy birthday wishes - thank you email after successful event - formal invitation email for event

  • Garr255
    Legendary
    *
    Offline Offline

    Activity: 938
    Merit: 1000


    What's a GPU?


    View Profile
    May 19, 2012, 04:04:47 AM
     #458
    How much does it cost to walk around and pick through everything? And do you know if they have any old computers? I could always use another c64 Smiley
    “First they ignore you, then they laugh at you, then they fight you, then you win.”  -- Mahatma Gandhi

    Average time between signing on to bitcointalk: Two weeks. Please don't expect responses any faster than that!

  • twoBitBasher
    Member
    **
    Offline Offline

    Activity: 85
    Merit: 10



    View Profile
    May 19, 2012, 10:47:47 AM
     #459
    Quote from: rjk on May 19, 2012, 01:20:05 AM
    Not really, I just got the fan controller but haven't had a chance to install it. I'd like to do it tomorrow, but that's the day of the state auction, and I sure as heck am not going to miss that. http://www.das.ohio.gov/Divisions/GeneralServices/Surplus/WarehouseNextAuction.aspx Grin

    Mmmm serrrrrvers Tongue
    If you think my comments have benefitted you it would be nice to hear thanks Smiley

    Doge: DMnfgNp1HQSjtTZ1HcWiYtMwoGP5xcYDcz

  • rjk (OP)
    Sr. Member
    ****
    Offline Offline

    Activity: 462
    Merit: 250


    1ngldh


    View Profile
    May 19, 2012, 07:05:44 PM
     #460
    Quote from: Garr255 on May 19, 2012, 04:04:47 AM
    How much does it cost to walk around and pick through everything? And do you know if they have any old computers? I could always use another c64 Smiley
    How about an IBM Selectric II? Grin
    Mining Rig Extraordinaire - the Trenton BPX6806 18-slot PCIe backplane [PICS] Dead project is dead, all hail the coming of the mighty ASIC!
    Pages: « 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 [23] 24 25 26 »
      Print  
Page 22
Viewing Page: 23

IRLBTC™ © 2025 IRLBTC.com